Fact evaluation model training and fact evaluation

Abstract

Disclosed in the present description are a fact evaluation model training method, and a fact evaluation method and device. The fact evaluation model training method comprises: generating a sample text pair having a fact evaluation label on the basis of reference text; inputting the sample text pair into an initial fact evaluation model for model training, determining fact evaluation probability distribution information of the sample text pair, and on the basis of the fact evaluation probability distribution information, outputting a fact evaluation result of the sample text pair; then determining a model loss value on the basis of the fact evaluation label and the fact evaluation result, and finally using the model loss value to adjust model parameters of the initial fact evaluation model until the initial fact evaluation model completes model training and a target fact evaluation model is obtained.

Description

Fact assessment model training, fact assessment Technical Field

[0001] This specification relates to the field of natural language processing technology, and in particular to a method for training a fact assessment model, a method for fact assessment, and corresponding devices. Background Art

[0002] In recent years, the rapid development of large models has made them a focal point in artificial intelligence, particularly in natural language processing. However, when using large models to generate text content, the generated text can sometimes be inconsistent with the reference text it feeds. This makes it difficult to effectively assess the consistency between the generated text and the reference text it feeds. Summary of the Invention

[0003] This specification provides a fact assessment model training method, a fact assessment method and a device, and the technical solution is as follows.

[0004] In a first aspect, the present specification provides a method for training a fact evaluation model, the method comprising: creating an initial fact evaluation model, obtaining a reference text, and generating a sample text pair with a factual evaluation label based on the reference text; inputting the sample text pair into the initial fact evaluation model for model training, determining the factual evaluation probability distribution information of the sample text pair based on the initial fact evaluation model, and outputting the factual evaluation result of the sample text pair based on the factual evaluation probability distribution information; determining a model loss value based on the factual evaluation label and the factual evaluation result, and using the model loss value to adjust the model parameters of the initial fact evaluation model until the initial fact evaluation model completes model training to obtain a target fact evaluation model.

[0005] In the second aspect, this specification provides a fact evaluation method, which includes: obtaining a reference text and a target generated text for a target large model, the reference text being the input text of the target large model, and the target generated text being the output text of the target large model; obtaining a target text pair based on the reference text and the target generated text, inputting the target text pair into a target fact evaluation model, determining target factuality evaluation probability distribution information of the target text pair based on the target fact evaluation model, and outputting the target factuality evaluation result of the target text pair based on the target factuality evaluation probability distribution information.

[0006] In a third aspect, the present specification provides a fact evaluation model training device, which includes: a creation module, suitable for creating an initial fact evaluation model, obtaining a reference text, and generating a sample text pair with a factual evaluation label based on the reference text; a model training module, suitable for inputting the sample text pair into the initial fact evaluation model for model training, determining the factual evaluation probability distribution information of the sample text pair based on the initial fact evaluation model, and outputting the factual evaluation result of the sample text pair based on the factual evaluation probability distribution information; a model generation module, suitable for determining a model loss value based on the factual evaluation label and the factual evaluation result, and using the model loss value to adjust the model parameters of the initial fact evaluation model until the initial fact evaluation model completes model training and obtains the target fact evaluation model.

[0007] In a fourth aspect, the present specification provides a fact evaluation device, which includes: an acquisition module, suitable for acquiring a target reference text and a target generated text for a target large model, wherein the target reference text is the input text of the target large model, and the target generated text is the output text of the target large model; a factuality evaluation module, suitable for obtaining a target text pair based on the target reference text and the target generated text, inputting the target text pair into a target fact evaluation model, determining the target factuality evaluation probability distribution information of the target text pair based on the target fact evaluation model, and outputting the target factuality evaluation result of the target text pair based on the target factuality evaluation probability distribution information.

[0008] In a fifth aspect, this specification provides a computer storage medium, wherein the computer storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded by a processor and executing the above-mentioned method steps.

[0009] In a sixth aspect, this specification provides an electronic device that may include a processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the above-mentioned method steps.

[0010] In a seventh aspect, this specification provides a computer program product, which stores at least one instruction, and the at least one instruction is loaded by a processor to execute any one of the above method steps.

[0011] The beneficial effects brought about by the technical solutions provided in some embodiments of this specification include at least: generating sample text pairs with factual evaluation labels based on reference texts, so that the generated sample text pairs are close to the text data in real scenarios, and then training the initial fact evaluation model based on the sample text pairs. Since the sample text pairs have factual evaluation labels, the initial fact evaluation model can fully learn the factual evaluation information of the sample text pairs and continuously adjust the model parameters to obtain the target fact evaluation model, so that the factual consistency between the text content generated by the large model and the reference text input by the large model can be evaluated.

[0012] At the same time, the fact assessment model training method provided in this manual effectively avoids the use of randomly generated text pairs for model training, and circumvents the bias in the data set and potential grammatical errors in traditional methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions in this specification or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0014] FIG1 is a schematic diagram of a scenario of a fact evaluation model training system provided in this specification.

[0015] FIG2 is a flow chart of a method for training a fact assessment model provided in an embodiment of this specification.

[0016] FIG3 is a flow chart of a method for generating sample text pairs with factual evaluation labels based on reference texts according to an embodiment of this specification.

[0017] FIG4 is a flow chart of another method for training a fact assessment model provided in an embodiment of this specification.

[0018] FIG5 is a schematic diagram of a process for determining factual evaluation probability distribution information of a sample text pair provided by an embodiment of this specification.

[0019] FIG6 is a schematic diagram of a process for determining a factual evaluation result provided in an embodiment of this specification.

[0020] FIG7 is a flow chart of a fact evaluation method provided in an embodiment of this specification.

[0021] FIG8 is a flow chart of another fact evaluation method provided in an embodiment of this specification.

[0022] FIG9 is a schematic diagram of the structure of the fact evaluation model training device of this specification.

[0023] FIG10 is a schematic diagram of the structure of the fact evaluation device of this specification.

[0024] FIG11 is a structural block diagram of an electronic device provided in an embodiment of this specification.

[0025] FIG12 is a schematic diagram of the structure of an operating system and user space provided in an embodiment of this specification.

[0026] FIG13 is an architectural diagram of the Android operating system in FIG12 .

[0027] FIG14 is an architectural diagram of the IOS system in FIG12 . DETAILED DESCRIPTION

[0028] The following will clearly and completely describe the technical solutions in this specification in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments in this specification, not all of them. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this specification.

[0029] In the description of this specification, it should be understood that the terms "first", "second", etc. are used for descriptive purposes only and should not be understood as indicating or implying relative importance. In the description of this specification, it should be noted that, unless otherwise expressly specified and limited, "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices. For those of ordinary skill in the art, the specific meanings of the above terms in this specification can be understood according to the specific circumstances. In addition, in the description of this specification, unless otherwise specified, "multiple" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0030] The present specification is described in detail below with reference to specific embodiments.

[0031] Please refer to Figure 1, which is a schematic diagram of a scenario of a fact evaluation model training system provided in this specification. As shown in Figure 1, the fact evaluation model training system may include at least a client cluster and a service platform 100.

[0032] The client cluster may include at least one client, as shown in FIG1 , specifically including client 1 corresponding to user 1, client 2 corresponding to user 2, ..., client n corresponding to user n, where n is an integer greater than 0.

[0033] Each client in the client cluster can be an electronic device with communication capabilities, including but not limited to wearable devices, handheld devices, personal computers, tablet computers, in-vehicle devices, smartphones, computing devices, or other processing devices connected to a wireless modem. Electronic devices may be called different names in different networks, such as user equipment, access terminal, subscriber unit, subscriber station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication device, user agent or user device, cellular phone, cordless phone, personal digital assistant (PDA), electronic devices in 5G network or future evolution network, etc.

[0034] The service platform 100 can be a separate server device, such as a rack-mounted, blade, tower, or cabinet-mounted server device, or a workstation, mainframe computer, or other hardware device with strong computing capabilities; it can also be a server cluster composed of multiple servers. The servers in the service cluster can be symmetrically composed, wherein each server has equivalent functions and status in the transaction link, and each server can provide services to the outside world independently. The independent service can be understood as not requiring the assistance of other servers.

[0035] In one or more embodiments of the present specification, the service platform 100 may establish a communication connection with at least one client in the client cluster, and complete data interaction during the fact assessment model training process based on the communication connection, such as online transaction data interaction. For example, the service platform 100 may provide factual assessment services to the client based on the target fact assessment model obtained by the fact assessment model training method of the present specification; for example, the service platform 100 may obtain training data from the client, such as the first training data.

[0036] It should be noted that the service platform 100 and at least one client in the client cluster establish a communication connection through a network for interactive communication, wherein the network can be a wireless network or a wired network, the wireless network includes but is not limited to a cellular network, a wireless local area network, an infrared network or a Bluetooth network, and the wired network includes but is not limited to Ethernet, a universal serial bus (USB) or a controller area network. In one or more embodiments of the specification, technologies and / or formats including Hypertext Markup Language (HTML) and Extensible Markup Language (XML) are used to represent data exchanged over the network (such as a target compressed package). In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), and Internet Protocol Security (IPsec) can also be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above-mentioned data communication technologies.

[0037] The fact assessment model training system embodiment provided in this specification and the fact assessment model training method in one or more embodiments are of the same concept. The execution subject corresponding to the fact assessment model training method involved in one or more embodiments of the specification can be the above-mentioned service platform 100; the execution subject corresponding to the fact assessment model training method involved in one or more embodiments of the specification can also be the electronic device corresponding to the client, which is specifically determined based on the actual application environment. The implementation process of the fact assessment model training system embodiment can be found in the following method embodiment in detail, and will not be repeated here.

[0038] Based on the scenario diagram shown in FIG1 , the fact assessment model training method provided by one or more embodiments of this specification is introduced in detail below.

[0039] Please refer to Figure 2, which is a flowchart illustrating a method for training a fact assessment model according to an embodiment of this specification. This method can be implemented using a computer program and run on a fact assessment model training device based on the von Neumann architecture. This computer program can be integrated into an application or run as a standalone tool application. The fact assessment model training device can be a service platform.

[0040] Specifically, the fact evaluation model training method includes the following steps.

[0041] S202: Create an initial fact evaluation model, obtain reference text, and generate sample text pairs with factual evaluation labels based on the reference text.

[0042] The initial factual assessment model can be an untrained basic generative large model or an incompletely trained generative large model. Here, the initial factual assessment model can perform a preliminary factual assessment of the text content. Factual assessment can be an evaluation of the authenticity and accuracy of the text content generated by the large model. Factual assessment has broad application value in many fields. For example, in the field of news reporting, factual assessment of news generated by large models is a key task to identify false news and misleading information. In legal proceedings, evaluating the authenticity and reliability of evidence compiled and summarized based on large models helps improve litigation efficiency. In scientific research, factual assessment of experimental data compiled based on large models helps ensure the reliability of research results.

[0043] Traditional factual assessment methods often rely on manual review and expert judgment, which has certain limitations. On the one hand, manual review is costly, time-consuming, and difficult to process with massive amounts of information; on the other hand, expert judgment is easily influenced by subjective consciousness and pre-existing cognition, which may lead to biased assessment results.

[0044] After creating the initial factual assessment model, reference text can be obtained. Reference text includes but is not limited to news reports, official announcements, scientific research data, and various documents. After obtaining the reference text, sample text pairs are generated based on the reference text. The sample text pairs can include the reference text and sample text generated based on the reference text, as well as factual assessment labels used to indicate whether the reference text and sample text are identical in factuality. Specifically, the sample text can include positive text and negative text. Positive text has the same factuality as the reference text, while negative text has factuality that contradicts the reference text.

[0045] To prevent the generated sample text from deviating from the text form of the actual scenario, resulting in deviations from the actual scenario, such as grammatical errors, and mismatches between the sample text and the reference text and the factual evaluation labels, sample text can be generated for the reference text based on the prompt words of the large model and the standardized large model. Specifically, positive text and negative text can be generated based on the same large model, or based on the prompt words of different large models; of course, positive text and negative text can also be generated based on different large models.

[0046] Here, the factuality evaluation labels include reference positive factuality evaluation labels and reference negative factuality evaluation labels. When the factuality evaluation label is a reference positive factuality evaluation label, it indicates that the factuality of the sample text pair is the same, that is, the sample text has the same factuality as the reference text; when the factuality evaluation label is a reference negative factuality evaluation label, it indicates that the factuality of the sample text pair is inconsistent, that is, the sample text has inconsistent factuality with the reference text. It is easy to understand that the reference positive factuality evaluation labels and the reference negative factuality evaluation labels can be represented by quantitative numbers.

[0047] S204: Input the sample text pair into the initial fact evaluation model for model training, determine factual evaluation probability distribution information of the sample text pair based on the initial fact evaluation model, and output factual evaluation results of the sample text pair based on the factual evaluation probability distribution information.

[0048] After the sample text pair is input into the initial fact assessment model, the initial fact assessment model learns the text features, entity features, and semantic features of the reference text and the sample text in the sample text pair. Text features can be linguistic features in the text, including features in vocabulary, grammar, and syntax. Entity features can be entities that appear in the text, such as names of people, places, organizations, time, numbers, etc. Semantic features refer to the meaning or significance of a word or phrase in a specific context. They can include the literal meaning of the word or phrase, as well as the extended meaning of the word or phrase in the context.

[0049] The initial fact evaluation model compares the text features, entity features, and semantic features of the reference text with the text features, entity features, and semantic features of the sample text to determine the factual evaluation probability distribution information of the sample text pair.

[0050] Specifically, the initial fact evaluation model can determine the first text feature vector, first entity feature vector and first semantic feature vector of the reference text based on the text features, entity features and semantic features of the reference text, and determine the second text feature vector, second entity feature vector and second semantic feature vector of the sample text based on the text features, entity features and semantic features of the sample text.

[0051] Afterwards, the first vector similarity between the first text feature vector and the second text feature vector is calculated, the second vector similarity between the first entity feature vector and the second entity feature vector is calculated, and the third vector similarity between the first semantic feature vector and the second semantic feature vector is calculated. Based on the first vector similarity, the second vector similarity and the third vector similarity, the factuality evaluation probability distribution information of the reference text and the sample text in the sample text pair is determined, that is, the first probability that the reference text and the sample text are the same factual texts and the second probability that the reference text and the sample text are contradictory factual texts.

[0052] After obtaining the probability distribution information for the factual evaluation of the sample text pair, the factual evaluation probability distribution information is analyzed to determine the factual evaluation result of the sample text pair. The factual evaluation result includes a sample positive factual evaluation label and a sample negative factual evaluation label. As is easy to understand, the sample positive factual evaluation label and the sample negative factual evaluation label can be represented by quantitative numbers.

[0053] The model base of the initial fact assessment model can adopt a generative model, such as T5 (Text-to-Text Transfer Transformer model). When the sample text pairs are input into the initial fact assessment model for model training, prompt words can be constructed for the sample text pairs. For example, the input of the initial fact assessment model is: "Is the following assertion consistent with the context? Assertion: <generated text> Context: <source text>". Here, the generated text can be the sample text in the sample text pair corresponding to the reference text, and the source text can be the reference text.

[0054] S206: Determine a model loss value based on the factual evaluation label and the factual evaluation result, and use the model loss value to adjust the model parameters of the initial fact evaluation model until the initial fact evaluation model completes model training to obtain a target fact evaluation model.

[0055] Among them, a first parameter corresponding to the factual evaluation label and a second parameter corresponding to the factual evaluation result are determined, and a model loss function is constructed based on the first parameter and the second parameter. Then, the quantitative numbers corresponding to the factual evaluation label and the factual evaluation result are substituted into the model loss function to calculate the model loss value. After obtaining the model loss value, the model parameters of the initial fact evaluation model are adjusted based on the model loss value. Then, the initial fact evaluation model is trained again based on the sample text until the model loss value converges. At this time, the initial fact evaluation model completes the model training and obtains the target fact evaluation model.

[0056] In this specification, sample text pairs with factual evaluation labels are generated based on reference texts, so that the generated sample text pairs are close to the text data in real scenarios. Then, the initial fact evaluation model is trained based on the sample text pairs. Since the sample text pairs have factual evaluation labels, the initial fact evaluation model can fully learn the factual evaluation information of the sample text pairs and continuously adjust the model parameters to obtain the target fact evaluation model, so that the factual consistency between the text content generated by the large model and the reference text input by the large model can be evaluated.

[0057] At the same time, the fact assessment model training method provided in this manual effectively avoids the use of randomly generated text pairs for model training, and circumvents the bias in the data set and potential grammatical errors in traditional methods.

[0058] Please refer to Figure 3, which is a flow chart of generating a sample text pair with a factual evaluation label based on a reference text according to an embodiment of this specification. As shown in Figure 3, generating a sample text pair with a factual evaluation label based on a reference text in S202 includes the following steps.

[0059] S302: performing a process of generating identical factual data on the reference text to obtain a positive text, and performing a process of generating contradictory factual data on the reference text to obtain a negative text.

[0060] Among them, the reference text is rewritten by integrating at least three dimensions: text features, entity features and semantic features, so that the obtained positive text and the reference text are consistent in factuality in at least these three dimensions, thereby achieving the same factual data generation processing for the reference text.

[0061] The text features can specifically be key text features. The importance weights corresponding to each language feature in the reference text can be calculated, and then the importance weights of each language feature in the reference text are sorted in descending order, and the language features corresponding to the preset proportion of importance weights arranged in front are extracted as key text features. In this way, positive text is generated based on key text features rather than all text features, which reduces the generation constraints of positive text, so that the factuality of the positive text can be kept consistent with that of the reference text as much as possible, while generating as much positive text as possible.

[0062] On the contrary, in the process of generating contradictory factual data for the reference text, the reference text is rewritten through at least one of the three dimensions of text features, entity features and semantic features, so that the facts of the obtained negative text and the reference text remain inconsistent in at least one of the three dimensions, thereby realizing the generation of contradictory factual data for the reference text.

[0063] S304: Generate a positive text pair with a positive factual evaluation label based on the positive text and the reference text, and generate a negative text pair with a negative factual evaluation label based on the negative text and the reference text.

[0064] Among them, a first text pair is obtained based on the positive text and the reference text. Since the positive text is obtained by performing the same factual data generation process on the reference text, the first text pair can be marked with a positive factual evaluation label to obtain a positive text pair.

[0065] Similarly, a second text pair is obtained based on the negative text and the reference text. Since the negative text is obtained by generating contradictory factual data based on the reference text, the second text pair can be marked with a negative factual evaluation label to obtain a negative text pair.

[0066] Based on positive text pairs, the initial fact evaluation model can fully learn the factual evaluation information of positive text pairs and continuously adjust the model parameters in the dimension of factual consistency; similarly, based on negative text pairs, the initial fact evaluation model can fully learn the factual evaluation information of negative text pairs and continuously adjust the model parameters in the dimension of factual contradiction.

[0067] S306: Obtaining sample text pairs with factual evaluation labels based on the positive text pairs and the negative text pairs.

[0068] Among them, the factual evaluation labels include positive factual evaluation labels and negative factual evaluation labels, and the sample text pairs include positive text pairs and negative text pairs. The obtained positive text pairs and negative texts are put into the same data set to obtain a factual data set, and each element in the factual data set is a sample text pair.

[0069] Training the initial fact evaluation model based on sample text pairs can enable the initial fact evaluation model to fully learn the fact evaluation information of positive text pairs and negative text pairs in the dimension of factual consistency and the dimension of factual contradiction, so as to continuously adjust the model parameters and thereby improve the accuracy of the fact evaluation results of the target fact evaluation model.

[0070] In the embodiments provided in this specification, the reference text is processed for identical factual data generation and contradictory factual data generation, respectively, thereby obtaining a positive text that is consistent with the reference text in terms of factuality, and a negative text that is contradictory to the reference text in terms of factuality. Subsequently, a positive text pair is obtained through the positive text and the reference text, and a negative text pair is obtained through the negative text and the reference text, so that the initial fact evaluation model can fully learn the factual evaluation information of the positive text pair and the negative text pair in terms of the factual consistency dimension and the factual contradiction dimension, thereby improving the accuracy of the factual evaluation results of the obtained target fact evaluation model.

[0071] In an embodiment provided in this specification, in S302, the reference text is subjected to the same factual data generation processing to obtain a positive text, including: performing the same factual rewriting processing on the reference text based on the positive text generation model to obtain the positive text; wherein the positive text generation model is adapted for the same factual rewriting processing scenario based on the first basic large language generation model.

[0072] Among them, the generation capability of the large model corresponding to the forward text generation model can make the factual consistency of the generated forward text more reasonable, and at the same time, closer to the real text application scenario, while avoiding the deviation of the data set and potential grammatical errors in traditional methods.

[0073] During the process of rewriting the reference text using the forward text generation model to identically represent the facts, the forward text is generated by maintaining the factual consistency between the rewritten text and the reference text. Specifically, the forward text generation model can ensure the factual consistency between the generated forward text and the reference text from at least three dimensions: comprehensive text features, entity features, and semantic features.

[0074] At the same time, in the process of adapting the first basic large language generative model for the same factual rewriting processing scenario to obtain a positive text generation model, the adaptation can be performed by adjusting the parameters of the first basic large language generative model and modifying the corresponding prompt (prompt word), so that by constructing a specific prompt for the reference text, a positive text that is consistent with the factuality of the reference text can be output for the same factual rewriting processing scenario. Here, the first basic large language generative model can be a large model such as GPT (Generative Pre-Trained Transformer model)-3.5-turbo.

[0075] In the embodiments provided in this specification, based on the generation capability of the large model corresponding to the forward text generation model, the factual consistency between the generated forward text and the reference text can be made more reasonable, so that the generated forward text can be close to the real text application scenario, while avoiding the deviations in the data set and potential grammatical errors in traditional methods.

[0076] In an embodiment provided in this specification, in S302, a reference text is subjected to contradictory factual data generation processing to obtain a negative text, including: performing contradictory factual rewriting processing on the reference text based on a negative text generation model to obtain a negative text; wherein the negative text generation model is adapted for the contradictory factual rewriting processing scenario based on the second basic language generative model.

[0077] Among them, the generation capability of the large model corresponding to the negative text generation model can make the factual contradiction of the generated negative text more reasonable, while being close to the real text application scenario. It can also avoid the deviation of the data set and potential grammatical errors in traditional methods.

[0078] During the process of rewriting a reference text to contradict facts using the negative text generation model, the negative text is generated by changing the factual consistency of the rewritten text with the reference text. Specifically, the positive text generation model can ensure that the generated negative text is factually inconsistent with the reference text based on at least one of three dimensions: comprehensive text features, entity features, and semantic features.

[0079] At the same time, in the process of adapting the second basic language generative model for the contradictory factual rewriting processing scenario to obtain a negative text generation model, the second basic language generative model can be adapted by adjusting the parameters and modifying the corresponding prompt (prompt word) of the model. By constructing a specific prompt for the reference text, a negative text that contradicts the factuality of the reference text can be output for the contradictory factual rewriting processing scenario. Here, the second basic language generative model can also be a large model such as GPT (Generative Pre-Trained Transformer model)-3.5-turbo.

[0080] In the embodiments provided in this specification, based on the generation capability of the large model corresponding to the negative text generation model, the factual contradiction between the generated negative text and the reference text can be made more reasonable, so that the generated negative text can be close to the real text application scenario, while avoiding the deviation of the data set and potential grammatical errors in the traditional method.

[0081] It should be understood that the positive text generation model and the negative text generation model can be different models or the same model.

[0082] Please refer to Figure 4, which is a flow chart of another fact assessment model training method provided by an embodiment of this specification. As shown in Figure 4, in the embodiment provided by this specification, when the positive text generation model and the negative text generation model are the same model, the first basic large language generation model and the second basic large language generation model are the same. At this time, the first basic large language generation model or the second basic large language generation model can be adapted for the contradictory factual rewriting processing scenario at the same time, as well as for the same factual rewriting processing scenario, so that the large model can output a negative text that is contradictory to the factuality of the reference text based on the prompt words of the contradictory factual rewriting processing scenario, and the large model can output a positive text that is consistent with the factuality of the reference text based on the prompt words of the same factual rewriting processing scenario.

[0083] Afterwards, a second text pair is obtained based on the negative text and the reference text. Since the negative text is obtained by generating contradictory factual data based on the reference text, the second text pair can be marked with a negative factual evaluation label to obtain a negative text pair. The obtained positive text pair and negative text are put into the same dataset to obtain a factual dataset. Then, the initial fact evaluation model is trained based on the sample text pairs in the factual dataset to obtain a target fact evaluation model.

[0084] In an embodiment provided in the present specification, the above embodiment performs same factual rewriting processing on the reference text based on the forward text generation model to obtain the forward text, including: using at least one of the same factual word order rearrangement processing, same factual active and passive conversion processing, same factual sentence fusion processing, same factual text abbreviation processing, and same factual synonym replacement processing based on the forward text generation model to rewrite the reference text to obtain the forward text.

[0085] Among them, rearranging the word order of the reference text with the same facts can be to rearrange the phrases, sentences or paragraphs in the reference text to keep the authenticity of the information consistent with the reference text, that is, to adjust the word order in accordance with logic or expression habits. When performing this rewriting, attention should be paid to maintaining the consistency of facts and information in the text, and avoiding changing the meaning or significance of the text.

[0086] Performing the same factual active-passive conversion on the reference text can involve converting the factual content in the reference text from active to passive. Specifically, this involves converting the active voice in the reference text to the passive voice, or vice versa. This conversion ensures that the converted forward text still accurately conveys the meaning of the reference text.

[0087] Merging identical factual sentences within a reference text involves fusing the sentence structures within the reference text, fusing multiple parallel sentences. This can reduce the length of the text and improve its conciseness and readability. This is a common type of text transformation in real-world scenarios. Similarly, when performing this type of rewriting, care should be taken to maintain the consistency of the facts and information within the text and avoid altering the text's meaning or significance.

[0088] Abbreviation of identical factual text within a reference text can be performed on one or more paragraphs within the reference text that contain repetitive content and require abbreviation to reduce length or improve efficiency. When abbreviating, methods such as deleting duplicate sentences, simplifying sentence structures, combining similar concepts, and using standardized abbreviations can be employed to preserve key information and reduce redundancy. The abbreviated text should maintain the same factual content as the original text while remaining as clear and understandable as possible.

[0089] Factual synonym replacement of a reference text involves replacing certain content in the reference text with identical, factual, and synonymous content. Specifically, this process may involve replacing certain words or phrases in the text, such as verbs, adverbs, or nouns, to better convey the same meaning or concept. Again, when performing this type of rewriting, care should be taken to maintain the consistency of the facts and information in the text and avoid changing the meaning or significance of the text.

[0090] Furthermore, when rewriting the reference text, some details may be omitted. The omission of some details does not affect the factual consistency between the reference text and the source text. It should be noted that factual consistency between the reference text and the source text means that the content of the source text can be found in the reference text, not that the content of the reference text and the source text are identical. The content of the reference text can include content from the source text.

[0091] In the embodiments provided in this specification, rewriting the reference text through the forward text generation model is essentially performing text interpretation on the reference text using the forward text generation model, thereby obtaining a forward text that is consistent with the facts of the reference text.

[0092] In an embodiment provided in the present specification, the above embodiment performs fact-contradictory rewriting processing on a reference text based on a negative text generation model to obtain a negative text, including: rewriting the reference text to obtain a negative text based on the negative text generation model using at least one of fact-contradictory numerical transformation processing, fact-contradictory antonym replacement processing, and fact-contradictory subject-object swapping processing.

[0093] Rewriting a reference text to contradict facts can involve modifying or changing facts in the reference text to make them contrary to or contradictory to the original facts in the reference text. Rewriting a reference text to contradict facts can involve distorting, exaggerating, minimizing, or fabricating facts in the reference text to alter the meaning or impact of the reference text.

[0094] Performing a contradictory numerical transformation on a reference text actually involves editing the numerical values ​​in the reference text to be contradictory to the numerical values ​​in the reference text, such as transforming and rewriting the numerical values ​​(such as dates) in the reference text, such as changing January 1 to January 2.

[0095] Performing antonym replacement of contradictory facts on the reference text can be understood as replacing antonyms of verbs, adverbs, or nouns in the reference text, that is, performing part-of-speech antonym editing. Performing subject-object swapping of contradictory facts on the reference text can be understood as swapping the subject and object in the reference text.

[0096] In the embodiments provided in this specification, rewriting the reference text through the negative text generation model is essentially rewriting the reference text in a factually contradictory manner using the negative text generation model, thereby obtaining a negative text that is factually contradictory to the reference text.

[0097] Please refer to Figure 5, which is a flow chart of determining factual evaluation probability distribution information of a sample text pair according to an embodiment of this specification. Specifically, determining factual evaluation probability distribution information of a sample text pair based on an initial fact evaluation model in S204 includes the following steps.

[0098] S502: Determine a first probability that the sample text pair is the same factual text based on the initial fact evaluation model.

[0099] Among them, the initial fact assessment model can determine the first text feature vector, first entity feature vector and first semantic feature vector of the reference text based on the text features, entity features and semantic features of the reference text, and determine the second text feature vector, second entity feature vector and second semantic feature vector of the sample text based on the text features, entity features and semantic features of the sample text.

[0100] Calculate a first vector similarity between the first text feature vector and the second text feature vector, calculate a second vector similarity between the first entity feature vector and the second entity feature vector, calculate a third vector similarity between the first semantic feature vector and the second semantic feature vector, and determine a first probability that the sample text pair is the same factual text based on the first vector similarity, the second vector similarity, and the third vector similarity.

[0101] S504: Determine a second probability that the sample text pair is a contradictory factual text based on the initial fact evaluation model.

[0102] Among them, the initial fact evaluation model can also determine the second probability that the sample text pair is a contradictory factual text based on the first vector similarity, the second vector similarity and the third vector similarity; or, the second probability that the sample text pair is a contradictory factual text can be calculated based on the first probability that the sample text pair is the same factual text, and in this case, the sum of the first probability and the second probability can be 1.

[0103] S506: Obtain factual evaluation probability distribution information based on the first probability and the second probability.

[0104] Among them, after obtaining the first probability and the second probability, the first probability is used to represent the probability that the reference text and the sample text in the sample text pair are the same factual texts; the second probability is used to represent the probability that the reference text and the sample text in the sample text pair are contradictory factual texts.

[0105] The factuality evaluation probability distribution information records a first probability that the reference text and the sample text in the sample text pair are mutually identical factual texts, and a second probability that the reference text and the sample text in the sample text pair are mutually contradictory factual texts.

[0106] In the embodiments provided in this specification, a first probability that the sample text pair is the same factual text and a second probability that the sample text pair is contradictory factual text are determined based on the initial fact evaluation model, thereby obtaining factual evaluation probability distribution information based on the first probability and the second probability, thereby facilitating determination of the factual evaluation result of the sample text pair based on the factual evaluation probability distribution information.

[0107] Please refer to Figure 6, which is a flowchart of determining a factuality evaluation result according to an embodiment of this specification. As shown in Figure 6, outputting the factuality evaluation result of the sample text pair based on the factuality evaluation probability distribution information in S204 includes the following steps.

[0108] S602: Compare the first probability and the second probability.

[0109] The first probability is used to represent the probability that the reference text and the sample text in the sample text pair are mutually identical factual texts; the second probability is used to represent the probability that the reference text and the sample text in the sample text pair are mutually contradictory factual texts.

[0110] Therefore, by comparing the first probability and the second probability, the tendency of the reference text and the sample text in the sample text pair to be mutually identical factual texts and the tendency of the reference text and the sample text to be mutually contradictory factual texts can be determined.

[0111] S604: When the first probability is greater than or equal to the second probability, output the positive factuality evaluation result of the sample.

[0112] Among them, when the first probability is greater than or equal to the second probability, it indicates that the tendency of the reference text and the sample text in the sample text pair to be the same factual text is greater than or equal to the tendency to be contradictory factual texts. At this time, it can be considered that the factual evaluation results corresponding to the reference text and the sample text in the sample text pair are positive factual evaluation results of the sample.

[0113] S606: When the first probability is less than the second probability, output the negative factual evaluation result of the sample.

[0114] Among them, when the first probability is less than the second probability, it indicates that the reference text and the sample text in the sample text pair have a lower tendency to be mutually identical factual texts than to be mutually contradictory factual texts. At this time, it can be considered that the factual evaluation results corresponding to the reference text and the sample text in the sample text pair are sample negative factual evaluation results.

[0115] In the embodiments provided in this specification, by comparing the first probability and the second probability, when the first probability is greater than or equal to the second probability, it indicates that the tendency of the reference text and the sample text in the sample text pair to be the same factual text is greater than or equal to the tendency of them to be contradictory factual texts, and the corresponding factual evaluation result is a sample positive factual evaluation result; when the first probability is less than the second probability, it indicates that the tendency of the reference text and the sample text in the sample text pair to be the same factual text is less than the tendency of them to be contradictory factual texts, and the corresponding factual evaluation result is a sample negative factual evaluation result.

[0116] Please refer to Figure 7, which is a flowchart illustrating a fact assessment method provided in an embodiment of this specification. This method can be implemented using a computer program and run on a fact assessment device based on the von Neumann architecture. This computer program can be integrated into an application or run as a standalone tool application.

[0117] Specifically, the method includes the following steps.

[0118] S702: Obtain a target reference text and a target generated text for a target large model, where the target reference text is the input text of the target large model, and the target generated text is the output text of the target large model.

[0119] After receiving the target reference text, the target macromodel outputs a generated target text based on the target reference text. The target macromodel is used to generate the target generated text based on the target reference text in response to user instructions. For example, the target macromodel can extract the target reference text to generate the target generated text; or the target macromodel can answer the user's question based on the target reference text to generate the target generated text. There are no restrictions on the type of text processing performed by the target macromodel.

[0120] S704: Obtain a target text pair based on the target reference text and the target generated text, input the target text pair into a target fact evaluation model, determine target factuality evaluation probability distribution information of the target text pair based on the target fact evaluation model, and output the target factuality evaluation result of the target text pair based on the target factuality evaluation probability distribution information.

[0121] Among them, the input text of the target large model and the output text of the target large model are combined into a target text pair, that is, the target text pair is obtained based on the target reference text and the target generated text. After obtaining the target text pair, the target text pair is input into the target fact evaluation model. The training process of the target fact evaluation model can refer to steps S202 to S206, and will not be elaborated here.

[0122] After the target text pair is input into the target fact evaluation model, the target fact evaluation model can determine the first text feature vector, the first entity feature vector and the first semantic feature vector of the target reference text based on the text features, entity features and semantic features of the target reference text, and determine the second text feature vector, the second entity feature vector and the second semantic feature vector of the target generated text based on the text features, entity features and semantic features of the target generated text.

[0123] Afterwards, the first vector similarity between the first text feature vector and the second text feature vector is calculated, the second vector similarity between the first entity feature vector and the second entity feature vector is calculated, and the third vector similarity between the first semantic feature vector and the second semantic feature vector is calculated. Based on the first vector similarity, the second vector similarity and the third vector similarity, the factual evaluation probability distribution information of the target reference text and the target generated text in the target text pair is determined, that is, the first probability that the target reference text and the target generated text are the same factual texts and the second probability that the target reference text and the target generated text are contradictory factual texts.

[0124] After obtaining the target factual evaluation probability distribution information of the target text pair, the target factual evaluation probability distribution information is analyzed to determine the target factual evaluation result of the target text pair. The target factual evaluation result includes a target positive factual evaluation label and a target negative factual evaluation label. The target positive factual evaluation label indicates that the target reference text and the target generated text are the same factual texts, and the target negative factual evaluation label indicates that the target reference text and the target generated text are contradictory factual texts.

[0125] In the embodiments provided herein, the target fact assessment model is derived by continuously adjusting model parameters based on the initial fact assessment model's thorough learning of factual assessment information for sample text pairs. This allows the target fact assessment model to assess the factual consistency between the output text of the large model and the input text of the large model. Therefore, the target fact assessment model can accurately determine the target factual assessment probability distribution information for the sample text pairs, thereby determining the target factual assessment result for the target text pairs.

[0126] Please refer to Figure 8, which is a flow chart of another fact evaluation method provided in an embodiment of this specification. In the embodiment provided in this specification, the fact evaluation method may further include the following steps.

[0127] S802: Determine a first probability that the target text pair is the same factual text based on the target factuality evaluation probability distribution information.

[0128] The factual evaluation probability distribution information includes a first probability that the target reference text and the target generated text are the same factual text and a second probability that the target reference text and the target generated text are contradictory factual texts.

[0129] Therefore, a first probability that the target text pair is the same factual text can be obtained based on the target factuality evaluation probability distribution information. The first probability is used to characterize the probability that the target reference text and the target generated text in the target text pair are the same factual text.

[0130] S804: Determine a fact consistency score of the target text pair based on the first probability.

[0131] Among them, since the first probability is used to represent the probability that the target reference text and the target generated text in the target text pair are the same factual text, the first probability can be converted into a score to obtain the factual consistency score of the target text pair.

[0132] It is easy to understand that the greater the factual consistency score of the target text pair, the higher the probability that the target reference text and the target generated text in the target text pair are the same factual text; the lower the factual consistency score of the target text pair, the lower the probability that the target reference text and the target generated text in the target text pair are the same factual text.

[0133] In the embodiment provided in this specification, a fact consistency score of the target text pair is determined based on the first probability, so that the fact consistency of the target text pair is quantified based on the fact consistency score.

[0134] The following will describe in detail the fact assessment model training device provided in the embodiments of this specification in conjunction with Figure 9. It should be noted that the fact assessment model training device shown in Figure 9 is used to execute the method of the embodiments shown in Figures 1 to 6 of this specification. For ease of explanation, only the parts relevant to this specification are shown. For specific technical details not disclosed, please refer to the embodiments shown in Figures 1 to 6 of this specification.

[0135] Please refer to Figure 9, which is a structural diagram of the fact evaluation model training device of this specification. The fact evaluation model training device 1 can be implemented as all or part of the user terminal through software, hardware or a combination of both. According to some embodiments, the fact evaluation model training device 1 includes a creation module 11, a model training module 12 and a model generation module 13, wherein: the creation module 11 is suitable for creating an initial fact evaluation model, obtaining a reference text, and generating a sample text pair with a factual evaluation label based on the reference text; the model training module 12 is suitable for inputting the sample text pair into the initial fact evaluation model for model training, determining the factual evaluation probability distribution information of the sample text pair based on the initial fact evaluation model, and outputting the factual evaluation result of the sample text pair based on the factual evaluation probability distribution information; the model generation module 13 is suitable for determining the model loss value based on the factual evaluation label and the factual evaluation result, and using the model loss value to adjust the model parameters of the initial fact evaluation model until the initial fact evaluation model completes the model training to obtain the target fact evaluation model.

[0136] Optionally, the creation module 11 includes: a first generation unit, suitable for performing the same factual data generation processing on the reference text to obtain positive text, and performing the contradictory factual data generation processing on the reference text to obtain negative text; a second generation unit, suitable for generating a positive text pair with positive factual evaluation labels based on the positive text and the reference text, and generating a negative text pair with negative factual evaluation labels based on the negative text and the reference text; a sample text pair determination unit, suitable for obtaining a sample text pair with factual evaluation labels based on the positive text pair and the negative text pair.

[0137] Optionally, the first generation unit is also suitable for performing the same factual rewriting processing on the reference text based on the forward text generation model to obtain the forward text; wherein the forward text generation model is adapted for the same factual rewriting processing scenario based on the first basic large language generation model.

[0138] Optionally, the first generation unit is also suitable for rewriting the reference text based on the forward text generation model by adopting at least one of the same factual word order rearrangement processing, same factual active and passive conversion processing, same factual sentence fusion processing, same factual text abbreviation processing, and same factual synonym replacement processing to obtain the forward text.

[0139] Optionally, the first generation unit is also suitable for performing a contradictory factual rewriting process on the reference text based on a negative text generation model to obtain a negative text; wherein the negative text generation model is adapted for the contradictory factual rewriting processing scenario based on the second basic language generation model.

[0140] Optionally, the first generation unit is further adapted to rewrite the reference text based on the negative text generation model by adopting at least one of contradictory factual numerical transformation processing, contradictory factual antonym replacement processing, and contradictory factual subject-object swapping processing to obtain negative text.

[0141] Optionally, the model training module 12 includes: a first probability determination unit, suitable for determining a first probability that the sample text pair is the same factual text based on the initial fact evaluation model; a second probability determination unit, suitable for determining a second probability that the sample text pair is contradictory factual text based on the initial fact evaluation model; and a probability distribution determination unit, suitable for obtaining factual evaluation probability distribution information based on the first probability and the second probability.

[0142] Optionally, the model training module 12 includes: a comparison unit, suitable for comparing the first probability and the second probability; a first judgment unit, suitable for outputting a positive factual evaluation result of the sample when the first probability is greater than or equal to the second probability; and a second judgment unit, suitable for outputting a negative factual evaluation result of the sample when the first probability is less than the second probability.

[0143] It should be noted that the fact assessment model training device provided in the above embodiment only uses the division of the above functional modules as an example when executing the fact assessment model training method. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the fact assessment model training device provided in the above embodiment and the fact assessment model training method embodiment belong to the same concept. The implementation process is detailed in the method embodiment and will not be repeated here.

[0144] The following will provide a detailed description of the fact assessment device provided in the embodiments of this specification, in conjunction with Figure 10. It should be noted that the fact assessment device shown in Figure 10 is used to execute the method of the embodiments shown in Figures 7 and 8 of this specification. For ease of explanation, only the parts relevant to this specification are shown. For specific technical details not disclosed, please refer to the embodiments shown in Figures 7 and 8 of this specification.

[0145] Please refer to Figure 10, which is a structural diagram of the fact evaluation device of this specification. The fact evaluation device 2 can be implemented as all or part of the user terminal through software, hardware or a combination of both. According to some embodiments, the fact evaluation device 2 includes an acquisition module 21 and a factual evaluation module 22, wherein: the acquisition module 21 is suitable for acquiring a target reference text and a target generated text for a target large model, the target reference text is the input text of the target large model, and the target generated text is the output text of the target large model; the factual evaluation module 22 is suitable for obtaining a target text pair based on the target reference text and the target generated text, inputting the target text pair into a target fact evaluation model, determining target factual evaluation probability distribution information of the target text pair based on the target fact evaluation model, and outputting a target factual evaluation result of the target text pair based on the target factual evaluation probability distribution information.

[0146] Optionally, the fact evaluation device 2 also includes: a first probability determination module, adapted to determine a first probability that the target text pair is the same factual text based on the target factuality evaluation probability distribution information; and a fact consistency score determination module, adapted to determine a fact consistency score of the target text pair based on the first probability.

[0147] It should be noted that the fact assessment device provided in the above embodiment, when executing the fact assessment method, is merely illustrated by the division of the aforementioned functional modules. In actual applications, the aforementioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the fact assessment device provided in the above embodiment and the fact assessment method embodiment are based on the same concept. The implementation process is detailed in the method embodiment and will not be repeated here.

[0148] The above serial numbers in this specification are for description only and do not represent the advantages or disadvantages of the embodiments.

[0149] This specification also provides a computer storage medium, which can store multiple instructions, and the instructions are suitable for being loaded by a processor and executed by the fact evaluation model training method or fact evaluation method of the embodiments shown in Figures 1 to 8 above. The specific execution process can be found in the specific description of the embodiments shown in Figures 1 to 8, and will not be repeated here.

[0150] This specification also provides a computer program product, which stores at least one instruction, and the at least one instruction is loaded by the processor and executed by the fact assessment model training method or the fact assessment method of the embodiments shown in Figures 1 to 8 above. The specific execution process can be found in the specific description of the embodiments shown in Figures 1 to 8, and will not be repeated here.

[0151] Please refer to Figure 11, which is a block diagram of the structure of an electronic device provided in an embodiment of this specification. The electronic device described in this specification may include one or more of the following components: a processor 110, a memory 120, an input device 130, an output device 140, and a bus 150. The processor 110, the memory 120, the input device 130, and the output device 140 may be connected via the bus 150.

[0152] The processor 110 may include one or more processing cores. The processor 110 utilizes various interfaces and circuits to connect various components within the electronic device. It executes instructions, programs, code sets, or instruction sets stored in the memory 120, as well as accesses data stored in the memory 120, to perform various functions of the electronic device and process data. Optionally, the processor 110 may be implemented using at least one of the following hardware forms: a digital signal processing (DSP), a field-programmable gate array (FPGA), or a programmable logic array (PLA). The processor 110 may integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. The CPU primarily processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem handles wireless communications. It is understood that the modem may not be integrated into the processor 110 and may be implemented separately via a communications chip.

[0153] The memory 120 may include a random access memory (RAM) or a read-only memory (ROM). Optionally, the memory 120 includes a non-transitory computer-readable storage medium. The memory 120 may be used to store instructions, programs, codes, code sets, or instruction sets. The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The operating system may be an Android system, including a system deeply developed based on the Android system, an IOS system developed by Apple, including a system deeply developed based on the IOS system or other systems. The data storage area may also store data created by the electronic device during use, such as a phone book, audio and video data, chat record data, etc.

[0154] Refer to Figure 12, which is a structural diagram of an operating system and user space provided in an embodiment of this specification. The memory 120 can be divided into an operating system space and a user space. The operating system runs in the operating system space, and native and third-party applications run in the user space. In order to ensure that different third-party applications can achieve better operating results, the operating system allocates corresponding system resources to different third-party applications. However, different application scenarios in the same third-party application also have different requirements for system resources. For example, in the local resource loading scenario, the third-party application has higher requirements for disk reading speed; in the animation rendering scenario, the third-party application has higher requirements for GPU performance. The operating system and the third-party application are independent of each other, and the operating system often cannot perceive the current application scenario of the third-party application in a timely manner, resulting in the operating system being unable to perform targeted system resource adaptation according to the specific application scenario of the third-party application.

[0155] In order for the operating system to distinguish the specific application scenarios of third-party applications, it is necessary to open up data communication between third-party applications and the operating system so that the operating system can obtain the current scenario information of third-party applications at any time, and then perform targeted system resource adaptation based on the current scenario.

[0156] See Figure 13, which is an architectural diagram of the Android operating system shown in Figure 12. Taking the Android operating system as an example, the programs and data stored in memory 120 are shown in Figure 13. Memory 120 may store a Linux kernel layer 320, a system runtime library layer 340, an application framework layer 360, and an application layer 380. The Linux kernel layer 320, the system runtime library layer 340, and the application framework layer 360 belong to the operating system space, while the application layer 380 belongs to the user space. The Linux kernel layer 320 provides underlying drivers for various hardware components of electronic devices, such as display drivers, audio drivers, camera drivers, Bluetooth drivers, Wi-Fi drivers, and power management. The system runtime library layer 340 provides key feature support for the Android system through several C / C++ libraries. For example, the SQLite library provides database support, the OpenGL / ES library provides 3D drawing support, and the Webkit library provides browser kernel support. The system runtime library layer 340 also includes the Android runtime, which primarily provides core libraries that allow developers to write Android applications in Java. The application framework layer 360 provides various APIs that may be used when building applications. Developers can also use these APIs to build their own applications, such as activity management, window management, view management, notification management, content provider management, package management, call management, resource management, and location management. The application layer 380 runs at least one application. These applications can be native applications that come with the operating system, such as contacts, SMS, clock, and camera applications, or third-party applications developed by third-party developers, such as games, instant messaging programs, and photo enhancement programs.

[0157] See Figure 14, which is an architectural diagram of the iOS system in Figure 12. Taking the iOS operating system as an example, the programs and data stored in the memory 120 are shown in Figure 14. The iOS system includes: a core operating system layer 420 (Core OS layer), a core services layer 440 (Core Services layer), a media layer 460 (Media layer), and a touchable layer 480 (Cocoa Touch Layer). The core operating system layer 420 includes the operating system kernel, drivers, and underlying program frameworks. These underlying program frameworks provide functions closer to the hardware for use by the program frameworks located in the core services layer 440. The core services layer 440 provides system services and / or program frameworks required by applications, such as the foundation framework, account framework, advertising framework, data storage framework, network connection framework, geolocation framework, motion framework, etc. The media layer 460 provides applications with audio-visual interfaces, such as graphics and image-related interfaces, audio technology-related interfaces, video technology-related interfaces, and wireless playback (AirPlay) interfaces for audio and video transmission technology. The touchable layer 480 provides various commonly used interface-related frameworks for application development. It is responsible for user touch interaction operations on electronic devices, such as local notification services, remote push services, advertising frameworks, game tool frameworks, message user interface (UI) frameworks, UIKit frameworks, and map frameworks.

[0158] In the frameworks shown in FIG14 , those relevant to most applications include, but are not limited to, the Foundation framework in the core services layer 440 and the UIKit framework in the touchable layer 480. The Foundation framework provides many basic object classes and data types, offering fundamental system services for all applications and having nothing to do with the UI. The classes provided by the UIKit framework are the foundational UI class library for creating touch-based user interfaces. iOS applications can use the UIKit framework to provide their UIs, providing the application infrastructure for building user interfaces, drawing, handling user interaction events, responding to gestures, and so on.

[0159] Among them, the method and principle of implementing data communication between third-party applications and the operating system in the IOS system can be referred to the Android system, and this manual will not go into details here.

[0160] Among them, the input device 130 is used to receive input instructions or data, and the input device 130 includes but is not limited to a keyboard, a mouse, a camera, a microphone or a touch device. The output device 140 is used to output instructions or data, and the output device 140 includes but is not limited to a display device and a speaker. In one example, the input device 130 and the output device 140 can be combined, and the input device 130 and the output device 140 are touch screen displays, which are used to receive touch operations on or near the touch screen using any suitable object such as a finger or a touch pen, and to display the user interface of each application. The touch screen display is usually provided on the front panel of the electronic device. The touch screen display can be designed as a full screen, a curved screen or a special-shaped screen. The touch screen display can also be designed as a combination of a full screen and a curved screen, or a combination of a special-shaped screen and a curved screen, which is not limited in this specification.

[0161] In addition, those skilled in the art will understand that the structures of the electronic devices shown in the above figures do not limit the electronic devices. The electronic devices may include more or fewer components than shown, or may combine certain components, or arrange the components differently. For example, the electronic devices may also include radio frequency circuits, input units, sensors, audio circuits, wireless fidelity (WiFi) modules, power supplies, Bluetooth modules, and other components, which are not described in detail here.

[0162] In this specification, the execution entity of each step can be the electronic device described above. Optionally, the execution entity of each step is the operating system of the electronic device. The operating system can be Android, iOS, or other operating systems, which is not limited in this specification.

[0163] The electronic device of this specification may also be equipped with a display device, which may be any device capable of realizing a display function, such as a cathode ray tube display (CR), a light-emitting diode display (LED), an electronic ink screen, a liquid crystal display (LCD), a plasma display panel (PDP), etc. The user may use the display device on the electronic device 101 to view displayed text, images, videos and other information. The electronic device may be a smart phone, a tablet computer, a gaming device, an AR (Augmented Reality) device, a car, a data storage device, an audio playback device, a video playback device, a notebook, a desktop computing device, a wearable device such as an electronic watch, electronic glasses, an electronic helmet, an electronic bracelet, an electronic necklace, electronic clothing and the like.

[0164] In the electronic device shown in Figure 11, which can be a terminal, the processor 110 can be used to call the fact evaluation model training program stored in the memory 120, and specifically perform the following operations: create an initial fact evaluation model, obtain reference text, and generate a sample text pair with a factual evaluation label based on the reference text; input the sample text pair into the initial fact evaluation model for model training, determine the factual evaluation probability distribution information of the sample text pair based on the initial fact evaluation model, and output the factual evaluation result of the sample text pair based on the factual evaluation probability distribution information; determine the model loss value based on the factual evaluation label and the factual evaluation result, and use the model loss value to adjust the model parameters of the initial fact evaluation model until the initial fact evaluation model completes model training to obtain the target fact evaluation model.

[0165] Optionally, when the processor 110 generates a sample text pair with a factual evaluation label based on a reference text, it specifically performs the following steps: performing identical factual data generation processing on the reference text to obtain a positive text, and performing contradictory factual data generation processing on the reference text to obtain a negative text; generating a positive text pair with a positive factual evaluation label based on the positive text and the reference text, and generating a negative text pair with a negative factual evaluation label based on the negative text and the reference text; and obtaining a sample text pair with a factual evaluation label based on the positive text pair and the negative text pair.

[0166] Optionally, when the processor 110 performs the same factual data generation processing on the reference text and obtains the positive text, it specifically executes: performing the same factual rewriting processing on the reference text based on the positive text generation model to obtain the positive text; wherein, the positive text generation model is adapted for the same factual rewriting processing scenario based on the first basic large language generation model.

[0167] Optionally, the processor 110 performs same-fact rewriting processing on the reference text based on the forward text generation model. When the forward text is obtained, the processor 110 specifically performs: based on the forward text generation model, at least one of the following processing is used to rewrite the reference text to obtain the forward text.

[0168] Optionally, when the processor 110 performs contradictory factual data generation processing on the reference text and obtains negative text, it specifically executes: performing contradictory factual rewriting processing on the reference text based on the negative text generation model to obtain negative text; wherein, the negative text generation model is adapted for the contradictory factual rewriting processing scenario based on the second basic language generative model.

[0169] Optionally, the processor 110 performs a contradictory factual rewriting process on the reference text based on the negative text generation model. When the negative text is obtained, the processor 110 specifically performs: based on the negative text generation model, at least one of contradictory factual numerical transformation processing, contradictory factual antonym replacement processing, and contradictory factual subject-object exchange processing is used to rewrite the reference text to obtain the negative text.

[0170] Optionally, when the processor 110 determines the factual evaluation probability distribution information of the sample text pair based on the initial fact evaluation model, it specifically performs: determining a first probability that the sample text pair is the same factual text based on the initial fact evaluation model; determining a second probability that the sample text pair is contradictory factual text based on the initial fact evaluation model; and obtaining the factual evaluation probability distribution information based on the first probability and the second probability.

[0171] Optionally, when the processor 110 executes the output of the factual evaluation result of the sample text pair based on the factual evaluation probability distribution information, it specifically performs: comparing the first probability and the second probability, when the first probability is greater than or equal to the second probability, outputting the positive factual evaluation result of the sample; when the first probability is less than the second probability, outputting the negative factual evaluation result of the sample.

[0172] In addition, the processor 110 can also be used to call the fact evaluation program stored in the memory 120, and perform the following specific operations: obtain the reference text and target generated text for the target large model, the reference text is the input text of the target large model, and the target generated text is the output text of the target large model; obtain the target text pair based on the reference text and the target generated text, input the target text pair into the target fact evaluation model, determine the target fact evaluation probability distribution information of the target text pair based on the target fact evaluation model, and output the target fact evaluation result of the target text pair based on the target fact evaluation probability distribution information.

[0173] Optionally, the processor 110 is further adapted to determine a first probability that the target text pair is the same factual text based on the target factuality evaluation probability distribution information; and determine a fact consistency score of the target text pair based on the first probability.

[0174] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware through a computer program. The program can be stored in a computer-readable storage medium, and when executed, the program can include the processes in the above-described method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory, or a random access memory.

[0175] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in the embodiments of this specification are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the object characteristics, interactive behavior characteristics, and user information involved in this specification are all obtained with full authorization.

[0176] The above disclosure is only a preferred embodiment of this specification, and certainly cannot be used to limit the scope of rights of this specification. Therefore, equivalent changes made according to the claims of this specification are still within the scope covered by this specification.

Claims

1. A fact assessment model training method, the method comprising: Creating an initial factual evaluation model, obtaining reference text, and generating sample text pairs with factual evaluation labels based on the reference text; Inputting the sample text pair into the initial fact evaluation model for model training, determining factual evaluation probability distribution information of the sample text pair based on the initial fact evaluation model, and outputting a factual evaluation result of the sample text pair based on the factual evaluation probability distribution information; A model loss value is determined based on the factual evaluation label and the factual evaluation result, and the model loss value is used to adjust the model parameters of the initial fact evaluation model until the initial fact evaluation model completes model training to obtain a target fact evaluation model.

2. The method according to claim 1, wherein generating a sample text pair with a factual evaluation label based on the reference text comprises: Performing a process of generating identical factual data on the reference text to obtain a positive text, and performing a process of generating contradictory factual data on the reference text to obtain a negative text; generating a positive text pair with a positive factual evaluation label based on the positive text and the reference text, and generating a negative text pair with a negative factual evaluation label based on the negative text and the reference text; A sample text pair with a factual evaluation label is obtained based on the positive text pair and the negative text pair.

3. The method according to claim 2, wherein the step of generating the same factual data as the reference text to obtain the forward text comprises: The reference text is rewritten with the same facts based on a forward text generation model to obtain a forward text; wherein the forward text generation model is adapted for the same factual rewriting processing scenario based on the first basic large language generation model.

4. The method according to claim 3, wherein the step of rewriting the reference text based on the forward text generation model to obtain the forward text comprises: Based on the positive text generation model, the reference text is rewritten by adopting at least one of the same factual word order rearrangement processing, the same factual active-passive conversion processing, the same factual sentence fusion processing, the same factual text abbreviation processing, and the same factual synonym replacement processing to obtain the positive text.

5. The method according to claim 2, wherein the generating of contradictory factual data on the reference text to obtain a negative text comprises: The reference text is rewritten based on a negative text generation model to obtain a negative text; wherein the negative text generation model is adapted for a contradictory factual rewriting processing scenario based on a second basic language generation model.

6. The method according to claim 5, wherein the step of rewriting the reference text to contradict facts based on the negative text generation model to obtain the negative text comprises: Based on the negative text generation model, at least one of the contradictory factual numerical transformation processing, contradictory factual antonym replacement processing, and contradictory factual subject-object exchange processing is adopted to rewrite the reference text to obtain a negative text.

7. The method according to claim 1, wherein determining the factuality evaluation probability distribution information of the sample text pair based on the initial fact evaluation model comprises: Determining a first probability that the sample text pair is the same factual text based on the initial fact evaluation model; Determining a second probability that the sample text pair is a contradictory factual text based on the initial fact evaluation model; Factual evaluation probability distribution information is obtained based on the first probability and the second probability.

8. The method according to claim 7, wherein outputting the factuality evaluation result of the sample text pair based on the factuality evaluation probability distribution information comprises: Comparing the first probability with the second probability, and outputting a positive factual evaluation result of the sample when the first probability is greater than or equal to the second probability; When the first probability is less than the second probability, a negative factual evaluation result of the sample is output.

9. A fact assessment method, comprising: Obtaining a reference text and a target generated text for a target large model, wherein the reference text is the input text of the target large model and the target generated text is the output text of the target large model; A target text pair is obtained based on the reference text and the target generated text, the target text pair is input into a target fact evaluation model, target factuality evaluation probability distribution information of the target text pair is determined based on the target fact evaluation model, and a target factuality evaluation result of the target text pair is output based on the target factuality evaluation probability distribution information.

10. The method according to claim 9, further comprising: Determining a first probability that the target text pair is the same factual text based on the target factuality evaluation probability distribution information; A fact consistency score for the target text pair is determined based on the first probability.

11. A fact assessment model training device, comprising: A creation module, adapted to create an initial factual evaluation model, obtain reference text, and generate sample text pairs with factual evaluation labels based on the reference text; a model training module adapted to input the sample text pair into the initial fact assessment model for model training, determine factuality assessment probability distribution information of the sample text pair based on the initial fact assessment model, and output a factuality assessment result of the sample text pair based on the factuality assessment probability distribution information; The model generation module is suitable for determining a model loss value based on the factual evaluation label and the factual evaluation result, and using the model loss value to adjust the model parameters of the initial fact evaluation model until the initial fact evaluation model completes model training to obtain a target fact evaluation model.

12. A fact assessment device, comprising: an acquisition module, adapted to acquire a target reference text and a target generated text for a target macromodel, wherein the target reference text is an input text of the target macromodel, and the target generated text is an output text of the target macromodel; A factuality evaluation module is suitable for obtaining a target text pair based on the target reference text and the target generated text, inputting the target text pair into a target factual evaluation model, determining target factuality evaluation probability distribution information of the target text pair based on the target factual evaluation model, and outputting a target factuality evaluation result of the target text pair based on the target factual evaluation probability distribution information.

13. A computer storage medium storing a plurality of instructions, wherein the instructions are suitable for being loaded by a processor and executing the method steps according to any one of claims 1 to 8 or 9 or 10.

14. A computer program product, the computer program product storing at least one instruction, wherein the at least one instruction is loaded by a processor and executes the method steps according to any one of claims 1 to 8 or 9 or 10.

15. An electronic device comprising: A processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the method steps according to any one of claims 1 to 8 or 9 or 10.

Images