Model generation result processing method, device, electronic device and storage medium

Abstract

The present disclosure provides a method, device, electronic device and storage medium for processing model generation results, and relates to the fields of artificial intelligence technology such as machine learning and natural language processing. The specific implementation scheme is: the text generation result of the generative large model is disassembled to obtain multiple result logic units; each result logic unit includes a fragment in the text generation result; each fragment can independently identify a premise or conclusion in the logical reasoning relationship of the text generation result; the text generation result is a response result generated by the generative large model based on the text input information; based on multiple result logic units, a logical reasoning graph that can characterize the logical reasoning relationship between multiple result logic units is generated; based on the logical reasoning graph, it is determined whether the logical reasoning of the text generation result generated by the generative large model is correct. The technology disclosed in the present disclosure can efficiently and accurately determine whether the logical reasoning of the generative large model is correct.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, specifically to the field of artificial intelligence technology such as machine learning and natural language processing, and more particularly to a method, device, electronic device and storage medium for processing model generation results. Background Art

[0002] With the widespread application of generative large models, the effectiveness evaluation of generative large models has also become a very important technology.

[0003] The effect evaluation of generative large models refers to determining whether the text generation results of generative large models are correct. The effect evaluation of generative large models is different from the task evaluation of traditional models in that the text generation results of generative large models are not unique and fixed, so the effect evaluation cannot be simply evaluated by whether the string matches. Summary of the invention

[0004] The present invention provides a method, device, electronic device and storage medium for processing model generation results.

[0005] According to one aspect of the present disclosure, a method for processing a model generation result is provided, comprising:

[0006] The text generation result of the generative large model is disassembled to obtain a plurality of result logic units; each of the result logic units includes a fragment in the text generation result; each of the fragments can independently identify a premise or conclusion in the logical reasoning relationship of the text generation result; the text generation result is a response result generated by the generative large model based on text input information;

[0007] Based on the multiple result logic units, generating a logical reasoning graph capable of characterizing the logical reasoning relationship between the multiple result logic units;

[0008] Based on the logical reasoning graph, it is evaluated whether the logical reasoning used by the generative large model to generate the text generation result is correct.

[0009] According to another aspect of the present disclosure, there is provided a device for processing a model generation result, comprising:

[0010] A disassembly module is used to disassemble the text generation result of the generative large model to obtain multiple result logic units; each of the result logic units includes a fragment in the text generation result; each of the fragments can independently identify a premise or conclusion in the logical reasoning relationship of the text generation result; the text generation result is a response result generated by the generative large model based on text input information;

[0011] A generating module, configured to generate, based on the plurality of result logic units, a logical reasoning graph capable of representing a logical reasoning relationship between the plurality of result logic units;

[0012] An evaluation module is used to evaluate whether the logical reasoning of the text generation result generated by the generative large model is correct based on the logical reasoning diagram.

[0013] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0014] at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any possible implementation manner and the aspects described above.

[0017] According to yet another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method of the above-mentioned aspect and any possible implementation manner.

[0018] According to yet another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the computer program implements the above-mentioned aspects and any possible implementation method.

[0019] According to the technology disclosed in the present invention, it is possible to efficiently and accurately determine whether the logical reasoning of the generative large model is correct, thereby effectively improving the evaluation efficiency of the generative large model.

[0020] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0022] Figure 1 is a schematic diagram according to a first embodiment of the present disclosure;

[0023] Figure 2 is a schematic diagram according to a second embodiment of the present disclosure;

[0024] Figure 3 It is a logical reasoning diagram provided by the present disclosure;

[0025] Figure 4 is a schematic diagram according to a third embodiment of the present disclosure;

[0026] Figure 5 is a schematic diagram of a logical reasoning graph generated by this embodiment;

[0027] Fig. 6A , Figure 6B and Figure 6C They are Figure 5 The logical reasoning graph shown is split into two subgraphs;

[0028] Figure 7 is a schematic diagram according to a fourth embodiment of the present disclosure;

[0029] Figure 8 is a schematic diagram according to a fifth embodiment of the present disclosure;

[0030] Fig. 9 The present invention is a block diagram of a device for implementing the method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0032] Obviously, the described embodiments are only part of the embodiments of the present disclosure, but not all of them. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present disclosure.

[0033] It should be noted that the terminal devices involved in the embodiments of the present disclosure may include but are not limited to mobile phones, personal digital assistants (PDAs), wireless handheld devices, tablet computers and other smart devices; display devices may include but are not limited to personal computers, televisions and other devices with display functions.

[0034] In addition, the term "and / or" in this article is only a description of the association relationship between the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0035] Figure 1 is a schematic diagram according to the first embodiment of the present disclosure; Figure 1 As shown, this embodiment provides a method for processing model generation results, which may specifically include the following steps:

[0036] S101, disassembling the text generation result of the generative large model to obtain multiple result logic units;

[0037] Each result logic unit in this embodiment includes a fragment in the text generation result; each fragment can independently identify a premise or conclusion in the logical reasoning relationship of the text generation result; the text generation result is a response result generated by the generative large model based on the text input information.

[0038] The executor of the method for processing the model generation results of this embodiment may be a device for processing the model generation results, which may be an independent electronic device, or may be a software-based application to evaluate the generation effect of the generative large model.

[0039] The generative large model of this embodiment may also be called a generative language model (General Language Model; GLM), or a generative large language model.

[0040] In this embodiment, taking the use of a generative large model in the field of text processing as an example, when in use, text input information is input into the generative large model, and the generative large model can generate and output text generation results based on the text input information. In actual application scenarios, in order to improve the effect of the generative large model generating text generation results, it is necessary to perform some corresponding processing on the generation results of the generative large model to achieve efficient and accurate determination of whether the logical reasoning of the generative large model is correct, and then the generative large model can be accurately and effectively evaluated. And based on the evaluation results, the generative large model can be optimized in turn to further improve the text generation effect of the generative large model.

[0041] Since the text generation results of the generative large model are usually long and may include multiple sentences, the fragment in each result logic unit can be a sentence in the text generation result, or two or more consecutive sentences. That is, the fragment must be a continuous paragraph in the text generation result, and cannot be a splicing of two or more discontinuous paragraphs.

[0042] Specifically, in this embodiment, the principle of decomposing the text generation result is to ensure that each result logic unit after decomposition is a premise or conclusion when performing logical relationship reasoning on the text generation result.

[0043] S102, based on the multiple result logic units, generating a logic reasoning graph capable of representing the logic reasoning relationship between the multiple result logic units;

[0044] The logical reasoning graph of this embodiment is a directed acyclic graph formed by multiple result logic units according to logical reasoning relationships.

[0045] S103. Based on the logical reasoning diagram, determine whether the logical reasoning of the text generation result generated by the generative large model is correct.

[0046] Since the generation of the logical reasoning graph is based on the logical reasoning relationship between multiple result logical units, the logical reasoning relationship between multiple result logical units in the logical reasoning graph can be used to evaluate whether the logical reasoning of the text generation result generated by the generative large model is correct, and then the generative large model can be evaluated from the logical reasoning dimension. That is, if the logical reasoning of the text generation result generated by the generative large model is correct, it means that the logical reasoning of the generative large model is correct; if the logical reasoning of the text generation result generated by the generative large model is incorrect, it means that the logical reasoning of the generative large model is incorrect.

[0047] The method for processing the model generation result of this embodiment is to obtain multiple result logic units by disassembling the text generation result, generate a logical reasoning graph of multiple result logic units, and further determine whether the logical reasoning of the text generation result generated by the generative large model is correct based on the logical reasoning graph, thereby effectively and accurately determining whether the logical reasoning of the generative large model is correct. Therefore, by adopting the technical solution of this embodiment, the text generation result of the generative large model can be effectively evaluated from the dimension of logical reasoning, thereby effectively and accurately evaluating the generative large model. Moreover, the technical solution of this embodiment can run fully automatically without manual processing, which can greatly reduce labor costs, improve evaluation speed, and effectively improve the evaluation efficiency of the generative large model.

[0048] Figure 2 is a schematic diagram according to the second embodiment of the present disclosure; the processing method of the model generation result of this embodiment, in the above Figure 1 Based on the technical solutions of the embodiments shown in the figure, the technical solutions of the present disclosure are further described in more detail. Figure 2 As shown, the method for processing the model generation result of this embodiment may specifically include the following steps:

[0049] S201, using a pre-trained logic disassembly model to disassemble the text generation result of the generative large model to obtain multiple result logic units;

[0050] This step is the above Figure 1An implementation of step S101 of the illustrated embodiment. In this implementation, a pre-trained logic disassembly model is used to automatically disassemble the text generation result. In specific implementation, the text generation result is input into the logic disassembly model, and the logic disassembly model can directly output a plurality of corresponding result logic units based on the input text generation result.

[0051] The logic disassembly model of this embodiment can collect multiple sets of training data before training, each set of training data includes training corpus and multiple training logic units obtained by manually disassembling the training corpus. The principle of disassembly is also to ensure that each training logic unit after disassembly is a premise or conclusion when performing logical relationship reasoning on the training corpus. Then, the logic disassembly model is trained using multiple sets of training data so that the logic disassembly model learns the ability of disassembly.

[0052] In this embodiment, by adopting a logical decomposition model to intelligently decompose the text generation result, the accuracy of multiple decomposed result logical units can be effectively improved, and the decomposition efficiency can be improved.

[0053] Additionally, optionally, Figure 1 Step S101 in the illustrated embodiment may also decompose the text generation result of the generative large model according to a preset decomposition strategy to obtain a plurality of result logic units.

[0054] For example, the preset disassembly strategy can be configured to ensure that each result logic unit after disassembly is a premise or conclusion in the logical reasoning relationship of the text generation result.

[0055] In specific implementation, the text generation result can be split according to the sentence granularity first, and then all the segmentation methods can be traversed to obtain multiple segmentation results of the text generation result. In each segmentation method, a sentence can be divided into a segment individually, or two or more adjacent continuous sentences can be divided into a segment. Then, each segmented segment in each segmentation result can be tested to see whether it can be used as a premise or a conclusion in the logical reasoning relationship of the text generation result. Finally, from all the segmentation results, a segmentation result in which all the segmented segments can be used as a premise or a conclusion is selected as the final required segmentation result, and multiple result logic units are obtained accordingly.

[0056] By disassembling according to the preset disassembly strategy, multiple result logic units can be obtained efficiently and accurately.

[0057] S202, based on multiple result logic units, using a pre-trained logic reasoning graph generation model to generate a logic reasoning graph that can represent the logic reasoning relationship between the multiple result logic units;

[0058] In this embodiment, a logical reasoning graph generation model can be pre-trained. When used, multiple result logic units are input into the logical reasoning graph generation model. The logical reasoning graph generation model can generate a logical reasoning graph composed of multiple result logic units based on the input information.

[0059] The logic reasoning graph generation model can also be trained using multiple sets of training data during training, and each training data may include multiple training logic units and training logic reasoning graphs annotated based on multiple training logic units. By using multiple sets of training data to train the logic reasoning graph generation model, the logic reasoning graph generation model can learn the ability to generate corresponding training logic reasoning graphs based on multiple training logic units.

[0060] When in use, multiple result logic units are input into the logic reasoning graph generation model trained in the above manner. At this time, the logic reasoning graph generation model can generate and output a logic reasoning graph that can characterize the logical reasoning relationship between multiple result logic units.

[0061] The above-mentioned method of intelligently generating a logical reasoning graph according to a logical reasoning graph generation model can efficiently and accurately generate logical reasoning graphs of multiple result logic units.

[0062] S203, decomposing the logic reasoning graph into multiple two-layer subgraphs; each two-layer subgraph identifies a logic reasoning step;

[0063] Specifically, when disassembling the second-level subgraph, the smallest logical relationship subgraph, i.e., the second-level subgraph, is split according to the reasoning relationship in the logical reasoning graph. Figure 3 is a logical reasoning diagram provided by the present disclosure. According to the disassembly process of this embodiment, Figure 3 The logical reasoning graph shown is decomposed into two two-layer subgraphs, such as the two-layer subgraph composed of A, B and C, and the two-layer subgraph composed of C, D and E. Each two-layer subgraph corresponds to a logical reasoning step. In this way, all two-layer subgraphs in the logical reasoning graph can be decomposed.

[0064] S204, determining whether the logical reasoning of each second-layer subgraph is correct;

[0065] Specifically, when judging, a pre-trained reasoning model can be used to infer whether the logical reasoning of each two-layer subgraph is correct. For example, when in use, each two-layer subgraph can be input into the reasoning model in order from top to bottom and from left to right in the logical reasoning graph. Then the reasoning model determines in turn whether the logical reasoning represented by each input two-layer subgraph is correct. The reasoning model of this embodiment can also be implemented using a large language model (Large Language Model; LLM). Optionally, in this embodiment, it is also possible to determine whether the reasoning of each two-layer subgraph is correct by setting a certain strategy, and it is also possible to accurately determine whether the logical reasoning of each two-layer subgraph is correct.

[0066] S205. In response to the logical reasoning of the multiple second-layer subgraphs being correct, determine that the logical reasoning of the text generation result generated by the generative large model is correct.

[0067] In other words, as long as the logical reasoning of one of the multiple second-layer subgraphs is incorrect, it is considered that the logical reasoning of the text generation result generated by the generative large model is incorrect, and then it is determined that the logical reasoning of the generative large model is incorrect.

[0068] Steps S203-S205 of this embodiment are as described above. Figure 1 A specific implementation of step S103 of the illustrated embodiment.

[0069] Specifically, in this implementation, the logical reasoning graph is decomposed into multiple two-layer sub-graphs. By determining whether the single logical reasoning step of each two-layer sub-graph is correct, it is determined whether the logical reasoning of the text generation result generated by the generative large model is correct, and then it is determined whether the logical reasoning generated by the generative large model is correct.

[0070] In this embodiment, through this method, only one two-layer subgraph is judged each time, that is, only a single logical reasoning step is judged at a time. This can effectively reduce the difficulty of judging whether the logical reasoning of the text generation result generated by the generative large model is correct, and effectively improve the accuracy of judging whether the logical reasoning of the generative large model is correct. This can effectively improve the accuracy and efficiency of the evaluation of the text generation result of the generative large model, and can greatly improve the evaluation effect of the generative large model.

[0071] The processing method of the model generation result of this embodiment adopts the above-mentioned method, and can automatically realize the evaluation of the generative large model by accurately judging whether the logical reasoning of the generative large model is correct, saving time and labor, effectively shortening the evaluation speed, and improving the evaluation efficiency of the generative large model. Moreover, by judging the logical reasoning of each second-layer subgraph of the logical reasoning graph of the text generation result, the logical reasoning of the text generation result is judged, and then the text generation result of the generative large model is evaluated, which can effectively improve the accuracy and efficiency of the evaluation.

[0072] Figure 4 is a schematic diagram according to the third embodiment of the present disclosure; the processing method of the model generation result of this embodiment, in the above Figure 1 Based on the technical solutions of the embodiments shown in the figure, the technical solutions of the present disclosure are further described in more detail. Figure 4 As shown, the method for processing the model generation result of this embodiment may specifically include the following steps:

[0073] S401, using a pre-trained logic decomposition model, respectively decomposing the text input information and the text generation result of the generative large model to obtain a plurality of input logic units and a plurality of result logic units;

[0074] Each input logic unit includes a fragment of the text input information, and each fragment can independently identify a premise or conclusion in the logical reasoning relationship of the text input information. Each result logic unit includes a fragment of the text generation result; each fragment corresponds to a premise or conclusion in the logical reasoning relationship that can independently identify the text generation result. For specific implementation methods, please refer to the above Figure 2 The relevant records of step S201 of the illustrated embodiment are not repeated here.

[0075] S402, based on the multiple result logic units and with reference to the multiple input logic units, generating a logic reasoning graph capable of representing the logic reasoning relationship between the multiple result logic units;

[0076] That is to say, the generation of the logical reasoning graph of the multiple result logic units in this embodiment is not only based on the multiple result logic units, but also refers to the multiple input logic units.

[0077] For example, when step S402 is specifically implemented, it may include the following steps:

[0078] (1) based on multiple input logical units and multiple result logical units, retrieve the most relevant examples from a preset example database;

[0079] (2) Based on multiple input logic units and multiple result logic units, as well as multiple input example logic units, multiple result example logic units and corresponding example logic reasoning graphs, a pre-trained other generative large model is used to generate logic reasoning graphs corresponding to multiple result logic units.

[0080] In the preset example database of this embodiment, multiple groups of examples may be stored. Each group of examples may include multiple input example logic units corresponding to input example information, multiple result example logic units corresponding to result example information, and example logical reasoning graphs corresponding to multiple result example logic units.

[0081] In this embodiment, the information in the example database is also in text form, that is, the input example information in each group of examples is text input example information, and the result example information is text result example information. That is, the input example information can be the text input information of the generative large model, and the result example information can be the text generation result generated by the generative large model. Multiple input example logic units and multiple result example logic units can be obtained by disassembling multiple input logic units and multiple result logic units in the above step S401. The example logical reasoning diagram of this embodiment can be manually annotated based on multiple result example logic units.

[0082] During the specific search, in this embodiment, the most relevant examples can be retrieved in a preset example database based on multiple input logic units and multiple result logic units. The most relevant examples here can be considered to be examples corresponding to multiple input example logic units and multiple result example logic units that are most relevant in terms of the overall contextual semantics of the multiple input logic units and the multiple result logic units.

[0083] Then, multiple input logic units and multiple result logic units, as well as the various parts of information included in the most relevant examples, can be spliced ​​and input into the generative big model together, so that other pre-trained generative big models can generate example logical reasoning graphs corresponding to multiple result example logical units based on the generation method of multiple result example logical units in the most relevant examples. In this embodiment, the pre-trained other generative big models are not the generative big models to be evaluated in this embodiment, but other generative big models that have already been trained.

[0084] Optionally, in this embodiment, each group of examples in the example database may only include a plurality of result example logic units corresponding to the result example information and example logical reasoning graphs corresponding to the plurality of result example logic units.

[0085] Correspondingly, during specific retrieval, the most relevant examples can be retrieved in a preset example database based only on multiple result logic units; and based on multiple result example logic units and corresponding example logic reasoning graphs, other pre-trained generative large models can be used to generate logic reasoning graphs corresponding to multiple result logic units.

[0086] However, compared with the above-mentioned example database that includes multiple input example logic units corresponding to the input example information and the corresponding technical solution, the accuracy of this implementation is slightly inferior.

[0087] In this embodiment, by retrieving the most relevant examples and using other generative large models to intelligently generate logical reasoning graphs, logical reasoning graphs of multiple result logical units can be accurately and efficiently generated.

[0088] S403, decomposing the logic reasoning graph into multiple two-layer subgraphs; each two-layer subgraph identifies a logic reasoning step;

[0089] S404, determining whether the logical reasoning of each second-layer subgraph is correct;

[0090] S405. In response to the fact that the reasoning of the plurality of second-layer subgraphs is correct, it is determined that the logical reasoning of the text generation result generated by the generative large model is correct.

[0091] The specific implementation of steps S403-S405 of this embodiment can refer to the above Figure 2 The description of steps S203 - S205 of the illustrated embodiment will not be repeated here.

[0092] The method for processing the model generation results of this embodiment, by adopting the above-mentioned manner, can further more accurately and efficiently generate logical reasoning diagrams of multiple result logical units, which can further improve the accuracy of judging whether the logical reasoning of the generative large model is correct, and thus can effectively improve the accuracy and efficiency of the evaluation of the generative large model.

[0093] The following is a specific example to describe the evaluation method of the generative large model disclosed in the present invention.

[0094] For example, in this example, the text input information query can be: The police caught a thief and caught four suspects: A, B, C, and D. Their confessions are: A said: I didn't steal it. B said: A stole it. C said: It wasn't me. D said: B stole it. It is known that only one of them told the truth. Who is the thief?

[0095] The text input information is input into the generative large model, and the text generation result response generated by the generative large model is: According to the question, "B said: A stole it; A said: I didn't steal it", then one of A and B must have told the truth, then C and D both lied, that is, C lied, and C is the thief.

[0096] For example, according to step S401, the text input information query is decomposed to obtain multiple input logic units, including: the first input logic unit [query-unit-1], the police caught a thief and arrested four suspects: A, B, C, and D; their confessions were; the second input logic unit [query-unit-2], A said: I didn’t steal it; the third input logic unit [query-unit-3], B said: A stole it; the fourth input logic unit [query-unit-4], C said: It wasn’t me; the fifth input logic unit [query-unit-5], D said: B stole it; the sixth input logic unit [query-unit-6], it is known that only one of them told the truth; the seventh input logic unit [query-unit-7], who is the thief?

[0097] Similarly, according to step S401, the text generation result response is disassembled to obtain multiple result logic units, including: the first result logic unit [response-unit-1], B said: A stole it; the second result logic unit [response-unit-2], A said: I didn’t steal it; the third result logic unit [response-unit-3], then one of A and B must have told the truth; the fourth input logic unit [response-unit-4], then C and D are both lying; the fifth input logic unit [response-unit-5], that is, C lied; the sixth input logic unit [response-unit-6], C is the thief.

[0098] Next, a logical reasoning diagram of multiple result logic units may be generated according to the method of step S302.

[0099] In this embodiment, the process of logical reasoning is not a chain or tree structure, but more like a directed acyclic graph. The process of logical reasoning represents the reasoning process of the problem, and each node of the generated logical reasoning graph is a minimum logical unit, that is, a result logical unit. Each secondary subgraph in the logical reasoning graph represents a minimum reasoning process.

[0100] The example database of this embodiment may specifically be an In-Context-Learning (ICL) database.

[0101] Specifically, according to the specific implementation of the above step S402, searching in the ICL database can be called ICL searching. During the specific search, multiple input logic units and multiple result logic units are input, and the most relevant examples are retrieved in the ICL database according to the input multiple input logic units and multiple result logic units. And they are spliced ​​into prompts in a one-shot manner, and the spliced ​​prompts are spliced ​​from multiple input example logic units, multiple result example logic units, and example logic reasoning diagrams in the most relevant examples.

[0102] Then, multiple input logic units, multiple result logic units, and spliced ​​prompts are used to generate logical reasoning graphs corresponding to multiple result logic units using a generative large model. For example, Figure 5 It is a schematic diagram of the logical reasoning diagram generated by this embodiment.

[0103] Finally, based on Figure 5 The obtained logical reasoning graph is logically judged. In practical applications, it is difficult to judge the correctness of the entire logical reasoning graph at one time. Therefore, the implementation method of the above steps S403-S405 can be referred to. The logical reasoning graph is first disassembled into two-layer subgraphs, each of which represents a logical reasoning step. Only one two-layer subgraph is judged each time, that is, only a single logical reasoning step is judged at a time. This method can reduce the overall judgment difficulty and greatly improve the judgment effect.

[0104] The specific judgment process can be implemented using a logical reasoning model based on LLM. LLM is used to judge the correctness of the logical reasoning steps of each second-level subgraph in the logical reasoning graph from bottom to top. If each logical reasoning step is correct, the entire logical reasoning process is correct. Otherwise, it is determined that there are logical problems in the logical reasoning process. This method can not only give a correctness judgment on the logical reasoning of logical reasoning problems, but also accurately locate the steps where the errors occur.

[0105] For example, Fig. 6A , Figure 6B and Figure 6C They are Figure 5 The logical reasoning graph shown is split into two subgraphs. Fig. 6A , Figure 6B and Figure 6CIf the logical reasoning steps of the three second-level subgraphs are correct, it can be determined that the logical reasoning of the text generation results generated by the generative large model is correct, and the generative large model has been evaluated.

[0106] The above method of the embodiment of the present disclosure is to evaluate a text generation result generated by a generative large model. In actual application scenarios, the generative large model can perform multiple generation tasks within a period of time. Specifically, it can be evaluated whether the logical reasoning of the text generation result generated by the generative large model is correct each time the generation task is executed in accordance with the above method of the embodiment of the present disclosure. Furthermore, it can be counted whether the accuracy rate of this period of time reaches a preset ratio threshold, such as 95%, 98% or other ratios. If it is reached, it is determined that the accuracy of the generative large model is high and the effect is good; otherwise, it can be considered that the accuracy and effect of the generative large model are poor.

[0107] Logic is the most important dimension that reflects the logical reasoning ability of a generative large model. It is a necessary ability to solve some complex logical reasoning tasks. If the logical problems in the model text generation results can be found in the evaluation stage, then the effect evaluation and effect optimization of the logical reasoning problem will play a vital role. Compared with other dimensions, logic is a manifestation of higher intelligence, which makes the evaluation of logic extremely difficult. Traditional methods are difficult to achieve good results. Therefore, the processing method of model generation results proposed in this disclosure evaluates the generative large model from a logical perspective. It is an automatic evaluation framework for logical reasoning of generative large models, which can more effectively evaluate the logical logic of the generative large model, thereby effectively improving the evaluation accuracy and evaluation effect of the logical reasoning of the generative large model.

[0108] The method for processing the above-mentioned model generation results of the embodiment of the present disclosure is to first cut the logical units, then generate a logical reasoning graph, and obtain each two-layer subgraph in the logical reasoning graph, and then evaluate it with each two-layer subgraph as the granularity, that is, the minimum logical reasoning step. This can greatly reduce the difficulty of determining whether the logical reasoning of the generative large model is correct, that is, it can effectively reduce the difficulty of evaluating the generative large model, thereby greatly improving the evaluation effect of the generative large model.

[0109] Moreover, the method for processing the above-mentioned model generation results of the embodiment of the present disclosure uses natural language to represent the entire reasoning process, which is more universal and can be applied to more types of problems.

[0110] Moreover, the processing method of the above-mentioned model generation results of the embodiment of the present invention, by generating a logical reasoning diagram, represents the logical reasoning process of the overall logical reasoning, which can not only effectively detect the logical reasoning errors of the text generation results, but also detect the lack of logical reasoning and the inversion of logical reasoning, and can be used for data logic repair.

[0111] Moreover, the processing method of the above-mentioned model generation result of the embodiment of the present disclosure can accurately determine whether the logical reasoning of the generative large model is correct, and then automatically realize the evaluation of the generative large model, saving time and effort, effectively shortening the evaluation speed, and effectively improving the evaluation efficiency. Moreover, by generating a logical reasoning graph for evaluation, and by evaluating the logical reasoning of each second-layer subgraph in the minimum logical unit, that is, the logical reasoning graph, it is possible to further effectively improve the accuracy of whether the logical reasoning of the generative large model is correct, and then effectively improve the accuracy of the evaluation of the generative large model and improve the evaluation efficiency.

[0112] In the above-mentioned embodiments of the present disclosure, the application in the field of text processing is taken as an example. In actual applications, the technical solution of the present disclosure can also be applied in the field of speech processing. For example, speech information can be collected first, and corresponding text information can be obtained through speech recognition. Then, based on the text information, the above-mentioned technical solution of the present disclosure is adopted to realize the processing of the generation result of the model, and then efficiently and accurately determine whether the logical reasoning of the generative large model is correct, and finally realize accurate and effective evaluation of the generative large model.

[0113] Figure 7 is a schematic diagram according to a fourth embodiment of the present disclosure; Figure 7 As shown, this embodiment provides a model generation result processing device 700, which is applied to the field of text processing, including:

[0114] Disassembly module 701, used for disassembling the text generation result of the generative large model to obtain multiple result logic units; each result logic unit includes a fragment in the text generation result; each fragment can independently identify a premise or conclusion in the logical reasoning relationship of the text generation result; the text generation result is a response result generated by the generative large model based on text input information;

[0115] A generating module 702, configured to generate, based on the plurality of result logic units, a logical reasoning graph capable of representing a logical reasoning relationship between the plurality of result logic units;

[0116] The evaluation module 703 is used to evaluate whether the logical reasoning of the text generation result generated by the generative large model is correct based on the logical reasoning diagram.

[0117] The device 700 for processing the model generation result of this embodiment, by adopting the above-mentioned modules, realizes the implementation principle and technical effect of the processing of the model generation result, which is the same as the implementation of the above-mentioned related method embodiments. For details, please refer to the records of the above-mentioned related method embodiments, which will not be repeated here.

[0118] Figure 8 is a schematic diagram according to the fifth embodiment of the present disclosure; Figure 8 As shown, this embodiment provides a processing device 800 for model generation results, including the above Figure 7 Modules with the same name and function are shown as: a disassembly module 801 , a generation module 802 and an evaluation module 803 .

[0119] In the model generation result processing device 800 of this embodiment, the disassembly module 801 is used to:

[0120] A pre-trained logic disassembly model is used to disassemble the text generation result of the generative large model to obtain multiple result logic units.

[0121] Further optionally, in one embodiment of the present disclosure, the generating module 802 is used to:

[0122] Based on the multiple result logic units, a pre-trained logic reasoning graph generation model is used to generate a logic reasoning graph that can characterize the logic reasoning relationship between the multiple result logic units.

[0123] Further optionally, in one embodiment of the present disclosure, the disassembly module 801 is further used to:

[0124] The text input information of the generative large model is disassembled to obtain multiple input logic units; each of the input logic units includes a fragment in the text input information, and each of the fragments can independently identify a premise or conclusion in a logical reasoning relationship.

[0125] Further optionally, in one embodiment of the present disclosure, the generating module 802 is used to:

[0126] Based on the multiple result logic units and with reference to the multiple input logic units, the logic reasoning graph capable of representing the logic reasoning relationship between the multiple result logic units is generated.

[0127] Further optionally, in one embodiment of the present disclosure, the generating module 802 is used to:

[0128] Based on the multiple input logic units and the multiple result logic units, the most relevant examples are retrieved in a preset example database; the example database includes multiple groups of examples, each of the examples includes multiple input example logic units corresponding to the input example information, multiple result example logic units corresponding to the result example information, and example logical reasoning graphs corresponding to the multiple result example logic units;

[0129] Based on the multiple input logic units and the multiple result logic units, as well as the multiple input example logic units, the multiple result example logic units and the corresponding example logic reasoning graphs, other pre-trained generative large models are used to generate the logic reasoning graphs corresponding to the multiple result logic units.

[0130] Further optionally, if Figure 8 As shown, in one embodiment of the present disclosure, the evaluation module 803 includes:

[0131] The disassembling unit 8031 ​​is used to disassemble the logic reasoning graph into multiple two-layer subgraphs; each of the two-layer subgraphs identifies a logic reasoning step;

[0132] A determination unit 8032, used to determine whether the logical reasoning of each of the second-layer subgraphs is correct;

[0133] The determination unit 8033 is used to determine that the logical reasoning of the generative large model to generate the text generation result is correct in response to the logical reasoning of the multiple second-layer sub-graphs being correct.

[0134] The device 800 for processing the model generation results of this embodiment, by adopting the above-mentioned modules, realizes the implementation principle and technical effect of the processing of the model generation results, which is the same as the implementation of the above-mentioned related method embodiments. For details, please refer to the records of the above-mentioned related method embodiments, which will not be repeated here.

[0135] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0136] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0137] Fig. 9 A schematic block diagram of an example device 900 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0138] like Fig. 9As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0139] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0140] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as the above-mentioned methods of the present disclosure. For example, in some embodiments, the above-mentioned methods of the present disclosure may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the above-mentioned methods of the present disclosure described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to perform the above-mentioned methods of the present disclosure in any other appropriate manner (e.g., by means of firmware).

[0141] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0142] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0143] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0144] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0145] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0146] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0147] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0148] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for processing model generation results, applied to the field of text processing, comprising: The text generation results of the generative large model are disassembled to obtain multiple result logic units; Each of the result logic units includes a fragment in the text generation result; Each segment in the text generation result can independently identify a premise or conclusion in the logical reasoning relationship of the text generation result; The text generation result is a response result generated by the generative large model based on the text input information; Based on the multiple result logic units, generating a logical reasoning graph capable of characterizing the logical reasoning relationship between the multiple result logic units; Based on the logical reasoning graph, determining whether the logical reasoning of the generative large model to generate the text generation result is correct; Wherein, based on the multiple result logic units, generating a logical reasoning graph capable of characterizing the logical reasoning relationship between the multiple result logic units includes: Based on multiple input logic units and the multiple result logic units, the most relevant examples are retrieved in a preset example database; the example database includes multiple groups of examples, each of the examples includes multiple input example logic units corresponding to input example information, multiple result example logic units corresponding to result example information, and example logical reasoning graphs corresponding to the multiple result example logic units; the multiple input logic units are obtained by disassembling the text input information; based on the multiple input logic units and the multiple result logic units, as well as the multiple input example logic units, the multiple result example logic units and the corresponding example logical reasoning graphs, other pre-trained generative big models are used to generate the logical reasoning graphs corresponding to the multiple result logic units; the other generative big models are different from the generative big model.

2. The method according to claim 1, wherein: The text generation results of the generative large model are disassembled to obtain multiple result logic units, including: A pre-trained logic disassembly model is used to disassemble the text generation result of the generative large model to obtain multiple result logic units.

3. The method according to claim 1, wherein: Based on the multiple result logic units, generating a logical reasoning graph capable of characterizing the logical reasoning relationship between the multiple result logic units, including: Based on the multiple result logic units, a pre-trained logic reasoning graph generation model is used to generate a logic reasoning graph that can characterize the logic reasoning relationship between the multiple result logic units.

4. The method according to claim 1, wherein: Each of the input logic units includes a segment of the text input information, and each segment of the text input information can independently identify a premise or conclusion in a logical reasoning relationship.

5. The method according to any one of claims 1 to 4, wherein: Based on the logical reasoning graph, evaluating whether the logical reasoning of the text generation result generated by the generative large model is correct includes: Decomposing the logic reasoning graph into a plurality of two-layer subgraphs; each of the two-layer subgraphs identifies a logic reasoning step; Determining whether the logical reasoning of each of the second-layer subgraphs is correct; In response to the logical reasoning of the multiple second-layer subgraphs being correct, it is determined that the logical reasoning of the generative large model in generating the text generation result is correct.

6. A processing device for model generation results, applied to the field of text processing, comprising: A disassembly module is used to disassemble the text generation results of the generative large model to obtain multiple result logic units; Each of the result logic units includes a fragment in the text generation result; Each segment in the text generation result can independently identify a premise or conclusion in the logical reasoning relationship of the text generation result; The text generation result is a response result generated by the generative large model based on the text input information; A generating module, configured to generate, based on the plurality of result logic units, a logical reasoning graph capable of representing a logical reasoning relationship between the plurality of result logic units; An evaluation module, used for evaluating whether the logical reasoning of the text generation result generated by the generative large model is correct based on the logical reasoning graph; The generating module is used for: Based on the multiple input logic units and the multiple result logic units, the most relevant examples are retrieved in a preset example database; the example database includes multiple groups of examples, each of the examples includes multiple input example logic units corresponding to the input example information, multiple result example logic units corresponding to the result example information, and example logical reasoning graphs corresponding to the multiple result example logic units; the multiple input logic units are obtained by disassembling the text input information; Based on the multiple input logic units and the multiple result logic units, as well as the multiple input example logic units, the multiple result example logic units and the corresponding example logic reasoning graphs, other pre-trained generative big models are used to generate the logic reasoning graphs corresponding to the multiple result logic units; the other generative big models are different from the generative big model.

7. The device according to claim 6, wherein: The disassembly module is used for: A pre-trained logic disassembly model is used to disassemble the text generation result of the generative large model to obtain multiple result logic units.

8. The device according to claim 6, wherein: The generating module is used for: Based on the multiple result logic units, a pre-trained logic reasoning graph generation model is used to generate a logic reasoning graph that can characterize the logic reasoning relationship between the multiple result logic units.

9. The device according to claim 6, wherein: Each of the input logic units includes a segment of the text input information, and each segment of the text input information can independently identify a premise or conclusion in a logical reasoning relationship.

10. The device according to any one of claims 6 to 9, wherein: The evaluation module comprises: A disassembling unit, used for disassembling the logic reasoning graph into a plurality of two-layer subgraphs; each of the two-layer subgraphs identifies a logic reasoning step; A determination unit, used to determine whether the logical reasoning of each of the second-layer subgraphs is correct; A determination unit is used to determine that the logical reasoning of the generative large model to generate the text generation result is correct in response to the logical reasoning of the multiple second-layer sub-graphs being correct.

11. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 5.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-5.

13. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 5.

Images