Method, device, electronic device and storage medium for processing model generation results

Abstract

To provide a method, a device, electronic equipment, and a storage medium for processing a model generation result that efficiently and accurately determine whether logical inference of a generative large-scale model is correct.SOLUTION: A method for processing a model generative result comprises a step of decomposing a text generation result of a generative large-scale-model to acquire a plurality of result logical units. Each result logical unit includes a fragment in the text generation result, each fragment can independently mark one premise or one conclusion in a logical inference relation of the text generation result, and the text generation result is a response generated by the generative large-scale model on the basis of text input information. The method also comprises steps of: generating, on the basis of the plurality of result logical units, a logical inference diagram that a characterizes logical inference relationship between the plurality of result logical units; and determining, on the basis of the logical inference diagram, whether the logical inference by which the generative large-scale model generates the text generation result is correct.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to the field of computer technology, specifically to the field of artificial intelligence technology such as machine learning and natural language processing, and in particular to a method, device, electronic device, and storage medium for processing model generation results. [Background technology]

[0002] With the widespread application of generative large-scale models, the effectiveness evaluation of generative large-scale models has become a very important technology.

[0003] Effectiveness evaluation of a large-scale generative model involves determining whether the text generation results of the large-scale generative model are correct. Unlike task evaluation of conventional models, the text generation results of a large-scale generative model are not uniquely fixed, so it is not possible to simply evaluate whether strings match during effectiveness evaluation. Summary of the Invention [Problem to be solved by the invention]

[0004] The present disclosure provides a method, apparatus, electronic device, and storage medium for processing model generation results. [Means for solving the problem]

[0005] According to one aspect of the present disclosure, there is provided a method for processing model generation results, comprising: A step of decomposing the text generation result of the generative large-scale model to obtain a plurality of result logical units, each of the result logical units including a fragment in the text generation result, each of the fragments being capable of independently marking one premise or conclusion in a logical inference relationship of the text generation result, and the text generation result being a response result generated by the generative large-scale model based on text input information; generating a logic inference diagram based on the plurality of result logic units, the logic inference diagram characterizing logic inference relationships between the plurality of result logic units; and evaluating whether the logical inference by which the generative large-scale model generates the text generation result is correct based on the logical inference diagram.

[0006] According to another aspect of the present disclosure, there is provided an apparatus for processing model generation results, comprising: a decomposition module for decomposing a text generation result of a generative large-scale model to obtain a plurality of result logical units, each of the result logical units including a fragment in the text generation result, each of the fragments being capable of independently marking one premise or conclusion in a logical inference relationship of the text generation result, and the text generation result being a response result generated by the generative large-scale model based on text input information; a generating module for generating a logical inference diagram based on the plurality of resultant logical units, the logical inference diagram characterizing logical inference relationships between the plurality of resultant logical units; and an evaluation module for evaluating whether the logical inference by which the generative formula large-scale model generates the text generation result is correct based on the logical inference diagram.

[0007] According to another aspect of the present disclosure, there is provided an electronic device, comprising: at least one processor; a memory communicatively coupled to the at least one processor; The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to perform the method of the aspect and any possible implementation.

[0008] According to another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium having stored thereon computer instructions that cause the computer to perform the method of the aspect and any possible implementation thereof.

[0009] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the method of the aspect and any possible implementation manner.

[0010] Based on the technology disclosed herein, it is possible to efficiently and accurately determine whether the logical inference of a large-scale generative formula model is correct, and further, it is possible to effectively improve the evaluation efficiency of a large-scale generative formula model.

[0011] It should be understood that the contents described herein are not intended to identify key or important features of the embodiments of the present disclosure, nor should they be used to limit the scope of the present disclosure. Other features of the present disclosure can be readily understood through the following specification. [Brief explanation of the drawings]

[0012] The drawings are for a better understanding of the present application and are not intended to limit the present application. [Figure 1] FIG. 1 is a schematic diagram according to a first embodiment of the present disclosure. [Figure 2] FIG. 10 is a schematic diagram according to a second embodiment of the present disclosure. [Figure 3] FIG. 1 is a logical reasoning diagram provided by the present disclosure. [Figure 4] FIG. 10 is a schematic diagram according to a third embodiment of the present disclosure. [Figure 5] 1 is a schematic diagram of a logical inference diagram generated by the present embodiment. [Figure 6A] These are two-layer sub-diagrams of the logical reasoning diagram division shown in FIG. 5, respectively. [Figure 6B] These are two-layer sub-diagrams of the logical reasoning diagram division shown in FIG. 5, respectively. [Figure 6C] These are two-layer sub-diagrams of the logical reasoning diagram division shown in FIG. 5, respectively. [Figure 7] FIG. 10 is a schematic diagram of a fourth embodiment of the present disclosure. [Figure 8] FIG. 10 is a schematic diagram of a fifth embodiment of the present disclosure. [Figure 9]FIG. 1 is a block diagram of an electronic device for implementing a method according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, exemplary embodiments of the present application will be described based on the drawings. For ease of understanding, various details of the embodiments of the present application are included and should be considered as merely examples. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of brevity, the following description will omit descriptions of well-known functions and structures.

[0014] Obviously, the described embodiments are some of the embodiments of the present application, but not all of the embodiments, and all other embodiments that a person skilled in the art can obtain without creative effort according to the embodiments of the present application all belong to the scope of protection of the present application.

[0015] Note that terminal devices related to the embodiments of the present application may include, but are not limited to, smart devices such as mobile phones, personal digital assistants (PDAs), wireless handheld devices, and tablet computers. Display devices may include, but are not limited to, devices with display capabilities such as personal computers and televisions.

[0016] Furthermore, the term "and / or" in this specification describes only the relation between related objects and indicates that three types of relations can exist, for example, A and / or B can indicate three cases: only A exists, A and B exist simultaneously, or only B exists. It should be understood that the symbol " / " generally indicates that the related objects before and after it are in an "or" relation.

[0017] 1 is a schematic diagram according to a first embodiment of the present disclosure. As shown in FIG. 1, this embodiment provides a method for processing model generation results, which specifically can include the following steps: S101, decompose the text generation result of the generative large-scale model to obtain multiple result logical units; Each result logical unit in this embodiment includes a fragment in the text generation result, and each fragment can independently mark one premise or conclusion in the logical inference relationship of the text generation result, and the text generation result is a response result generated by the generative large-scale model based on the text input information.

[0018] The entity that executes the model generation result processing method of this embodiment may be a device that processes the model generation results, and the device may be an independent electronic device or an application using a software platform, which evaluates the generation effect of the generative large-scale model.

[0019] The generative large-scale model of this embodiment is also called a general language model (GLM), and is also called a generative large-scale language model.

[0020] In this embodiment, the generative large-scale model is used in the field of text processing. When used, text input information is input to the generative large-scale model, and the generative large-scale model can generate and output a text generation result based on the text input information. In actual application scenarios, in order to improve the effectiveness of the generative large-scale model in generating text generation results, several corresponding processes need to be performed on the generation results of the generative large-scale model, which can efficiently and accurately determine whether the logical inference of the generative large-scale model is correct, and can also perform accurate and effective evaluation of the generative large-scale model. Furthermore, based on the evaluation results, the generative large-scale model can be conversely optimized to further improve the text generation effectiveness of the generative large-scale model.

[0021] Because the text generation results of a generative large-scale model are typically long and can contain multiple sentences, a fragment in each result logical unit can be a single sentence in the text generation result, or two or more consecutive sentences. That is, a fragment in the text generation result is always a single, continuous segment, not a concatenation of two or more intermittent segments.

[0022] Specifically, in this embodiment, the principle of decomposing the text generation result ensures that each resulting logical unit after decomposition serves as a premise or a conclusion when performing logical relational inference on the text generation result.

[0023] S102, generating a logical inference diagram based on the plurality of result logical units, the logical inference diagram characterizing logical inference relationships between the plurality of result logical units;

[0024] The logical inference diagram of this embodiment is a directed acyclic diagram in which a plurality of result logical units are configured according to logical inference relationships.

[0025] S103, based on the logical inference diagram, determine whether the logical inference that the generative formula large-scale model generates the text generation result is correct.

[0026] Since the logical inference diagram is generated based on the logical inference relationships between multiple result logical units, it is possible to evaluate whether the logical inference used by the generative large-scale model to generate a text generation result is correct by referring to the logical inference relationships between multiple result logical units in the logical inference diagram, and further, it is possible to realize evaluation of the generative large-scale model from the logical inference dimension. That is, if the logical inference used by the generative large-scale model to generate a text generation result is correct, it indicates that the logical inference of the generative large-scale model is correct, and if the logical inference used by the generative large-scale model to generate a text generation result is incorrect, it indicates that the logical inference of the generative large-scale model is incorrect.

[0027] The method for processing model generation results in this embodiment decomposes the text generation result to obtain multiple result logical units, generates a logical inference diagram for the multiple result logical units, and then determines whether the logical inference used by the generative large-scale model to generate the text generation result is correct based on the logical inference diagram, thereby more effectively and accurately determining whether the logical inference is correct for the generative large-scale model. Therefore, the technical solution in this embodiment can efficiently and accurately evaluate the text generation results of a generative large-scale model from the perspective of logical inference, more efficiently and accurately realizing the effective evaluation of the generative large-scale model. In addition, the technical solution in this embodiment can be fully automated, eliminating the need for manual processing, significantly reducing labor costs, improving evaluation speed, and effectively improving the evaluation efficiency of generative large-scale models.

[0028] 2 is a schematic diagram of a second embodiment of the present disclosure. The method for processing model generation results of this embodiment is based on the technical solution of the embodiment shown in FIG. 1 above, and the technical solution of this disclosure will be further described in more detail. As shown in FIG. 2, the method for processing model generation results of this embodiment may specifically include the following steps: S201, using a pre-trained logic decomposition model to decompose the text generation result of the generative large-scale model to obtain multiple result logic units; This step is an implementation of step S101 in the embodiment shown in Figure 1. In this implementation, a pre-trained logical decomposition model is used to automatically decompose the text generation result. In specific implementation, the text generation result is input to the logical decomposition model, which directly outputs multiple corresponding result logical units based on the input text generation result.

[0029] In this embodiment, the logic decomposition model can collect multiple groups of training data before training, each group of training data including a training corpus and multiple training logic units obtained by manually decomposing the training corpus. The principle of decomposition is to ensure that each training logic unit after decomposition becomes a premise or a conclusion when logical relational inference is performed on the training corpus. Then, the logic decomposition model can be trained using multiple groups of training data, so that the logic decomposition model can learn decomposition capabilities.

[0030] In this embodiment, a logical decomposition model is used to perform intelligent decomposition on the text generation results, which can effectively improve the accuracy of multiple result logical units of the decomposition and improve the decomposition efficiency.

[0031] Alternatively, step S101 of the embodiment shown in FIG. 1 can also decompose the text generation result of the generative formula large-scale model according to a preset decomposition policy to obtain multiple result logical units.

[0032] For example, a preset decomposition policy can be set to ensure that each resulting logical unit after decomposition is one premise or one conclusion in the logical inference relationship of the text generation result.

[0033] In concrete implementation, the text generation result is first divided according to the sentence granularity, and then all fragment division methods are traversed to obtain various segmentation results for the text generation result. Each fragment division method can divide a sentence individually into a fragment, or divide two or more adjacent consecutive sentences into a fragment. Next, each fragment in each segmentation result is detected to determine whether it can be used as a premise or a conclusion in the logical reasoning relationship of the text generation result. Finally, from all the segmentation results, a segmentation result in which all the decomposed fragments form a premise or a conclusion can be selected as the required decomposition result, and the corresponding multiple result logical units can be obtained.

[0034] By decomposing according to a preset decomposition policy, multiple resultant logical units can be obtained efficiently and accurately.

[0035] S202, based on the plurality of result logic units, using a pre-trained logic diagram generation model to generate a logic diagram characterizing the logic inference relationships between the plurality of result logic units; In this embodiment, one logical inference diagram generation model is trained in advance, and when used, multiple result logical units can be input to the logical inference diagram generation model, and the logical inference diagram generation model can generate a logical inference diagram composed of multiple result logical units based on the input information.

[0036] During training, the logical reasoning diagram generation model can further be obtained by training using multiple groups of training data, each of which includes multiple training logic units and training logical reasoning diagrams labeled based on the multiple training logic units. By training the logical reasoning diagram generation model using multiple groups of training data, the logical reasoning diagram generation model can learn the ability to generate corresponding training logical reasoning diagrams based on the multiple training logical units.

[0037] In use, a plurality of result logical units are input to the logical inference diagram generation model trained by the above-described method, and the logical inference diagram generation model can generate and output a logical inference diagram that characterizes the logical inference relationships between the plurality of result logical units.

[0038] The method for intelligently generating a logical inference diagram based on the above logical inference diagram generation model can efficiently and accurately generate logical inference diagrams for multiple result logic units.

[0039] S203, decomposing the logical reasoning diagram into multiple two-layer sub-diagrams, each two-layer sub-diagram marking one logical reasoning step; Specifically, when decomposing a two-layer subdiagram, the smallest logical relationship subdiagram, i.e., the two-layer subdiagram, is divided according to the inference relationship in the logical reasoning diagram. For example, FIG. 3 is a logical reasoning diagram provided by the present disclosure. According to the decomposition process of this embodiment, the logical reasoning diagram shown in FIG. 3 can be decomposed into two two-layer subdiagrams, for example, a two-layer subdiagram consisting of A, B, and C, and a two-layer subdiagram consisting of C, D, and E. Each two-layer subdiagram corresponds to one logical reasoning stage. This method can be used to decompose all two-layer subdiagrams in the logical reasoning diagram.

[0040] S204, determining whether the logical inference of each two-layer sub-diagram is correct; Specifically, during the judgment process, a pre-trained reasoning model can be used to infer whether the logical inference of each two-layer subgraph is correct. For example, during use, each two-layer subgraph can be input into the reasoning model sequentially from top to bottom and left to right in the logical inference diagram. Furthermore, the reasoning model sequentially determines whether the logical inference characterized by each input two-layer subgraph is correct. The reasoning model of this embodiment can also be implemented using a large language model (LLM). Alternatively, this embodiment can set a certain policy to determine whether the inference of each two-layer subgraph is correct, and can accurately determine whether the logical inference of each two-layer subgraph is correct.

[0041] S205, in response to the logical inferences of the plurality of two-layer sub-diagrams all being correct, the generative large-scale model determines that the logical inference that generates the text generation result is correct.

[0042] In other words, unless the logical reasoning of one of the multiple two-layer subdiagrams is correct, the logical reasoning by which the generative large-scale model generates the text generation result is deemed to be incorrect, and furthermore, the logical reasoning of the generative large-scale model is determined to be incorrect.

[0043] Steps S203-S205 of this embodiment are a specific implementation of step S103 of the embodiment shown in FIG.

[0044] Specifically, in this implementation method, a logical inference diagram is decomposed into multiple two-layer subdiagrams, and a single logical inference stage of each two-layer subdiagram is determined to be correct or not, thereby determining whether the logical inference used by the generative large-scale model to generate the text generation result is correct or not, and further determining whether the logical inference generated by the generative large-scale model is correct or not.

[0045] In this embodiment, this method judges only one two-layer sub-diagram each time, that is, it judges whether only a single logical inference stage is correct at a time. This effectively reduces the difficulty of judging whether the logical inference of the text generation result generated by the generative large-scale model is correct, effectively improves the accuracy of judging whether the logical inference of the generative large-scale model is correct, and further, effectively improves the accuracy and evaluation efficiency of the text generation result of the generative large-scale model, thereby significantly improving the evaluation effect of the generative large-scale model.

[0046] The model generation result processing method of this embodiment uses the above-mentioned method to accurately determine whether the logical inference of the generative large-scale model is correct, thereby automatically realizing the evaluation of the generative large-scale model, saving time and effort, effectively shortening the evaluation speed, and improving the evaluation efficiency of the generative large-scale model. Furthermore, by determining the logical inference of each two-layer sub-diagram decomposed by the logical inference diagram of the text generation result, the logical inference of the text generation result can be determined, and the text generation result of the generative large-scale model can be evaluated, effectively improving the evaluation accuracy and evaluation efficiency.

[0047] 4 is a schematic diagram of a third embodiment of the present disclosure. The method for processing the model generation results of this embodiment is based on the technical solution of the embodiment shown in FIG. 1 above, and the technical solution of this disclosure will be further described in more detail. As shown in FIG. 4, the method for processing the model generation results of this embodiment may specifically include the following steps: S401, using a pre-trained logic decomposition model to respectively decompose the text input information and the text generation result of the generative large-scale model to obtain a plurality of input logic units and a plurality of result logic units; Each input logic unit includes a fragment in the text input information, and each fragment can independently mark one premise or conclusion in the logical inference relationship of the text input information. Each result logic unit includes a fragment in the text generation result, and each fragment can independently mark one premise or conclusion in the logical inference relationship of the text generation result. For specific implementation methods, please refer to the relevant description of step S201 in the embodiment shown in Figure 2 above, and further description will be omitted here.

[0048] S402, based on the plurality of result logic units, generating a logic inference diagram characterizing the logic inference relationships between the plurality of result logic units with reference to the plurality of input logic units; That is, in the generation of a logical inference diagram for a plurality of result logic units in this embodiment, it is necessary to simultaneously refer to not only a plurality of result logic units but also a plurality of input logic units.

[0049] For example, when step S402 is specifically implemented, the following (1) searching for the most relevant samples in a pre-defined sample database based on a plurality of input logical units and a plurality of result logical units; (2) generating a logic inference diagram corresponding to the plurality of result logic units based on the plurality of input logic units and the plurality of result logic units, and using a sample logic inference diagram corresponding to the plurality of input sample logic units, the sum of the plurality of result sample logic units, and other pre-trained generative large-scale models.

[0050] The preset sample database of this embodiment may store multiple groups of samples, each of which may include a plurality of input sample logic units corresponding to input sample information, a plurality of result sample logic units corresponding to result sample information, and sample logic inference diagrams corresponding to the plurality of result sample logic units.

[0051] In this embodiment, the information in the sample database is also all in text format, i.e., the input sample information in each group of samples is text input sample information, and the result sample information is text result sample information. That is, the input sample information may be the text input information of the generative large-scale model, and the result sample information may be the text generation result generated by the generative large-scale model. The multiple input sample logical units and the multiple result sample logical units can be obtained using the multiple input logical unit and multiple result logical unit decomposition methods of step S401 described above. The sample logical inference diagram in this embodiment can be manually labeled based on the multiple result sample logical units.

[0052] Specifically, in this embodiment, a search can be performed based on a plurality of input logical units and a plurality of result logical units to search for a sample with the highest relevance in a pre-defined sample database, where the sample with the highest relevance can be considered to be a sample corresponding to a plurality of input sample logical units and a plurality of result sample logical units that are most relevant in terms of the overall context of the plurality of input logical units and the plurality of result logical units.

[0053] Next, the multiple input logical units and the multiple result logical units, and each partial information contained in the most relevant sample are combined and input to the generative large-scale model, and another pre-trained generative large-scale model generates a sample logical inference diagram generation method based on the multiple result sample logical units in the most relevant sample, and generates a sample logical inference diagram corresponding to the multiple result sample logical units. In this embodiment, the other pre-trained generative large-scale model is not the generative large-scale model evaluated in this embodiment, but another generative large-scale model that has already been trained.

[0054] Alternatively, in this embodiment, each group of samples in the sample database may only include a plurality of result sample logical units corresponding to the result sample information, and a sample logical inference diagram corresponding to the plurality of result sample logical units.

[0055] In response to this case, when specifically searching, the most relevant sample can be searched for in a pre-set sample database based only on the multiple result logical units, and a logical inference diagram corresponding to the multiple specific result sample logical units can be generated using another pre-trained generative large-scale model based on the sample logical inference diagram corresponding to the multiple result logical units.

[0056] However, this implementation method has poor accuracy compared with the corresponding multiple input sample logical units and corresponding technical solutions in which the input sample information is simultaneously included in the sample database mentioned above.

[0057] In this embodiment, the method of searching for the most relevant samples and intelligently generating logical inference diagrams using other generative large-scale models can accurately and efficiently generate logical inference diagrams for multiple result logical units.

[0058] S403, decomposing the logical reasoning diagram into multiple two-layer sub-diagrams, each two-layer sub-diagram marking one logical reasoning step; S404, determining whether the logical inference of each two-layer sub-diagram is correct; S405, in response to the inferences of the multiple two-layer sub-diagrams all being correct, the generative large-scale model determines that the logical inference that generates the text generation result is correct.

[0059] For the specific implementation of steps S403-S405 in this embodiment, reference can be made to the description of steps S203-S205 in the embodiment shown in FIG. 2, and the description will be omitted here.

[0060] The model generation result processing method of this embodiment can use the above-mentioned method to more accurately and efficiently generate logical inference diagrams of multiple result logical units, and can further improve the accuracy of determining whether the logical inference of the generative large-scale model is correct, and can further effectively improve the accuracy and evaluation efficiency of the generative large-scale model.

[0061] The following provides a specific example to explain the method for evaluating large-scale generative models disclosed herein.

[0062] For example, in this example, the text input information query is as follows: The police caught a thief and arrested four suspects, A, B, C, and D. Their statements were: A said: I didn't steal. B said: A stole. C said: It wasn't me. D said: B stole. Only one of them is known to have told the truth. Who is the thief? The text input information is input into a generative large-scale model, and the text generation result response generated by the generative large-scale model is, according to the subject line, "Part B said: Part A stole. Part A said: I did not steal." In this case, one of Part A and Part B must be telling the truth, and Part C and Part D must both be lying, i.e., Part C is lying and is the thief.

[0063] For example, based on step S401, the multiple input logical units obtained by decomposing the text input information query include: a first input logical unit (query-unit-1): The police caught a thief and arrested four suspects, A, B, C, and D. Their statements are; a second input logical unit (query-unit-2): Person A said: I did not steal; a third input logical unit (query-unit-3): Person B said: A stole; a fourth input logical unit (query-unit-4): Person C said: It was not me; a fifth input logical unit (query-unit-5): Person D said: B stole; and a sixth input logical unit (query-unit-6): It is known that only one of them told the truth; a seventh input logical unit (query-unit-7): Who is the thief? Similarly, based on step S401, the multiple result logical units obtained by decomposing the text generation result response include: the first result logical unit (response-unit-1), what B said: A stole; the second result logical unit (response-unit-2), what A said: I did not steal; the third result logical unit (response-unit-3), so one of A and B must be telling the truth; the fourth input logical unit (response-unit-4), so C and D are both lying; the fifth input logical unit (response-unit-5), that is, C is lying; the sixth input logical unit (response-unit-6), C is a thief.

[0064] Then, a logic inference diagram of the multiple result logic units can be generated based on the method of step S302.

[0065] In this embodiment, the logical reasoning process is not a chain or tree structure, but a directed acyclic diagram. The logical reasoning process represents the reasoning process of the problem, and each node of the generated logical reasoning diagram is a minimum logical unit, i.e., a result logical unit. Each second-order sub-diagram in the logical reasoning diagram represents a minimum reasoning process.

[0066] Specifically, the sample database of this embodiment may be an In-Context-Learning (ICL) database.

[0067] Specifically, searching the ICL database based on the specific implementation of step S402 can be called an ICL search. Specifically, when searching, a plurality of input logical units and a plurality of result logical units are input, and the most relevant sample is searched for in the ICL database based on the input plurality of input logical units and a plurality of result logical units. The most relevant sample among the combined prompt words is generated by combining a plurality of input sample logical units, a plurality of result sample logical units, and a sample logical reasoning diagram in the combined prompt word in a one-shot manner.

[0068] Next, a plurality of input logical units, a plurality of result logical units, and a combined prompt word (prompt) are used to generate a logical inference diagram corresponding to the plurality of result logical units using a generative large-scale model. For example, Figure 5 is a schematic diagram of a logical inference diagram generated by this embodiment.

[0069] Finally, a logical judgment is made based on the logical reasoning diagram obtained in Figure 5. In practical applications, it is relatively difficult to judge the correctness of the entire logical reasoning diagram at once, so we can refer to the implementation method of steps S403-S405 mentioned above: first, the logical reasoning diagram is decomposed into two-layer sub-diagrams, each two-layer sub-diagram represents one logical reasoning stage, and only one two-layer sub-diagram is judged each time, that is, only one logical reasoning stage is judged to be correct or not at a time, which reduces the overall difficulty of judgment and greatly improves the judgment effect.

[0070] The specific judgment process can be realized using a logical reasoning model based on the LLM implementation. Using the LLM, the correctness of the logical reasoning steps of each two-layer subdiagram in the logical reasoning diagram is judged from bottom to top. If each logical reasoning step is correct, the entire logical reasoning process is judged to be correct; if not, it is judged that there is a logic problem in the logical reasoning process. This method not only judges the correctness of the logical reasoning given a logical reasoning problem, but also accurately identifies the step where the error occurred.

[0071] For example, Figures 6A, 6B, and 6C are two-layer sub-diagrams obtained by dividing the logical reasoning diagram shown in Figure 5. If the logical reasoning stages of the three two-layer sub-diagrams in Figures 6A, 6B, and 6C are all determined to be correct, it can be determined that the logical reasoning of the text generation result generated by the generative large-scale model is correct, thereby realizing the evaluation of the generative large-scale model.

[0072] The above-described method of the embodiment of the present disclosure evaluates a text generation result generated based on a generative large-scale model. In a practical application scenario, the generative large-scale model may perform multiple generation tasks within one segment time. Specifically, the above-described method of the embodiment of the present disclosure can evaluate whether the logical reasoning used by the generative large-scale model to generate a text generation result is correct each time the generation task is performed. Furthermore, it can be statistically determined whether the accuracy rate of the segment time reaches a predetermined proportional threshold, such as 95%, 98%, or other proportionality. If so, it is determined that the accuracy rate of the generative large-scale model is high and effective; if not, it is determined that the accuracy rate and effectiveness of the generative large-scale model are poor.

[0073] Logic is the most important dimension that embodies the logical reasoning ability of a generative large-scale model and is a necessary ability for solving some complex logical reasoning tasks. If logic problems are found in the model text generation results during the evaluation stage, it can play an important role in the effectiveness evaluation and optimization of the logical reasoning problems. Compared with other dimensions, logic is an embodiment of higher intelligence, and logic evaluation is extremely difficult, and traditional methods have difficulty achieving relatively good results. Therefore, the model generation result processing method provided by the present disclosure evaluates generative large-scale models from the perspective of logic, and is an automatic logical reasoning evaluation framework for generative large-scale models, which can more effectively evaluate the logical reasoning of generative large-scale models and effectively improve the evaluation accuracy and evaluation effect of logical reasoning of generative large-scale models.

[0074] The processing method for the above-mentioned model generation results in the embodiments of the present disclosure first divides the logical units, then generates a logical inference diagram, obtains each two-layer sub-diagram in the logical inference diagram, and then evaluates each two-layer sub-diagram at a granularity, i.e., the smallest logical inference stage, which can greatly reduce the difficulty of determining whether the logical inference of the generative large-scale model is correct, i.e., can effectively reduce the difficulty of evaluating the generative large-scale model, and further greatly improve the evaluation effect of the generative large-scale model.

[0075] Furthermore, the method for processing the model generation results described above in the embodiments of the present disclosure uses natural language to express the entire reasoning process, which is more versatile and can be applied to more types of problems.

[0076] In addition, the above-described method for processing the model generation results in the embodiments of the present disclosure generates a logical inference diagram to represent the overall logical inference process, which can not only effectively detect logical inference errors in the text generation results, but also detect situations where logical inference is missing or reversed, and can be used for data logic repair.

[0077] In addition, the above-described model generation result processing method of the embodiments of the present disclosure can accurately determine whether the logical inference of a generative large-scale model is correct, and can automatically evaluate the generative large-scale model, saving time and effort, effectively shortening the evaluation speed, and effectively improving the evaluation efficiency. Furthermore, by evaluating using the method of generating a logical inference diagram and evaluating the logical inference of the smallest logical unit, i.e., each two-layer sub-diagram of the logical inference diagram, the accuracy of determining whether the logical inference of a generative large-scale model is correct can be further effectively improved, and the accuracy of evaluating the generative large-scale model can be effectively improved, thereby improving the evaluation efficiency.

[0078] The above-described embodiments of the present disclosure are applied to the field of text processing as an example, but in actual applications, the technical solution of the present disclosure can also be applied to the field of speech processing. For example, speech information can be first collected and the corresponding text information can be obtained through speech recognition. Then, the above-described technical solution of the present disclosure can be used to process the model generation results based on the text information, and can efficiently and accurately determine whether the logical reasoning of the generative large-scale model is correct, and finally, can accurately and effectively evaluate the generative large-scale model.

[0079] 7 is a schematic diagram according to a fourth embodiment of the present disclosure. As shown in FIG. 7, this embodiment provides a model generation result processing device 700, which is applied in the field of text processing, and includes: a decomposition module 701, a generation module 702, and an evaluation module 703; The decomposition module 701 is used to decompose the text generation result of the generative large-scale model to obtain a plurality of result logical units, each of which includes a fragment in the text generation result, and each of which can independently mark one premise or conclusion in the logical inference relationship of the text generation result, and the text generation result is a response result generated by the generative large-scale model based on text input information; a generating module 702 for generating, based on the plurality of resultant logic units, a logic inference diagram that characterizes logic inference relationships between the plurality of resultant logic units; The evaluation module 703 is used to evaluate whether the logical inference by which the generative large-scale model generates the text generation result is correct based on the logical inference diagram.

[0080] The model generation result processing device 700 of this embodiment uses the above-mentioned modules to realize the implementation principle and technical effect of processing the model generation result, which is the same as the implementation of the above-mentioned related method embodiment. For details, please refer to the description of the above-mentioned related method embodiment, and the description will be omitted here.

[0081] 8 is a schematic diagram according to a fifth embodiment of the present disclosure. As shown in FIG. 8, this embodiment provides a model generation result processing device 800, which includes modules with the same names and functions as those shown in FIG. 7 above: a decomposition module 801, a generation module 802, and an evaluation module 803.

[0082] In the model generation result processing device 800 of this embodiment, the decomposition module 801: A pre-trained logic decomposition model is used to decompose the text generation results of the generative large-scale model to obtain multiple result logic units.

[0083] Further optionally, in one embodiment of the present disclosure, the generating module 802 Based on the plurality of resultant logic units, a pre-trained logic diagram generation model is used to generate a logic diagram characterizing the logic relationships between the plurality of resultant logic units.

[0084] Further optionally, in one embodiment of the present disclosure, the decomposition module 801 further comprises: The generative large-scale model is used to decompose the textual input information to obtain multiple input logic units, each of which includes a fragment in the textual input information, and each of which can independently mark one premise or conclusion in a logical inference relationship.

[0085] Further optionally, in one embodiment of the present disclosure, the generating module 802 Based on the plurality of result logic units, the logic diagram is used to generate the logic inference diagram characterizing the logic inference relationships between the plurality of result logic units with reference to the plurality of input logic units.

[0086] Further optionally, in one embodiment of the present disclosure, the generating module 802 Searching for a most relevant sample in a pre-defined sample database according to the plurality of input logical units and the plurality of result logical units, the sample database including a plurality of groups of samples, each of the samples including a plurality of input sample logical units corresponding to input sample information, a plurality of result sample logical units corresponding to result sample information, and a sample logic inference diagram corresponding to the plurality of result sample logical units; It is used to generate the logic inference diagram corresponding to the plurality of result logic units using another pre-trained generative large-scale model based on the plurality of input logic units and the plurality of result logic units, the plurality of input sample logic units, the plurality of result sample logic units, and the corresponding sample logic inference diagram.

[0087] Further optionally, as shown in FIG. 8 , in one embodiment of the present disclosure, the evaluation module 803 includes: a decomposition unit 8031, a judgment unit 8032, and a determination unit 8033; The decomposition unit 8031 ​​is used to decompose the logic reasoning diagram into a plurality of two-layer sub-diagrams, each of which marks one logic reasoning step; The determining unit 8032 is used to determine whether the logical reasoning of each of the two-layer subgraphs is correct; The decision unit 8033 is used to decide that the logical inference by which the generative large-scale model generates the text generation result is correct in response to the logical inferences of the multiple two-layer sub-graphs all being correct.

[0088] The model generation result processing device 800 of this embodiment uses the above-mentioned modules to realize the implementation principle and technical effect of processing the model generation result, which is the same as the implementation of the above-mentioned related method embodiment. For details, please refer to the description of the above-mentioned related method embodiment, and the description will be omitted here.

[0089] In the technical solution disclosed herein, the acquisition, storage, application, etc. of relevant user personal information shall all comply with the provisions of relevant laws and regulations and shall not violate public order and morals.

[0090] According to embodiments of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium, and a computer program product.

[0091] 9 is a block diagram of an electronic device 900 for implementing an embodiment of the present disclosure. The electronic device is intended to represent various types of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The electronic device may also represent various types of mobile devices, such as personal digital assistants, mobile phones, smartphones, wearable devices, and other similar computing devices. The components, their connections and relationships, and their functions illustrated herein are merely examples and are not intended to limit the description herein and / or the practice of the present disclosure as claimed.

[0092] 9, the device 900 includes a computing unit 901, which can perform various appropriate operations and processes based on a computer program stored in a read-only memory (ROM) 902 or loaded from a storage unit 908 into a random access memory (RAM) 903. The RAM 903 can also store various programs and data required for the device 900 to operate. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0093] Multiple components within device 900 are connected to I / O interface 905, including input units 906 such as a keyboard, mouse, etc., output units 907 such as various types of displays, speakers, etc., storage units 908 such as a disk, optical disk, etc., and communication units 909 such as a network card, modem, wireless communication transceiver, etc. The communication units 909 enable device 900 to exchange information / data with other devices via computer networks such as the Internet and / or various telecommunication networks.

[0094] The computing unit 901 is a general-purpose and / or special-purpose processing component equipped with various processing and computational capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, computing units that execute various machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 901 performs the various methods and processes described above, such as the method for constructing a terrain map. For example, in some embodiments, the method for constructing a terrain map can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 908. In some embodiments, some or all of the computer program is loaded and / or installed into the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, it can perform one or more steps of the methods described above. Alternatively, in other embodiments, the computing unit 901 can be configured to execute the method for constructing a terrain map through any other suitable manner (e.g., via firmware).

[0095] Various implementations of the systems and techniques described herein may be realized in digital electronic circuitry systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-a-chip systems (SOCs), chip programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include being implemented in one or more computer programs that can be executed and / or interpreted by a programmable system that includes at least one programmable processor, which may be a special-purpose or general-purpose programmable processor, and that can receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0096] Program codes for implementing the methods of the present disclosure can be written using any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus such that, when executed by the processor or controller, the functions / acts specified in the flowcharts and / or block diagrams are performed. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a separate software package and partially on a remote machine, or entirely on a remote machine or server.

[0097] In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain or store a program for use with or in connection with an instruction execution system, device, or apparatus. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium includes, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0098] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user, and a keyboard and pointing device (e.g., a mouse or trackball) through which a user can provide input to the computer. Other types of devices can also be used to provide interaction with a user; for example, feedback provided to the user can be any form of sensing feedback (e.g., visual feedback, auditory feedback, or haptic feedback) and can receive input from the user in any form (including acoustic input, voice input, and tactile input).

[0099] The systems and techniques described herein can be implemented in a computing system including a back-end component (e.g., a data server), a computing system including a middleware component (e.g., an application server), a computing system including a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user interacts with an implementation of the systems and techniques described herein), or any combination of such back-end, middleware, and front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0100] The computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The relationship between the client and the server is created by computer programs running on corresponding computers and having a client-server relationship to each other. The server may be a cloud server, a server in a distributed system, or a server that combines blockchains.

[0101] It should be understood that steps can be rearranged, added, or deleted using the various types of flows shown above. For example, the steps described in the present disclosure may be performed in parallel, sequentially, or in a different order, but this specification is not limited thereto as long as the technical solution disclosed in the present disclosure can achieve the desired results.

[0102] The above specific implementation methods do not constitute limitations on the scope of protection of the present disclosure. Those skilled in the art may make various modifications, combinations, subcombinations, and substitutions based on design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principle of the present disclosure shall fall within the scope of protection of the present disclosure.

Claims

1. 1. A method for processing computer generated model results, applied in the field of text processing, comprising: A step of decomposing the text generation result of the generative large-scale model to obtain a plurality of result logical units, each of the result logical units including a fragment in the text generation result, each of the fragments being capable of independently marking one premise or conclusion in a logical inference relationship of the text generation result, and the text generation result being a response result generated by the generative large-scale model based on text input information; generating a logic inference diagram based on the plurality of result logic units, the logic inference diagram characterizing logic inference relationships between the plurality of result logic units; and determining whether the logical inference that generated the text generation result of the generative formula large-scale model is correct based on the logical inference diagram. How to process the model generation results.

2. The step of decomposing the text generation result of the generative large-scale model to obtain a plurality of result logical units includes: using a pre-trained logic decomposition model to decompose the text generation results of the generative large-scale model to obtain a plurality of result logic units; The method for processing model generation results according to claim 1 .

3. generating a logical inference diagram characterizing logical inference relationships between the plurality of resultant logical units based on the plurality of resultant logical units, generating a logic diagram characterizing a logic relationship between the plurality of result logic units using a pre-trained logic diagram generation model based on the plurality of result logic units; The method for processing model generation results according to claim 1 .

4. Before generating a logical inference diagram characterizing logical inference relationships between the plurality of result logical units based on the plurality of result logical units, the method for processing the model generation results includes: further comprising decomposing the textual input information of the generative large-scale model to obtain a plurality of input logic units, each of the input logic units including a fragment in the textual input information, each of the fragments being capable of independently marking one premise or conclusion in a logical inference relationship; The method for processing model generation results according to claim 1 .

5. generating a logical inference diagram characterizing logical inference relationships between the plurality of resultant logical units based on the plurality of resultant logical units, generating the logical inference diagram based on the plurality of result logic units and with reference to the plurality of input logic units, the logical inference diagram characterizing logical inference relationships between the plurality of result logic units; 5. A method for processing model generation results according to claim 4.

6. generating the logical inference diagram based on the plurality of result logic units and referencing the plurality of input logic units, the logical inference diagram characterizing logical inference relationships between the plurality of result logic units, Searching for the most relevant sample in a pre-defined sample database based on the plurality of input logical units and the plurality of result logical units, wherein the sample database includes a plurality of groups of samples, each of the samples including a plurality of input sample logical units corresponding to input sample information, a plurality of result sample logical units corresponding to result sample information, and a sample logic inference diagram corresponding to the plurality of result sample logical units; and generating the logic inference diagram corresponding to the plurality of result logic units using another pre-trained generative large-scale model based on the plurality of input logic units and the plurality of result logic units, the plurality of input sample logic units, the plurality of result sample logic units, and the corresponding sample logic inference diagram.

6. A method for processing model generation results according to claim 5.

7. The step of determining whether the logical inference by which the generative large-scale model generates the text generation result is correct based on the logical inference diagram includes: decomposing the logic diagram into a plurality of two-layer sub-diagrams, each of the two-layer sub-diagrams marking one logic step; determining whether the logical inference of each of the two-layer subgraphs is correct; and determining, in response to the logical inferences of the plurality of two-layer subgraphs being all correct, that the logical inferences by which the generative large-scale model generates the text generation result are correct. A method for processing model generation results according to any one of claims 1 to 6.

8. A device for processing model generation results, applied in the field of text processing, comprising: a decomposition module for decomposing a text generation result of a generative large-scale model to obtain a plurality of result logical units, each of the result logical units including a fragment in the text generation result, each of the fragments being capable of independently marking one premise or conclusion in a logical inference relationship of the text generation result, and the text generation result being a response result generated by the generative large-scale model based on text input information; a generating module for generating a logical inference diagram based on the plurality of resultant logical units, the logical inference diagram characterizing logical inference relationships between the plurality of resultant logical units; and an evaluation module for determining whether the logical inference by which the generative large-scale model generates the text generation result is correct based on the logical inference diagram. A processing unit for model generation results.

9. The decomposition module comprises: A pre-trained logic decomposition model is used to decompose the text generation results of the generative large-scale model to obtain a plurality of result logic units. The apparatus for processing model generation results according to claim 8 .

10. The generation module: and generating a logic diagram characterizing a logic relationship between the plurality of result logic units using a pre-trained logic diagram generation model based on the plurality of result logic units. The apparatus for processing model generation results according to claim 8 .

11. The decomposition module further comprises: used to decompose a textual input of a generative large-scale model to obtain a plurality of input logic units, each of which includes a fragment in the textual input, and each of which can independently mark one premise or conclusion in a logical inference relationship; The apparatus for processing model generation results according to claim 8 .

12. The generation module: based on the plurality of result logic units, and with reference to the plurality of input logic units, to generate the logic inference diagram, which characterizes the logic inference relationships between the plurality of result logic units; The apparatus for processing model generation results according to claim 11 .

13. The generation module: Searching for a most relevant sample in a pre-defined sample database according to the plurality of input logical units and the plurality of result logical units, the sample database including a plurality of groups of samples, each of the samples including a plurality of input sample logical units corresponding to input sample information, a plurality of result sample logical units corresponding to result sample information, and a sample logic inference diagram corresponding to the plurality of result sample logical units; and generating the logic inference diagram corresponding to the plurality of result logic units using another pre-trained generative large-scale model based on the plurality of input logic units and the plurality of result logic units, the plurality of input sample logic units, the plurality of result sample logic units, and the corresponding sample logic inference diagram. The apparatus for processing model generation results according to claim 12.

14. The evaluation module includes: a decomposition unit for decomposing the logic diagram into a plurality of two-layer sub-diagrams, each of the two-layer sub-diagrams marking one logic diagram step; a judging unit for judging whether the logical inference of each of the two-layer sub-graphs is correct; a determining unit for determining that the logical inferences of the plurality of two-layer subgraphs are all correct, and determining that the logical inferences of the generative large-scale model that generates the text generation result are correct; A device for processing model generation results according to any one of claims 8 to 13.

15. An electronic device, at least one processor; a memory communicatively coupled to the at least one processor; The memory stores instructions executable by the at least one processor, the instructions, when executed by the at least one processor, causing the at least one processor to perform a method according to any one of claims 1 to 6. electronic equipment.

16. A non-transitory computer-readable storage medium having computer instructions stored thereon, comprising: The computer instructions cause the computer to perform a method according to any one of claims 1 to 6. A non-transitory computer-readable storage medium.

17. A computer program comprising: The computer program, when executed by a processor, implements the method according to any one of claims 1 to 6. Computer program.

Images