Method, device, electronic equipment, and storage medium for processing model generation result

By decomposing text generation results into logical units and generating reasoning diagrams, the method effectively evaluates the logical reasoning of large-scale generative models, enhancing evaluation efficiency and accuracy.

JP2024157003AActive Publication Date: 2024-11-06BEIJING BAIDU NETCOM SCI & TECH CO LTD
View PDF 9 Cites 0 Cited by

Patent Information

Application Number
JP2024135737
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-12-20
Filing Date
2024-08-15
Publication Date
2024-11-06
Estimated Expiration
2044-08-15

AI Technical Summary

Technical Problem

The evaluation of large-scale generative models is challenging due to the non-uniquely fixed nature of their text generation results, making it difficult to determine the correctness of logical reasoning within these models.

Method used

A method and apparatus for decomposing text generation results into logical units, generating a logical reasoning diagram, and evaluating the correctness of the logical reasoning using these units, facilitated by pre-trained models and logical decomposition and diagram generation techniques.

Benefits of technology

Enables efficient and accurate evaluation of the logical reasoning of generative large-scale models, improving evaluation efficiency and accuracy while reducing manual effort and time.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024157003000001_ABST
    Figure 2024157003000001_ABST
Patent Text Reader

Abstract

To provide a method, a device, electronic equipment, and a storage medium for processing a model generation result that efficiently and accurately determine whether logical inference of a generative large-scale model is correct.SOLUTION: A method for processing a model generative result comprises a step of decomposing a text generation result of a generative large-scale-model to acquire a plurality of result logical units. Each result logical unit includes a fragment in the text generation result, each fragment can independently mark one premise or one conclusion in a logical inference relation of the text generation result, and the text generation result is a response generated by the generative large-scale model on the basis of text input information. The method also comprises steps of: generating, on the basis of the plurality of result logical units, a logical inference diagram that a characterizes logical inference relationship between the plurality of result logical units; and determining, on the basis of the logical inference diagram, whether the logical inference by which the generative large-scale model generates the text generation result is correct.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to the field of computer technology, specifically to the field of artificial intelligence technology such as machine learning and natural language processing, and in particular to a method, device, electronic device, and storage medium for processing model generation results. [Background technology]

[0002] As generative large-scale models are widely applied, evaluating their effectiveness has become a very important technique.

[0003] Effectiveness evaluation of a large-scale generative model means judging whether the text generation results of the large-scale generative model are correct or not. Unlike task evaluation of conventional models, the effectiveness evaluation of a large-scale generative model is not uniquely fixed in the text generation results of the large-scale generative model, so it is not easy to evaluate whether the character strings match during the effectiveness evaluation. Summary of the Invention [Problem to be solved by the invention]

[0004] The present disclosure provides a method, an apparatus, an electronic device, and a storage medium for processing model generation results. [Means for solving the problem]

[0005] According to one aspect of the present disclosure, there is provided a method for processing model generation results, comprising: A step of decomposing the text generation result of the generative large-scale model to obtain a plurality of result logic units, each of the result logic units including a fragment in the text generation result, each of the fragments being capable of independently marking one premise or conclusion in a logical inference relationship of the text generation result, the text generation result being a response result generated by the generative large-scale model based on text input information; generating a logic inference diagram based on the plurality of result logic units, the logic inference diagram characterizing logic inference relationships between the plurality of result logic units; and evaluating, based on the logical inference diagram, whether the logical inference by which the generative large-scale model generates the text generation result is correct.

[0006] According to another aspect of the present disclosure, there is provided an apparatus for processing model generation results, comprising: A decomposition module for decomposing a text generation result of a generative large-scale model to obtain a plurality of result logical units, each of the result logical units including a fragment in the text generation result, each of the fragments being capable of independently marking one premise or conclusion in a logical inference relationship of the text generation result, the text generation result being a response result generated by the generative large-scale model based on text input information; a generating module for generating a logical inference diagram based on the plurality of result logic units, the logical inference diagram characterizing logical inference relationships between the plurality of result logic units; and an evaluation module for evaluating whether the logical inference by which the generative large-scale model generates the text generation result is correct based on the logical inference diagram.

[0007] According to another aspect of the present disclosure, there is provided an electronic device, comprising: At least one processor; a memory in communication with the at least one processor; The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to perform the methods of the aspects and any possible implementations.

[0008] According to another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium having computer instructions stored thereon, the computer instructions causing the computer to perform the method of the aspect and any possible implementations thereof.

[0009] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the method of the aspect and any possible implementation manner.

[0010] Based on the technology disclosed herein, it is possible to efficiently and accurately determine whether the logical inference of a large-scale generative formula model is correct, and further, it is possible to effectively improve the evaluation efficiency of a large-scale generative formula model.

[0011] It should be understood that the contents described herein are not intended to identify key or important features of the embodiments of the present disclosure, nor should they be used to limit the scope of the present disclosure. Other features of the present disclosure can be readily understood through the following specification. [Brief description of the drawings]

[0012] The drawings are for a better understanding of the present application and are not intended to limit the present application. [Figure 1] FIG. 1 is a schematic diagram according to a first embodiment of the present disclosure. [Diagram 2] FIG. 2 is a schematic diagram according to a second embodiment of the present disclosure. [Diagram 3] FIG. 1 is a logical reasoning diagram provided by the present disclosure. [Figure 4] FIG. 11 is a schematic diagram according to a third embodiment of the present disclosure. [Diagram 5] FIG. 2 is a schematic diagram of a logical inference diagram generated by the present embodiment. [Figure 6A] These are two-layer sub-diagrams of the logical reasoning diagram division shown in FIG. [Figure 6B] These are two-layer sub-diagrams of the logical reasoning diagram division shown in FIG. [Figure 6C] These are two-layer sub-diagrams of the logical reasoning diagram division shown in FIG. [Figure 7] FIG. 13 is a schematic diagram of a fourth embodiment of the present disclosure. [Figure 8] FIG. 13 is a schematic diagram of a fifth embodiment of the present disclosure. [Figure 9]FIG. 1 is a block diagram of an electronic device for implementing a method according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, exemplary embodiments of the present application will be described based on the drawings. For ease of understanding, various details of the embodiments of the present application are included, and they should be regarded as merely examples. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of simplicity, the following description omits the description of well-known functions and structures.

[0014] Apparently, the described embodiments are some embodiments of the present application, but not all embodiments. According to the embodiments of the present application, all other embodiments that a person skilled in the art can obtain without creative labor all belong to the scope of protection of the present application.

[0015] In addition, the terminal device related to the embodiment of the present application may include, but is not limited to, a mobile phone, a personal digital assistant (PDA (registered trademark)), a wireless handheld device, a tablet computer, and other smart devices. The display device may include, but is not limited to, a personal computer, a television, and other devices having a display function.

[0016] Additionally, the term "and / or" in this specification describes only the relation between related objects and indicates that three types of relations can exist, for example, A and / or B can indicate three cases: only A exists, A and B exist simultaneously, or only B exists. It should be understood that the sign " / " generally indicates that the related objects before and after it are in an "or" relationship.

[0017] 1 is a schematic diagram according to a first embodiment of the present disclosure. As shown in FIG. 1, the present embodiment provides a method for processing model generation results, which specifically can include the following steps: S101, decomposing the text generation result of the generative large-scale model to obtain multiple result logical units; Each result logic unit in this embodiment includes a fragment in the text generation result, and each fragment can independently mark one premise or conclusion in the logical reasoning relationship of the text generation result, and the text generation result is a response result generated by the generative large-scale model based on the text input information.

[0018] The entity that executes the model generation result processing method of this embodiment may be a processing device for the model generation results, which may be an independent electronic device or an application using a software base, and evaluates the generation effect of the generative large-scale model.

[0019] The generative large-scale model of this embodiment is also called a generative language model (General Language Model; GLM), and is also called a generative large-scale language model.

[0020] In this embodiment, the generative large-scale model is used in the text processing field as an example. When used, text input information is input to the generative large-scale model, and the generative large-scale model can generate and output text generation results according to the text input information. In actual application scenarios, in order to improve the effect of the generative large-scale model generating text generation results, some corresponding processing needs to be performed on the generation results of the generative large-scale model, which can realize the efficient and accurate determination of whether the logical inference of the generative large-scale model is correct, and can also perform accurate and effective evaluation of the generative large-scale model. Furthermore, based on the evaluation result, the generative large-scale model can be conversely optimized to further improve the text generation effect of the generative large-scale model.

[0021] The text generation results of generative large-scale models are usually long and can contain multiple sentences. A fragment in each result logical unit can be a single sentence in the text generation result, or two or more consecutive sentences. That is, a fragment in the text generation result is always a single continuous segment, not a concatenation of two or more intermittent segments.

[0022] Specifically, in this embodiment, the principle of decomposing the text generation result is to ensure that each result logical unit after decomposition serves as a premise or a conclusion when performing logical relational inference on the text generation result.

[0023] S102, generating a logical inference diagram based on the plurality of result logical units, the logical inference diagram characterizing logical inference relationships between the plurality of result logical units;

[0024] The logical inference diagram of this embodiment is a directed acyclic diagram in which a plurality of result logical units are constructed according to logical inference relationships.

[0025] S103, based on the logical inference diagram, determining whether the logical inference for generating the text generation result by the generative formula large-scale model is correct.

[0026] Since the logical inference diagram is generated based on the logical inference relationships between multiple result logical units, it is possible to evaluate whether the logical inference with which the generative large-scale model generates the text generation result is correct by referring to the logical inference relationships between multiple result logical units in the logical inference diagram, and further, it is possible to realize evaluation of the generative large-scale model from the logical inference dimension. That is, if the logical inference with which the generative large-scale model generates the text generation result is correct, it indicates that the logical inference of the generative large-scale model is correct, and if the logical inference with which the generative large-scale model generates the text generation result is incorrect, it indicates that the logical inference of the generative large-scale model is incorrect.

[0027] The method for processing the model generation result of this embodiment decomposes the text generation result to obtain a plurality of result logical units, generates a logical inference diagram of the plurality of result logical units, and then determines whether the logical inference of the generative large-scale model generating the text generation result is correct based on the logical inference diagram, thereby more effectively realizing the accurate determination of whether the logical inference of the generative large-scale model is correct. Therefore, the technical solution of this embodiment can be used to effectively evaluate the text generation result of the generative large-scale model efficiently and accurately from the level of logical inference, and more efficiently and accurately realize the effective evaluation of the generative large-scale model. In addition, the technical solution of this embodiment does not require manual processing, and can be fully automated, which can greatly reduce labor costs, improve the evaluation speed, and effectively improve the evaluation efficiency of the generative large-scale model.

[0028] FIG. 2 is a schematic diagram according to a second embodiment of the present disclosure. The method for processing the model generation result of this embodiment is based on the technical solution of the embodiment shown in FIG. 1 above, and the technical solution of this disclosure will be further described in more detail. As shown in FIG. 2, the method for processing the model generation result of this embodiment can specifically include the following steps: S201, using a pre-trained logic decomposition model to decompose the text generation result of the generative large-scale model to obtain a plurality of result logic units; This step is an implementation of step S101 in the embodiment shown in Figure 1. In this implementation, a pre-trained logic decomposition model is used to automatically decompose the text generation result. In a specific implementation, the text generation result is input to the logic decomposition model, which directly outputs a number of corresponding result logic units according to the input text generation result.

[0029] The logic decomposition model in this embodiment can collect multiple groups of training data before training, where each group of training data includes a training corpus and multiple training logic units obtained by manually decomposing the training corpus. The principle of decomposition ensures that each training logic unit after decomposition is a premise or a conclusion when performing logical relational reasoning on the training corpus. Then, the logic decomposition model can be trained using multiple groups of training data, so that the logic decomposition model can learn the ability of decomposition.

[0030] In this embodiment, a logical decomposition model is used to perform intelligent decomposition on the text generation result, so that the accuracy of multiple result logical units of the decomposition can be effectively improved, and the decomposition efficiency can be improved.

[0031] Optionally, step S101 of the embodiment shown in FIG. 1 can also decompose the text generation result of the generative formula large-scale model according to a preset decomposition policy to obtain multiple result logical units.

[0032] For example, a preset decomposition policy can be set to ensure that each resulting logical unit after decomposition is one premise or one conclusion in the logical inference relationship of the text generation result.

[0033] In concrete implementation, the text generation result can be firstly divided according to the sentence granularity, and then all fragment division methods can be traversed to obtain various division results of the text generation result. In each fragment division method, a sentence can be individually divided into a fragment, or two or more adjacent consecutive sentences can be divided into a fragment. Then, each division fragment in each division result can be detected to detect whether it can be a premise or a conclusion in the logical reasoning relationship of the text generation result. Finally, from all the division results, a division result in which all the decomposed fragments are all a premise or a conclusion can be selected as the required decomposition result, and the corresponding multiple result logical units can be obtained.

[0034] By decomposing according to a preset decomposition policy scheme, multiple result logical units can be obtained efficiently and accurately.

[0035] S202, based on the plurality of result logic units, using a pre-trained logic diagram generation model to generate a logic diagram characterizing a logic inference relationship between the plurality of result logic units; In this embodiment, one logical inference diagram generation model is trained in advance, and when used, multiple result logical units can be input to the logical inference diagram generation model, and the logical inference diagram generation model can generate a logical inference diagram composed of multiple result logical units based on the input information.

[0036] During training, the logical reasoning diagram generation model can further be obtained by training using multiple groups of training data, each of which can include multiple training logic units and training logical reasoning diagrams labeled based on the multiple training logic units. By training the logical reasoning diagram generation model using multiple groups of training data, the logical reasoning diagram generation model can learn the ability to generate corresponding training logical reasoning diagrams based on the multiple training logical units.

[0037] In use, a plurality of result logical units are input to the logical inference diagram generation model trained by the above-mentioned method, and the logical inference diagram generation model can generate and output a logical inference diagram that characterizes the logical inference relationships between the plurality of result logical units.

[0038] The method for intelligently generating a logical reasoning diagram based on the above logical reasoning diagram generation model can efficiently and accurately generate logical reasoning diagrams for multiple result logic units.

[0039] S203, decomposing the logical reasoning diagram into multiple two-layer sub-diagrams, each two-layer sub-diagram marking one logical reasoning step; Specifically, when decomposing a two-layer sub-diagram, the smallest logical relationship sub-diagram, i.e., the two-layer sub-diagram, is divided according to the reasoning relationship in the logical reasoning diagram. For example, FIG. 3 is a logical reasoning diagram provided by the present disclosure. According to the decomposition process of this embodiment, the logical reasoning diagram shown in FIG. 3 can be decomposed into two two-layer sub-diagrams, for example, a two-layer sub-diagram composed of A, B, and C, and a two-layer sub-diagram composed of C, D, and E. Each two-layer sub-diagram corresponds to one logical reasoning stage. Through this method, all two-layer sub-diagrams in the logical reasoning diagram can be decomposed.

[0040] S204, determining whether the logical inference of each two-layer sub-diagram is correct; Specifically, during the judgment, the pre-trained reasoning model can be used to infer whether the logical inference of each two-layer sub-diagram is correct. For example, during use, each two-layer sub-diagram can be input into the reasoning model in order from top to bottom and left to right in the logical inference diagram. Furthermore, the reasoning model sequentially judges whether the logical inference characterized by each input two-layer sub-diagram is correct. The reasoning model of this embodiment can also be realized using a large language model (LLM). Alternatively, in this embodiment, by setting a certain policy, it can also be judged whether the inference of each two-layer sub-diagram is correct, and it can also be accurately judged whether the logical inference of each two-layer sub-diagram is correct.

[0041] S205, in response to the logical inferences of the plurality of two-layer sub-diagrams all being correct, the generative large-scale model determines that the logical inference that generates the text generation result is correct.

[0042] In other words, unless the logical reasoning of one of the multiple two-layer subdiagrams is correct, the logical reasoning by which the generative large-scale model generates a text generation result is deemed to be incorrect, and further, the logical reasoning of the generative large-scale model is determined to be incorrect.

[0043] Steps S203-S205 of this embodiment are a specific implementation of step S103 of the embodiment shown in FIG.

[0044] Specifically, in this implementation method, a logical inference diagram is decomposed into multiple two-layer sub-diagrams, and a single logical inference stage of each two-layer sub-diagram is determined to be correct or not, thereby determining whether the logical inference used by the generative large-scale model to generate a text generation result is correct or not, and further determining whether the logical inference generated by the generative large-scale model is correct or not.

[0045] In this embodiment, the method judges only one two-layer sub-diagram each time, that is, only judges whether a single logical reasoning stage is correct at a time, which can effectively reduce the difficulty of judging whether the logical reasoning of the text generation result generated by the generative large-scale model is correct, effectively improve the accuracy of judging whether the logical reasoning of the generative large-scale model is correct, and further, can effectively improve the accuracy and evaluation efficiency of the text generation result of the generative large-scale model, and greatly improve the evaluation effect of the generative large-scale model.

[0046] The method for processing the model generation result of this embodiment uses the above-mentioned method to accurately judge whether the logical inference of the generative large-scale model is correct, thereby automatically realizing the evaluation of the generative large-scale model, saving time and labor, effectively shortening the evaluation speed, and improving the evaluation efficiency of the generative large-scale model. In addition, the logical inference of each two-layer sub-diagram decomposed by the logical inference diagram of the text generation result is judged to realize the judgment of the logical inference of the text generation result, and further, the evaluation of the text generation result of the generative large-scale model is realized, and the evaluation accuracy and evaluation efficiency can be effectively improved.

[0047] FIG. 4 is a schematic diagram according to a third embodiment of the present disclosure. The method for processing the model generation result of this embodiment is based on the technical solution of the embodiment shown in FIG. 1 above, and the technical solution of this disclosure will be further described in more detail. As shown in FIG. 4, the method for processing the model generation result of this embodiment can specifically include the following steps: S401, using a pre-trained logic decomposition model to respectively decompose the text input information and the text generation result of the generative large-scale model to obtain a plurality of input logic units and a plurality of result logic units; Each input logic unit includes a fragment in the text input information, and each fragment can independently mark one premise or conclusion in the logical reasoning relationship of the text input information. Each result logic unit includes a fragment in the text generation result, and each fragment can independently mark one premise or conclusion in the logical reasoning relationship of the text generation result. For a specific implementation manner, please refer to the relevant description of step S201 in the embodiment shown in Figure 2 above, and the description will be omitted here.

[0048] S402, based on the plurality of result logic units, generating a logic inference diagram characterizing a logic inference relationship between the plurality of result logic units with reference to the plurality of input logic units; That is, in the generation of a logical inference diagram of a plurality of result logic units in this embodiment, it is necessary to simultaneously refer to not only a plurality of result logic units but also a plurality of input logic units.

[0049] For example, when step S402 is specifically realized, the following (1) searching for the most relevant samples in a predefined sample database based on a plurality of input logical units and a plurality of result logical units; (2) generating a logic inference diagram corresponding to the plurality of result logic units based on the plurality of input logic units and the plurality of result logic units, and using a sample logic inference diagram corresponding to the plurality of input sample logic units, the sum of the plurality of result sample logic units, and other pre-trained generative large-scale models.

[0050] In the preset sample database of this embodiment, multiple groups of samples can be stored, and each group of samples can include a plurality of input sample logic units corresponding to input sample information, a plurality of result sample logic units corresponding to result sample information, and a sample logic inference diagram corresponding to the plurality of result sample logic units.

[0051] In this embodiment, the information in the sample database is also all in text format, that is, the input sample information in each group of samples is text input sample information, and the result sample information is text result sample information. That is, the input sample information can be the text input information of the generative large-scale model, and the result sample information can be the text generation result generated by the generative large-scale model. The multiple input sample logical units and the multiple result sample logical units can be obtained respectively by using the multiple input logical units and multiple result logical units decomposition method of step S401 mentioned above. The sample logical reasoning diagram of this embodiment can be manually labeled according to the multiple result sample logical units.

[0052] Specifically, in this embodiment, a most relevant sample can be searched for in a pre-defined sample database based on a plurality of input logical units and a plurality of result logical units, where the most relevant sample can be regarded as a sample corresponding to a plurality of input sample logical units and a plurality of result sample logical units that are most relevant in the overall context semantics of the plurality of input logical units and the plurality of result logical units.

[0053] Then, the multiple input logical units and the multiple result logical units, and each partial information included in the most relevant sample are combined, and the generative large-scale model is input together, and the pre-trained other generative large-scale model generates a sample logical inference diagram generation method based on the multiple result sample logical units in the most relevant sample, and generates a sample logical inference diagram corresponding to the multiple result sample logical units. In this embodiment, the pre-trained other generative large-scale model is not the generative large-scale model evaluated in this embodiment, but another generative large-scale model that has already been trained.

[0054] Alternatively, in this embodiment, each group of samples in the sample database may only include a plurality of result sample logic units corresponding to the result sample information, and a sample logic inference diagram corresponding to the plurality of result sample logic units.

[0055] In response to this case, when performing a specific search, the most relevant sample can be searched for in a pre-set sample database based only on the multiple result logical units, and a logical inference diagram corresponding to the multiple result logical units can be generated using another pre-trained generative large-scale model based on the sample logical inference diagram corresponding to the specific multiple result sample logical units.

[0056] However, this implementation has poor accuracy compared with the corresponding multiple input sample logical units and corresponding technical solutions in which the input sample information is simultaneously included in the above-mentioned sample database.

[0057] In this embodiment, by searching for the most relevant samples and using other generative large-scale models to intelligently generate logical inference diagrams, logical inference diagrams for multiple result logical units can be generated accurately and efficiently.

[0058] S403, decomposing the logical reasoning diagram into multiple two-layer sub-diagrams, each two-layer sub-diagram marking one logical reasoning step; S404, determining whether the logical inference of each two-layer sub-diagram is correct; S405, in response to the inferences of the multiple two-layer sub-graphs all being correct, the generative large-scale model determines that the logical inference that generates the text generation result is correct.

[0059] For the specific implementation of steps S403-S405 in this embodiment, reference may be made to the description of steps S203-S205 in the embodiment shown in FIG. 2, and the description will be omitted here.

[0060] The model generation result processing method of this embodiment can use the above-mentioned method to more accurately and efficiently generate logical inference diagrams of multiple result logical units, and can further improve the accuracy of determining whether the logical inference of the generative large-scale model is correct, and can effectively improve the evaluation accuracy and evaluation efficiency of the generative large-scale model.

[0061] The following provides a specific example to explain the method for evaluating generative large-scale models disclosed herein.

[0062] For instance, in the example, the text input information query is as follows: The police caught a thief and arrested four suspects, A, B, C, and D. Their statements are; A said: I did not steal. B said: A stole. C said: Not me. D said: B stole. Only one of them is known to have told the truth. Who is the thief? The text input information is input into a generative large-scale model, and the text generation result response generated by the generative large-scale model is, according to the subject line, "B said: A stole it. A said: I did not steal." Therefore, one of A and B must be telling the truth, and C and D must both be lying, i.e. C is lying and is a thief.

[0063] For example, based on step S401, the multiple input logic units obtained by decomposing the text input information query include: a first input logic unit (query-unit-1), the police caught a thief, and caught four suspects A, B, C, and D. Their statements are; a second input logic unit (query-unit-2), what A said: I did not steal; a third input logic unit (query-unit-3), what B said: A stole; a fourth input logic unit (query-unit-4), what C said: It was not me; a fifth input logic unit (query-unit-5), what D said: B stole; and a sixth input logic unit (query-unit-6), it is known that only one of them told the truth; a seventh input logic unit (query-unit-7), who is the thief?. Similarly, based on step S401, the multiple result logical units obtained by decomposing the text generation result response include: the first result logical unit (response-unit-1), what B said: A stole; the second result logical unit (response-unit-2), what A said: I did not steal; the third result logical unit (response-unit-3), so one of A and B must be telling the truth; the fourth input logical unit (response-unit-4), so C and D are all lying; the fifth input logical unit (response-unit-5), i.e. C is lying; and the sixth input logical unit (response-unit-6), C is a thief.

[0064] Then, a logic inference diagram of the multiple result logic units can be generated based on the method of step S302.

[0065] In this embodiment, the logical reasoning process is not a chain or tree structure, but a directed acyclic diagram. The logical reasoning process represents the reasoning process of the problem, and each node of the generated logical reasoning diagram is a minimum logical unit, i.e., a result logical unit. Each second-order sub-diagram in the logical reasoning diagram represents a minimum reasoning process.

[0066] Specifically, the sample database of this embodiment may be an In-Context-Learning (ICL) database.

[0067] Specifically, searching the ICL database based on the specific implementation of step S402 above can be called ICL search. Specifically, when searching, a plurality of input logical units and a plurality of result logical units are input, and the most relevant sample is searched for in the ICL database based on the input plurality of input logical units and the plurality of result logical units. A one-shot method is used to combine a plurality of input sample logical units, a plurality of result sample logical units, and a sample logical reasoning diagram in the most relevant sample in the combined prompt word.

[0068] Then, the input logic units, the result logic units, and the combined prompt words are used to generate a logic inference diagram corresponding to the result logic units using a generative large-scale model. For example, FIG. 5 is a schematic diagram of a logic inference diagram generated by this embodiment.

[0069] Finally, a logical judgment is made based on the logical reasoning diagram obtained in Fig. 5. In practical application, it is relatively difficult to judge the correctness of the entire logical reasoning diagram at one time, so the implementation method of steps S403-S405 mentioned above can be referred to: first, the logical reasoning diagram is decomposed into two-layer sub-diagrams one by one, each two-layer sub-diagram represents one logical reasoning stage, and only one two-layer sub-diagram is judged each time, that is, only whether a single logical reasoning stage is correct or not is judged at one time, which reduces the overall judgment difficulty and greatly improves the judgment effect.

[0070] The specific judgment process can be realized by using a logical reasoning model based on the LLM realization. The LLM is used to judge the correctness of the logical reasoning steps of each two-layer sub-diagram in the logical reasoning diagram in order from bottom to top, and if each logical reasoning step is correct, the entire logical reasoning process is judged to be correct, otherwise, it is judged that there is a logicality problem in the logical reasoning process. This method can not only judge the correctness of the logical reasoning that provides the logical reasoning problem, but also accurately position the step where the error occurs.

[0071] For example, Figures 6A, 6B, and 6C are two-layer sub-diagrams obtained by dividing the logical reasoning diagram shown in Figure 5. If it is determined that the logical reasoning stages of the three two-layer sub-diagrams in Figures 6A, 6B, and 6C are all correct, it can be determined that the logical reasoning of the text generation result generated by the generative large-scale model is correct, thereby realizing the evaluation of the generative large-scale model.

[0072] The above-mentioned method of the embodiment of the present disclosure evaluates one text generation result generated based on the generative large-scale model. In a practical application scenario, within one segment time, the generative large-scale model can execute multiple generation tasks. Specifically, according to the above-mentioned method of the embodiment of the present disclosure, when executing the generation task every time, it can evaluate whether the logical reasoning that the generative large-scale model generates the text generation result is correct. Furthermore, it can be statistically determined whether the accuracy rate of the stage time reaches a preset proportional threshold, such as 95%, 98%, or other proportionality, and if so, it is determined that the accuracy rate of the generative large-scale model is high and the effect is good; otherwise, it can be determined that the accuracy rate and effect of the generative large-scale model are not good.

[0073] Logic is the most important dimension that embodies the good or bad of the logical reasoning ability of a generative large-scale model, and is a necessary ability to solve some complex logical reasoning tasks. If a logical problem is found in the model text generation result in the evaluation stage, it can play an important role in the effect evaluation and optimization of logical reasoning problems. Compared with other dimensions, logicality is the embodiment of higher intelligence, and the difficulty of logicality evaluation is extremely high, and traditional methods are difficult to achieve relatively good results. Therefore, the processing method of the model generation result provided by the present disclosure is to evaluate the generative large-scale model from the perspective of logicality, which is an automatic evaluation framework of logical reasoning for generative large-scale models, and can more effectively evaluate the logical reasoning of the generative large-scale model, and can effectively improve the evaluation accuracy and evaluation effect of logical reasoning of the generative large-scale model.

[0074] The processing method for the above-mentioned model generation results in the embodiments of the present disclosure first divides logical units, then generates a logical reasoning diagram, obtains each two-layer sub-diagram in the logical reasoning diagram, and then evaluates each two-layer sub-diagram at a granularity, i.e., the minimum logical reasoning stage, which can greatly reduce the difficulty of determining whether the logical reasoning of the generative large-scale model is correct, i.e., can effectively reduce the difficulty of evaluating the generative large-scale model, and further greatly improve the evaluation effect of the generative large-scale model.

[0075] In addition, the above-described model generation result processing method of the embodiment of the present disclosure uses natural language to express the entire reasoning process, which is more versatile and can be applied to more types of problems.

[0076] In addition, the above-mentioned model generation result processing method of the embodiment of the present disclosure generates a logical reasoning diagram to represent the logical reasoning process of the overall logical reasoning, which can not only effectively detect logical reasoning errors in the text generation result, but also detect situations of missing logical reasoning and reversal of logical reasoning, and can be used for data logic repair.

[0077] In addition, the above-mentioned model generation result processing method of the embodiment of the present disclosure can accurately determine whether the logical inference of the generative large-scale model is correct, and can automatically realize the evaluation of the generative large-scale model, saving time and labor, effectively shortening the evaluation speed, and effectively improving the evaluation efficiency. In addition, by evaluating by the method of generating a logical inference diagram and by evaluating the logical inference of the smallest logical unit, i.e., each two-layer sub-diagram of the logical inference diagram, the accuracy of whether the logical inference of the generative large-scale model is correct can be further effectively improved, and the accuracy of the evaluation of the generative large-scale model can be effectively improved, and the evaluation efficiency can be improved.

[0078] The above embodiment of the present disclosure is applied to the field of text processing as an example, but in actual application, the technical solution of the present disclosure can also be applied to the field of voice processing. For example, voice information can be collected first, and corresponding text information can be obtained through voice recognition. Then, the above technical solution of the present disclosure can be used according to the text information to process the model generation result, and can efficiently and accurately determine whether the logical reasoning of the generative large-scale model is correct, and finally, can accurately and effectively evaluate the generative large-scale model.

[0079] 7 is a schematic diagram according to a fourth embodiment of the present disclosure. As shown in FIG. 7, this embodiment provides a model generation result processing apparatus 700, which is applied in the field of text processing, and includes: a decomposition module 701, a generation module 702, and an evaluation module 703; The decomposition module 701 is used to decompose the text generation result of the generative large-scale model to obtain a plurality of result logical units, each of the result logical units including a fragment in the text generation result, each of the fragments can independently mark one premise or conclusion in the logical inference relationship of the text generation result, and the text generation result is a response result generated by the generative large-scale model based on the text input information; a generating module 702 for generating, based on the plurality of result logic units, a logic diagram characterizing a logic relationship between the plurality of result logic units; The evaluation module 703 is used to evaluate whether the logical inference by which the generative large-scale model generates the text generation result is correct based on the logical inference diagram.

[0080] The model generation result processing device 700 of this embodiment realizes the implementation principle and technical effect of processing the model generation result using the above-mentioned modules, which is the same as the implementation of the embodiment of the related method described above. For details, please refer to the description of the embodiment of the related method described above, and the description will be omitted here.

[0081] 8 is a schematic diagram according to a fifth embodiment of the present disclosure. As shown in FIG. 8, this embodiment provides a model generation result processing device 800, which includes modules with the same names and functions as those shown in FIG. 7 above: a decomposition module 801, a generation module 802, and an evaluation module 803.

[0082] In the present embodiment, the model generation result processing device 800 includes a decomposition module 801: A pre-trained logical decomposition model is used to decompose the text generation result of the generative large-scale model to obtain multiple result logical units.

[0083] Further optionally, in one embodiment of the present disclosure, the generating module 802 is Based on the plurality of result logic units, a pre-trained logic diagram generation model is used to generate a logic diagram characterizing logic relationships among the plurality of result logic units.

[0084] Further optionally, in one embodiment of the present disclosure, the decomposition module 801 further comprises: A generative large-scale model is used to decompose textual input information to obtain a number of input logic units, each of which includes a fragment in the textual input information, and each of which can independently mark one premise or conclusion in a logical reasoning relationship.

[0085] Further optionally, in one embodiment of the present disclosure, the generating module 802 is Based on the plurality of result logic units, the plurality of input logic units is used to generate the logic inference diagram, which characterizes the logic inference relationships between the plurality of result logic units with reference to the plurality of input logic units.

[0086] Further optionally, in one embodiment of the present disclosure, the generating module 802 is Searching for a most relevant sample in a predefined sample database according to the plurality of input logic units and the plurality of result logic units, the sample database including a plurality of groups of samples, each of the samples including a plurality of input sample logic units corresponding to input sample information, a plurality of result sample logic units corresponding to result sample information, and a sample logic inference diagram corresponding to the plurality of result sample logic units; Based on the plurality of input logic units and the plurality of result logic units, the plurality of input sample logic units, the plurality of result sample logic units, and the corresponding sample logic inference diagrams, another pre-trained generative large-scale model is used to generate the logic inference diagram corresponding to the plurality of result logic units.

[0087] Further optionally, as shown in FIG. 8 , in one embodiment of the present disclosure, the evaluation module 803 includes: a decomposition unit 8031, a judgment unit 8032, and a decision unit 8033; The decomposition unit 8031 ​​is used for decomposing the logical reasoning diagram into a number of two-layer sub-diagrams, each of which marks one logical reasoning step; A judgment unit 8032 is used to judge whether the logical inference of each of the two-layer subgraphs is correct or not; A decision unit 8033 is used to decide that the logical inference by which the generative large-scale model generates the text generation result is correct in response to the logical inferences of the multiple two-layer sub-graphs all being correct.

[0088] The model generation result processing device 800 of this embodiment realizes the implementation principle and technical effect of processing the model generation result using the above-mentioned modules, which is the same as the implementation of the embodiment of the related method described above. For details, please refer to the description of the embodiment of the related method described above, and the description will be omitted here.

[0089] In the technical solution disclosed herein, the acquisition, storage, application, etc. of relevant user personal information shall conform to the provisions of relevant laws and regulations and shall not violate public order and morals.

[0090] According to an embodiment of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium, and a computer program product.

[0091] As shown in FIG. 9, a block diagram of an electronic device 900 for implementing an embodiment of the present disclosure is shown. The electronic device is intended to represent various types of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various types of mobile devices, such as personal digital assistants, mobile phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure as described herein and / or claimed.

[0092] 9, the device 900 includes a computing unit 901, which can perform various appropriate operations and processes based on a computer program stored in a read-only memory (ROM) 902 or loaded from a storage unit 908 into a random access memory (RAM) 903. The RAM 903 can also store various programs and data required for the device 900 to operate. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0093] Several components in device 900 are connected to an I / O interface 905, including input units 906 such as a keyboard, mouse, etc., output units 907 such as various types of displays, speakers, etc., storage units 908 such as disks, optical disks, etc., and communication units 909 such as a network card, modem, wireless communication transceiver, etc. The communication units 909 enable device 900 to exchange information / data with other devices via computer networks such as the Internet and / or various telecommunication networks.

[0094] The computing unit 901 is a general-purpose and / or dedicated processing component with various processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, computing units that execute various machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 901 executes the various methods and processes described above, such as the method of constructing a terrain map. For example, in some embodiments, the method of constructing a terrain map can be realized as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 908. In some embodiments, some or all of the computer program is loaded and / or installed in the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the above-mentioned methods can be performed. Alternatively, in other embodiments, the computing unit 901 can be configured to execute the method of constructing a terrain map via any other suitable manner (e.g., by firmware).

[0095] Various implementations of the systems and techniques described herein may be realized in digital electronic circuitry systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system on a chip (SOCs), programming programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include being implemented in one or more computer programs that may be executed and / or interpreted by a programmable system that includes at least one programmable processor, which may be an application specific or general purpose programmable processor, and that may receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device.

[0096] The program codes for implementing the methods of the present disclosure can be written using any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general purpose computer, a special purpose computer, or other programmable data processing apparatus such that the functions / operations specified in the flowcharts and / or block diagrams are implemented when the program codes are executed by the processor or controller. The program codes can be executed entirely on the machine, partially on the machine, partially on the machine as a separate software package and partially on a remote machine, or entirely on a remote machine or server.

[0097] In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain or store a program for use with or in combination with an instruction execution system, device, or apparatus. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium includes, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the above. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above.

[0098] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user, and a keyboard and pointing device (e.g., a mouse or trackball) by which a user can provide input to the computer. Other types of devices can also be used to provide interaction with a user, for example, feedback provided to the user can be any form of sensing feedback (e.g., visual feedback, auditory feedback, or haptic feedback) and can receive input from the user in any form (including acoustic input, speech input, and haptic input).

[0099] The systems and techniques described herein may be implemented in a computing system including a back-end component (e.g., a data server), or a middleware component (e.g., an application server), or a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user interacts with the implementation of the systems and techniques described herein), or any combination of such back-end, middleware, and front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0100] The computer system may include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The relationship between the client and the server is created by computer programs running on corresponding computers and having a client-server relationship to each other. The server may be a cloud server, a server of a distributed system, or a server for combining blockchains.

[0101] It should be understood that steps can be rearranged, added, or deleted using various types of flows shown above. For example, each step described in the present disclosure can be performed in parallel, sequentially, or in a different order, but is not limited herein as long as the technical solution disclosed in the present disclosure can achieve the desired results.

[0102] The above specific implementation methods do not constitute a limitation of the protection scope of the present disclosure. Those skilled in the art can make various modifications, combinations, subcombinations, and substitutions based on design requirements and other factors. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present disclosure should be included in the protection scope of the present disclosure.

Claims

1. A method for processing model generation results, applied in the field of text processing, comprising the steps of: A step of decomposing the text generation result of the generative large-scale model to obtain a plurality of result logic units, each of the result logic units including a fragment in the text generation result, each of the fragments being capable of independently marking one premise or conclusion in a logical inference relationship of the text generation result, the text generation result being a response result generated by the generative large-scale model based on text input information; generating a logic inference diagram based on the plurality of result logic units, the logic inference diagram characterizing logic inference relationships between the plurality of result logic units; and determining whether the logical inference generating the text generation result of the generative formula large-scale model is correct based on the logical inference diagram. How to process the model generation results.

2. The step of decomposing the text generation result of the generative large-scale model to obtain a plurality of result logical units includes: Using a pre-trained logic decomposition model to decompose the text generation results of the generative large-scale model to obtain a plurality of result logic units; A method for processing model generation results according to claim 1.

3. generating a logical inference diagram based on the plurality of result logic units, the logical inference diagram characterizing logical inference relationships between the plurality of result logic units, generating a logic diagram based on the plurality of result logic units, the logic diagram characterizing a logic relationship between the plurality of result logic units, using a pre-trained logic diagram generation model; A method for processing model generation results according to claim 1.

4. Prior to the step of generating a logical inference diagram based on the plurality of result logic units, the logical inference diagram characterizing logical inference relationships between the plurality of result logic units, the method further comprises: and decomposing the textual input of the generative large-scale model to obtain a plurality of input logic units, each of the input logic units including a fragment in the textual input, each of the fragments being capable of independently marking one premise or conclusion in a logical reasoning relationship. A method for processing model generation results according to claim 1.

5. generating a logical inference diagram based on the plurality of result logic units, the logical inference diagram characterizing logical inference relationships between the plurality of result logic units, generating a logic inference diagram based on the plurality of result logic units and with reference to the plurality of input logic units, the logic inference diagram characterizing logic inference relationships between the plurality of result logic units; A method for processing model generation results according to claim 4.

6. generating a logical inference diagram based on the plurality of result logic units and characterizing logical inference relationships between the plurality of result logic units with reference to the plurality of input logic units, comprising: Searching for a most relevant sample in a predefined sample database according to the plurality of input logic units and the plurality of result logic units, the sample database including a plurality of groups of samples, each of the samples including a plurality of input sample logic units corresponding to input sample information, a plurality of result sample logic units corresponding to result sample information, and a sample logic inference diagram corresponding to the plurality of result sample logic units; and generating the logic inference diagram corresponding to the plurality of result logic units using another pre-trained generative large-scale model based on the plurality of input logic units and the plurality of result logic units, the plurality of input sample logic units, the plurality of result sample logic units, and the corresponding sample logic inference diagram. A method for processing model generation results according to claim 5.

7. The step of determining whether the logical inference by which the generative large-scale model generates the text generation result is correct based on the logical inference diagram includes: decomposing the logic diagram into a plurality of two-layer sub-diagrams, each of the two-layer sub-diagrams marking one logic reasoning step; determining whether the logical inference of each of the two-layer subgraphs is correct; determining, in response to the logical inferences of the plurality of two-layer subgraphs all being correct, that the logical inferences by which the generative large-scale model generates the text generation result are correct; A method for processing model generation results according to any one of claims 1 to 6.

8. A device for processing model generation results, applied in the field of text processing, comprising: A decomposition module for decomposing a text generation result of a generative large-scale model to obtain a plurality of result logical units, each of the result logical units including a fragment in the text generation result, each of the fragments being capable of independently marking one premise or conclusion in a logical inference relationship of the text generation result, the text generation result being a response result generated by the generative large-scale model based on text input information; a generating module for generating a logical inference diagram based on the plurality of result logic units, the logical inference diagram characterizing logical inference relationships between the plurality of result logic units; and an evaluation module for determining whether a logical inference for generating the text generation result from the generative formula large-scale model is correct based on the logical inference diagram. A device for processing model generation results.

9. The decomposition module includes: A pre-trained logic decomposition model is used to decompose the text generation result of the generative formula large-scale model to obtain a plurality of result logic units. The apparatus for processing model generation results according to claim 8.

10. The generation module includes: and generating a logic diagram characterizing a logic inference relationship between the plurality of result logic units using a pre-trained logic inference diagram generation model based on the plurality of result logic units. The apparatus for processing model generation results according to claim 8.

11. The decomposition module further comprises: A generative large-scale model is used to decompose a textual input to obtain a plurality of input logic units, each of which includes a fragment in the textual input, and each of which can independently mark one premise or conclusion in a logical reasoning relationship. The apparatus for processing model generation results according to claim 8.

12. The generation module includes: based on the plurality of result logic units and with reference to the plurality of input logic units, to generate the logic inference diagram, the logic inference diagram characterizing the logic inference relationships between the plurality of result logic units. A device for processing model generation results according to claim 11.

13. The generation module includes: Searching for a most relevant sample in a predefined sample database according to the plurality of input logic units and the plurality of result logic units, the sample database including a plurality of groups of samples, each of the samples including a plurality of input sample logic units corresponding to input sample information, a plurality of result sample logic units corresponding to result sample information, and a sample logic inference diagram corresponding to the plurality of result sample logic units; According to the plurality of input logic units and the plurality of result logic units, the plurality of input sample logic units, the plurality of result sample logic units, and the corresponding sample logic inference diagrams, another pre-trained generative large-scale model is used to generate the logic inference diagrams corresponding to the plurality of result logic units. A device for processing model generation results according to claim 12.

14. The evaluation module includes: a decomposition unit for decomposing the logic diagram into a plurality of two-layer sub-diagrams, each of the two-layer sub-diagrams marking one logic diagram step; a judging unit for judging whether the logical inference of each of the two-layer sub-graphs is correct; a determining unit for determining that the logical inferences of the plurality of two-layer sub-graphs are all correct, the logical inferences of the generative large-scale model generating the text generation result are correct; A device for processing a model generation result according to any one of claims 8 to 13.

15. An electronic device, At least one processor; a memory in communication with the at least one processor; The memory stores instructions executable by the at least one processor, the instructions, when executed by the at least one processor, causing the at least one processor to perform a method according to any one of claims 1 to 6. electronic equipment.

16. A non-transitory computer-readable storage medium having computer instructions stored thereon, comprising: The computer instructions cause the computer to carry out a method according to any one of claims 1 to 6. A non-transitory computer-readable storage medium.

17. A computer program comprising: The computer program, when executed by a processor, implements the method according to any one of claims 1 to 6. Computer program.

Citation Information

Patent Citations

  • Text generation method, device and apparatus and storage medium

    CN112380848A

  • Language model training method and device based on knowledge graph

    CN116796004A

  • Language generation method and device and model construction method and device based on computer database

    CN116842049A

  • Knowledge graph relation reasoning method based on large model

    CN117114094A

  • Acquisition device, acquisition method, and acquisition program

    JP2020204966A