Method and device for generating reasoning result, medium and electronic equipment

Abstract

The invention provides a reasoning result generation method and device, a computer storage medium and electronic equipment, and relates to the technical field of artificial intelligence. Comprising the steps of obtaining a to-be-reasoned problem, generating a reasoning example set composed of a target number of reasoning examples corresponding to the to-be-reasoned problem, inputting the to-be-reasoned problem and the reasoning example set into a preset large language model, and enabling the large language model to perform reasoning answering on the to-be-reasoned problem based on each reasoning example in the reasoning example set, obtaining a first reasoning result corresponding to the to-be-reasoned problem; based on a large language model, determining a target reasoning problem matched with the first reasoning result; and searching the target reasoning problem through a search engine, and correcting the first reasoning result according to a search result to obtain a corrected second reasoning result. The generation of inaccurate reasoning results of the model can be reduced, and the fact correctness is improved, so that the model performance is improved, the credibility and effectiveness of the application are improved, and the credibility of the user is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a method for generating an inference result, a device for generating an inference result, a computer storage medium, and an electronic device. Background Art

[0002] With the rapid development of artificial intelligence, Large Language Model (LLM) can handle a variety of complex natural language tasks, such as text classification, intelligent question and answer, human-computer dialogue, etc., with its powerful intent recognition ability and deep understanding of the meaning of language text. Thinking chain is a prompting method that helps to improve the logical reasoning ability of LLM model. It can decompose complex problems into multiple logically coherent intermediate steps, which in turn helps LLM model to solve complex tasks more effectively and improve the performance of LLM model.

[0003] However, an important flaw of the thought chain is its lack of factual correctness, which makes the LLM model have a certain tendency to fabricate facts and details, resulting in the generation of inaccurate reasoning content, which seriously affects the performance of the LLM model, weakens the credibility and effectiveness of the application, and thus reduces the trust of users. Summary of the invention

[0004] The present disclosure provides a method for generating inference results, a device for generating inference results, a computer storage medium and an electronic device, thereby reducing the generation of inaccurate inference results of a model, improving the correctness of facts, thereby improving model performance, enhancing the credibility and effectiveness of applications, and thereby improving user trust.

[0005] In a first aspect, an embodiment of the present disclosure provides a method for generating a reasoning result, the method comprising: obtaining a problem to be reasoned, generating a target number of reasoning examples corresponding to the problem to be reasoned, obtaining a reasoning example set, each reasoning example in the reasoning example set being used to characterize an example of obtaining a corresponding reasoning result from the problem to be reasoned through at least one intermediate reasoning step; inputting the problem to be reasoned and the reasoning example set into a preset large language model, so that the large language model performs reasoning and answering on the problem to be reasoned based on each reasoning example in the reasoning example set, and obtains a first reasoning result corresponding to the problem to be reasoned; based on the large language model, determining a target reasoning problem that matches the first reasoning result; searching for the target reasoning problem through a search engine, and correcting the first reasoning result according to the search results, to obtain a corrected second reasoning result.

[0006] In an optional embodiment of the present disclosure, a reasoning example set consisting of a target number of reasoning examples corresponding to the problem to be reasoned is generated, including: determining an uncertainty score of the large language model for the reasoning problem; generating a number of reasoning examples corresponding to the uncertainty score to obtain a reasoning example set.

[0007] In an optional embodiment of the present disclosure, determining an uncertainty score of a large language model for a reasoning problem includes:

[0008] Based on the large language model, multiple reasoning results corresponding to the problem to be reasoned are generated;

[0009] Determine the degree of uncertainty of multiple inference results based on the large language model to obtain the uncertainty score of the large language model for the reasoning problem; or, determine the proportion of the same inference results from multiple inference results, and determine the uncertainty score of the large language model for the reasoning problem based on the proportion of the same inference results; or, determine the variance values ​​of multiple inference results, and determine the uncertainty score of the large language model for the reasoning problem based on the variance values; or, determine the entropy values ​​of multiple inference results, and determine the uncertainty score of the large language model for the reasoning problem based on the entropy values.

[0010] In an optional embodiment of the present disclosure, the method further includes: decomposing the reasoning problem to be reasoned based on a thinking chain prompt template to obtain multiple sub-problems corresponding to the problem to be reasoned, so as to generate a target number of reasoning examples corresponding to each sub-problem corresponding to the problem to be reasoned.

[0011] In an optional embodiment of the present disclosure, a problem to be reasoned and a set of reasoning examples are input into a preset large language model, so that the large language model performs reasoning and answers to the problem to be reasoned based on each reasoning example in the set of reasoning examples, and obtains a first reasoning result corresponding to the problem to be reasoned, including: rewriting the problem to be reasoned multiple times, and inputting the multiple rewritten problems to be reasoned and the set of reasoning examples into the large language model respectively, so that the large language model performs reasoning and answers to the multiple rewritten problems to be reasoned based on each reasoning example in the set of reasoning examples, and obtains multiple initial reasoning results; performing consistency verification on the multiple initial reasoning results according to preset detection rules to obtain consistency verification results, and determining the first reasoning result according to the consistency verification results.

[0012] In an optional embodiment of the present disclosure, a consistency check of the inference results is performed on a plurality of initial inference results according to a preset detection rule to obtain a consistency check result, and a first inference result is determined according to the consistency check result, including:

[0013] Performing statistical analysis of the inference conclusions of the multiple initial inference results to achieve consistency verification, if the inference conclusions corresponding to the multiple initial inference results are consistent, then determining any initial inference result among the multiple initial inference results as the first inference result;

[0014] Alternatively, consistency check is implemented by respectively performing semantic analysis on multiple initial reasoning results based on embedded similar sentences, and if the semantics corresponding to the multiple initial reasoning results are the same, any initial reasoning result among the multiple initial reasoning results is determined as the first reasoning result;

[0015] Alternatively, the verification result of consistency verification achieved by statistical analysis of reasoning conclusions of multiple initial reasoning results is weighted and merged with the verification result of consistency verification achieved by semantic analysis of multiple initial reasoning results based on embedded similar sentences to obtain the first reasoning result.

[0016] In an optional embodiment of the present disclosure, a first reasoning result is corrected according to the search results to obtain a corrected second reasoning result, including: if the search results contain at least numerical information, numerically matching the numerical information in the search results with the numerical information of the first reasoning result, so that when the numerical values ​​do not match, the first reasoning result is corrected based on the numerical information in the search results to obtain a corrected second reasoning result; if the search results contain at least text information, determining whether the semantic information of the search results is consistent with the semantic information of the first reasoning result based on a large language model, so that when the semantic information does not match, the first reasoning result is corrected based on the semantic information of the search results to obtain a corrected second reasoning result.

[0017] In a second aspect, an embodiment of the present disclosure provides a device for generating a reasoning result, the device comprising: a reasoning example generation module, used to obtain a problem to be reasoned, generate a target number of reasoning examples corresponding to the problem to be reasoned, and obtain a reasoning example set, each reasoning example in the reasoning example set is used to represent an example of obtaining a corresponding reasoning result of the problem to be reasoned through at least one intermediate reasoning step; a reasoning answer module, used to input the problem to be reasoned and the reasoning example set into a preset large language model, so that the large language model performs reasoning and answering on the problem to be reasoned based on each reasoning example in the reasoning example set, and obtains a first reasoning result corresponding to the problem to be reasoned; a problem determination module, used to determine a target reasoning problem matching the first reasoning result based on the large language model; and a reasoning result correction module, used to search for the target reasoning problem through a search engine, and correct the first reasoning result according to the search result to obtain a corrected second reasoning result.

[0018] In an optional embodiment of the present disclosure, the reasoning example generation module is specifically used to determine the uncertainty score of the large language model for the reasoning problem; generate a number of reasoning examples corresponding to the uncertainty score, and obtain a reasoning example set.

[0019] In an optional embodiment of the present disclosure, the reasoning example generation module is specifically used to generate multiple reasoning results corresponding to the problem to be reasoned based on the large language model;

[0020] Determine the uncertainty of multiple inference results based on the large language model, and obtain the uncertainty score of the large language model for the inference problem;

[0021] Alternatively, the reasoning example generation module is specifically used to generate multiple reasoning results corresponding to the problem to be reasoned based on the large language model; determine the proportion of the same reasoning results from the multiple reasoning results, and determine the uncertainty score of the large language model for the reasoning problem based on the proportion of the same reasoning results;

[0022] Alternatively, the reasoning example generation module is specifically used to generate multiple reasoning results corresponding to the problem to be reasoned based on the large language model; determine the variance values ​​of the multiple reasoning results, and determine the uncertainty score of the large language model for the reasoning problem based on the variance values;

[0023] Alternatively, the reasoning example generation module is specifically used to generate multiple reasoning results corresponding to the problem to be reasoned based on the large language model; determine the entropy values ​​of the multiple reasoning results, and determine the uncertainty score of the large language model for the reasoning problem based on the entropy value.

[0024] In an optional embodiment of the present disclosure, the device further includes a problem decomposition module, which is used to decompose the reasoning problem to be reasoned based on a thinking chain prompt template to obtain multiple sub-problems corresponding to the problem to be reasoned, so as to generate a target number of reasoning examples corresponding to each sub-problem corresponding to the problem to be reasoned.

[0025] In an optional embodiment of the present disclosure, a reasoning and answering module is used to rewrite a problem to be reasoned for multiple times, and input the multiple rewritten problems to be reasoned and a set of reasoning examples into a large language model respectively, so that the large language model performs reasoning and answers to the multiple rewritten problems to be reasoned based on each reasoning example in the set of reasoning examples to obtain multiple initial reasoning results; the consistency of the reasoning results is checked on the multiple initial reasoning results according to preset detection rules to obtain a consistency check result, and the first reasoning result is determined according to the consistency check result.

[0026] In an optional embodiment of the present disclosure, the reasoning answer module is used to perform statistical analysis of the reasoning conclusions of the multiple initial reasoning results to achieve consistency verification, and if the reasoning conclusions corresponding to the multiple initial reasoning results are consistent, any initial reasoning result among the multiple initial reasoning results is determined as the first reasoning result;

[0027] Alternatively, the reasoning answer module is used to implement consistency verification by performing semantic analysis on the multiple initial reasoning results based on the embedded similar sentences, and if the semantics corresponding to the multiple initial reasoning results are the same, then any initial reasoning result among the multiple initial reasoning results is determined as the first reasoning result;

[0028] Alternatively, the reasoning solution module is used to perform statistical analysis of the reasoning conclusions of multiple initial reasoning results to achieve consistency verification, and to weightedly merge the verification results of the consistency verification achieved by semantic analysis of the multiple initial reasoning results based on embedded similar sentences to obtain a first reasoning result.

[0029] In an optional embodiment of the present disclosure, the inference result correction module is used to, if the search results contain at least numerical information, numerically match the numerical information in the search results with the numerical information of the first inference result, so as to correct the first inference result based on the numerical information in the search results when the numerical values ​​do not match, to obtain a corrected second inference result; if the search results contain at least text information, determine whether the semantic information of the search results is consistent with the semantic information of the first inference result based on a large language model, so as to correct the first inference result based on the semantic information of the search results when the semantic information does not match, to obtain a corrected second inference result.

[0030] In a third aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, which implements the above method for generating reasoning results when executed by a processor.

[0031] In a fourth aspect, an embodiment of the present disclosure provides an electronic device, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the above method for generating reasoning results by executing the executable instructions.

[0032] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, which is executed by a processor to implement the above method for generating reasoning results.

[0033] The technical solution disclosed in this disclosure has the following beneficial effects:

[0034] The above-mentioned method for generating reasoning results obtains a problem to be reasoned, generates a target number of reasoning examples corresponding to the problem to be reasoned, and obtains a reasoning example set, wherein each reasoning example in the reasoning example set is used to represent an example of obtaining a corresponding reasoning result of the problem to be reasoned through at least one intermediate reasoning step; the problem to be reasoned and the reasoning example set are input into a preset large language model, so that the large language model performs reasoning and answering on the problem to be reasoned based on each reasoning example in the reasoning example set, and obtains a first reasoning result corresponding to the problem to be reasoned; based on the large language model, a target reasoning problem matching the first reasoning result is determined; the target reasoning problem is searched through a search engine, and the first reasoning result is corrected according to the search result to obtain a corrected second reasoning result.

[0035] On the one hand, the reasoning examples generated in the method include intermediate reasoning steps generated based on the thinking chain, thereby forming a factual reasoning chain, so that the large language model generates reasoning results with reasoning processes according to the intermediate steps in the reasoning examples, which helps to improve the interpretability of the model, so that users can more clearly and intuitively understand the processing logic of the model in the process of implementing reasoning. On the other hand, the introduction of reasoning examples into the large language model so that the large language model can reason and answer the reasoning problems based on each reasoning example in the reasoning example set helps to improve the performance of the large language model in complex reasoning tasks, and reduce the large language model when receiving the content to be reasoned input by the user. The inability to accurately understand the user's intention, so as to improve the accuracy of the model-generated reasoning results. On the other hand, by introducing external knowledge through a search engine to verify the authenticity of the reasoning results generated by the model, the content of the reasoning results generated by the model can be better controlled, thereby ensuring that the content of the generated reasoning results is consistent with the external knowledge, achieving the technical effect of reducing the generation of inaccurate text and improving the factual correctness of the reasoning results.

[0036] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and are used together with the specification to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0038] Figure 1 An architectural diagram showing a system for generating an inference result according to the exemplary embodiment;

[0039] Figure 2A flowchart showing a method for generating an inference result according to the exemplary embodiment;

[0040] Figure 3 A flowchart showing a method for generating multiple execution reasoning results according to the exemplary embodiment;

[0041] Figure 4 A flow chart showing a method for generating a complete reasoning result in this exemplary embodiment;

[0042] Figure 5 A flowchart showing another method for generating a complete reasoning result in this exemplary embodiment;

[0043] Figure 6 A schematic diagram of the structure of a device for generating an inference result in this exemplary embodiment is shown;

[0044] Figure 7 The structure diagram of an electronic device in this exemplary embodiment is schematically shown. DETAILED DESCRIPTION

[0045] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, exemplary embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the exemplary embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0046] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0047] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the steps. For example, some steps may be decomposed, while some steps may be combined or partially combined, so the actual execution order may change according to the actual situation.

[0048] In order to help those skilled in the art better understand the technical solution of the present disclosure, the relevant contents involved in the technical solution of the present disclosure are introduced below.

[0049] 1) Chain of Thought (CoT): In the field of artificial intelligence, CoT is a series of intermediate reasoning steps. Through CoT, the large model can decompose a complex problem into logically coherent sub-problems and solve them according to the steps of process processing, thereby significantly improving the performance of the large model.

[0050] 2) Large Language Model (LLM): refers to a deep learning model that is trained using a large amount of unlabeled text data. It can generate natural language text or understand the meaning of language text. Through the LLM model, users can interact with it through natural language to perform various complex tasks, such as intelligent question and answer, text classification, summarization, translation, and intelligent chat.

[0051] With the rapid development of artificial intelligence, the LLM model can handle a variety of complex natural language tasks, such as text classification, intelligent question and answer, and human-computer dialogue, with its powerful intent recognition ability and deep understanding of the meaning of language text. The thinking chain is a prompting method that helps to improve the logical reasoning ability of the LLM model. It can decompose complex problems into multiple logically coherent intermediate steps, which in turn helps the LLM model to solve complex tasks more effectively and improve the performance of the LLM model. Through the thinking chain prompt, explainable factual reasoning chains can be generated. These factual reasoning chains can be used to assist the LLM model in generating more accurate reasoning results, thereby improving the reliability and processing performance of the LLM model in processing complex reasoning tasks. Among them, complex reasoning tasks include tasks such as mathematical applications, common sense reasoning, and symbolic reasoning.

[0052] However, an important defect of the thought chain is that it lacks factual correctness, which makes the LLM model have a certain tendency to fabricate facts and details, resulting in inaccurate reasoning content, which seriously affects the performance of the LLM model, weakens the credibility and effectiveness of the application, and reduces the trust of users. And the inventors found that the above problem is not easy to solve, because the decoding process of the LLM model is autoregressive, and inaccurate reasoning content is inevitably generated in the process. Therefore, when generating reasoning content based on the thought chain, ensuring the factual correctness of the thought chain is one of the key technologies to improve the quality of reasoning content generated by the LLM model.

[0053] In view of the above technical problems, the exemplary embodiment of the present disclosure proposes a method for generating reasoning results, which can generate reasoning results for the questions to be reasoned input by the user, especially for the complex questions to be reasoned input by the user. That is, the reasoning examples generated in the method include intermediate reasoning steps generated based on the thinking chain, thereby forming a factual reasoning chain, so that the large language model generates a reasoning result with a reasoning process according to the intermediate steps in the reasoning example, thereby helping to improve the interpretability of the model, so that the user can more clearly and intuitively understand the processing logic of the model in the process of implementing the reasoning. On the other hand, the reasoning example is introduced into the large language model, so that the large language model can reason and answer the reasoning problem based on each reasoning example in the reasoning example set, which helps to improve the performance of the large language model in complex reasoning tasks, and reduce the large language model when receiving the content to be reasoned input by the user. The user's intention cannot be accurately understood, so as to improve the accuracy of the reasoning result generated by the model. On the other hand, by introducing external knowledge through the search engine to verify the authenticity of the reasoning result generated by the model, the content of the reasoning result generated by the model can be better controlled, thereby ensuring that the generated reasoning result content is consistent with the external knowledge, achieving the technical effect of reducing the generation of inaccurate text and improving the factual correctness of the reasoning result.

[0054] The following is an example of an application scenario architecture of a method and device for generating an inference result proposed in the present disclosure. For example, the method and device can be applied to Figure 1 In the system architecture of the exemplary application environment shown.

[0055] Figure 1 A system architecture diagram for generating inference results in one application scenario of this exemplary embodiment is shown as follows: Figure 1 As shown, in this scenario, a terminal device 101 and a server 102 may be included.

[0056] The terminal device 101 can be, for example, a mobile phone, a tablet computer (PAD), a laptop computer, a desktop computer, a smart TV, a smart vehicle-mounted device, a smart wearable device, a smart TV, and an aircraft, and any other device that involves realizing tasks such as intelligent question and answer, common sense, and symbolic reasoning. The terminal device 101 can be installed with a target application, and the target application can have functions such as inputting questions to be reasoned and displaying reasoning results, such as instant messaging applications, music applications, game applications, video applications, short video applications, navigation applications, news applications, and shopping applications, etc., and the embodiments of the present disclosure do not impose any special restrictions on this. The application involved in the embodiments of the present application can be a software client, or it can be a client such as a web page or a small program. The server 102 is a server corresponding to software or a web page, a small program, etc., and does not limit the specific type of the client.

[0057] The server 102 may be a background server of the target application, used to provide corresponding background services for it, such as reasoning result generation service, etc. It may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, i.e., content delivery networks (CDN), as well as big data and artificial intelligence platforms, but is not limited thereto.

[0058] It should be noted that the method for generating recommendation reasons in the embodiment of the present application can be executed by the terminal device 101 or the server 102 alone, or can be executed jointly by the server 102 and the terminal device 101. When executed by the terminal device 101 or the server 102 alone, the application process of the large language model can be implemented by the terminal device 101 or the server 102 alone, or the above process can also be performed by the server 102. When executed jointly by the server 102 and the terminal device 101, the server 102 can train the large language model, and then deploy the trained large language model to the terminal device 101, and the terminal device 101 can implement the recommendation reason generation process, or part of the model training or application process can be implemented by the terminal device 101, and part of the process can be implemented by the server 102, and the two can cooperate to implement the model training or application process. In actual application, specific configuration can be made according to the situation, and this application does not make specific limitations here.

[0059] Among them, the server 102 and the terminal device 101 may include one or more processors, memories, and interactive I / O interfaces, etc. In addition, the server 102 may also be configured with a database, which may be used to store the model parameters obtained by training, etc. Among them, the memories of the server 102 and the terminal device 101 may also store the program instructions required to be executed in the method for generating a recommendation reason provided in the embodiment of the present application, and these program instructions, when executed by the processor, can be used to implement the model training process or the recommendation reason generation process provided in the embodiment of the present application.

[0060] It should be noted that when the method for generating a recommendation reason provided in the embodiment of the present disclosure is executed by the server 102 or the terminal device 101 alone, the above-mentioned application scenario may also only include a single device of the server 102 or the terminal device 101, or the server 102 and the terminal device 101 may also be considered to be the same device. Of course, in actual application, when the method for generating a recommendation reason provided in the embodiment of the present disclosure is jointly executed by the server 102 and the terminal device 101, the server 102 and the terminal device 101 may also be the same device, that is, the server 102 and the terminal device 101 may be different functional modules of the same device, or virtual devices virtualized from the same physical device.

[0061] In the embodiment of the present application, the terminal device 101 and the server 102 can be directly or indirectly connected to each other through one or more networks 103. The network 103 can be a wired network or a wireless network. For example, the wireless network can be a mobile cellular network or a wireless fidelity (Wireless-Fidelity, WIFI) network. Of course, it can also be other possible networks, which are not limited in the embodiment of the present application. It should be noted that Figure 1 The figure is only an example. In fact, the number of terminal devices and servers is not limited and is not specifically limited in the embodiments of the present application.

[0062] For example, in an exemplary embodiment, the target application may be carried by the terminal device 101 so that the user can input the problem to be inferred from the target application interface. The server 102 may obtain the problem to be inferred, generate a reasoning example set consisting of a target number of reasoning examples corresponding to the problem to be inferred, and each reasoning example in the reasoning example set is used to characterize an example of obtaining a corresponding reasoning result from the problem to be inferred through at least one intermediate reasoning step; the server 102 inputs the problem to be inferred and the reasoning example set into a preset large language model, so that the large language model performs reasoning and answering on the problem to be inferred based on each reasoning example in the reasoning example set, and obtains a first reasoning result corresponding to the problem to be inferred; based on the large language model, determine the target reasoning problem matching the first reasoning result; search the target reasoning problem through a search engine, and modify the first reasoning result according to the search results to obtain a modified second reasoning result. Among them, the data such as the large language model and model parameters may be stored in a database associated with the server 102 (not shown in the figure).

[0063] Below, in combination with the application scenarios described above, the method provided by the exemplary embodiment of the present application is described with reference to the accompanying drawings. It should be noted that it is easy for those skilled in the art to understand that the above application scenarios are only shown to facilitate understanding of the spirit and principle of the present application, and the implementation of the present application is not limited in this respect. It should be noted that the following method can be executed by the above terminal device or server, or by the terminal device and the server together. Here, the terminal device or server is specifically used as the execution subject for example.

[0064] Figure 2 A flowchart showing a method for generating an inference result in this exemplary embodiment is shown in FIG. Figure 2 As shown, it is a schematic diagram of the training process of the target language model provided in the embodiment of the present application. The method can be executed by a computer device, which can be Figure 1 The terminal device or server shown in the figure, the specific implementation process of the method includes the following steps S201-S204:

[0065] Step S201: obtain a problem to be reasoned, generate a target number of reasoning examples corresponding to the problem to be reasoned, and obtain a reasoning example set, wherein each reasoning example in the reasoning example set is used to represent an example of obtaining a corresponding reasoning result of the problem to be reasoned through at least one intermediate reasoning step.

[0066] Step S202: input the problem to be inferred and the set of reasoning examples into a preset large language model, so that the large language model can reason and answer the problem to be inferred based on each reasoning example in the set of reasoning examples, and obtain a first reasoning result corresponding to the problem to be inferred.

[0067] Step S203: Determine a target reasoning question that matches the first reasoning result based on the large language model.

[0068] Step S204: Search the target reasoning problem through a search engine, and modify the first reasoning result according to the search results to obtain a modified second reasoning result.

[0069] In the above disclosure Figure 2In the technical solution provided, a problem to be inferred is obtained, and a target number of reasoning examples corresponding to the problem to be inferred is generated to obtain a set of reasoning examples, wherein each reasoning example in the set of reasoning examples is used to represent an example of obtaining a corresponding reasoning result from the problem to be inferred through at least one intermediate reasoning step; the problem to be inferred and the set of reasoning examples are input into a preset large language model, so that the large language model performs reasoning and answers to the problem to be inferred based on each reasoning example in the set of reasoning examples, and obtains a first reasoning result corresponding to the problem to be inferred; based on the large language model, a target reasoning problem matching the first reasoning result is determined; the target reasoning problem is searched through a search engine, and the first reasoning result is corrected according to the search results to obtain a corrected second reasoning result. On the one hand, the reasoning examples generated in the method contain intermediate reasoning steps generated based on the thinking chain, thereby forming a factual reasoning chain, so that the large language model generates a reasoning result with a reasoning process according to the intermediate steps in the reasoning example, thereby helping to improve the interpretability of the model, so that users can more clearly and intuitively understand the processing logic of the model in the process of implementing reasoning. On the other hand, introducing reasoning examples into the large language model so that the large language model can reason and answer the reasoning questions based on each reasoning example in the reasoning example set will help improve the performance of the large language model in complex reasoning tasks and reduce the large language model's inability to accurately understand the user's intention when receiving the content to be reasoned input by the user, so as to improve the accuracy of the reasoning results generated by the model. On the other hand, by introducing external knowledge through the search engine to verify the authenticity of the reasoning results generated by the model, the content of the reasoning results generated by the model can be better controlled, thereby ensuring that the content of the generated reasoning results is consistent with the external knowledge, achieving the technical effect of reducing the generation of inaccurate text and improving the factual correctness of the reasoning results.

[0070] The following will be combined with specific embodiments to Figure 2 The specific implementation methods of each step in the embodiment shown are described in detail:

[0071] In step S201, a problem to be reasoned is obtained, and a target number of reasoning examples corresponding to the problem to be reasoned are generated to obtain a reasoning example set, wherein each reasoning example in the reasoning example set is used to represent an example of obtaining a corresponding reasoning result of the problem to be reasoned through at least one intermediate reasoning step.

[0072] The problem to be inferred may be a problem input by a user, and in particular, may be a problem with high complexity input by a user, such as a mathematical application reasoning problem, a common sense reasoning problem, a symbolic reasoning problem, a question-answering reasoning problem, etc. A problem with high complexity to be inferred may be, for example, a problem whose complexity is greater than a preset complexity threshold. For example, the text length of the problem to be inferred is greater than or equal to a length threshold, or the amount of text content involved in the problem to be inferred is greater than a quantity threshold, etc.

[0073] It should be understood that the user can directly input the text of the question to be reasoned in the application interface, or the user can input the question to be reasoned by voice, and convert the voice-input question to be reasoned into text through a speech-to-text model, and the speech-to-text model can be, for example, a Whisper model, a Buzz model, an Aura model or other models, and the embodiments of the present disclosure do not impose any special restrictions on this.

[0074] Exemplarily, after the user inputs a question to be reasoned, the server may obtain the question to be reasoned to generate a target number of reasoning examples corresponding to the question to be reasoned, thereby obtaining a set of reasoning examples.

[0075] The reasoning example set includes multiple reasoning examples, each of which includes a problem to be reasoned, a reasoning result, and at least one intermediate reasoning step. It should be noted that the at least one intermediate reasoning step may be generated based on a thought chain.

[0076] For example, by generating a target number of reasoning examples, the intermediate reasoning steps in the reasoning examples can be displayed, so that each step from the problem to be reasoned to obtaining the final reasoning result can be clearly listed. This can not only clearly display the thought process of solving the problem to be reasoned to improve the interpretability of model processing, so that users can more clearly understand the overall thinking process of the large language model in solving reasoning tasks; it can also allow the large language model to generate corresponding reasoning results with reference to the steps in the reasoning examples.

[0077] When executing step S201 to generate a target number of reasoning examples corresponding to the problem to be reasoned and obtain a set of reasoning examples, the number of generated reasoning examples can be determined by the following embodiments:

[0078] In an optional embodiment of the present disclosure, the target number is preconfigured, so that reasoning examples corresponding to the problem to be reasoned are generated according to the preconfigured target number.

[0079] In another optional embodiment, a corresponding number of reasoning examples may be generated based on the uncertainty of the problem to be reasoned, that is, the uncertainty score of the large language model for the reasoning problem is determined; and the number of reasoning examples corresponding to the uncertainty score is generated to obtain a set of reasoning examples.

[0080] Exemplarily, a large language model may be used to treat the uncertainty score of the reasoning problem, so as to adaptively generate a corresponding number of reasoning examples through the uncertainty score to form a reasoning example set.

[0081] It should be understood that if the uncertainty score of the large language model for reasoning problems is higher, the corresponding understanding of the reasoning problems is weaker, and the accuracy of the corresponding generated reasoning results is lower. For this reason, it is necessary to generate more reasoning examples to help the large language model better understand the problem to be reasoned. Conversely, if the uncertainty score is lower, the large language model's understanding of the reasoning problem is more thorough, and the corresponding generated reasoning results are more accurate, then fewer reasoning examples can be generated.

[0082] When determining the uncertainty score of the large language model for the reasoning problem in the above embodiment, the uncertainty score of the large language model for the reasoning problem can be determined based on one or a combination of the following embodiments:

[0083] Embodiment 1:

[0084] In an optional embodiment of the present disclosure, multiple reasoning results corresponding to the problem to be reasoned are generated based on the large language model; the uncertainty levels of the multiple reasoning results are determined based on the large language model to obtain an uncertainty score of the large language model for the reasoning problem.

[0085] Exemplarily, in this embodiment, after a large language model generates multiple inference results corresponding to the problem to be inferred, a query request can be sent to the large language model, and the large language model returns the degree of uncertainty of the multiple inference results after receiving the query request, thereby obtaining the uncertainty score of the large language model for the inference problem. For example, if the conclusions corresponding to the multiple inference results generated by the large language model are all different, it indicates that the large language model does not understand the problem to be inferred, and the corresponding uncertainty score is also relatively high.

[0086] In this embodiment, the uncertainty degree of the reasoning results generated by the large language model can be used to infer the uncertainty score of the large language model for the reasoning problem, so that when the uncertainty score is high, more reasoning examples can be generated to assist the large language model in understanding the reasoning problem, and when the uncertainty score is low, more reasoning examples can be generated to save resources.

[0087] Embodiment 2:

[0088] In an optional embodiment of the present disclosure, multiple inference results corresponding to the problem to be inferred are generated based on a large language model; the proportion of identical inference results is determined from the multiple inference results, and an uncertainty score of the large language model for the problem to be inferred is determined based on the proportion of identical inference results.

[0089] Exemplarily, for multiple inference results, the proportion of inference results with the same calculation results among the multiple inference results can be calculated. For example, there are 10 inference results corresponding to the problem to be inferred based on the large language model, and the total score of the corresponding uncertainty score is 10 points. When 8 of the calculated inference results are the same, it can be determined that the uncertainty score of the large language model for the problem to be inferred is 2 points.

[0090] In this embodiment, the higher the proportion of identical inference results, the lower the uncertainty score of the corresponding large language model for the reasoning problem, which indicates that the large language model has a stronger understanding of the problem to be reasoned, and a smaller number of reasoning examples can be generated. Conversely, if the lower the proportion of identical inference results, the higher the uncertainty score of the corresponding large language model for the reasoning problem, which indicates that the large language model has a weaker understanding of the problem to be reasoned, a larger number of reasoning examples can be generated to assist the large language model in understanding the intention of the reasoning problem.

[0091] Embodiment 3:

[0092] In an optional embodiment of the present disclosure, multiple inference results corresponding to the problem to be inferred are generated based on the large language model; variance values ​​of the multiple inference results are determined, and an uncertainty score of the large language model for the problem to be inferred is determined based on the variance values.

[0093] Exemplarily, when calculating the variance values ​​of multiple inference results, it is implemented based on the following formula (1):

[0094]

[0095] In formula (1), Var(X) represents the variance of multiple inference results; X i represents the i-th inference result, where i ranges from i∈[1,N] and N is the total number of inference results; X represents the average value of multiple inference results.

[0096] It should be understood that the average value of multiple inference results is usually for the case where the inference results contain numerical values, so the uncertainty score is calculated by taking the average value and variance value of the numerical values.

[0097] In this embodiment, the larger the variance value, the higher the uncertainty score of the large language model for the problem to be reasoned, which means that the large language model has a weaker understanding of the problem to be reasoned. Therefore, more reasoning examples can be generated to assist the large language model in understanding the problem to be reasoned.

[0098] Embodiment 4:

[0099] In an optional embodiment of the present disclosure, multiple reasoning results corresponding to the problem to be reasoned are generated based on the large language model; entropy values ​​of the multiple reasoning results are determined, and an uncertainty score of the large language model for the problem to be reasoned is determined based on the entropy values.

[0100] For example, when calculating the entropy values ​​of multiple inference results, it can be implemented based on the following formula (2):

[0101]

[0102] In formula (2), H(x) is the entropy value of multiple inference results; p(x j ) represents the probability that the inference result belongs to the jth category, the value range of j is j∈[1,n], and n is the total number of types; argmax means taking the largest value as the entropy of the problem to be inferred.

[0103] In this embodiment, the higher the entropy value, the higher the uncertainty score of the large language model for the problem to be reasoned, indicating that the large language model has a weaker understanding of the problem to be reasoned. Therefore, more reasoning examples need to be generated to assist the large language model in understanding the problem to be reasoned.

[0104] In this embodiment, the intermediate reasoning steps in the reasoning examples are generated through thought chains, which is a prompting method that helps to improve the logical reasoning ability of the large language model. It breaks down complex problems into multiple logically coherent intermediate steps, thereby helping to improve the reliability of processing tasks and the performance of the large language model. By adaptively generating a corresponding number of reasoning examples based on the uncertainty score of the current problem to be reasoned by the large language model, the prompting effect of the thought chain can be higher, which in turn helps to improve the performance of the large language model in complex reasoning tasks.

[0105] In another optional embodiment of the present disclosure, after obtaining the problem to be inferred, the problem to be inferred can be decomposed through the thinking chain prompt template to obtain multiple sub-problems corresponding to the problem to be inferred, so as to generate reasoning examples corresponding to the target number of sub-problems for each sub-problem corresponding to the problem to be inferred based on the above embodiment. Figure 4 , this disclosure will not go into details here.

[0106] In step S202, the problem to be inferred and the set of reasoning examples are input into a preset large language model, so that the large language model performs reasoning and answering on the problem to be inferred based on each reasoning example in the set of reasoning examples, and obtains a first reasoning result corresponding to the problem to be inferred.

[0107] Exemplarily, after obtaining the problem to be inferred and generating a set of reasoning examples in step S201, the problem to be inferred and the set of reasoning examples can be input into a preset large language model, and the large language model can be guided by a thinking chain prompt template to reason and answer the problem to be inferred through each reasoning example in the set of reasoning examples, thereby obtaining the reasoning process and reasoning results related to the problem to be processed.

[0108] When executing the step S202 in which the problem to be inferred and the set of inference examples are input into the preset large language model, in order to improve the accuracy of the generated inference results and enhance the reliability of the model, the large language model can be used multiple times to perform inference and answer the questions to obtain multiple inference results.

[0109] The following will be combined Figure 3 The above process of using a large language model multiple times for reasoning and answering is exemplified.

[0110] Figure 3 A flowchart showing a method for generating a multiple execution reasoning result in this exemplary embodiment is shown; Figure 3 As shown, the above method includes the following steps S301 to S304:

[0111] Step S301: rewrite the reasoning question multiple times.

[0112] Among them, question rewriting can be content expansion, format conversion, etc. of the problem to be reasoned. Among them, format conversion is a format that is easier to search and understand the intention, and content expansion is to enrich the content of the problem to be reasoned.

[0113] For example, in order to increase the richness and randomness of the generated reasoning results, the problem to be reasoned can be rewritten multiple times to obtain multiple rewritten problems to be reasoned. When rewriting the problem to be reasoned, the Transformer model can be used to rewrite the problem to be reasoned input by the user. Of course, other methods can also be used for rewriting, and the embodiments of the present disclosure do not impose any special restrictions on this.

[0114] Step S302: input the multiple rewritten questions to be reasoned and the reasoning example set into the large language model respectively, so that the large language model can reason and answer the multiple rewritten questions to be reasoned based on each reasoning example in the reasoning example set, and obtain multiple initial reasoning results.

[0115] In an optional embodiment of the present disclosure, in the process of implementing reasoning solutions through a large language model, random parameters may be set so that the reasoning results generated each time through the large language model have a certain degree of randomness.

[0116] Step S303: performing consistency check on multiple initial reasoning results according to preset detection rules to obtain consistency check results.

[0117] Exemplarily, in order to improve the accuracy of inference results generated by a large language model and the model performance, the consistency of inference results can be checked for multiple initial inference results according to preset detection rules to ensure that the obtained inference results are consistent.

[0118] Step S304: determine the first reasoning result according to the consistency check result.

[0119] When executing step S303 and step S304, the above-mentioned preset detection rules will be exemplarily described below in conjunction with specific embodiments:

[0120] Embodiment 1:

[0121] In an optional embodiment of the present disclosure, a statistical analysis of the inference conclusions of multiple initial inference results is performed to implement consistency verification. If the inference conclusions corresponding to the multiple initial inference results are consistent, any initial inference result among the multiple initial inference results is determined as the first inference result.

[0122] Exemplarily, a statistical analysis may be performed on multiple initial inference results generated by a large language model. If the inference conclusions corresponding to the multiple initial inference results are statistically consistent, any one of the multiple initial inference results is determined as the first inference result.

[0123] Of course, it is also possible to determine that the reasoning conclusions corresponding to multiple initial reasoning results are consistent when the reasoning conclusions corresponding to the initial reasoning results greater than the target number threshold are consistent, and discard the initial reasoning results that are inconsistent with the reasoning conclusions corresponding to the initial reasoning results greater than the target number threshold.

[0124] Optionally, when performing consistency verification through statistical analysis, in order to improve the verification efficiency, each initial reasoning result can be decomposed and analyzed based on syntactic analysis, dependency analysis and other methods to extract the key concepts, entities and their mutual relationships in each initial reasoning result, and then perform statistical analysis on the above-extracted entities and mutual relationships.

[0125] For example, if the question to be inferred is about the attribute information of a certain thing, such as "Is white sugar sweet?", then the description of the attribute in the inference conclusions corresponding to the generated multiple initial inference results should be the same, that is, "white sugar is sweet"; if they are not the same, for example, "white sugar is bitter", then the inference result of "white sugar is bitter" can be discarded.

[0126] In this embodiment, by performing statistical analysis on multiple initial inference results generated by a large language model, the most frequent inference conclusions are identified, that is, most of the initial inference results point to the same inference conclusion, which can be retained, and a small number of initial inference results pointing to other inference conclusions are discarded, which helps to improve the factual correctness of the generated inference results, thereby improving the model performance and credibility.

[0127] Embodiment 2:

[0128] In an optional embodiment of the present disclosure, consistency verification is implemented by performing semantic analysis on multiple initial reasoning results based on embedded similar sentences. If the semantics corresponding to the multiple initial reasoning results are the same, any initial reasoning result among the multiple initial reasoning results is determined as the first reasoning result.

[0129] For example, the semantic information of each initial reasoning result can also be captured by sentence embedding. Since sentences embed similar reasoning conclusions, the semantic content is also relatively similar. For example, the embedded sentence "Today is a sunny day" will be very similar to the sentence of the reasoning result "The weather is nice today". Even if the words are different, the semantic information they express is similar. By determining whether the semantics corresponding to multiple initial reasoning results are the same, if they are the same, the corresponding consistency test results are consistent.

[0130] It can be understood that, when the semantics corresponding to the initial reasoning results greater than the target number threshold are the same, it is also possible to determine that the reasoning conclusions corresponding to multiple initial reasoning results are consistent, and at the same time discard the initial reasoning results that have different semantics from the initial reasoning results greater than the target number threshold, so as to provide the accuracy of the generated reasoning results.

[0131] Embodiment 3:

[0132] In an optional embodiment of the present disclosure, a statistical analysis of inference conclusions of multiple initial inference results is performed to achieve a consistency check, and a weighted combination of the consistency check results of semantic analysis performed on the multiple initial inference results based on embedded similar sentences is performed to obtain a first inference result.

[0133] Exemplarily, the inference result obtained by statistical analysis in the above embodiment 1 and the inference result obtained by semantic analysis of multiple initial inference results based on embedded similar sentences in the above embodiment 2 can be weighted and merged to obtain the final first inference result.

[0134] In step S203, based on the large language model, a target reasoning question matching the first reasoning result is determined.

[0135] Exemplarily, after obtaining the first reasoning result, a target reasoning question related to the first reasoning result may be generated by calling a large language model, so that the target reasoning question may be subsequently searched and verified through a search engine.

[0136] Optionally, when executing step S203, the first reasoning result can be disassembled to obtain words and sentences containing key information and entities. For example, the long sentences in the first reasoning result are segmented into short sentences, and the segmented short sentences are classified through the Transformer model to determine whether they contain independent facts or arguments. If so, they are retained, otherwise, the short sentences that do not contain facts or arguments are discarded. For short sentences containing facts or arguments, information extraction technology can also be used to extract key information and facts, such as time, place, people, numbers, attributes, etc. Finally, based on the content of the disassembled short sentences and the extracted key information and facts, the large language model is called to generate target reasoning problems related to them.

[0137] In step S204, the target reasoning problem is searched through a search engine, and the first reasoning result is corrected according to the search results to obtain a corrected second reasoning result.

[0138] Exemplarily, after obtaining the target reasoning problem, the target reasoning problem can be searched through a search engine to verify the generated reasoning results by introducing external knowledge, so as to better control the reasoning content generated by the large language model. Correcting the first reasoning result through the search results can ensure that the reasoning results generated by the model are consistent with the external knowledge obtained by the search engine, so as to reduce the generation of inaccurate text, thereby improving the factual correctness of the reasoning results generated by the model.

[0139] In the process of correcting the first reasoning result according to the search results in step S204, in order to improve efficiency, the search results may verify the first reasoning result based on one or more combinations of the following embodiments:

[0140] In an optional embodiment, if the search results contain at least numerical information, the numerical information in the search results is numerically matched with the numerical information of the first reasoning result.

[0141] Exemplarily, if the numerical values ​​do not match, the first reasoning result can be updated and corrected based on the numerical information in the search results to obtain a corrected second reasoning result; conversely, if the numerical values ​​match, there is no need to update and correct the first reasoning result, and the first reasoning result can be directly used as the final output reasoning result to be displayed to the user.

[0142] In an optional embodiment of the present disclosure, if the search result contains at least text information, it is determined based on the large language model whether the semantic information of the search result is consistent with the semantic information of the first inference result.

[0143] Exemplarily, if the semantic information does not match, the first reasoning result can be corrected based on the semantic information of the search results to obtain a corrected second reasoning result; conversely, when the semantic information matches, there is no need to update or correct the first reasoning result, and the first reasoning result can be directly used as the final output reasoning result to be displayed to the user.

[0144] In addition, based on any of the above embodiments, when it is determined that the first inference result is to be corrected according to the search results, in an optional embodiment of the present disclosure, the corrected inference result can be input into the large language model, and the inference result output based on the large language model can be used as the final output second inference result to be displayed to the user.

[0145] In this embodiment, the inference result after the first inference result is corrected by the search results is input into the large language model again to output the final second inference result. The parameters of the large language model can be corrected to improve the ability of the large language model to handle complex inference tasks.

[0146] The following will refer to Figure 4 , Figure 5 The entire process of generating an inference result of the method for generating an inference result according to the exemplary embodiment of the present disclosure is described in detail.

[0147] Figure 4 A flowchart showing a method for generating a complete reasoning result in this exemplary embodiment is shown; Figure 4 As shown, the above method includes the following steps S401 to S408:

[0148] In step S401, a question to be inferred input by a user is obtained.

[0149] Step S402: Decompose the reasoning problem based on the thought chain prompt template to obtain multiple sub-problems.

[0150] Step S403: adaptively generate a corresponding number of reasoning examples based on the uncertainty scores of the large language model for multiple sub-questions.

[0151] Exemplarily, in this embodiment, a corresponding number of reasoning examples may be adaptively generated for the uncertainty score of each sub-problem.

[0152] Step S404: Generate multiple first reasoning results.

[0153] Step S405: performing consistency check on the multiple initial reasoning results according to a preset detection rule to eliminate reasoning results with low consistency from the multiple first reasoning results.

[0154] Step S406: Generate a target reasoning question that matches the first reasoning result.

[0155] Step S407: Search the target reasoning problem through a search engine, and modify the first reasoning result according to the search results to obtain a modified reasoning result.

[0156] Step S408: Input the corrected inference result into the large language model, output the second inference result, and finally display the second inference result to the user.

[0157] Furthermore, the present disclosure also provides another method for generating an inference result: Figure 5 A flowchart showing another method for generating a complete reasoning result in this exemplary embodiment; Figure 5 As shown, in step S401, the question to be inferred input by the user is obtained.

[0158] Step S402: Decompose the reasoning problem based on the thought chain prompt template to obtain multiple sub-problems. Step S403: Adaptively generate a corresponding number of reasoning examples according to the uncertainty scores of the multiple sub-problems by the large language model. Step S404: Generate multiple first reasoning results.

[0159] Then, for each first reasoning result, step S501 is executed to determine whether the consistency of the first reasoning result is low. If so, step S502 is executed to discard the first reasoning result. If not, step S406 is executed to generate a target reasoning question matching the first reasoning result.

[0160] Then continue to execute step S503, search the target reasoning problem through the search engine. In step S504, determine whether the first reasoning result needs to be updated. If it does not need to be updated, execute step S505, output and display the reasoning result to the user; in this embodiment, the first reasoning result is displayed. If it needs to be updated, execute step 506 to modify the first reasoning result according to the search result to obtain the modified reasoning result.

[0161] Step S408: input the corrected inference result into the large language model, and output the second inference result. Finally, execute S505: output and display the inference result to the user. In this embodiment, the corrected second inference result is displayed.

[0162] It should be noted that although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0163] In order to implement the above-mentioned method for generating an inference result, an apparatus for generating an inference result is provided in one embodiment of the present disclosure. Figure 6 A schematic architecture diagram of a device for generating an inference result is shown.

[0164] The inference result generating device 600 may include an inference example generating module 601 , an inference answering module 602 , a problem determining module 603 and an inference result correcting module 604 .

[0165] Among them, the reasoning example generation module 601 is used to obtain the problem to be reasoned, generate the target number of reasoning examples corresponding to the problem to be reasoned, and obtain a reasoning example set, each reasoning example in the reasoning example set is used to represent an example of obtaining a corresponding reasoning result of the problem to be reasoned through at least one intermediate reasoning step; the reasoning answer module 602 is used to input the problem to be reasoned and the reasoning example set into a preset large language model, so that the large language model performs reasoning and answering on the problem to be reasoned based on each reasoning example in the reasoning example set, and obtains a first reasoning result corresponding to the problem to be reasoned; the problem determination module 603 is used to determine a target reasoning problem that matches the first reasoning result based on the large language model; the reasoning result correction module 604 is used to search for the target reasoning problem through a search engine, correct the first reasoning result according to the search result, and obtain a corrected second reasoning result.

[0166] In an optional embodiment of the present disclosure, the reasoning example generation module 601 is specifically used to determine the uncertainty score of the large language model for the reasoning problem; generate a number of reasoning examples corresponding to the uncertainty score, and obtain a reasoning example set.

[0167] In an optional embodiment of the present disclosure, the reasoning example generation module 601 is specifically used to generate multiple reasoning results corresponding to the problem to be reasoned based on the large language model; determine the uncertainty degree of the multiple reasoning results based on the large language model, and obtain the uncertainty score of the large language model for the reasoning problem; or, the reasoning example generation module 601 is specifically used to generate multiple reasoning results corresponding to the problem to be reasoned based on the large language model; determine the proportion of the same reasoning results from the multiple reasoning results, and determine the uncertainty score of the large language model for the reasoning problem based on the proportion of the same reasoning results; or, the reasoning example generation module 601 is specifically used to generate multiple reasoning results corresponding to the problem to be reasoned based on the large language model; determine the variance values ​​of the multiple reasoning results, and determine the uncertainty score of the large language model for the reasoning problem based on the variance values; or, the reasoning example generation module 601 is specifically used to generate multiple reasoning results corresponding to the problem to be reasoned based on the large language model; determine the entropy values ​​of the multiple reasoning results, and determine the uncertainty score of the large language model for the reasoning problem based on the entropy values.

[0168] In an optional embodiment of the present disclosure, the device further includes a problem decomposition module, which is used to decompose the reasoning problem to be reasoned based on a thinking chain prompt template to obtain multiple sub-problems corresponding to the problem to be reasoned, so as to generate a target number of reasoning examples corresponding to each sub-problem corresponding to the problem to be reasoned.

[0169] In an optional embodiment of the present disclosure, the reasoning and answering module 602 is used to rewrite the problem to be reasoned for multiple times, and input the multiple rewritten problems to be reasoned and the reasoning example set into the large language model respectively, so that the large language model performs reasoning and answers to the multiple rewritten problems to be reasoned based on each reasoning example in the reasoning example set to obtain multiple initial reasoning results; the consistency of the reasoning results is checked on the multiple initial reasoning results according to preset detection rules to obtain a consistency check result, and the first reasoning result is determined according to the consistency check result.

[0170] In an optional embodiment of the present disclosure, the reasoning answer module 602 is used to perform a statistical analysis of the reasoning conclusions on multiple initial reasoning results to achieve consistency verification. If the reasoning conclusions corresponding to the multiple initial reasoning results are consistent, any initial reasoning result among the multiple initial reasoning results is determined as the first reasoning result; or, the reasoning answer module 602 is used to perform a semantic analysis on the multiple initial reasoning results based on embedded similar sentences to achieve consistency verification. If the semantics corresponding to the multiple initial reasoning results are the same, any initial reasoning result among the multiple initial reasoning results is determined as the first reasoning result; or, the reasoning answer module 602 is used to perform a statistical analysis of the reasoning conclusions on the multiple initial reasoning results to achieve consistency verification, and weightedly merge the verification results of the semantic analysis on the multiple initial reasoning results based on embedded similar sentences to achieve consistency verification, to obtain the first reasoning result.

[0171] In an optional embodiment of the present disclosure, the inference result correction module 604 is used to, if the search results contain at least numerical information, numerically match the numerical information in the search results with the numerical information of the first inference result, so as to correct the first inference result based on the numerical information in the search results when the numerical values ​​do not match, and obtain a corrected second inference result; if the search results contain at least text information, determine whether the semantic information of the search results is consistent with the semantic information of the first inference result based on the large language model, so as to correct the first inference result based on the semantic information of the search results when the semantic information does not match, and obtain a corrected second inference result.

[0172] The device 600 for generating inference results provided in the embodiments of the present disclosure can execute the technical solution of the method for generating inference results in any of the above-mentioned embodiments, and its implementation principle and beneficial effects are similar to those of the method for generating inference results. Please refer to the implementation principle and beneficial effects of the method for generating inference results, and no further details will be given here.

[0173] In the exemplary embodiments of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible implementations, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary embodiments of the present invention described in the above "Exemplary Method" section of the present specification.

[0174] The program product for implementing the above method according to an embodiment of the present invention may adopt a portable compact disc read-only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto, and in this document, a readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device.

[0175] The program product may adopt any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0176] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0177] The program code contained on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, radio frequency (RF), etc., or any suitable combination of the foregoing.

[0178] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet Service Provider).

[0179] In an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided. It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as a system, method or program product. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation combining hardware and software aspects, which may be collectively referred to herein as a "circuit", "module" or "system".

[0180] Refer to the following Figure 7 An electronic device 700 according to this embodiment of the present invention will be described. Figure 7 The electronic device 700 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0181] like Figure 7 As shown, the electronic device 700 is in the form of a general-purpose computing device. The components of the electronic device 700 may include, but are not limited to: the at least one processing unit 710 mentioned above, the at least one storage unit 720 mentioned above, a bus 730 connecting different system components (including the storage unit 720 and the processing unit 710), and a display unit 740. The storage unit stores program code, which can be executed by the processing unit 710, so that the processing unit 710 executes the steps of various exemplary embodiments of the present invention described in the above "Exemplary Method" section of this specification. For example, the processing unit 710 may execute as follows Figure 2 Steps S201 to S204 shown in FIG.

[0182] The storage unit 720 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 7201 and / or a cache storage unit 7202 , and may further include a read-only storage unit (ROM) 7203 .

[0183] The storage unit 720 may also include a program / utility 7204 having a set (at least one) of program modules 7205, such program modules 7205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0184] Bus 730 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0185] The electronic device 700 may also communicate with one or more external devices 1000 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 700, and / or communicate with any device that enables the electronic device 700 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 750. Furthermore, the electronic device 700 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 760. As shown, the network adapter 760 communicates with other modules of the electronic device 700 via a bus 730. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, disk arrays (Redundant Arrays of Independent Disks, RAID) systems, tape drives, and data backup storage systems.

[0186] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0187] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0188] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0189] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0190] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for generating an inference result, characterized in that: include: Obtaining a problem to be reasoned, generating a target number of reasoning examples corresponding to the problem to be reasoned, and obtaining a reasoning example set, wherein each reasoning example in the reasoning example set is used to represent an example of obtaining a corresponding reasoning result by performing at least one intermediate reasoning step on the problem to be reasoned; Inputting the problem to be inferred and the set of reasoning examples into a preset large language model, so that the large language model performs reasoning and answering on the problem to be inferred based on each reasoning example in the set of reasoning examples, and obtains a first reasoning result corresponding to the problem to be inferred; Determining, based on the large language model, a target reasoning question matching the first reasoning result; The target reasoning problem is searched through a search engine, and the first reasoning result is corrected according to the search results to obtain a corrected second reasoning result.

2. The method according to claim 1, characterized in that The generating of a set of reasoning examples consisting of a target number of reasoning examples corresponding to the problem to be reasoned includes: Determining an uncertainty score of the large language model for the problem to be reasoned; Generate a number of reasoning examples corresponding to the uncertainty score to obtain a reasoning example set.

3. The method according to claim 2, characterized in that Determining the uncertainty score of the large language model for the problem to be inferred includes: Based on the large language model, generating multiple reasoning results corresponding to the problem to be reasoned; Determining the uncertainty degree of the multiple inference results based on the large language model, and obtaining an uncertainty score of the large language model for the problem to be inferred; Alternatively, determining a proportion of identical inference results from the multiple inference results, and determining an uncertainty score of the large language model for the problem to be inferred based on the proportion of identical inference results; Alternatively, determining variance values ​​of the multiple inference results, and determining an uncertainty score of the large language model for the problem to be inferred based on the variance values; Alternatively, entropy values ​​of the multiple inference results are determined, and an uncertainty score of the large language model for the problem to be inferred is determined based on the entropy values.

4. The method according to claim 1, characterized in that The method further comprises: The problem to be inferred is decomposed based on the thinking chain prompt template to obtain multiple sub-problems corresponding to the problem to be inferred, so as to generate a target number of reasoning examples corresponding to each sub-problem corresponding to the problem to be inferred.

5. The method according to claim 1, characterized in that Inputting the problem to be inferred and the set of reasoning examples into a preset large language model, so that the large language model performs reasoning and answering on the problem to be inferred based on each reasoning example in the set of reasoning examples, and obtaining a first reasoning result corresponding to the problem to be inferred, including: Rewriting the problem to be inferred multiple times, and inputting the multiple rewritten problems to be inferred and the reasoning example set into the large language model respectively, so that the large language model performs reasoning and answers to the multiple rewritten problems to be inferred based on each reasoning example in the reasoning example set, and obtains multiple initial reasoning results; The consistency of the inference results is checked on the multiple initial inference results according to a preset detection rule to obtain a consistency check result, and the first inference result is determined according to the consistency check result.

6. The method according to claim 5, characterized in that Performing consistency check on the multiple initial reasoning results according to a preset detection rule to obtain a consistency check result, and determining the first reasoning result according to the consistency check result, including: Performing statistical analysis of the inference conclusions of the multiple initial inference results to achieve consistency verification, and if the inference conclusions corresponding to the multiple initial inference results are consistent, determining any initial inference result among the multiple initial inference results as the first inference result; Alternatively, consistency check is implemented by respectively performing semantic analysis on the multiple initial reasoning results based on embedded similar sentences, and if the semantics corresponding to the multiple initial reasoning results are the same, any initial reasoning result among the multiple initial reasoning results is determined as the first reasoning result; Alternatively, the verification result of consistency check achieved by statistical analysis of the reasoning conclusions of the multiple initial reasoning results is weighted and merged with the verification result of consistency check achieved by semantic analysis of the multiple initial reasoning results based on embedded similar sentences to obtain the first reasoning result.

7. The method according to claim 1, characterized in that The step of correcting the first reasoning result according to the search result to obtain a corrected second reasoning result includes: If the search results contain at least numerical information, numerically match the numerical information in the search results with the numerical information of the first reasoning result, so that when the numerical values ​​do not match, the first reasoning result is corrected based on the numerical information in the search results to obtain a corrected second reasoning result; If the search results contain at least text information, then whether the semantic information of the search results is consistent with the semantic information of the first reasoning result is determined based on the large language model; if the semantic information is inconsistent, the first reasoning result is corrected based on the semantic information of the search results to obtain a corrected second reasoning result.

8. A device for generating an inference result, characterized in that: The device comprises: A reasoning example generation module, used to obtain a problem to be reasoned, generate a target number of reasoning examples corresponding to the problem to be reasoned, and obtain a reasoning example set, wherein each reasoning example in the reasoning example set is used to represent an example of obtaining a corresponding reasoning result by passing the problem to be reasoned through at least one intermediate reasoning step; A reasoning and answering module, used for inputting the problem to be reasoned and the reasoning example set into a preset large language model, so that the large language model performs reasoning and answering on the problem to be reasoned based on each reasoning example in the reasoning example set, and obtains a first reasoning result corresponding to the problem to be reasoned; a problem determination module, configured to determine, based on the large language model, a target reasoning problem matching the first reasoning result; The inference result correction module is used to search the target inference problem through a search engine, and correct the first inference result according to the search result to obtain a corrected second inference result.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for generating an inference result according to any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to execute the method for generating an inference result according to any one of claims 1 to 7 by executing the executable instructions.

Images