LLM output result verification method and device, storage medium and equipment

Abstract

The embodiment of the invention discloses an LLM output result verification method, which comprises the following steps: after an LLM obtains an output result, identifying the type of a reasoning task of the output result obtained by the LLM, if the type of the reasoning task is a logic judgment type, obtaining the reasoning process of the output result and the output result according to the LLM, and verifying the correctness of the output result. If the task is the inference task of the information acquisition type, generating a to-be-verified proposition according to the input data and the output result, and verifying the correctness of the output result according to the to-be-verified proposition. According to the method, through different types of reasoning tasks performed by the LLM, whether the output result obtained by the LLM is a correct output result or not can be checked by adopting a corresponding method, and whether illusion appears in the process of obtaining the output result by reasoning of the LLM can also be judged.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a method, apparatus, storage medium, and device for verifying LLM output results. Background Art

[0002] With the development of artificial intelligence (AI) technology, large language models (LLM) have been widely used in various fields.

[0003] However, due to the limitations of training data, LLM may suffer from problems such as lack of knowledge, knowledge errors, or outdated knowledge. These problems may cause LLM to experience "hallucinations" when performing tasks, thereby reducing the accuracy and reliability of LLM's task execution.

[0004] Therefore, how to accurately and reliably execute tasks based on LLM has become an urgent problem to be solved. Summary of the Invention

[0005] The embodiments of this specification provide a method, device, storage medium, and electronic device for verifying LLM output results to partially solve the problems existing in the above-mentioned prior art.

[0006] The embodiments of this specification adopt the following technical solutions: This specification provides a method for verifying LLM output results, the method comprising: Get the output results of LLM based on input data; determining a type of reasoning task performed by the LLM to obtain the output result; When it is determined that the type is a logic judgment type, obtaining the reasoning process of the LLM to obtain the output result according to the input data, and verifying the correctness of the output result based on the reasoning process and the output result; When it is determined that the type is an information acquisition type, a question to be verified is generated based on the output result and the input data, and the correctness of the output result is verified based on the question to be verified. This specification provides a device for verifying LLM output results, the device comprising: The acquisition module is used to obtain the output results obtained by LLM based on the input data; a task type determination module, configured to determine the type of reasoning task performed by the LLM to obtain the output result; a first verification module configured to, when determining that the type is a logic judgment type, obtain the reasoning process by which the LLM obtains the output result based on the input data, and verify the correctness of the output result based on the reasoning process and the output result; The second verification module is used to generate a proposition to be verified according to the output result and the input data when it is determined that the type is an information acquisition type, and verify the correctness of the output result according to the proposition to be verified.

[0007] This specification provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned LLM output result verification method.

[0008] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the method for verifying the LLM output result described above is implemented.

[0009] At least one of the above technical solutions adopted in the embodiments of this specification can achieve the following beneficial effects: The embodiments of this specification disclose a method for verifying an LLM output result. After the LLM obtains an output result, the method identifies the type of reasoning task for which the LLM obtains the output result. If it is a logical judgment type reasoning task, the correctness of the output result is verified based on the reasoning process and the output result obtained by the LLM. If it is an information acquisition type reasoning task, a proposition to be verified is generated based on the input data and the output result, and the correctness of the output result is verified based on the proposition to be verified. The above method can verify whether the output result obtained by the LLM is a correct output result by using corresponding methods based on different types of reasoning tasks performed by the LLM, and can also determine whether the LLM has "hallucinations" in the process of reasoning to obtain the output result. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The exemplary embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation of this specification. In the drawings: Figure 1 A flow chart of a method for verifying LLM output results provided in an embodiment of this specification; Figure 2 A flow chart of a HOP execution method provided in an embodiment of this specification; Figure 3 A flow chart of a method for generating a HOP according to an embodiment of this specification; Figure 4A schematic diagram of a device for verifying LLM output results provided in an embodiment of this specification; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this specification. DETAILED DESCRIPTION

[0011] To make the objectives, technical solutions, and advantages of this specification more clear, the following will clearly and completely describe the technical solutions of this specification in conjunction with the specific embodiments of this specification and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this specification.

[0012] The technical solutions provided by the embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0013] Figure 1 A flowchart of a method for verifying LLM output results provided in an embodiment of this specification specifically includes the following steps: S100: Obtain the output result obtained by the LLM based on the input data.

[0014] In the embodiments of this specification, Figure 1 The device used to verify the output of the LLM in the method shown can be any electronic device, such as a personal computer, a mobile phone, a tablet computer, a server, or a server cluster consisting of several servers. The following description only uses a server as an example.

[0015] In the embodiments of this specification, an LLM is used to receive input data and perform reasoning based on the input data to produce an output result. Upon obtaining an output result, the LLM not only outputs the output result but also outputs the reasoning process that led the LLM to the output result. The reasoning process described in the embodiments of this specification is a natural language reasoning process.

[0016] The server can obtain the above output results obtained by the LLM and the reasoning process based on which the LLM obtains the above output results.

[0017] S102: Determine the type of reasoning task performed by the LLM to obtain the output result.

[0018] In an embodiment of the present specification, the server may identify the type of reasoning task performed by the LLM to obtain the output result based on the output result obtained by the LLM, the input data based on which the LLM obtains the output result, and / or the above-mentioned reasoning process.

[0019] Since the types of reasoning tasks performed by LLM are generally divided into two types: logical judgment type and information acquisition type, in the embodiment of this specification, the server can identify the type of reasoning task performed by the LLM to obtain the output result through the LLM itself.

[0020] Specifically, for logical reasoning, the LLM can determine whether the reasoning task it is performing involves at least one of subjective evaluation, logical analysis, weighing pros and cons, and inference based on existing knowledge. If so, the reasoning task performed by the LLM is determined to be a logical reasoning task; otherwise, it is not a logical reasoning task. Alternatively, the LLM can also determine whether the output result requires the LLM to integrate knowledge, application rules, or expression tendencies from a preset knowledge base, rather than directly retrieving facts, and whether the output result is Boolean data (i.e., binary data of either 0 or 1) or one of a limited number of predetermined results (i.e., multi-category data). If so, the reasoning task performed by the LLM is determined to be a logical reasoning task; otherwise, it is not a logical reasoning task.

[0021] For information acquisition-type reasoning, the LLM can determine whether the reasoning task it is performing directly extracts objective facts, definitions, or specific information from a preset knowledge base and / or input data and / or training data. If so, the reasoning task performed by the LLM can be determined to be information acquisition-type; otherwise, it is not. Alternatively, the LLM can determine whether the output result is a fragment that clearly exists in the preset knowledge base and / or input data and / or training data, without the need for subjective processing by the LLM. If so, the reasoning task performed by the LLM can be determined to be information acquisition-type; otherwise, it is not.

[0022] When the LLM identifies that the reasoning task it is performing is a logical reasoning task, it can verify the output result in step S104. When the LLM identifies that the reasoning task it is performing is an information acquisition task, it can verify the output result in step S106. In other words, there is no logical dependency between steps S104 and S106; they can be executed separately. The present embodiment does not limit the order in which steps S104 and S106 are executed.

[0023] S104: When it is determined that the type is a logic judgment type, the reasoning process of the LLM to obtain the output result according to the input data is obtained, and the correctness of the output result is verified based on the reasoning process and the output result.

[0024] In an embodiment of the present specification, when the reasoning task performed by the LLM to obtain the above-mentioned output result is a reasoning task of the logical judgment type, the server can still verify the correctness of the above-mentioned output result obtained by the LLM through the LLM, that is, the above-mentioned output result can be self-checked by the LLM. Specifically, the server can input a preset first prompt information to the LLM, so that the LLM, under the guidance of the first prompt information, relies on the LLM's own reasoning ability to determine whether the output result is correct under the conditions of the above-mentioned input data, that is, to perform an initial verification of the correctness of the output result. When the LLM outputs the initial verification result, it can also output the reason for obtaining the initial verification result. If the LLM fails the initial verification of the output result, it can be directly determined that the correctness verification of the output result has failed.

[0025] In the embodiment of this specification, when LLM passes the initial verification of the output result, in order to further improve the accuracy of the verification of the output result, the reasoning process performed by LLM to obtain the above output result can still be continued to perform further final verification on the correctness of the output result.

[0026] Specifically, if the LLM passes the initial verification of the output result, a preset second prompt message may be input to the LLM. Under the guidance of the second prompt message, the LLM determines whether the reason for the LLM's initial verification of the output result matches the reasoning process based on which the LLM obtained the output result. If so, the final verification of the output result may be determined to have passed. Otherwise, the final verification of the output result may be determined to have failed.

[0027] For example, suppose the input data is an email, and the LLM's reasoning task is to determine whether it is a phishing email. The LLM's pre-set knowledge base rule for identifying phishing emails is "the presence of the keyword 'salary' or similar in the email content." The LLM's output is "This email is a phishing email," and the reasoning is "Because the keyword 'salary' is in the email content, this email is a phishing email." Considering this example from a bystander's perspective, the LLM's reasoning and output are essentially correct.

[0028] When verifying the correctness of the above output result, the correctness of the output result is initially verified by the LLM. The initial verification result is passed, so the reason why the initial verification result output by the LLM is passed is obtained.

[0029] If the reason output by the LLM is "because the email contains the keyword "salary"," and this reason matches the above reasoning process of the LLM, it means that the reasoning process of the LLM is stable and no hallucination occurs. The above output result is credible and passes the correctness verification.

[0030] However, if the reason output by the LLM is "because the email content format is confusing", which does not match the above reasoning process of the LLM, it means that the reasoning process of the LLM is unstable and there is a high possibility of hallucination. Although we can know that the above output result is actually correct from the perspective of an observer, during the verification process, it can still be determined that because the LLM's reasoning process is unstable, the above output result is not credible and fails the correctness verification.

[0031] Furthermore, when determining whether the reason why the LLM passed the initial verification matches the above-mentioned reasoning process, the semantic similarity between the reason and the reasoning process can be specifically determined. If the similarity is greater than a preset threshold, it can be determined that the reason matches the reasoning process. If the similarity is not greater than the preset threshold, it can be determined that the reason does not match the reasoning process. When determining the semantic similarity between the reason and the reasoning process, the keywords contained in the reason and the keywords contained in the reasoning process can be determined respectively first, and then the semantic similarity between the reason and the reasoning process can be determined based on the keywords contained in the reason and the keywords contained in the reasoning process. Of course, the semantic similarity between the reason and the reasoning process can also be determined by other methods. For example, the semantic similarity between the reason and the reasoning process can still be determined by the above-mentioned LLM.

[0032] S106: When it is determined that the type is an information acquisition type, a proposition to be verified is generated according to the output result and the input data, and the correctness of the output result is verified according to the proposition to be verified.

[0033] In the embodiment of the present specification, when the reasoning task performed by the LLM to obtain the above output result is an information acquisition type reasoning task, the server can also verify the above output result through the LLM in step S106.

[0034] Specifically, the server may first re-input the above-mentioned output results and input data into the LLM, and input the preset third prompt information into the LLM, so that the LLM, under the guidance of the third prompt information, generates a positive proposition based on the above-mentioned output results and input data. The logic of the positive proposition conforms to the logical relationship of obtaining the output result from the input data, and then determines the negation of the positive proposition as the proposition to be verified.

[0035] As an example of generating a proposition to be verified, the server can generate a question based on the input data and an answer to the question based on the output result. The question-answer pair consisting of the question and the answer is a positive proposition that conforms to the logical relationship between the input data and the output result. For example, if the input data is a command line "xxx...", the reasoning task required by the LLM is to extract the Uniform Resource Locator (URL) from the command line, and the output data of the LLM is "abc", then the generated positive proposition can be "The URL extracted from the command line "xxx..." is "abc"", "The command line "xxx..." contains the URL "abc"", and any other proposition that conforms to the logical relationship between the input data and the output result.

[0036] After obtaining the positive proposition, the server can input the preset fourth prompt information into the LLM, so that the LLM, under the guidance of the fourth prompt information, generates the negation of the positive proposition as the proposition to be verified. Among them, the negation is the complement of the positive proposition, and its logic is completely contradictory and mutually exclusive with the positive proposition. Continuing with the above example, after obtaining the positive proposition "the command line "xxx..." contains the URL "abc"", the LLM can determine that the negation of the positive proposition is "the command line "xxx..." does not contain the URL "abc"", and use the negation as the proposition to be verified.

[0037] Finally, the server may input the proposition to be verified into the LLM again, and also input the preset fifth prompt information into the LLM, and use the fifth prompt information to guide the LLM to judge the correctness of the proposition to be verified. If the LLM determines that the proposition to be verified is incorrect, then it is determined that the correctness verification of the above-mentioned output result has passed; if the LLM determines that the proposition to be verified is correct, then it is determined that the correctness verification of the above-mentioned output result has failed.

[0038] Continuing with the above example, since the logic of the positive proposition directly reflects the logical relationship that the input data can produce the output result, the negation completely contradicts the logic of the positive proposition and reflects the logical relationship that the input data cannot produce the output result. Therefore, since the LLM has already obtained the output result based on the input data, the reasoning process based on which the LLM performs the reasoning task should conform to the logic of the positive proposition. However, if the LLM now believes that the proposition to be verified (i.e., the negation) is correct, that is, the LLM believes that the output result cannot be obtained from the input data, which is completely inconsistent with the LLM's previous reasoning process, then it can be determined that the LLM's logic is confused and likely an illusion. The output result obtained by the LLM is unreliable, and therefore the correctness check of the output result is determined to fail. On the contrary, if the LLM now believes that the proposition to be verified (i.e., the negated proposition mentioned above) is incorrect, that is, the LLM still believes that the output result can be obtained from the input data, that is, the current logic of the LLM is completely consistent with the logic when the output result was obtained before, then it can be determined that the logic of the LLM is stable, and the above-mentioned output result obtained is credible, so it is determined that the correctness check of the output result has passed.

[0039] The above Figure 1 The verification method shown can be implemented by an agent deployed on a server.

[0040] In addition, in the embodiment of this specification, the input data of the above-mentioned input LLM can specifically be the calculation logic of the special operator corresponding to the operator name contained in the knowledge high-level program (HOP), as well as the input parameters of the special operator. Among them, the execution method of HOP is as follows Figure 2 shown.

[0041] Figure 2 The flowchart of a HOP execution method provided in the embodiment of this specification specifically includes the following steps: S200: Running a running environment of a preset type of computer program code and loading business data.

[0042] The HOP described in this specification refers to a predefined type of computer program code, such as Python. The HOP includes operator names that conform to the function name format requirements of the predefined type of computer program code. However, the computational logic of the operator corresponding to the operator name is defined in natural language. These operators are referred to as special operators below.

[0043] The server can first obtain the HOP and run the operating environment of the preset type of computer program code, and then load the business data required by the HOP. For example, when the HOP is Python code, the server can pull up the Python operating environment and load the business data.

[0044] The HOP is generated by the LLM based on a business logic process corresponding to a certain business. The business logic process is described in natural language. The generation process of the HOP will be described later.

[0045] S202: Based on the service data, run the HOP through the running environment.

[0046] In the embodiments of this specification, the framework code of the HOP is written by the above-mentioned preset type of computer program code. Even if the code contains special operators, the operator name of the special operator also conforms to the function name format requirements of the preset type of computer program code. Therefore, from a formal point of view, the HOP is essentially the preset type of computer code. The server can run the HOP based on the loaded business data, using the business data as the input parameter of the entire HOP, and by pulling up the running environment of the preset type of computer program code.

[0047] S204: When the statement containing the operator name in the HOP is run, the pre-trained LLM is called.

[0048] When the server runs through the execution environment and reaches a statement in the HOP that contains the operator name of the special operator, the pre-trained LLM can be called.

[0049] S206: Input the business data and / or the intermediate result obtained by currently running the HOP into the LLM as the input parameters of the special operator corresponding to the operator name, so that the LLM executes the calculation logic of the special operator corresponding to the operator name according to the input parameters and the preset knowledge base to determine the execution result of the special operator.

[0050] Since the calculation logic of the special operator is defined by natural language, the running environment of the preset type of computer program code cannot normally execute the special operator. Therefore, the server can input the business data and the intermediate result obtained by currently running the HOP as the input parameters of the special operator into the LLM, so that the LLM can execute the calculation logic of the special operator according to the input parameters and the knowledge in the preset knowledge base to obtain the execution result of the special operator, which is Figure 1 The output result of LLM in step S100 is shown.

[0051] S208: Check the correctness of the execution result.

[0052] Since the calculation logic of the special operator is defined in natural language and needs to be executed by LLM, in order to avoid inaccurate execution of the special operator defined in natural language by LLM, the correctness of the execution result can be checked to determine whether LLM correctly executes the calculation logic of the special operator defined in natural language.

[0053] Among them, the method of checking the correctness of the execution result obtained by executing the special operator on the LLM in step S208 is the method included in this specification. Figure 1 Verification methods including those shown.

[0054] S210: When the correctness check passes, continue to execute the statement according to the execution result of the operator.

[0055] If the correctness check is passed, the LLM returns the execution result of the special operator to the execution environment of the preset type of computer program code, and the execution environment continues to execute the statement until the HOP is completely executed.

[0056] Specifically, when the above Figure 1 When the verification method shown verifies the correctness of the output result of the LLM executing the special operator, the LLM can return the output result as the execution result of the special operator to the running environment, so that the running environment continues to execute the statement according to the execution result of the special operator.

[0057] Among them, the above Figure 2 The generation process of the HOP shown can be as follows Figure 3 shown. Figure 3 The flow chart of the HOP generation method provided in the embodiment of this specification specifically includes the following steps: S300: Acquire a business logic process described in natural language.

[0058] In the embodiments of this specification, Figure 3 The device for generating codes by the method shown can be any electronic device, such as a computer, a server, or a server cluster consisting of multiple servers. Figure 2 The devices for executing HOP in the method shown may be the same or different. For the convenience of description, the following description is still based on the same server as an example.

[0059] In order to generate code for executing a certain business, the server can first obtain the business logic process of the business. Specifically, the business logic process described in this specification includes a standard operating procedure (SOP). The SOP is a standard operating procedure for the business described in natural language. It is a document that details the work, workflow, and steps in the business. It provides standard process specifications within the organization of the business for reference and operation by internal personnel of the organization. In addition to containing the standardized process of the business, the SOP also includes clear responsibilities and authorities of each role in the business, emergency measures, quality standards and assessments, document management, and other content.

[0060] In the embodiments of this specification, the code generation method provided in this specification is described below by taking the above-mentioned business as the risk control business as an example.

[0061] When the above-mentioned business is a risk control business based on the recorded risk log, the business logic process of the risk control business (that is, the risk control logic process) can be described in the following natural language: The first step is to analyze the cmd field to determine whether the cmd field grants the file executable permission. If the cmd command line does not grant the file executable permission, the conclusion is that no action is required. The second step is to determine whether the installation package is a commonly used software installation package. If it is a commonly used installation software, the conclusion is that no action is required; The third step is to determine whether the cmd field has been manually analyzed and judged by security operations personnel within the past N days and determined to require no action. For example, if the chmod command line is judged to require no action, then no action is required for this risk log. The fourth step is to determine whether the risk log does not need to be handled or generates an attack alarm based on the results of the above three steps.

[0062] The risk log is a risk log of other businesses recorded by the user when performing other businesses, and the risk control business is a business that requires risk control of the other businesses based on the risk log.

[0063] S302: Input the business logic process into the pre-trained LLM.

[0064] After the server obtains the business logic process described above, it can input the business logic process described in natural language into a pre-trained LLM. This LLM can be pre-deployed on the server, or it can be deployed on other devices. If deployed on other devices, the server can send the business logic process to the other devices, allowing them to input the received business logic process into the LLM. The following description uses the LLM deployed on the server as an example.

[0065] In order to avoid the LLM from generating "hallucinations" and code that is inconsistent with the actual business logic process, and to improve the accuracy and reliability of the LLM generated code, in the embodiment of this specification, the server directly inputs the actual business logic process obtained in step S300 into the LLM, so that the LLM generates code for executing the business based on the business logic process.

[0066] S304: Identify the steps included in the business logic process and the logical relationships between the steps through the LLM.

[0067] Generally, a business logic process consists of several steps arranged according to certain logical relationships. For example, in the example above, the first step is executed first. If the first step evaluates to "no," the result is output directly, indicating no action is required. If the result is "yes," the second step is executed. If the second step evaluates to "yes," the result is output directly, indicating no action is required. Otherwise, the third step is executed. If the third step evaluates to "yes," the result is output directly, indicating no action is required. Otherwise, an attack alert is generated. Therefore, the logical relationships between the different steps in a business logic process constitute the basic framework of the process. Specifically, the execution of each step requires the use of knowledge specific to the business logic process, including the concepts and attributes of the entities required in the process, as well as the relationships between them. For example, in the example above, the first step requires determining whether the "cmd" field grants executable permissions to a file. The knowledge required to execute this first step includes at least the concept of the "cmd" field, the concept of file executable permissions, and the relationship between the "cmd" field and file executable permissions (i.e., which "cmd" field indicates that the file has been granted executable permissions, and which "cmd" field indicates that the file has not been granted executable permissions).

[0068] It can be seen from this that a business logic process is essentially composed of the logical relationship between each step and the knowledge required to execute each step. Therefore, to generate code for executing the business logic process, LLM needs to identify the steps contained in the business logic process and the logical relationship between the steps based on the business logic process described in natural language.

[0069] Specifically, since identifying the steps and logical relationships between the steps in a business logic process may also require a knowledge base corresponding to the business logic process, in the embodiments of this specification, the knowledge base corresponding to the business logic process may be pre-injected into the pre-trained LLM. The knowledge base may include at least one of the following: The knowledge graph corresponding to the business logic process includes the concept description of each entity required in the business logic process (used to describe what the entity is, such as the concept of cmd command line in the above example), the attributes of each entity (used to describe all possible attributes of the entity in the business logic), and the relationship between each entity; The calling interface of the tool required in the business logic process includes API or SDK, etc. For example, in the second step of the above example, it is necessary to determine whether the installation package is an installation package of commonly used software. In this case, it is necessary to call the tool to obtain a list of installation packages of commonly used software. This requires calling the tool through the tool's calling interface.

[0070] After injecting the above-mentioned knowledge base containing the knowledge graph corresponding to the business logic process and / or the calling interface of the required tools into the pre-trained LLM, when the server inputs the business logic process described in natural language to the LLM, the LLM can identify the semantics of the business logic process described in the natural language based on its own reasoning ability, and map the concepts in the business logic process into entities in the knowledge graph. Then, based on the semantics, the entire business logic process is split into several steps, and finally the logical relationship between the steps is inferred.

[0071] S306: Based on the logical relationship, generate a preset type of computer program code for representing the logical relationship between the steps as a framework code.

[0072] Because natural language itself has a certain degree of ambiguity and ambiguity, which is an inherent defect of any natural language, directly using natural language to express the logical relationship between the steps in the business logic process is very likely to lead to logical confusion among the steps. In particular, when people other than the person who wrote the business logic process read the business logic process, the ambiguity and ambiguity of natural language may lead to inaccurate understanding, which will affect the business itself.

[0073] Computer programming language, as a formalized and precise language, has clear semantics and strict logical structure, and can effectively avoid the above-mentioned problems of natural language. Therefore, in the embodiment of this specification, for the logical relationship between each step in the business logic process, the server can generate a preset type of computer program code for representing the logical relationship between each step as a framework code through LLM based on the logical relationship between each step identified in step S304.

[0074] The preset type of computer program code may be any type of computer program code, including but not limited to Python code. The following description will only take Python code as an example.

[0075] S308: For each step, generate at least one operator for executing the step according to the natural language used to describe the step in the business logic process and a preset knowledge base corresponding to the business logic process.

[0076] In the embodiment of this specification, the execution order of step S306 and step S308 may not be particular. Specifically, the server may execute step S306 through the LLM while also executing step S308 through the LLM.

[0077] As for the knowledge required for each step in the business logic process, natural language is more natural and more understandable than computer programming language. Therefore, in the embodiments of this specification, an operator defined in natural language is used to represent a step.

[0078] The computational logic of the operators described in the embodiments of this specification is defined in natural language, but the operator names of these operators still conform to the function name format requirements of the aforementioned predefined type of computer program code. Therefore, the operators defined in natural language in this specification are essentially custom functions within the predefined type of computer program code, which are high-level abstract expressions of the logic that implements the steps.

[0079] Specifically, for each step identified in the business logic process, the LLM can determine the knowledge required to execute the step, such as one or more entities in the knowledge graph, one or more attributes of the entity, the relationship between the entity and other entities, and / or the calling interface of the tool required for the step, based on the semantics of the natural language used to describe the step in the identified business logic process and the knowledge base corresponding to the business logic process. After determining the knowledge required to execute the step, at least one operator for executing the step can be generated based on the knowledge. The operator is defined in natural language and its name conforms to the function name format requirements of the aforementioned preset type of computer program code.

[0080] Continuing with the previous example, the first step is to determine whether the cmd field grants the file executable permission. Therefore, LLM can generate an operator name hop_judge that meets the Python code function name requirements and define the operator hop_judge as follows: def hop_judge Determine whether {cmd} grants file executable permissions As can be seen, the definition of the operator hop_judge is completely defined using natural language, "Judge whether {cmd} grants file executable permissions," making the operator more understandable. Operators that use natural language to define computational logic are special operators in HOP.

[0081] The above is just an example of expressing a step with one operator. Those skilled in the art should understand that in actual applications, for more complex steps, more operators (such as more than two operators) can be used to express the step.

[0082] S310: Generate a knowledge high-level program for executing the business logic process according to the framework code and the operator name of the operator generated for each step.

[0083] As explained above, a business logic process is essentially composed of the logical relationships between the steps and the knowledge required to execute each step. The logical relationships have been expressed using the generated framework code in step S306, and the knowledge required to execute each step has been expressed in step S308 using operators defined in natural language, whose operator names still conform to the preset type of computer program code function name format requirements. Therefore, LLM can combine the framework code and the operators corresponding to each step to generate a knowledge high-level program (HOP) for executing the business logic process.

[0084] Continuing with the above example, LLM combines the operators corresponding to each step into the framework code to generate the following HOP for executing the above risk control logic process: def evaluate(cmd): if hop_judge_a("Judge (cmd) whether to grant file executable permissions") #High-level judgment pkg = hop_get("Get the installation package used by (cmd)") if not pkg in hop_knowledge_retrieve("Common Installation Package") #High-level knowledge concept matching if not hop_judge_b("Check the history to see if (cmd) has been manually judged as not requiring action") return "Attack Alarm" return "No need to dispose" In the HOP above, the content enclosed by ("") is the natural language definition of the corresponding operator. These operators include hop_judge_a, hop_get, hop_knowledge_retrieve, and hop_judge_b. The framework code expressed in Python above shows the entire risk control logic process as follows: first, execute the hop_judge_a operator to determine whether (cmd) has file executable permissions. If so, execute the hop_get operator, obtain the installation package used by (cmd), and assign it to pkg. If not, directly output "No action required." After obtaining the installation package used by (cmd) and assigning it to pkg, execute the hop_knowledge_retrieve operator to obtain the commonly used installation package and determine whether pkg is not among the commonly used installation packages. If so, execute the hop_judge_b operator. If not, directly output "No action required." Execute the hop_judge_b operator to determine whether (cmd) has not been manually determined to require action in the past. If so, output "Attack Alarm"; otherwise, output "No action required."

[0085] It can be seen that the HOP represented by the Python code is completely consistent with the original intention of the risk control logic process described in natural language. Therefore, through the HOP, even if there is a change in the personnel in the department that performs the risk control business, when the HOP is read by other personnel (other than the personnel who wrote the original risk control logic process), the logical confusion of each step caused by the ambiguity and ambiguity of the natural language itself can be avoided. Moreover, although the special operators in the above HOP code are defined by natural language, the operator names are still function names in the Python code format. Therefore, the HOP code itself is composed of Python framework code referencing various Python-formatted operator names. From a formal point of view, it is still Python code. Therefore, the HOP code is not only readable by humans, but can also be used Figure 2 The method shown is directly executed in the Python execution environment to implement the above risk control logic process.

[0086] In the embodiments of this specification, in order for the LLM to accurately convert the business logic process described in the input natural language into the above-mentioned HOP, it is not enough to simply inject the knowledge base corresponding to the business logic process into the business LLM. The business LLM also needs to be fine-tuned and trained in the business scenario corresponding to the business logic process in advance.

[0087] Specifically, when fine-tuning the LLM, you can first obtain a sample logical process described in natural language. This sample logical process also includes several steps, referred to as sample steps below. This sample logical process and the aforementioned business logic process are business logic processes in the same business scenario. That is, the knowledge base corresponding to the sample logical process is exactly the same as the knowledge base corresponding to the aforementioned business logic process.

[0088] After obtaining the sample logic process, you can also Figure 3 In the steps S302 to S310 shown, the sample logic process is input into the LLM to be trained, and the sample steps contained in the sample logic process and the logical relationship between the sample steps are identified by the LLM to be trained. According to the logical relationship between the sample steps, a preset type of computer program code for representing the logical relationship between the sample steps is generated as the framework code to be optimized. Moreover, for each sample step, according to the natural language used to describe the sample step in the sample logic process and the knowledge base corresponding to the preset sample logic process, at least one operator for executing the sample step is generated as the operator to be optimized. The calculation logic of the operator to be optimized is also defined in natural language, and the operator name of the operator to be optimized also conforms to the function name format of the above-mentioned preset type of computer program code. Finally, according to the framework code to be optimized and the operator name of the operator to be optimized generated for each sample step, a HOP to be optimized for executing the sample logic process is generated. This process is the same as Figure 3 Steps S302 to S310 are identical and will not be described again here.

[0089] After the LLM to be trained generates the HOP to be optimized, the optimized HOP after adjusting the HOP to be optimized can be determined. Among them, since the HOP to be optimized is also the above-mentioned preset type of computer program code (such as Python code) in form, the HOP to be optimized can be manually adjusted so that the logical relationship between the sample steps expressed by the framework code to be optimized is completely consistent with the original sample logical process, and the natural language corresponding to the operator to be optimized defined by natural language contained therein is more in line with the knowledge in the knowledge base corresponding to the sample logical process, and the calculation logic of the operator to be optimized expressed is clearer and more accurate. Of course, the HOP to be optimized can also be adjusted by other trained LLMs, and this embodiment of the specification does not limit this.

[0090] After obtaining the optimized HOP, the supervised fine-tuning training (SFT) of the LLM to be trained can be performed based on the optimized HOP, that is, the optimized HOP is used as the label corresponding to the sample logic process, and used as the supervision signal to adjust the model parameters of the LLM to be trained, so that the model parameters of the LLM to be trained are adjusted to make the LLM adapt to generate various business logic processes in the business scenario. Figure 3 The device used for code generation can be the same device or a different device. Furthermore, since the embodiments of this specification require that the original reasoning capabilities of the LLM be maintained as much as possible and adapted only to the business scenario, when performing SFT on the LLM, not all model parameters of the LLM are adjusted; instead, only some of the model parameters within the LLM are adjusted. The adjusted model parameters may only include the LoRA model parameters within the LLM.

[0091] After the LLM is obtained through the above SFT training, the server can Figure 3 The method shown generates the HOP corresponding to the business logic process in this business scenario. Figure 2 When the method shown executes the HOP, since the HOP not only includes the above-mentioned preset type of computer program code (such as Python), but also includes special operators defined by natural language, the LLM obtained by the above-mentioned training is also required to assist in the execution when executing the HOP.

[0092] When the above-mentioned business scenario is a risk control business scenario, that is, the above-mentioned business logic process is a risk control logic process under the risk control business scenario, the loaded business data can be a pre-recorded risk log, which is a risk log of other businesses recorded by the user when performing other businesses. The risk control business is a business that needs to perform risk control on the other businesses based on the risk log.

[0093] pass Figure 2 After the step S200 shown pulls up the operating environment of the preset type of computer program code and loads the business data, the computer program can be run through the operating environment based on the business data. Figure 3 Generated HOP.

[0094] When running to the special operator defined by natural language in the HOP, it is necessary to call the above-mentioned LLM. Among them, since the framework of the HOP is still a preset type of computer program code, the operating environment using the preset type of computer program code can directly run the HOP. As for the special operator defined by natural language contained in the HOP, since the special operator is essentially a custom function in the preset type of computer program code, the HOP references the function name, which means executing the function corresponding to the function name, obtaining the execution result, and then continuing to execute the subsequent code according to the execution result. Therefore, when running to the operator name of the special operator defined by natural language in the HOP, it is necessary to call the pre-defined calculation logic of the operator. When the calculation logic of the operator is parsed by the operating environment, if it is parsed that the calculation logic is defined by natural language, the operating environment can directly call the above-mentioned LLM.

[0095] After calling the above LLM, you can Figure 2 In step S206 shown, the business data and / or the intermediate results obtained by currently running the HOP are input into the LLM as input parameters of the operator, so that the LLM executes the calculation logic of the operator according to the input parameters and the knowledge base corresponding to the above-mentioned business logic process to determine the execution result of the operator.

[0096] Continuing with the above example, for the operator hop_judge_a in the above HOP, its natural language defined calculation logic is "Judge whether (cmd) grants file executable permissions." In this case, business data, that is, the pre-recorded risk log, can be used as the operator's input parameter. The operator's natural language defined calculation logic and the input parameter are both input into the LLM. Prompt information can also be input into the LLM. Under the guidance of the prompt information, the LLM first identifies the cmd command line in the input parameter (i.e., the entity in the knowledge graph) based on the knowledge base corresponding to the risk control logic process. Then, based on the knowledge base and the identified cmd command line, the LLM determines the attributes of the cmd command line in the input parameter (i.e., the attributes of the entity in the knowledge graph). Based on the attributes, the LLM determines whether the cmd command line grants file executable permissions (i.e., the relationship between different entities in the knowledge graph). The judgment result is used as the output parameter of the operator hop_judge_a and returned to the Python runtime environment, allowing the Python runtime environment to continue executing subsequent code in the HOP.

[0097] For another example, for the operator hop_knowledge_retrieve in the above HOP, the calculation logic defined in natural language is "common installation packages". The calculation logic defined in natural language of the operator can be input into LLM, and prompt information can be input into LLM, so that the business LLM, under the guidance of the prompt information, first determines the tool for obtaining the list of commonly used installation packages based on the knowledge base corresponding to the risk control logic process, and then calls the tool through the tool's calling interface to obtain the commonly used installation packages through the called tool. Finally, the obtained list of commonly used installation packages is used as the output parameter of the operator hop_knowledge_retrieve and returned to the Python runtime environment, so that the Python runtime environment continues to execute subsequent code in the HOP to determine whether the intermediate result pkg of the current running of the HOP is not included in the list of commonly used installation packages.

[0098] Those skilled in the art will appreciate that the above description uses the risk control logic process as an example. In reality, the HOP generation methods provided in the embodiments of this specification can be used to generate and execute HOPs corresponding to business logic processes in any business scenario. Furthermore, the aforementioned preset computer program code is described using Python code as an example; the preset computer program code described in the embodiments of this specification can also be other types of computer program code.

[0099] exist Figure 2 In step S208 shown in FIG, since the calculation logic of the special operator is defined by natural language and LLM may have "hallucinations", in order to ensure the accuracy of LLM's execution of the special operator, the embodiment of this specification also needs to adopt Figure 1 The verification method shown performs a correctness check on the execution result obtained by the LLM executing the special operator. If the correctness check passes, the LLM can return the execution result to the running environment of the above-mentioned preset type of computer program code, so that the running environment continues to execute the HOP until the entire HOP is executed. If the correctness check fails, the calculation logic of the special operator can be re-executed by the LLM to re-determine the execution result of the special operator until the execution result obtained passes the correctness check, or the number of times the LLM executes the calculation logic of the special operator reaches a preset number.

[0100] If the number of times that the LLM executes the calculation logic of the special operator reaches a preset number, and the execution result of the special operator determined for the last time still fails to pass the correctness check, an assistance execution request can be generated by the LLM and output. The assistance execution request is used to request manual assistance to obtain the execution result of the special operator. The assistance execution request can be sent to a user device, and the user can input the execution result of the special operator to the user device according to the calculation logic and input parameters of the special operator carried in the assistance execution request, and the user device returns the execution result input by the user to the LLM. The LLM receives the execution result returned by the user device and uses the received execution result as an execution result that has passed the correctness check. It is also possible to perform supervised fine-tuning training on at least part of the model parameters in the LLM based on the calculation logic of the special operator and the execution result received from the user, such as only performing supervised fine-tuning training on the model parameters of LoRA in the LLM.

[0101] The above is a method for verifying LLM output results provided in an embodiment of this specification. Based on the same idea, this specification also provides corresponding devices, storage media, and electronic devices.

[0102] Figure 4 A schematic diagram of a device for verifying LLM output results provided in an embodiment of this specification, the device comprising: An acquisition module 401 is used to obtain an output result obtained by the LLM based on input data; A task type determination module 402 is used to determine the type of reasoning task performed by the LLM to obtain the output result; A first verification module 403 is configured to, when determining that the type is a logic judgment type, obtain the reasoning process of the LLM to obtain the output result based on the input data, and verify the correctness of the output result based on the reasoning process and the output result; The second verification module 404 is configured to generate a proposition to be verified based on the output result and the input data when it is determined that the type is an information acquisition type, and verify the correctness of the output result based on the proposition to be verified.

[0103] Optionally, the first verification module 403 is specifically used to perform an initial verification on the correctness of the output result through the LLM; if the LLM fails the initial verification of the output result, it is determined that the correctness verification of the output result fails; if the LLM passes the initial verification of the output result, the correctness of the output result is finally verified according to the reasoning process.

[0104] Optionally, the first verification module 403 is specifically used to obtain the reason why the LLM determines that the output result has passed the initial verification; determine whether the reason matches the reasoning process; if they match, determine that the final verification of the output result has passed; if they do not match, determine that the final verification of the output result has failed.

[0105] Optionally, the first verification module 403 is specifically used to determine the keywords contained in the cause and the keywords contained in the reasoning process respectively; determine the similarity between the cause and the reasoning process based on the keywords contained in the cause and the keywords contained in the reasoning process; if the similarity is greater than a preset threshold, determine that the cause matches the reasoning process; if the similarity is not greater than the preset threshold, determine that the cause does not match the reasoning process.

[0106] Optionally, the second verification module 404 is specifically used to generate a positive proposition based on the output result and the input data, and the logic of the positive proposition conforms to the logical relationship of obtaining the output result from the input data; and determine the negation of the positive proposition as the proposition to be verified.

[0107] Optionally, the second verification module 404 is specifically used to input the proposition to be verified into the LLM so as to determine the correctness of the proposition to be verified through the LLM; if the LLM determines that the proposition to be verified is incorrect, it is determined that the correctness verification of the output result has passed; if the LLM determines that the proposition to be verified is correct, it is determined that the correctness verification of the output result has failed.

[0108] Optionally, the input data includes: the calculation logic of the special operator corresponding to the operator name contained in the knowledge high-order program HOP, and the input parameters of the special operator; The calculation logic of the special operator is defined by natural language, and the HOP is a computer program code of a preset type; In which, the HOP is run through the running environment of the preset type of computer program code and based on business data. When running to the statement containing the operator name in the HOP, the calculation logic of the special operator is input into the LLM, and the business data and / or the intermediate results obtained by currently running the HOP are input into the LLM as input parameters of the special operator, so that the LLM executes the calculation logic of the special operator according to the input parameters and the preset knowledge base to determine the execution result of the operator as the output result of the LLM.

[0109] Optionally, the input data is input to the LLM each time the runtime environment runs to a statement in the HOP containing an operator name of a special operator; The device further comprises: The running module 405 is used to return the output result as the execution result of the special operator to the running environment through the LLM when the correctness of the output result is verified, so that the running environment continues to execute the statement according to the execution result of the special operator.

[0110] This specification also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it can be used to perform the above-mentioned LLM output result verification method.

[0111] based on Figure 1 The verification method of the LLM output result shown in the embodiment of this specification also provides Figure 5 The structural diagram of the electronic device shown in FIG. Figure 5 At the hardware level, the electronic device includes a processor, an internal bus, a network interface, memory, and non-volatile storage, and may also include other hardware required for its operations. The processor reads the corresponding computer program from the non-volatile storage into the internal memory and then runs it to implement the aforementioned LLM output verification method.

[0112] The foregoing is merely an example of the present invention and is not intended to limit the present invention. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be included within the scope of the claims of the present invention.

Claims

1. A method for verifying output results of a large language model (LLM), the method comprising: Get the output results of LLM based on input data; determining a type of reasoning task performed by the LLM to obtain the output result; When it is determined that the type is a logic judgment type, obtaining the reasoning process of the LLM to obtain the output result according to the input data, and verifying the correctness of the output result based on the reasoning process and the output result; When it is determined that the type is an information acquisition type, a proposition to be verified is generated according to the output result and the input data, and the correctness of the output result is verified according to the proposition to be verified.

2. The method according to claim 1, further comprising verifying the correctness of the output result based on the reasoning process and the output result, specifically comprising: Performing an initial verification of the correctness of the output result by the LLM; If the LLM fails the initial verification of the output result, it is determined that the correctness verification of the output result fails; If the LLM passes the initial verification of the output result, the correctness of the output result is finally verified according to the reasoning process.

3. The method according to claim 2, further comprising: performing a final verification on the correctness of the output result according to the reasoning process, specifically comprising: Obtaining the reason why the LLM determines that the output result has passed the initial verification; Determining whether the cause matches the reasoning process; If they match, it is determined that the final verification of the output result has passed; If there is no match, it is determined that the final verification of the output result fails.

4. The method according to claim 3, wherein determining whether the cause matches the reasoning process comprises: respectively determining keywords included in the cause and keywords included in the reasoning process; Determining the similarity between the cause and the reasoning process based on the keywords included in the cause and the keywords included in the reasoning process; If the similarity is greater than a preset threshold, determining that the cause matches the reasoning process; If the similarity is not greater than a preset threshold, it is determined that the cause does not match the reasoning process.

5. The method according to claim 1, generating a proposition to be verified based on the output result and the input data, specifically comprising: Generate a positive proposition based on the output result and the input data, wherein the logic of the positive proposition conforms to the logical relationship of obtaining the output result from the input data; Determine the negation of the positive proposition as the proposition to be verified.

6. The method according to claim 5, wherein the correctness of the output result is verified according to the proposition to be verified, specifically comprising: Inputting the proposition to be verified into the LLM to determine the correctness of the proposition to be verified through the LLM; If the LLM determines that the proposition to be verified is incorrect, then the correctness verification of the output result is determined to be passed; If the LLM determines that the proposition to be verified is correct, it determines that the correctness verification of the output result fails.

7. The method of claim 1, wherein the input data comprises: The calculation logic of the special operator corresponding to the operator name contained in the knowledge high-level program HOP, as well as the input parameters of the special operator; The calculation logic of the special operator is defined by natural language, and the HOP is a computer program code of a preset type; In which, the HOP is run through the running environment of the preset type of computer program code and based on business data. When running to the statement containing the operator name in the HOP, the calculation logic of the special operator is input into the LLM, and the business data and / or the intermediate results obtained by currently running the HOP are input into the LLM as input parameters of the special operator, so that the LLM executes the calculation logic of the special operator according to the input parameters and the preset knowledge base to determine the execution result of the operator as the output result of the LLM.

8. The method according to claim 7, wherein the input data is input into the LLM each time the execution environment runs a statement containing an operator name of a special operator in the HOP; The method further comprises: When the correctness of the output result is verified, the output result is returned to the execution environment as the execution result of the special operator through the LLM, so that the execution environment continues to execute the statement according to the execution result of the special operator.

9. A device for verifying output results of a large language model (LLM), comprising: The acquisition module is used to obtain the output results obtained by LLM based on the input data; a task type determination module, configured to determine the type of reasoning task performed by the LLM to obtain the output result; a first verification module configured to, when determining that the type is a logic judgment type, obtain the reasoning process by which the LLM obtains the output result based on the input data, and verify the correctness of the output result based on the reasoning process and the output result; The second verification module is used to generate a proposition to be verified according to the output result and the input data when it is determined that the type is an information acquisition type, and verify the correctness of the output result according to the proposition to be verified.

10. A computer-readable storage medium storing a computer program, wherein the computer program implements the method according to any one of claims 1 to 8 when executed by a processor.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 8 when executing the program.

Images