A method, device, storage medium and equipment for verifying LLM output results

Abstract

This specification discloses an embodiment of a method for verifying the output results of an LLM (Limited Linear Modulation). After the LLM obtains its output result, the method identifies the type of reasoning task from which the LLM obtained the output result. If it is a logical judgment type reasoning task, the correctness of the output result is verified based on the reasoning process and the output result obtained by the LLM. If it is an information acquisition type reasoning task, a proposition to be verified is generated based on the input data and the output result, and the correctness of the output result is verified based on the proposition to be verified. This method can verify whether the output result obtained by the LLM is correct by using appropriate methods for different types of reasoning tasks performed by the LLM, and can also determine whether the LLM experienced "illusion" during the reasoning process to obtain the output result.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a method, apparatus, storage medium and device for verifying LLM output results. Background Technology

[0002] With the development of artificial intelligence (AI) technology, large language models (LLM) have been widely applied in various fields.

[0003] However, LLM may encounter problems such as knowledge gaps, knowledge errors, or outdated knowledge due to limitations in training data. These problems may cause LLM to experience "illusions" when performing tasks, thereby reducing the accuracy and reliability of LLM in task execution.

[0004] Therefore, how to perform tasks accurately and reliably based on LLM has become an urgent problem to be solved. Summary of the Invention

[0005] This specification provides an embodiment of an LLM output verification method, apparatus, storage medium, and electronic device to partially solve the problems existing in the prior art.

[0006] The embodiments in this specification adopt the following technical solutions:

[0007] This specification provides a method for verifying LLM output results, the method comprising:

[0008] Obtain the output results of LLM based on the input data;

[0009] Determine the type of inference task performed by the LLM to obtain the output result;

[0010] When the type is determined to be a logical judgment type, the reasoning process of the LLM obtaining the output result based on the input data is obtained, and the correctness of the output result is verified based on the reasoning process and the output result.

[0011] When the type is determined to be an information acquisition type, a verification proposition is generated based on the output result and the input data, and the correctness of the output result is verified based on the verification proposition. This specification provides an LLM output result verification device, the device comprising:

[0012] The acquisition module is used to acquire the output results obtained by LLM based on the input data;

[0013] The task type determination module is used to determine the type of inference task performed by the LLM to obtain the output result;

[0014] The first verification module is used to, when the type is determined to be a logical judgment type, obtain the reasoning process of the LLM obtaining the output result based on the input data, and verify the correctness of the output result based on the reasoning process and the output result;

[0015] The second verification module is used to generate a verification proposition based on the output result and the input data when the type is determined to be an information acquisition type, and to verify the correctness of the output result based on the verification proposition.

[0016] This specification provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the verification method for the LLM output results described above.

[0017] This specification provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the verification method for the LLM output results described above.

[0018] The above-described at least one technical solution adopted in the embodiments of this specification can achieve the following beneficial effects:

[0019] This specification discloses an embodiment of a method for verifying the output results of an LLM (Limited Linear Modulation). After the LLM obtains its output result, the method identifies the type of reasoning task from which the LLM obtained the output result. If it is a logical judgment type reasoning task, the correctness of the output result is verified based on the reasoning process and the output result obtained by the LLM. If it is an information acquisition type reasoning task, a proposition to be verified is generated based on the input data and the output result, and the correctness of the output result is verified based on the proposition to be verified. This method can verify whether the output result obtained by the LLM is correct by using appropriate methods for different types of reasoning tasks performed by the LLM, and can also determine whether the LLM experienced "illusion" during the reasoning process to obtain the output result. Attached Figure Description

[0020] The accompanying drawings, which are included to provide a further understanding of this specification and form part of this specification, illustrate exemplary embodiments and are used to explain this specification, but do not constitute an undue limitation thereof. In the drawings:

[0021] Figure 1 A flowchart illustrating a method for verifying LLM output results provided in an embodiment of this specification;

[0022] Figure 2 A flowchart illustrating an HOP execution method provided in an embodiment of this specification;

[0023] Figure 3 A flowchart illustrating the HOP generation method provided in the embodiments of this specification;

[0024] Figure 4 A schematic diagram of an LLM output verification device provided in the embodiments of this specification;

[0025] Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this specification. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0027] The technical solutions provided in the various embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0028] Figure 1 A flowchart illustrating an LLM output verification method provided in this specification includes the following steps:

[0029] S100: Obtain the output results of LLM based on the input data.

[0030] In the embodiments of this specification, by means of... Figure 1 The device used to verify the LLM output results as shown can be any electronic device, such as a personal computer, mobile phone, tablet computer, server, or server cluster consisting of several servers. The following explanation uses a server as an example only.

[0031] In the embodiments of this specification, the LLM is used to receive input data and perform inference based on the input data to obtain an output result. After obtaining the output result, the LLM not only outputs the output result but also outputs the inference process by which it derived the output result. The inference process described in the embodiments of this specification is a natural language inference process.

[0032] The server can then obtain the above output results obtained by the LLM, as well as the reasoning process on which the LLM obtained the above output results.

[0033] S102: Determine the type of inference task performed by the LLM to obtain the output result.

[0034] In the embodiments of this specification, the server can identify the type of reasoning task performed by the LLM to obtain the output result based on the output result obtained by the LLM, the input data on which the LLM obtains the output result, and / or the reasoning process described above.

[0035] Since the inference tasks performed by LLMs are generally divided into two types: logical judgment type and information acquisition type, the server in this embodiment can identify the type of inference task performed by the LLM to obtain the output result through the LLM itself.

[0036] Specifically, for a logical judgment-type reasoning process, the LLM can determine whether its reasoning task involves at least one of subjective evaluation, logical analysis, weighing pros and cons, and inference based on existing knowledge. If so, the type of the reasoning task performed by the LLM is determined to be a logical judgment type; otherwise, it is not. Alternatively, the LLM can also determine whether the output requires the LLM to integrate knowledge, application rules, or expression tendencies from a pre-defined knowledge base, rather than directly retrieving facts, and whether the output is Boolean data (i.e., binary data of "either 0 or 1") or one of a finite number of possible results (i.e., multi-class data). If so, the type of reasoning task performed by the LLM is also determined to be a logical judgment type; otherwise, it is not.

[0037] For information acquisition-type reasoning processes, the LLM can determine whether its reasoning task directly extracts objective facts, definitions, or specific information from a pre-defined knowledge base and / or input data and / or training data. If so, the type of reasoning task performed by the LLM is determined to be information acquisition-type; otherwise, it is not. Alternatively, the LLM can also determine whether the output result is a clearly existing fragment in the pre-defined knowledge base and / or input data and / or training data, requiring no subjective processing by the LLM. If so, the type of reasoning task performed by the LLM is determined to be information acquisition-type; otherwise, it is not.

[0038] When the LLM identifies that the reasoning task it is performing is a logical judgment type reasoning task, it can verify the output result through step S104 below. When the LLM identifies that the reasoning task it is performing is an information retrieval type reasoning task, it can verify the output result through step S106 below. That is, there is no logical dependency between steps S104 and S106 below, and they can be executed separately. The embodiments in this specification do not limit the execution order of steps S104 and S106.

[0039] S104: When the type is determined to be a logical judgment type, obtain the reasoning process of the LLM to obtain the output result based on the input data, and verify the correctness of the output result based on the reasoning process and the output result.

[0040] In the embodiments of this specification, when the reasoning task performed by the LLM upon obtaining the above output result is a logical judgment type reasoning task, the server can still verify the correctness of the above output result obtained by the LLM through the LLM. That is, the server can perform a self-check on the above output result through the LLM. Specifically, the server can input a preset first prompt message to the LLM, so that the LLM, guided by the first prompt message, relies on its own reasoning ability to determine whether the output result is correct under the above input data conditions, that is, to perform an initial verification of the correctness of the output result. At the same time as the LLM outputs the initial verification result, it can also output the reason for obtaining the initial verification result. If the LLM fails the initial verification of the output result, it can be directly determined that the correctness verification of the output result has failed.

[0041] In the embodiments of this specification, when the initial verification of the output result by the LLM passes, in order to further improve the accuracy of the verification of the output result, the reasoning process based on the output result obtained by the LLM can continue to perform a further final verification of the correctness of the output result.

[0042] Specifically, if the LLM passes the initial verification of the output result, a preset second prompt message can be input into the LLM. Guided by this second prompt message, the LLM will determine whether the reason for passing the initial verification matches the reasoning process upon which the LLM obtained the output result. If they match, the final verification of the output result is considered passed; otherwise, the final verification is considered failed.

[0043] For example, suppose the input data is an email. The reasoning task required by the LLM is to determine whether this email is a phishing email. The rule in the LLM's pre-set knowledge base for identifying phishing emails is "the email content contains the keyword 'salary' or similar keywords." The LLM's output is "this email is a phishing email," and the reasoning process is "because the email content contains the keyword 'salary,' therefore this email is a phishing email." From an observer's perspective, both the LLM's reasoning process and output are essentially correct.

[0044] When verifying the correctness of the above output results, the LLM is first used to perform an initial verification of the correctness of the output results. The initial verification result is passed, thus obtaining the reason why the initial verification result of the LLM output is passed.

[0045] If the LLM outputs the reason "because the email contains the keyword 'salary'", and this reason matches the above reasoning process of the LLM, it indicates that the reasoning process of the LLM is stable and there is no illusion. The above output result is credible and passes the correctness verification.

[0046] If the LLM outputs the reason "because the email content format is messed up," this reason does not match the above reasoning process of the LLM, indicating that the reasoning process of the LLM is unstable and may be a hallucination. Although we, as observers, know that the above output is actually correct, in the verification process, we can still conclude that because the reasoning process of the LLM is unstable, the above output is not credible and the correctness verification fails.

[0047] Furthermore, when determining whether the reason for the initial verification success of the LLM matches the aforementioned reasoning process, the semantic similarity between the reason and the reasoning process can be determined. If the similarity is greater than a preset threshold, the reason and the reasoning process are considered to match; if the similarity is not greater than the preset threshold, the reason and the reasoning process are considered to not match. Specifically, when determining the semantic similarity between the reason and the reasoning process, the keywords contained in the reason and the keywords contained in the reasoning process can be determined first. Then, based on the keywords in the reason and the keywords contained in the reason, the semantic similarity between the reason and the reasoning process can be determined. Of course, other methods can also be used to determine the semantic similarity between the reason and the reasoning process; for example, the semantic similarity can still be determined using the aforementioned LLM.

[0048] S106: When the type is determined to be an information acquisition type, a proposition to be verified is generated based on the output result and the input data, and the correctness of the output result is verified based on the proposition to be verified.

[0049] In the embodiments of this specification, when the inference task performed by the LLM after obtaining the above output result is an information acquisition type inference task, the server can also use the LLM to verify the above output result in step S106.

[0050] Specifically, the server can first re-input the above output results and input data into the LLM, and input the preset third prompt information into the LLM. Under the guidance of the third prompt information, the LLM generates a positive proposition based on the above output results and input data. The logic of the positive proposition conforms to the logical relationship of obtaining the output result from the input data. Then, the negative proposition of the positive proposition is determined as the proposition to be verified.

[0051] As one embodiment of generating propositions to be verified, the server can generate a question based on input data and an answer based on the output result. The question-answer pair consisting of the question and the answer is a positive proposition that conforms to the logical relationship of obtaining the output result from the input data. For example, if the input data is a command line "xxx...", and the reasoning task required by LLM is to extract the Uniform Resource Locator (URL) from the command line, and the output data of LLM is "abc", then the generated positive propositions can be "The URL extracted from the command line "xxx..." is "abc"", "The command line "xxx..." contains the URL "abc", or any proposition that conforms to the logical relationship that the above input data can obtain the output result.

[0052] After obtaining the positive proposition, the server can input a preset fourth prompt into the LLM, causing the LLM to generate the negative proposition under the guidance of this fourth prompt, which will then be used as the proposition to be verified. The negative proposition is the complement of the positive proposition, and its logic is completely contradictory and mutually exclusive. Continuing with the previous example, after obtaining the positive proposition "Command line "xxx..." contains URL "abc", the LLM can determine that the negative proposition is "Command line "xxx..." does not contain URL "abc", and will use this negative proposition as the proposition to be verified.

[0053] Finally, the server can input the proposition to be verified into the LLM again, and also input the preset fifth prompt information into the LLM. The fifth prompt information guides the LLM to judge the correctness of the proposition to be verified. If the LLM determines that the proposition to be verified is incorrect, the correctness verification of the above output result is determined to be passed. If the LLM determines that the proposition to be verified is correct, the correctness verification of the above output result is determined to be failed.

[0054] Continuing with the previous example, the logic of the positive proposition directly reflects the logical relationship that the input data can yield the output result. The negative proposition, however, completely contradicts the logic of the positive proposition, reflecting the logical relationship that the input data cannot yield the output result. Therefore, since the LLM has already obtained the output result based on the input data, the reasoning process upon which the LLM performs this reasoning task should conform to the logic of the positive proposition. If the LLM now believes that the proposition to be verified (i.e., the negative proposition) is correct, meaning that the LLM believes that the input data cannot yield the output result, this completely contradicts the previous reasoning process of the LLM. Therefore, it can be concluded that the LLM's logic is confused, and it is very likely experiencing a hallucination. The output result obtained by the LLM is unreliable, and thus the verification of the correctness of the output result fails. Conversely, if the LLM now considers the proposition to be verified (i.e., the aforementioned negation proposition) to be incorrect, that is, if the LLM still believes that the output result can be obtained from the input data, meaning that the current logic of the LLM is completely consistent with the logic when the output result was obtained previously, then it can be determined that the logic of the LLM is stable, and the aforementioned output result obtained is reliable. Therefore, the correctness verification of the output result is confirmed to have passed.

[0055] The above are as follows Figure 1 The verification method shown can be implemented by an agent deployed on a server.

[0056] Furthermore, in the embodiments of this specification, the input data of the aforementioned input LLM can specifically be the computational logic of a special operator corresponding to the operator name contained in the Higher-Order Knowledge Program (HOP), as well as the input parameters of that special operator. The execution method of the HOP is as follows: Figure 2 As shown.

[0057] Figure 2 A flowchart of an HOP execution method provided in this specification embodiment specifically includes the following steps:

[0058] S200: A runtime environment for running computer program code of a preset type and loading business data.

[0059] The HOPs mentioned in this specification are computer program code of a preset type, such as Python. Each HOP contains an operator name, which also conforms to the function name format requirements of the preset type of computational program code. However, the computational logic of the operator corresponding to that operator name is defined through natural language; this operator will be referred to as a special operator below.

[0060] The server can first obtain the HOP and run the runtime environment of the computer program code of that preset type, and then load the business data required by the HOP. For example, when the HOP is Python code, the server can launch the Python runtime environment and load the business data.

[0061] The HOP is generated by LLM based on the business logic process corresponding to a certain business, which is described in natural language. The generation process of the HOP will be explained later.

[0062] S202: Based on the business data, run the HOP through the operating environment.

[0063] In the embodiments of this specification, the framework code of the HOP is written by the aforementioned preset type of computer program code. Even if the code contains special operators, the operator names of the special operators conform to the function name format requirements of the preset type of computer program code. Therefore, in terms of form, the HOP is essentially the preset type of computer program code. The server can run the HOP based on the loaded business data, using the business data as the input parameter of the entire HOP, and run the HOP through the runtime environment of the preset type of computer program code.

[0064] S204: When the execution reaches a statement in the HOP that contains the operator name, the pre-trained LLM is invoked.

[0065] When the server runs a statement containing the operator name of the special operator in the HOP through this runtime environment, it can invoke the pre-trained LLM mentioned above.

[0066] S206: Input the business data and / or the intermediate results obtained by running the HOP as input parameters to the LLM for the special operator corresponding to the operator name, so that the LLM executes the calculation logic of the special operator corresponding to the operator name according to the input parameters and the preset knowledge base, so as to determine the execution result of the special operator.

[0067] Because the computational logic of this special operator is defined by natural language, the runtime environment of the preset type of computer program code cannot execute this special operator normally. Therefore, the server can use the business data and the intermediate results obtained from running the current HOP as input parameters to the LLM for this special operator. The LLM then executes the computational logic of the special operator based on the input parameters and the knowledge in the preset knowledge base, obtaining the execution result of the special operator. This execution result is... Figure 1 The output of LLM in step S100 is shown.

[0068] S208: Perform a correctness check on the execution result.

[0069] Since the computational logic of special operators is defined in natural language and needs to be executed by LLM, in order to avoid the LLM executing the special operators defined in natural language inaccurately, the execution result can be checked for correctness to determine whether the LLM has correctly executed the computational logic of the special operator defined in natural language.

[0070] The method for verifying the correctness of the execution result obtained by executing special operators on the LLM in step S208 is included in this specification. Figure 1 The verification methods are shown in the figure.

[0071] S210: When the correctness check passes, continue executing the statement based on the execution result of the operator.

[0072] If the correctness check passes, the LLM returns the execution result of the special operator to the execution environment of the computer program code of the preset type, which then continues to execute the statement until the entire HOP has been executed.

[0073] Specifically, when using the above-mentioned methods... Figure 1 When the verification method shown verifies the correctness of the output of the special operator executed by the LLM, the LLM can return the output as the execution result of the special operator to the runtime environment, so that the runtime environment can continue to execute the statement based on the execution result of the special operator.

[0074] Among them, the above Figure 2 The process of generating the HOP shown can be as follows: Figure 3 As shown. Figure 3 The flowchart of the HOP generation method provided in the embodiments of this specification specifically includes the following steps:

[0075] S300: Obtain the business logic process described in natural language.

[0076] In the embodiments described in this specification, the following are employed: Figure 3 The device used to generate the code, as shown, can be any electronic device, such as a computer, server, or server cluster consisting of multiple servers. This device can be used with... Figure 2 The method shown can be performed on the same device or different devices. For ease of description, the following explanation will use the same server as an example.

[0077] To generate code for executing a specific business function, the server first obtains the business logic process for that function. Specifically, the business logic process described in this specification includes Standard Operating Procedures (SOPs). An SOP is a standard operating procedure for that business function, described in natural language. It is a document that details the work, workflow, and steps involved in that business function. It provides the organization's internal standard process specifications for that business function, for reference and adherence by internal personnel. In addition to the standardized process for that business function, the SOP also includes clearly defined responsibilities and authorities for each role within the business function, emergency measures, quality standards and assessments, document management, and other related content.

[0078] The following embodiments of this manual use the above-mentioned business as a risk control business as an example to illustrate the code generation method provided in this manual.

[0079] When the above-mentioned business involves risk control based on recorded risk logs, the business logic process of this risk control business (i.e., the risk control logic process) can be described in the following natural language:

[0080] The first step is to analyze the cmd field to determine whether the cmd field has been granted executable permissions. If the cmd command line has not been granted executable permissions, the conclusion is that no action is required.

[0081] The second step is to determine whether the installation package is for commonly used software. If it is, the conclusion is that no action is required.

[0082] The third step is to determine if the cmd field has been manually reviewed by security operations personnel within the past N days and has been determined to require no action. For example, if the chmod command line is determined to require no action, then the risk log does not require action.

[0083] The fourth step is to determine, based on the results of the above three steps, whether the risk log requires no action or generates an attack alert.

[0084] The aforementioned risk log is a risk log of other businesses recorded by the user when performing other businesses, and the aforementioned risk control business is the business that needs to perform risk control on the other businesses based on the risk log.

[0085] S302: Input the business logic process into the pre-trained LLM.

[0086] After obtaining the aforementioned business logic process, the server can input this naturally described business logic process into a pre-trained LLM. This LLM can be pre-deployed on the server, or it can be deployed on other devices. If deployed on other devices, the server can send the business logic process to those devices, allowing them to input the received business logic process into the LLM. The following explanation uses the deployment of the LLM on the server as an example.

[0087] To avoid the LLM generating code that is inconsistent with the actual business logic process due to "illusion," and to improve the accuracy and reliability of the LLM generated code, in this embodiment of the specification, the server directly inputs the actual business logic process obtained in step S300 into the LLM, so that the LLM generates code for executing the business based on the business logic process.

[0088] S304: Identify the steps included in the business logic process and the logical relationships between the steps through the LLM.

[0089] Generally, a business logic process consists of several steps arranged according to certain logical relationships. In the example above, the first step is executed. If the result of the first step is negative, no action is needed. If the result is positive, the second step is executed. If the result of the second step is positive, no action is needed. Otherwise, the third step is executed. If the result of the third step is positive, no action is needed. Otherwise, an attack alert is generated. Therefore, the logical relationships between different steps in a business logic process constitute its basic framework. Specifically, the execution of each step requires the knowledge corresponding to the business logic process, including the concepts, attributes, and relationships between the entities required in the process. For example, in the first step of the example above, it is necessary to determine whether the `cmd` field has been granted executable permissions. The knowledge required to execute this first step includes at least: the concept of the `cmd` field, the concept of executable permissions, and the relationship between the `cmd` field and executable permissions (i.e., which `cmd` fields indicate that executable permissions have been granted, and which indicate that executable permissions have not been granted).

[0090] Therefore, it can be seen that a business logic process is essentially composed of the logical relationships between each step and the knowledge required to execute each step. Thus, in order to generate code to execute the business logic process, LLM needs to identify the steps contained in the business logic process and the logical relationships between each step based on the business logic process described in natural language.

[0091] Specifically, since identifying the steps involved in the business logic process and the logical relationships between them may require a knowledge base corresponding to that business logic process, in this embodiment, the knowledge base corresponding to the business logic process can be pre-injected into the pre-trained LLM. This knowledge base may include at least one of the following:

[0092] The knowledge graph corresponding to this business logic process includes the conceptual descriptions of each entity required in the business logic process (used to describe what the entity is, such as the concept of the cmd command line in the example above), the attributes of each entity (used to describe all possible attributes of the entity in this business logic), and the relationships between the entities.

[0093] The business logic process requires the calling interfaces of tools, including APIs or SDKs. For example, in the second step of the above example, it is necessary to determine whether the installation package is a common software installation package. This requires calling a tool to obtain a list of common software installation packages, which requires calling the tool through its calling interface.

[0094] After injecting the knowledge base containing the knowledge graph corresponding to the business logic process and / or the calling interface of the required tools into the pre-trained LLM, when the server inputs the business logic process described in natural language into the LLM, the LLM can, based on its own reasoning ability, identify the semantics of the business logic process described in natural language, map the concepts in the business logic process to entities in the knowledge graph, and then, based on the semantics, break down the entire business logic process into several steps, and finally reason out the logical relationships between each step.

[0095] S306: Based on the logical relationship, generate computer program code of a preset type to represent the logical relationship between the steps, as framework code.

[0096] Because natural language inherently possesses a certain degree of ambiguity and vagueness—a flaw inherent in any natural language—directly using natural language to express the logical relationships between steps in a business logic process is highly likely to lead to logical confusion in those steps. In particular, when others other than the person who wrote the business logic process read it, their understanding may be inaccurate due to the ambiguity and vagueness of natural language, thus affecting the business itself.

[0097] As a formal and precise language, computer programming languages ​​have clear semantics and strict logical structures, which can effectively avoid the above-mentioned problems of natural languages. Therefore, in the embodiments of this specification, for the logical relationships between the steps in the business logic process, the server can generate computer program code of a preset type to represent the logical relationships between the steps based on the logical relationships between the steps identified in step S304 through the LLM, as the framework code.

[0098] The computer program code of this preset type can be any type of computer program code, including but not limited to Python code. The following explanation uses Python code as an example only.

[0099] S308: For each step, generate at least one operator for executing the step based on the natural language used to describe the step in the business logic process and the preset knowledge base corresponding to the business logic process.

[0100] In the embodiments described in this specification, the execution order of steps S306 and S308 is not important. Specifically, the server can execute step S306 and step S308 simultaneously through the LLM.

[0101] Since natural language is more natural and has better comprehensibility than computer programming languages, the knowledge required for each step in the business logic process is represented by an operator defined in natural language in the embodiments of this specification.

[0102] In this specification, the computational logic of the operators described in the embodiments is defined using natural language, but the operator names still conform to the function name format requirements of the aforementioned preset type of computer program code. Therefore, the operators defined using natural language in this specification are essentially user-defined functions in the preset type of computer program code, and these functions are high-level abstract expressions of the logic that implements the step.

[0103] Specifically, for each step identified from the business logic process, LLM can determine the knowledge required to execute that step based on the semantics of the natural language used to describe that step within the business logic process, and the knowledge base corresponding to that business logic process. This knowledge includes one or more entities in the knowledge graph, one or more attributes of that entity, the relationships between that entity and other entities, and / or the calling interface of the tools required for that step. Once the knowledge required to execute the step is determined, at least one operator for executing that step can be generated. This operator is defined in natural language, and its name conforms to the function name format requirements of the aforementioned preset type of computer program code.

[0104] Continuing with the previous example, the first step was to determine whether the cmd field had executable permissions. Therefore, LLM can generate an operator name hop_judge that meets the requirements of Python code function names, and define the operator hop_judge as follows:

[0105] def hop_judge

[0106] Determine if {cmd} has been granted executable permissions.

[0107] Therefore, the operator hop_judge is defined entirely using natural language, specifically the statement "determine whether {cmd} has executable permissions," making it more understandable. The operators described above, which use natural language to define their computational logic, are special operators in HOP.

[0108] The above is just an example of using one operator to express one step. Those skilled in the art should understand that in practical applications, for more complex steps, more operators (such as two or more operators) can be used to express the step.

[0109] S310: Based on the framework code and the operator name generated for each step, generate a higher-order knowledge program for executing the business logic process.

[0110] As explained above, a business logic process is essentially composed of the logical relationships between each step and the knowledge required to execute each step. The logical relationships have been expressed by the generated framework code in step S306, and the knowledge required to execute each step has been expressed by operators defined in natural language and whose operator names still conform to the preset type of computer program code function name format requirements in step S308. Therefore, LLM can combine the framework code and the operators corresponding to each step to generate a knowledge high-order program (HOP) for executing the business logic process.

[0111] Continuing with the previous example, LLM combines the operators corresponding to each step into the framework code, generating the following HOP for executing the above risk control logic process:

[0112] def evaluate(cmd):

[0113] if hop_judge_a("Check if (cmd) has been granted executable permissions") #Higher-order condition

[0114] pkg = hop_get("Get the installation package used by (cmd)")

[0115] If not pkg in hop_knowledge_retrieve("common installation packages") #Advanced knowledge concept matching

[0116] If not hop_judge_b("Check history, (cmd) whether it was manually judged as requiring no action in history")

[0117] Return "Attack Alert"

[0118] Return "No action required"

[0119] In the HOP above, the content within parentheses ("") defines the corresponding operators using natural language. These operators include hop_judge_a, hop_get, hop_knowledge_retrieve, and hop_judge_b. The entire risk control logic process, as expressed in the Python framework above, is as follows: First, the operator hop_judge_a is executed to determine if (cmd) has executable permissions. If yes, the operator hop_get is executed to retrieve the installation package used by (cmd) and assign it to pkg. Otherwise, "No action required" is output. After retrieving the installation package used by (cmd) and assigning it to pkg, the operator hop_knowledge_retrieve is executed to retrieve commonly used installation packages and determine if pkg is not among them. If yes, the operator hop_judge_b is executed. Otherwise, "No action required" is output. Finally, the operator hop_judge_b is executed to determine if (cmd) has not been manually judged as requiring no action in the history. If so, "Attack Alert" is output; otherwise, "No action required" is output.

[0120] As can be seen, the HOP represented by Python code above is completely consistent with the original risk control logic process described in natural language. Therefore, even if the personnel in the department performing risk control operations change, the HOP can still be read by other personnel (other than those who wrote the original risk control logic process) without causing logical confusion in each step due to the ambiguity and vagueness inherent in natural language. Furthermore, although the special operators in the HOP code are defined in natural language, the operator names are still function names in Python code format. Therefore, the HOP code itself is composed of Python framework code referencing various Python-formatted operator names. Formally, it is still Python code. Thus, this HOP code is not only human-readable but also usable by... Figure 2 The method shown is executed directly in the Python execution environment to implement the risk control logic process described above.

[0121] In the embodiments of this specification, in order for the LLM to accurately convert the business logic process described in the input natural language into the above-mentioned HOP, it is not enough to simply inject the knowledge base corresponding to the business logic process into the business LLM. It is also necessary to fine-tune and train the business LLM in advance under the business scenario corresponding to the business logic process.

[0122] Specifically, when fine-tuning the LLM, a sample logical process described in natural language can be obtained first. This sample logical process also contains several steps, referred to below as the sample step. This sample logical process and the aforementioned business logical process are business logical processes within the same business scenario; that is, the knowledge base corresponding to the sample logical process is exactly the same as the knowledge base corresponding to the aforementioned business logical process.

[0123] After obtaining the sample logic process, it can also be done through... Figure 3 Steps S302-S310, as shown, involve inputting the sample logic process into the LLM to be trained. The LLM identifies the sample steps within the sample logic process and the logical relationships between them. Based on these relationships, a preset type of computer program code representing the logical relationships between the sample steps is generated as the framework code to be optimized. For each sample step, based on the natural language used to describe that step in the sample logic process and the preset knowledge base corresponding to the sample logic process, at least one operator is generated to execute that step as an operator to be optimized. The computational logic of this operator is also defined using natural language, and the operator name conforms to the function name format of the preset type of computer program code. Finally, based on the framework code to be optimized and the operator names of the operators generated for each sample step, an optimized HOP is generated to execute the sample logic process. This process is similar to... Figure 3 The steps S302 to S310 shown are exactly the same, so they will not be repeated here.

[0124] After the LLM to be trained generates the HOP to be optimized, the optimized HOP after adjustments can be determined. Since the HOP to be optimized is also in the form of the aforementioned preset type of computer program code (such as Python code), it can be manually adjusted to ensure that the logical relationships between the sample steps expressed in the optimized framework code fully conform to the original sample logic process, and that the natural language corresponding to the operators to be optimized, defined by natural language, is more closely aligned with the knowledge in the knowledge base corresponding to the sample logic process, and that the computational logic of the operators to be optimized is clearer and more accurate. Of course, the HOP to be optimized can also be adjusted using other trained LLMs; this embodiment does not limit this approach.

[0125] After obtaining the optimized HOP, supervised fine-tuning (SFT) training can be performed on the LLM to be trained based on the optimized HOP. That is, the optimized HOP is used as the annotation corresponding to the sample logic process, and this serves as a supervision signal to adjust the model parameters in the LLM to be trained, so that the LLM can adapt to generating various business logic processes in the business scenario. The device used to train the LLM to be trained is the same as the aforementioned execution... Figure 3 The device used to generate the code shown can be the same device or different devices. Furthermore, because this embodiment aims to preserve the original inference capabilities of the LLM as much as possible, adapting it only to this business scenario, when performing SFT on the LLM, not all model parameters are adjusted; only a portion of the model parameters are adjusted. The adjusted model parameters may include only the LoRA model parameters within the LLM.

[0126] After obtaining the LLM through the SFT training described above, the server can then... Figure 3 The method shown generates the HOP corresponding to the business logic process in this business scenario. However, when using... Figure 2 When the method shown is used to execute the HOP, since the HOP not only contains computer program code of the above-mentioned preset type (such as Python), but also contains special operators defined by natural language, the LLM obtained by training above is also required to assist in the execution of the HOP.

[0127] When the above business scenario is a risk control business scenario, that is, when the above business logic process is a risk control logic process under the risk control business scenario, the loaded business data can be a pre-recorded risk log. This risk log is the risk log of other businesses recorded by the user when performing other businesses. The risk control business is the business that needs to perform risk control on the other businesses based on this risk log.

[0128] pass Figure 2 After step S200, as shown, launches the runtime environment of the preset type of computer program code and loads the business data, the program can be run based on the business data through the runtime environment. Figure 3 The generated HOP.

[0129] When the execution reaches a special operator defined in natural language within the HOP, the aforementioned LLM needs to be invoked. Since the framework of this HOP is still predefined type computer program code, the runtime environment using this predefined type of computer program code can directly run the HOP. As for the special operators defined in natural language within the HOP, these special operators are essentially user-defined functions within the predefined type of computer program code. The HOP references these function names, meaning it executes the function corresponding to that name, obtains the execution result, and then continues executing subsequent code based on that result. Therefore, when the execution reaches the operator name of a special operator defined in natural language within the HOP, the predefined computational logic of that operator needs to be invoked. When the runtime environment parses the computational logic of this operator, if it determines that the computational logic is defined in natural language, the runtime environment can directly invoke the aforementioned LLM.

[0130] After calling the LLM mentioned above, it can be accessed through Figure 2 In step S206, the business data and / or the intermediate results obtained from the current running of the HOP are used as input parameters to the LLM, so that the LLM executes the calculation logic of the operator according to the input parameters and the knowledge base corresponding to the above business logic process to determine the execution result of the operator.

[0131] Continuing with the previous example, for the operator hop_judge_a in the HOP above, its natural language-defined computational logic is "to determine whether (cmd) has been granted executable permissions." Business data, i.e., pre-recorded risk logs, can be used as the input parameter of this operator. The computational logic defined in the natural language of this operator, along with the input parameter, should be input into the LLM. A prompt message can also be input to the LLM, guiding it to first identify the cmd command line (i.e., an entity in the knowledge graph) in the input parameter based on the knowledge base corresponding to the risk control logic process. Then, based on the knowledge base and the identified cmd command line, it determines the attributes of the cmd command line in the input parameter (i.e., the attributes of the entity in the knowledge graph). Based on these attributes, it determines whether the cmd command line has been granted executable permissions (i.e., the relationship between different entities in the knowledge graph). The judgment result is used as the output parameter of the operator hop_judge_a, returning to the Python runtime environment, allowing the Python runtime environment to continue executing subsequent code in the HOP.

[0132] For example, for the operator hop_knowledge_retrieve in the above HOP, its natural language definition of computational logic is "common installation packages". The computational logic of this operator can be input into the LLM. Prompt information can also be input into the LLM, so that the business LLM, guided by the prompt information, first determines the tool for obtaining the list of common installation packages based on the knowledge base corresponding to the risk control logic process, then calls the tool through the calling interface to obtain the common installation packages, and finally uses the obtained list of common installation packages as the output parameter of the operator hop_knowledge_retrieve and returns it to the Python runtime environment, so that the Python runtime environment continues to execute the subsequent code in the HOP to determine whether the intermediate result pkg of the current HOP is not included in the list of common installation packages.

[0133] Those skilled in the art should understand that the above description uses the business logic process as an example of the risk control logic process. In fact, the HOP generation method provided in the embodiments of this specification can generate HOPs corresponding to business logic processes in any business scenario and run them accordingly. Furthermore, the above-mentioned preset type of computer program code is only illustrated using Python code as an example; the preset type of computer program code described in the embodiments of this specification can also be other types of computer program code.

[0134] exist Figure 2 In step S208, since the computational logic of the special operators is defined through natural language, and LLM may exhibit "illusions," this embodiment of the specification also requires the use of [a certain method] to ensure the accuracy of LLM execution of the special operators. Figure 1 The verification method shown verifies the correctness of the execution result obtained by the LLM executing the special operator. If the correctness verification passes, the LLM can return the execution result to the runtime environment of the computer program code of the above-mentioned preset type, so that the runtime environment can continue to execute the HOP until the entire HOP is executed. If the correctness verification fails, the LLM can re-execute the calculation logic of the special operator to redetermine the execution result of the special operator until the obtained execution result passes the correctness verification, or the number of times the LLM executes the calculation logic of the special operator reaches the preset number.

[0135] If the LLM executes the computational logic of the special operator a preset number of times, and the execution result of the last determined special operator still fails the correctness check, an assistance execution request can be generated and output by the LLM. This assistance execution request is used to request human assistance in obtaining the execution result of the special operator. This assistance execution request can be sent to a user device, whereby the user can input the execution result of the special operator according to the computational logic and input parameters carried in the assistance execution request. The user device then returns the user-input execution result to the LLM. The LLM receives the execution result returned by the user device and considers the received execution result as the execution result that has passed the correctness check. Furthermore, based on the computational logic of the special operator and the received user-input execution result, supervised fine-tuning training can be performed on at least some of the model parameters in the LLM, such as supervised fine-tuning training only on the LoRA model parameters in the LLM.

[0136] The above is an example of an LLM output verification method provided in this specification. Based on the same idea, this specification also provides corresponding devices, storage media and electronic devices.

[0137] Figure 4 This is a schematic diagram of an LLM output verification device provided in an embodiment of this specification. The device includes:

[0138] The acquisition module 401 is used to acquire the output results obtained by the LLM based on the input data;

[0139] Task type determination module 402 is used to determine the type of inference task performed by the LLM to obtain the output result;

[0140] The first verification module 403 is used to, when the type is determined to be a logical judgment type, obtain the reasoning process of the LLM obtaining the output result based on the input data, and verify the correctness of the output result based on the reasoning process and the output result;

[0141] The second verification module 404 is used to generate a proposition to be verified based on the output result and the input data when the type is determined to be an information acquisition type, and to verify the correctness of the output result based on the proposition to be verified.

[0142] Optionally, the first verification module 403 is specifically used to perform an initial verification of the correctness of the output result through the LLM; if the initial verification of the output result by the LLM fails, it is determined that the correctness verification of the output result fails; if the initial verification of the output result by the LLM passes, the correctness of the output result is finally verified according to the reasoning process.

[0143] Optionally, the first verification module 403 is specifically used to obtain the reason why the LLM determines that the initial verification of the output result is passed; determine whether the reason matches the reasoning process; if they match, determine that the final verification of the output result is passed; if they do not match, determine that the final verification of the output result is failed.

[0144] Optionally, the first verification module 403 is specifically used to determine the keywords contained in the cause and the keywords contained in the reasoning process respectively; determine the similarity between the cause and the reasoning process based on the keywords contained in the cause and the keywords contained in the reasoning process; if the similarity is greater than a preset threshold, determine that the cause and the reasoning process match; if the similarity is not greater than the preset threshold, determine that the cause and the reasoning process do not match.

[0145] Optionally, the second verification module 404 is specifically used to generate a positive proposition based on the output result and the input data, wherein the logic of the positive proposition conforms to the logical relationship of obtaining the output result from the input data; and determine the negative proposition of the positive proposition as the proposition to be verified.

[0146] Optionally, the second verification module 404 is specifically used to input the proposition to be verified into the LLM, so that the LLM can determine the correctness of the proposition to be verified; if the LLM determines that the proposition to be verified is incorrect, then the correctness verification of the output result is determined to be passed; if the LLM determines that the proposition to be verified is correct, then the correctness verification of the output result is determined to be failed.

[0147] Optionally, the input data includes: the computational logic of the special operators corresponding to the operator names contained in the knowledge high-order program HOP, and the input parameters of the special operators;

[0148] The computational logic of the special operator is defined through natural language, and the HOP is computer program code of a preset type;

[0149] The HOP is a computer program code execution environment of the preset type, running based on business data. When the execution reaches a statement in the HOP that contains the operator name, the calculation logic of the special operator is input to the LLM. The business data and / or the intermediate results obtained from the current execution of the HOP are used as input parameters of the special operator to the LLM. The LLM then executes the calculation logic of the special operator according to the input parameters and the preset knowledge base to determine the execution result of the operator, which is then used as the output result of the LLM.

[0150] Optionally, the input data is the data that the runtime environment inputs into the LLM each time it runs to a statement in the HOP that contains the name of a special operator;

[0151] The device further includes:

[0152] The execution module 405 is used to return the output result as the execution result of the special operator to the execution environment through the LLM when the verification of the correctness of the output result is passed, so that the execution environment can continue to execute the statement according to the execution result of the special operator.

[0153] This specification also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can be used to perform the verification method for the LLM output results provided above.

[0154] based on Figure 1 The verification method for the LLM output results shown in this specification is also provided in the embodiments. Figure 5 The diagram shows the structure of the electronic device. Figure 5 At the hardware level, the electronic device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to implement the aforementioned method for verifying the LLM output results.

[0155] The above description is merely an embodiment of this specification and is not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this specification.

Claims

1. A method for verifying the output of a large language model (LLM), the method comprising: Obtain the output results of LLM based on the input data; Determine the type of inference task performed by the LLM to obtain the output result; When the type is determined to be a logical judgment type, the reasoning process of the LLM obtaining the output result based on the input data is obtained, and the correctness of the output result is verified based on the reasoning process and the output result. When the type is determined to be an information acquisition type, a proposition to be verified is generated based on the output result and the input data, and the correctness of the output result is verified based on the proposition to be verified. The input data includes: the calculation logic of the special operators corresponding to the operator names contained in the knowledge high-order program HOP, and the input parameters of the special operators; The computational logic of the special operator is defined through natural language, and the HOP is computer program code of a preset type; The HOP is a computer program code execution environment of the preset type, running based on business data. When the execution reaches a statement in the HOP that contains the operator name, the calculation logic of the special operator is input to the LLM. The business data and / or the intermediate results obtained from the current execution of the HOP are used as input parameters of the special operator to the LLM. The LLM then executes the calculation logic of the special operator according to the input parameters and the preset knowledge base to determine the execution result of the operator, which is then used as the output result of the LLM.

2. The method as described in claim 1, wherein the correctness of the output result is verified based on the reasoning process and the output result, specifically including: The correctness of the output result is initially verified by the LLM. If the initial verification of the output result by the LLM fails, then the correctness verification of the output result is determined to have failed. If the LLM passes the initial verification of the output result, the correctness of the output result is then finally verified according to the reasoning process.

3. The method as described in claim 2, wherein the correctness of the output result is finally verified based on the reasoning process, specifically including: The LLM determines the reason why the initial verification of the output result passed; Determine whether the stated cause matches the stated reasoning process; If a match is found, the final verification of the output result is deemed successful. If there is no match, the final verification of the output result is deemed unsuccessful.

4. The method as described in claim 3, wherein determining whether the cause matches the reasoning process specifically includes: Identify the keywords contained in the reasons and the keywords contained in the reasoning process; Based on the keywords contained in the reason and the keywords contained in the reasoning process, determine the similarity between the reason and the reasoning process; If the similarity is greater than a preset threshold, the cause is determined to match the reasoning process. If the similarity is not greater than a preset threshold, it is determined that the cause does not match the reasoning process.

5. The method as described in claim 1, wherein generating a proposition to be verified based on the output result and the input data specifically includes: Based on the output and the input data, a positive proposition is generated, and the logic of the positive proposition conforms to the logical relationship of obtaining the output from the input data; The negation of the positive proposition is determined and designated as the proposition to be verified.

6. The method as described in claim 5, wherein verifying the correctness of the output result based on the proposition to be verified specifically includes: The proposition to be verified is input into the LLM so that the correctness of the proposition to be verified can be determined by the LLM. If the LLM determines that the proposition to be verified is incorrect, then the verification of the correctness of the output result is deemed to have passed. If the LLM determines that the proposition to be verified is correct, then it determines that the correctness verification of the output result fails.

7. The method as described in claim 1, wherein the input data is input to the LLM each time the runtime environment runs to a statement containing the operator name of a special operator in the HOP; The method further includes: When the correctness of the output result is verified, the LLM returns the output result as the execution result of the special operator to the runtime environment, so that the runtime environment can continue to execute the statement based on the execution result of the special operator.

8. A verification device for the output results of a large language model (LLM), the device comprising: The acquisition module is used to acquire the output results obtained by LLM based on the input data; The task type determination module is used to determine the type of inference task performed by the LLM to obtain the output result; The first verification module is used to, when the type is determined to be a logical judgment type, obtain the reasoning process of the LLM to obtain the output result based on the input data, and verify the correctness of the output result based on the reasoning process and the output result; The second verification module is used to generate a verification proposition based on the output result and the input data when the type is determined to be an information acquisition type, and to verify the correctness of the output result based on the verification proposition. The input data includes: the calculation logic of the special operators corresponding to the operator names contained in the knowledge high-order program HOP, and the input parameters of the special operators; The computational logic of the special operator is defined through natural language, and the HOP is computer program code of a preset type; The HOP is a computer program code execution environment of the preset type, running based on business data. When the execution reaches a statement in the HOP that contains the operator name, the calculation logic of the special operator is input to the LLM. The business data and / or the intermediate results obtained from the current execution of the HOP are used as input parameters of the special operator to the LLM. The LLM then executes the calculation logic of the special operator according to the input parameters and the preset knowledge base to determine the execution result of the operator, which is then used as the output result of the LLM.

9. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in any one of claims 1-7.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the method described in any one of claims 1-7.

Images