A method, apparatus, storage medium, and device for verifying LLM output results.

Abstract

This specification discloses an embodiment of a method for verifying the output result of an LLM (Limited Linear Modulation). After the business LLM obtains the output result, the method acquires the reasoning process by which the business LLM obtained the output result, and inputs the output result and the reasoning process into a pre-trained verification LLM. The verification LLM extracts the computational process from the reasoning process and verifies the correctness of each computational process executed by the business LLM. This method can utilize the verification LLM to verify whether each computational process executed by the business LLM during the reasoning process is correct, thereby determining whether the output result obtained by the business LLM is correct. It can also determine whether the business LLM experienced any "illusion" during the reasoning process to obtain the output result.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a method, apparatus, storage medium and device for verifying LLM output results. Background Technology

[0002] With the development of artificial intelligence (AI) technology, large language models (LLM) have been widely applied in various fields.

[0003] However, LLM may suffer from problems such as lack of knowledge, knowledge errors, or outdated knowledge due to limitations in training data. These problems may cause LLM to experience "illusions" when performing tasks, thereby reducing the accuracy and reliability of LLM in performing tasks.

[0004] Therefore, how to perform tasks accurately and reliably based on LLM has become an urgent problem to be solved. Summary of the Invention

[0005] This specification provides an embodiment of an LLM output verification method, apparatus, storage medium, and electronic device to partially solve the problems existing in the prior art.

[0006] The embodiments in this specification adopt the following technical solutions:

[0007] This specification provides a method for verifying LLM output results, the method comprising:

[0008] Obtain the output results obtained by the business LLM based on the input data, and the reasoning process by which the business LLM obtains the output results;

[0009] The output and the reasoning process are input into a pre-trained verification LLM;

[0010] The computational processes included in the inference process are extracted through the verification LLM;

[0011] By verifying the LLM, for each extracted operation process, the operation result obtained by the business LLM in executing the operation process is taken as the verification result, and the correctness of the business LLM in executing the operation process is verified based on the verification result.

[0012] This specification provides a verification device for LLM output results, the device comprising:

[0013] The acquisition module is used to acquire the output results obtained by the business LLM based on the input data, as well as the reasoning process by which the business LLM obtains the output results;

[0014] The input module is used to input the output results and the reasoning process into a pre-trained verification LLM;

[0015] An extraction module is used to extract the computational processes included in the inference process through the verification LLM;

[0016] The verification module is used to verify the correctness of the business LLM's execution of the operation process for each extracted operation process through the verification LLM, taking the operation result obtained by the business LLM from the operation process as the verification result, and verifying the correctness of the business LLM's execution of the operation process based on the verification result.

[0017] This specification provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the execution method of the aforementioned high-order knowledge program HOP.

[0018] This specification provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the verification method for the LLM output results described above.

[0019] The above-described at least one technical solution adopted in the embodiments of this specification can achieve the following beneficial effects:

[0020] This specification discloses an embodiment of a method for verifying the output result of an LLM (Limited Linear Modulation). After the business LLM obtains the output result, the method acquires the reasoning process by which the business LLM obtained the output result, and inputs the output result and the reasoning process into a pre-trained verification LLM. The verification LLM extracts the computational process from the reasoning process and verifies the correctness of each computational process executed by the business LLM. This method can utilize the verification LLM to verify whether each computational process executed by the business LLM during the reasoning process is correct, thereby determining whether the output result obtained by the business LLM is correct. It can also determine whether the business LLM experienced any "illusion" during the reasoning process to obtain the output result. Attached Figure Description

[0021] The accompanying drawings, which are included to provide a further understanding of this specification and form part of this specification, illustrate exemplary embodiments and are used to explain this specification, but do not constitute an undue limitation thereof. In the drawings:

[0022] Figure 1 A flowchart illustrating a method for verifying LLM output results provided in an embodiment of this specification;

[0023] Figure 2 A flowchart illustrating an HOP execution method provided in an embodiment of this specification;

[0024] Figure 3 A flowchart illustrating the HOP generation method provided in the embodiments of this specification;

[0025] Figure 4 A schematic diagram of an LLM output verification device provided in the embodiments of this specification;

[0026] Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this specification. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0028] The technical solutions provided in the various embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0029] Figure 1 A flowchart illustrating an LLM output verification method provided in this specification includes the following steps:

[0030] S100: Obtain the output results obtained by the business LLM based on the input data, and the reasoning process by which the business LLM obtains the output results.

[0031] In the embodiments of this specification, by means of... Figure 1 The device used to verify the output of the business LLM, as shown in the method, can be any electronic device, such as a personal computer, mobile phone, tablet, server, or server cluster consisting of several servers. The following explanation uses a server as an example only.

[0032] In the embodiments described in this specification, LLM is divided into business LLM and verification LLM.

[0033] The business LLM is used to receive input data, perform inference based on the input data, and obtain the output result. After obtaining the output result, the business LLM not only outputs the output result but also outputs the inference process that led to the output result. The inference process described in the embodiments of this specification is a natural language inference process. This natural language inference process includes not only text but also various forms of operators and formulas, such as using Markdown / LaTeX to represent operators, letters, and Roman numerals.

[0034] Verification LLM is used to verify the correctness of the above output results obtained from the business LLM and the reasoning process that yields those output results.

[0035] Therefore, the server can first obtain the output result obtained by the business LLM based on the input data, as well as the reasoning process by which the business LLM obtains the output result. The device that uses the business LLM to perform reasoning based on the input data to obtain the output result can be compared with devices that employ... Figure 1 The devices used to verify the output results of the business LLM as shown in the method can be the same or different, and the embodiments in this specification do not limit this.

[0036] S102: Input the output result and the reasoning process into the pre-trained verification LLM.

[0037] After the server obtains the output of the business LLM and the reasoning process for obtaining the output, it can input the output and the above reasoning process into a pre-trained verification LLM.

[0038] The verification LLM can be deployed on this server or on other devices. When the verification LLM is deployed on this server, the server can directly input the output result and the above-described reasoning process into the verification LLM. When the verification LLM is deployed on other devices, the server can send the output result and the above-described reasoning process to the other devices and instruct the other devices to input the output result and the above-described reasoning process into the verification LLM. This specification does not limit this, and the following description only uses the direct deployment of the verification LLM on this server as an example.

[0039] In addition, an agent can be pre-deployed on the server to perform the steps S100 to S102 described above.

[0040] S104: Extract the computational process included in the inference process through the verification LLM.

[0041] After inputting the output of the business LLM and the reasoning process that yielded that output into the verification LLM, the correctness of the output and the reasoning process can be verified through the verification LLM. This verification LLM requires at least fine-tuning training before it can perform verification; the specific training process will be described later.

[0042] When verifying the correctness of the output result and the reasoning process using the LLM verification, it can first extract the various operational processes included in the reasoning process. The operational processes described in the embodiments of this specification include logical operation processes and arithmetic operation processes.

[0043] Specifically, logical operations include, but are not limited to, AND, OR, NOT, greater than, less than, and equal to operations. Arithmetic operations include, but are not limited to, addition, subtraction, multiplication, division, exponentiation, and modulo operations.

[0044] S106: Through the verification LLM, for each extracted operation process, the operation result obtained by the business LLM in executing the operation process is taken as the verification result, and the correctness of the business LLM in executing the operation process is verified based on the verification result.

[0045] After the LLM extracts all the computational processes contained in the inference process, for each computational process, the computational result obtained by the business LLM in executing the computational process can be determined based on the inference process in natural language form, and this result is taken as the verification result. Then, the correctness of the business LLM in executing the computational process can be verified based on the verification result.

[0046] Specifically, this specification provides two methods for verifying the correctness of a calculation process: one is to verify using the inverse operation method, and the other is to verify by calling a calculator tool.

[0047] When verifying an extracted operation process using the inverse operation method, the verification LLM first determines the input parameters when the business LLM executes the operation process, and selects at least one input parameter as the comparison result. Then, based on the operation process and the corresponding verification result (i.e., the operation result of the operation process), the corresponding inverse operation process is determined. This inverse operation process is used to calculate the input parameter selected as the comparison result. Then, using the other input parameters (excluding the selected comparison result) and the verification result as the input parameters for the inverse operation process, the inverse operation process is executed to obtain its result. Finally, based on the result of the inverse operation process and the aforementioned comparison result, the correctness of the business LLM's execution of the above operation process is verified. Specifically, the verification LLM determines whether the result of the inverse operation process is the same as the aforementioned comparison result. If they are the same, the verification of the correctness of the business LLM's execution of the above operation process is considered passed; otherwise, the verification fails.

[0048] For example, suppose that in step S104, the verification LLM extracts an arithmetic operation 33-12=20 from the reasoning process of the business LLM. The verification LLM can then determine that the input parameters for this operation are 33 and 12, and the result to be verified is 20. Assuming that input parameter 12 is used as the comparison result, based on the other input parameter 33 and the result to be verified 20, the inverse operation used to calculate input parameter 12 can be determined to be 33-20. The verification LLM executes this inverse operation 33-20, and the result is 13, which differs from the comparison result 12. Therefore, it can be determined that the correctness verification of the business LLM's execution of the arithmetic operation 33-12=20 fails.

[0049] When verifying an extracted calculation process by calling a calculator tool, the verification LLM can first generate a preset type of computer program code to execute the calculation process as verification code. This preset type of computer program code includes, but is not limited to, Python code. After generating the verification code, the verification LLM can call the calculator tool corresponding to the preset type of computer program code. When the preset type of computer program code is Python code, the calculator tool can be a Python calculator tool. The verification LLM inputs the verification code into the called calculator tool, causing the calculator tool to execute the calculation process and obtain the calculation result output by the calculator tool, which serves as the comparison result. Finally, the verification LLM verifies the correctness of the business LLM's execution of the calculation process by comparing the result to be verified (i.e., the calculation result obtained by the business LLM executing the calculation process) with the above comparison result. Specifically, if the result to be verified obtained by the business LLM executing the calculation process is the same as the comparison result, the verification of the correctness of the business LLM's execution of the above calculation process is deemed to have passed; otherwise, the verification fails.

[0050] In the embodiments of this specification, the LLM verification can be performed by one or more of the above two verification methods to verify the correctness of a calculation process performed by the business LLM. However, when the LLM verification cannot call the calculator tool corresponding to the computer program code of the above-mentioned night photography type, the inverse operation method can only be used for verification.

[0051] For a calculation process extracted in step S104, if the verification LLM uses both of the above verification methods simultaneously, it can only be determined that the business LLM executed the calculation process correctly, i.e., the calculation result of the calculation process is correct, if both verification methods pass the verification. If only one verification method is used, it can be determined that the business LLM executed the calculation process correctly, i.e., the calculation result of the calculation process is correct, as long as the verification result of the used verification method passes the verification.

[0052] For the entire inference process by which the business LLM obtains the aforementioned output result, the entire inference process of the business LLM can only be determined to be correct if the verification results of all the operations included in the inference process pass verification. That is, the output result obtained by the business LLM is correct. If the verification results of at least one operation included in the inference process fail verification, the failed operation can be marked, and it can be directly determined that the inference process of the business LLM is incorrect, i.e., the output result obtained by the business LLM is incorrect. However, the verification LLM can continue to verify subsequent operations in the inference process to mark all failed operations. Subsequently, the business LLM can be trained based on all marked failed operations.

[0053] Furthermore, since the embodiments of this specification require the verification LLM to accurately extract the computational process from the reasoning process of the business LLM and accurately verify each extracted computational process, the verification LLM needs to be trained in advance. Specifically, when training the verification LLM to be trained, the sample output result obtained based on the sample input data and the sample reasoning process that obtains the sample output result can be obtained first, and the labeled computational process and the labeled verification process that verifies each labeled computational process can be determined. The LLM that obtains the sample output result based on the sample input data can be the aforementioned business LLM or other LLMs; this specification does not limit this, as long as the sample output result and the sample reasoning process that obtains the sample output result can be obtained. The method for annotating the actual computational processes involved in the sample inference process can be to manually annotate the actual computational processes involved in the sample inference process. The annotation verification process for verifying each annotated computational process can also be a manually calculated inverse computational process or a manually input prompt message. The prompt message is used to enable the verification LLM to be trained to generate computer program code of a preset type corresponding to each annotated computational process, and to call the calculator tool corresponding to the above-mentioned preset type of computer program code to execute the generated computer program code.

[0054] After obtaining the sample output results and sample inference process, these can be input into the verification LLM to be trained. The verification LLM then extracts the computational processes contained in the sample inference process, which are used as sample computational processes. Next, the verification LLM, for each extracted sample computational process, verifies the correctness of its execution based on the corresponding sample computational result contained in the sample inference process, and outputs the verification process as the sample verification process.

[0055] Finally, based on the first difference between the sample calculation process and the annotation calculation process, and the second difference between the sample verification process and the annotation verification process, at least some model parameters in the verification LLM to be trained are fine-tuned. Specifically, the loss value of the verification LLM to be trained can be determined based on the first and second differences, and at least some model parameters in the verification LLM to be trained are adjusted with the goal of reducing this loss value. The loss value is positively correlated with both the first and second differences.

[0056] Through the first difference mentioned above, the verification LLM can be trained to accurately extract each operation process from the inference process. Through the second difference mentioned above, the verification LLM can be trained to use appropriate verification methods to accurately verify the correctness of the operation process executed by the business LLM.

[0057] Furthermore, in the embodiments of this specification, the input data of the aforementioned input service LLM can specifically be the computation logic of a special operator corresponding to the operator name contained in the Higher-Order Knowledge Program (HOP), as well as the input parameters of that special operator. The execution method of the HOP is as follows: Figure 2 As shown.

[0058] Figure 2 A flowchart of an HOP execution method provided in this specification embodiment specifically includes the following steps:

[0059] S200: A runtime environment for running computer program code of a preset type and loading business data.

[0060] The HOPs mentioned in this specification are computer program code of a preset type, such as Python. Each HOP contains an operator name, which also conforms to the function name format requirements of this preset type of computational program code. However, the computational logic of the operator corresponding to that operator name is defined through natural language; this operator will be referred to as a special operator below.

[0061] The server can first obtain the HOP and run the runtime environment of the computer program code of that preset type, and then load the business data required by the HOP. For example, when the HOP is Python code, the server can launch the Python runtime environment and load the business data.

[0062] The HOP is generated by the business LLM based on the business logic process corresponding to a certain business. This business logic process is described in natural language. The generation process of the HOP will be explained later.

[0063] S202: Based on the business data, run the HOP through the operating environment.

[0064] In the embodiments of this specification, the framework code of the HOP is written by the aforementioned preset type of computer program code. Even if the code contains special operators, the operator names of the special operators conform to the function name format requirements of the preset type of computer program code. Therefore, in terms of form, the HOP is essentially the preset type of computer program code. The server can run the HOP based on the loaded business data, using the business data as the input parameter of the entire HOP, and run the HOP through the runtime environment of the preset type of computer program code.

[0065] S204: When the execution reaches a statement in the HOP that contains the operator name, the pre-trained business LLM is invoked.

[0066] When the server runs a statement containing the operator name of the special operator in the HOP through this runtime environment, it can invoke the pre-trained business LLM.

[0067] S206: Input the business data and / or the intermediate results obtained by the current running of the HOP as input parameters of the special operator corresponding to the operator name into the business LLM, so that the business LLM executes the calculation logic of the special operator corresponding to the operator name according to the input parameters and the preset knowledge base, so as to determine the execution result of the special operator.

[0068] Because the computational logic of this special operator is defined by natural language, the runtime environment of the preset type of computer program code cannot execute this special operator normally. Therefore, the server can use the business data and the intermediate results obtained from running the current HOP as input parameters to the business LLM. The business LLM then executes the computational logic of the special operator based on the input parameters and the knowledge in the preset knowledge base, obtaining the execution result of the special operator. This execution result is... Figure 1 The output results of the business LLM in step S100 are shown.

[0069] S208: Perform a correctness check on the execution result.

[0070] Since the computational logic of special operators is defined in natural language and needs to be executed by the business LLM, in order to avoid the business LLM from executing the special operators defined in natural language inaccurately, the execution result can be checked for correctness to determine whether the business LLM has correctly executed the computational logic of the special operator defined in natural language.

[0071] The method for verifying the correctness of the execution result obtained by executing special operators on the business LLM in step S208 is included in this specification. Figure 1 The methods shown include those for verifying LLMs.

[0072] S210: When the correctness check passes, continue executing the statement based on the execution result of the operator.

[0073] If the correctness check passes, the business LLM returns the execution result of the special operator to the execution environment of the computer program code of the preset type, and the execution environment continues to execute the statement until the entire HOP is executed.

[0074] Specifically, when using the above-mentioned methods... Figure 1 The verification method shown verifies the correctness of each operation process included in the reasoning process of the special operator executed by the business LLM. When all verifications pass, the output result (i.e., the execution result of the special operator) is returned to the business LLM. After receiving the verification result, the business LLM can determine that the correctness verification has passed, and then returns the output result as the execution result of the special operator to the runtime environment so that the runtime environment can continue to execute the statement according to the execution result of the special operator.

[0075] Among them, the above Figure 2 The process of generating the HOP shown can be as follows: Figure 3 As shown. Figure 3 The flowchart of the HOP generation method provided in the embodiments of this specification specifically includes the following steps:

[0076] S300: Obtain the business logic process described in natural language.

[0077] In the embodiments described in this specification, the following are employed: Figure 3 The device used to generate the code, as shown, can be any electronic device, such as a computer, server, or server cluster consisting of multiple servers. This device can be used with... Figure 2 The method shown can be performed on the same device or different devices. For ease of description, the following explanation will use the same server as an example.

[0078] To generate code for executing a specific business function, the server first obtains the business logic process for that function. Specifically, the business logic process described in this specification includes Standard Operating Procedures (SOPs). An SOP is a standard operating procedure for that business function, described in natural language. It is a document that details the work, workflow, and steps involved in that business function. It provides the organization's internal standard process specifications for that business function, for reference and adherence by internal personnel. In addition to the standardized process for that business function, the SOP also includes clearly defined responsibilities and authorities for each role within the business function, emergency measures, quality standards and assessments, document management, and other related content.

[0079] The following embodiments of this manual use the above-mentioned business as a risk control business as an example to illustrate the code generation method provided in this manual.

[0080] When the aforementioned business involves risk control based on recorded risk logs, the business logic process of this risk control business (i.e., the risk control logic process) can be described in the following natural language:

[0081] The first step is to analyze the cmd field to determine whether the cmd field has been granted executable permissions. If the cmd command line has not been granted executable permissions, the conclusion is that no action is required.

[0082] The second step is to determine whether the installation package is for commonly used software. If it is, the conclusion is that no action is required.

[0083] The third step is to determine if the cmd field has been manually reviewed by security operations personnel within the past N days and has been determined to require no action. For example, if the chmod command line is determined to require no action, then the risk log does not need to be processed.

[0084] The fourth step is to determine, based on the results of the above three steps, whether the risk log requires no action or generates an attack alert.

[0085] The aforementioned risk log is a risk log of other businesses recorded by the user when performing other businesses, and the aforementioned risk control business is the business that needs to perform risk control on the other businesses based on the risk log.

[0086] S302: Input the business logic process into the pre-trained business LLM.

[0087] After obtaining the aforementioned business logic process, the server can input this naturally described business logic process into a pre-trained business LLM. This business LLM can be pre-deployed on the server, or it can be deployed on other devices. If deployed on other devices, the server can send the business logic process to those devices, allowing them to input the received business logic process into the business LLM. The following explanation uses the deployment of the business LLM on the server as an example.

[0088] To avoid the business LLM generating code that does not match the actual business logic process due to "illusion," and to improve the accuracy and reliability of the generated code, in this embodiment of the specification, the server directly inputs the actual business logic process obtained in step S300 into the business LLM, so that the business LLM generates code for executing the business based on the business logic process.

[0089] S304: Identify the steps included in the business logic process and the logical relationships between the steps through the business LLM.

[0090] Generally, a business logic process consists of several steps arranged according to certain logical relationships. In the example above, the first step is executed. If the result of the first step is negative, no action is needed. If the result is positive, the second step is executed. If the result of the second step is positive, no action is needed. Otherwise, the third step is executed. If the result of the third step is positive, no action is needed. Otherwise, an attack alert is generated. Therefore, the logical relationships between different steps in a business logic process constitute its basic framework. Specifically, the execution of each step requires the knowledge corresponding to the business logic process, including the concepts, attributes, and relationships between the entities required in the process. For example, in the first step of the example above, it is necessary to determine whether the `cmd` field has been granted executable permissions. The knowledge required to execute this first step includes at least: the concept of the `cmd` field, the concept of executable permissions, and the relationship between the `cmd` field and executable permissions (i.e., which `cmd` fields indicate that executable permissions have been granted, and which indicate that executable permissions have not been granted).

[0091] Therefore, it can be seen that a business logic process is essentially composed of the logical relationships between each step and the knowledge required to execute each step. Thus, in order to generate code to execute the business logic process, the business LLM needs to identify the steps contained in the business logic process and the logical relationships between each step based on the business logic process described in natural language.

[0092] Specifically, since identifying the steps involved in the business logic process and the logical relationships between them may require a knowledge base corresponding to that business logic process, in this embodiment, the knowledge base corresponding to the business logic process can be pre-injected into a pre-trained business LLM. This knowledge base may include at least one of the following:

[0093] The knowledge graph corresponding to this business logic process includes the conceptual descriptions of each entity required in the business logic process (used to describe what the entity is, such as the concept of the cmd command line in the example above), the attributes of each entity (used to describe all possible attributes of the entity in this business logic), and the relationships between the entities.

[0094] The business logic process requires the calling interfaces of tools, including APIs or SDKs. For example, in the second step of the above example, it is necessary to determine whether the installation package is a common software installation package. This requires calling a tool to obtain a list of common software installation packages, which requires calling the tool through its calling interface.

[0095] After injecting the knowledge base containing the knowledge graph corresponding to the business logic process and / or the calling interface of the required tools into the pre-trained business LLM, when the server inputs the business logic process described in natural language into the business LLM, the business LLM can, based on its own reasoning ability, identify the semantics of the business logic process described in natural language, map the concepts in the business logic process to entities in the knowledge graph, and then, based on the semantics, break down the entire business logic process into several steps, and finally reason out the logical relationships between each step.

[0096] S306: Based on the logical relationship, generate computer program code of a preset type to represent the logical relationship between the steps, as framework code.

[0097] Because natural language inherently possesses a certain degree of ambiguity and vagueness—a flaw inherent in any natural language—directly using natural language to express the logical relationships between steps in a business logic process is highly likely to lead to logical confusion in those steps. In particular, when others other than the person who wrote the business logic process read it, their understanding may be inaccurate due to the ambiguity and vagueness of natural language, thus affecting the business itself.

[0098] As a formal and precise language, computer programming languages ​​have clear semantics and strict logical structures, which can effectively avoid the above-mentioned problems of natural languages. Therefore, in the embodiments of this specification, for the logical relationships between the steps in the business logic process, the server can generate computer program code of a preset type to represent the logical relationships between the steps based on the logical relationships between the steps identified in step S304 through the business LLM, as the framework code.

[0099] The computer program code of this preset type can be any type of computer program code, including but not limited to Python code. The following explanation uses Python code as an example only.

[0100] S308: For each step, generate at least one operator for executing the step based on the natural language used to describe the step in the business logic process and the preset knowledge base corresponding to the business logic process.

[0101] In the embodiments described in this specification, the execution order of steps S306 and S308 is not important. Specifically, the server can execute step S306 and step S308 simultaneously through the service LLM.

[0102] Since natural language is more natural and has better comprehensibility than computer programming languages, the knowledge required for each step in the business logic process is represented by an operator defined in natural language in the embodiments of this specification.

[0103] In this specification, the computational logic of the operators described in the embodiments is defined using natural language, but the operator names still conform to the function name format requirements of the aforementioned preset type of computer program code. Therefore, the operators defined using natural language in this specification are essentially user-defined functions in the preset type of computer program code, and these functions are high-level abstract expressions of the logic that implements the step.

[0104] Specifically, the business LLM can, for each step identified from the business logic process, determine the knowledge required to execute that step based on the semantics of the natural language used to describe that step within the identified business logic process, and the knowledge base corresponding to that business logic process. This knowledge includes, for example, one or more entities in the knowledge graph, one or more attributes of that entity, the relationships between that entity and other entities, and / or, the calling interface of the tools required for that step. Once the knowledge required to execute the step is determined, at least one operator for executing that step can be generated. This operator is defined in natural language, and its name conforms to the function name format requirements of the aforementioned preset type of computer program code.

[0105] Continuing with the previous example, the first step was to determine whether the cmd field had executable file permissions. Therefore, the business LLM can generate an operator name hop_judge that meets the requirements of Python code function names, and define the operator hop_judge as follows:

[0106] def hop_judge

[0107] Determine if {cmd} has been granted executable permissions.

[0108] Therefore, the operator hop_judge is defined entirely using natural language, specifically the statement "determine whether {cmd} has executable permissions," making it more understandable. The operators described above, which use natural language to define their computational logic, are special operators in HOP.

[0109] The above is just an example of using one operator to express one step. Those skilled in the art should understand that in practical applications, for more complex steps, more operators (such as two or more operators) can be used to express the step.

[0110] S310: Based on the framework code and the operator name generated for each step, generate a higher-order knowledge program for executing the business logic process.

[0111] As explained above, a business logic process is essentially composed of the logical relationships between each step and the knowledge required to execute each step. The logical relationships have been expressed by the generated framework code in step S306, and the knowledge required to execute each step has been expressed by operators defined in natural language and whose operator names still conform to the preset type of computer program code function name format requirements in step S308. Therefore, the business LLM can combine the framework code and the operators corresponding to each step to generate a knowledge high-order program (HOP) for executing the business logic process.

[0112] Continuing with the previous example, the business LLM integrates the operators corresponding to each step into the framework code, generating the following HOP for executing the above risk control logic process:

[0113] def evaluate(cmd):

[0114] if hop_judge_a("Check if (cmd) has been granted executable permissions") #Higher-order condition

[0115] pkg = hop_get("Get the installation package used by (cmd)")

[0116] If not pkg in hop_knowledge_retrieve("common installation packages") #Advanced knowledge concept matching

[0117] If not hop_judge_b("Check history, (cmd) whether it was manually judged as requiring no action in history")

[0118] Return "Attack Alert"

[0119] Return "No action required"

[0120] In the HOP above, the content within parentheses ("") defines the corresponding operators using natural language. These operators include hop_judge_a, hop_get, hop_knowledge_retrieve, and hop_judge_b. The entire risk control logic process, as expressed in the Python framework above, is as follows: First, the operator hop_judge_a is executed to determine if (cmd) has executable permissions. If yes, the operator hop_get is executed to retrieve the installation package used by (cmd) and assign it to pkg. Otherwise, "No action required" is output. After retrieving the installation package used by (cmd) and assigning it to pkg, the operator hop_knowledge_retrieve is executed to retrieve commonly used installation packages and determine if pkg is not among them. If yes, the operator hop_judge_b is executed. Otherwise, "No action required" is output. Finally, the operator hop_judge_b is executed to determine if (cmd) has not been manually judged as requiring no action in the history. If so, "Attack Alert" is output; otherwise, "No action required" is output.

[0121] As can be seen, the HOP represented by Python code above is completely consistent with the original risk control logic process described in natural language. Therefore, even if the personnel in the department performing risk control operations change, the HOP can still be read by other personnel (other than those who wrote the original risk control logic process) without causing logical confusion in each step due to the ambiguity and vagueness inherent in natural language. Furthermore, although the special operators in the HOP code are defined in natural language, the operator names are still function names in Python code format. Therefore, the HOP code itself is composed of Python framework code referencing various Python-formatted operator names. Formally, it is still Python code. Thus, this HOP code is not only human-readable but also usable by... Figure 2 The method shown is executed directly in the Python execution environment to implement the risk control logic process described above.

[0122] In the embodiments of this specification, in order for the business LLM to accurately convert the business logic process described in the input natural language into the above-mentioned HOP, it is not enough to simply inject the knowledge base corresponding to the business logic process into the business LLM. It is also necessary to fine-tune and train the business LLM in advance under the business scenario corresponding to the business logic process.

[0123] Specifically, when fine-tuning the business LLM, a sample logical process described in natural language can be obtained first. This sample logical process also contains several steps, referred to below as the sample step. This sample logical process and the aforementioned business logical process are business logical processes within the same business scenario; that is, the knowledge base corresponding to the sample logical process is exactly the same as the knowledge base corresponding to the aforementioned business logical process.

[0124] After obtaining the sample logic process, it can also be done through... Figure 3 Steps S302-S310, as shown, involve inputting the sample logic process into the business LLM to be trained. The business LLM identifies the sample steps within the sample logic process and the logical relationships between them. Based on these relationships, pre-defined computer program code representing these relationships is generated as the framework code to be optimized. For each sample step, at least one operator is generated to execute that step, based on the natural language used to describe it in the sample logic process and the pre-defined knowledge base corresponding to the sample logic process. This operator is used as the operator to be optimized. The computational logic of this operator is also defined using natural language, and its name conforms to the function name format of the pre-defined computer program code. Finally, based on the framework code to be optimized and the operator names generated for each sample step, an optimized HOP is generated to execute the sample logic process. This process is similar to... Figure 3 The steps S302 to S310 shown are exactly the same, so they will not be repeated here.

[0125] After the business LLM to be trained generates the HOP to be optimized, the optimized HOP after adjustments can be determined. Since the HOP to be optimized is also in the form of the aforementioned preset type of computer program code (such as Python code), it can be manually adjusted to ensure that the logical relationships between the sample steps expressed in the optimized framework code fully conform to the original sample logic process. Furthermore, the natural language corresponding to the operators to be optimized, defined by natural language, should be more closely aligned with the knowledge in the knowledge base corresponding to the sample logic process, and the computational logic of the operators to be optimized should be clearer and more accurate. Of course, the HOP to be optimized can also be adjusted using other trained business LLMs; this embodiment does not limit this approach.

[0126] After obtaining the optimized HOP, supervised fine-tuning (SFT) training can be performed on the business LLM to be trained based on the optimized HOP. That is, the optimized HOP is used as the annotation corresponding to the sample logic process, and this serves as a supervision signal to adjust the model parameters in the business LLM to be trained, so that the LLM can adapt to generating various business logic processes in the business scenario. The device used to train the LLM to be trained is the same as the aforementioned execution... Figure 3 The device used to generate the code shown can be the same device or different devices. Furthermore, because the embodiments in this specification aim to preserve the original inference capabilities of the business LLM as much as possible, adapting it only to this business scenario, when performing SFT on the business LLM, not all model parameters of the business LLM are adjusted; instead, only some model parameters are adjusted. The adjusted model parameters may only include the LoRA model parameters in the business LLM.

[0127] After obtaining the business LLM through the SFT training described above, the server can then... Figure 3 The method shown generates the HOP corresponding to the business logic process in this business scenario. However, when using... Figure 2 When the HOP is executed using the method shown, since the HOP not only contains computer program code of the aforementioned preset type (such as Python), but also contains special operators defined by natural language, the business LLM obtained above is also required to assist in the execution of the HOP.

[0128] When the above business scenario is a risk control business scenario, that is, when the above business logic process is a risk control logic process under the risk control business scenario, the loaded business data can be a pre-recorded risk log. This risk log is the risk log of other businesses recorded by the user when performing other businesses. The risk control business is the business that needs to perform risk control on the other businesses based on this risk log.

[0129] pass Figure 2 After step S200, as shown, launches the runtime environment of the preset type of computer program code and loads the business data, the program can be run based on the business data through the runtime environment. Figure 3 The generated HOP.

[0130] When the execution reaches a special operator defined in natural language within this HOP, the aforementioned business LLM needs to be invoked. Since the framework of this HOP is still a predefined type of computer program code, the runtime environment using this predefined type of computer program code can directly run the HOP. As for the special operators defined in natural language within this HOP, these special operators are essentially user-defined functions within the predefined type of computer program code. The HOP references these function names, meaning it executes the function corresponding to that name, obtains the execution result, and then continues executing subsequent code based on that result. Therefore, when the execution reaches the operator name of a special operator defined in natural language within this HOP, the predefined computational logic of that operator needs to be invoked. When the runtime environment parses the computational logic of this operator, if it determines that the computational logic is defined in natural language, the runtime environment can directly invoke the aforementioned business LLM.

[0131] After calling the above business LLM, it can be accessed through Figure 2 In step S206, the business data and / or the intermediate results obtained from the current running of the HOP are used as input parameters to the business LLM, so that the business LLM executes the calculation logic of the operator according to the input parameters and the knowledge base corresponding to the above business logic process to determine the execution result of the operator.

[0132] Continuing with the previous example, for the operator hop_judge_a in the HOP above, its natural language-defined computational logic is "to determine whether (cmd) has been granted executable permissions." Business data, i.e., pre-recorded risk logs, can be used as the input parameter of this operator. Both the computational logic defined in the natural language and the input parameter are input into the business LLM. A prompt message can also be input to the business LLM, guiding it to first identify the cmd command line (i.e., an entity in the knowledge graph) in the input parameter based on the knowledge base corresponding to the risk control logic process. Then, based on the knowledge base and the identified cmd command line, it determines the attributes of the cmd command line in the input parameter (i.e., the attributes of the entity in the knowledge graph). Based on these attributes, it determines whether the cmd command line has been granted executable permissions (i.e., the relationship between different entities in the knowledge graph). The judgment result is used as the output parameter of the operator hop_judge_a, returning to the Python runtime environment, allowing the Python runtime environment to continue executing subsequent code in the HOP.

[0133] For example, regarding the operator hop_knowledge_retrieve in the aforementioned HOP, its natural language-defined computational logic is "common installation packages". This computational logic can be input into the business LLM, along with a prompt message. Guided by this prompt, the business LLM first determines the tool for obtaining the list of common installation packages based on the knowledge base corresponding to the risk control logic process. Then, it calls the tool through its API to obtain the common installation packages. Finally, the obtained list of common installation packages is used as the output parameter of the operator hop_knowledge_retrieve and returned to the Python runtime environment. This allows the Python runtime environment to continue executing subsequent code in the HOP to determine whether the intermediate result pkg of the current HOP is not included in the list of common installation packages.

[0134] Those skilled in the art should understand that the above description uses the business logic process as an example of the risk control logic process. In fact, the HOP generation method provided in the embodiments of this specification can generate HOPs corresponding to business logic processes in any business scenario and run them accordingly. Furthermore, the above-mentioned preset type of computer program code is only illustrated using Python code as an example; the preset type of computer program code described in the embodiments of this specification can also be other types of computer program code.

[0135] exist Figure 2 In step S208, since the computational logic of the special operator is defined through natural language, and the business LLM may exhibit "illusion," this embodiment of the specification also requires the use of [a certain method] to ensure the accuracy of the business LLM's execution of the special operator. Figure 1 The verification method shown verifies the correctness of the execution result obtained by the business LLM executing the special operator. If the correctness verification passes, the business LLM can return the execution result to the runtime environment of the computer program code of the above-mentioned preset type, so that the runtime environment can continue to execute the HOP until the entire HOP is completed. If the correctness verification fails, the business LLM can re-execute the calculation logic of the special operator to redetermine the execution result of the special operator, until the obtained execution result passes the correctness verification, or the number of times the business LLM executes the calculation logic of the special operator reaches the preset number.

[0136] If the business LLM executes the computational logic of a special operator a preset number of times, and the execution result of the last determined special operator still fails the correctness check, then the business LLM can generate and output an assistance execution request. This assistance execution request is used to request human assistance in obtaining the execution result of the special operator. This assistance execution request can be sent to the user device, whereby the user can input the execution result of the special operator according to the computational logic and input parameters carried in the assistance execution request. The user device then returns the user-input execution result to the business LLM. The business LLM receives the execution result returned by the user device and considers the received execution result as the execution result that has passed the correctness check. Furthermore, based on the computational logic of the special operator and the received user-input execution result, supervised fine-tuning training can be performed on at least a portion of the model parameters in the business LLM, such as supervised fine-tuning training only on the LoRA model parameters in the business LLM.

[0137] The above is an example of an HOP execution method provided in this specification. Based on the same idea, this specification also provides corresponding devices, storage media and electronic devices.

[0138] Figure 4 This is a schematic diagram of an LLM output verification device provided in an embodiment of this specification. The device includes:

[0139] The acquisition module 401 is used to acquire the output result obtained by the business LLM based on the input data, and the reasoning process of the business LLM to obtain the output result;

[0140] Input module 402 is used to input the output result and the reasoning process into a pre-trained verification LLM;

[0141] Extraction module 403 is used to extract the computational process included in the inference process through the verification LLM;

[0142] The verification module 404 is used to verify the correctness of the business LLM's execution of the operation process for each extracted operation process through the verification LLM, taking the operation result obtained by the business LLM in executing the operation process as the verification result, and verifying the correctness of the business LLM in executing the operation process based on the verification result.

[0143] Optionally, the operation process includes at least one of logical operation process and arithmetic operation process.

[0144] Optionally, the verification module 404 is specifically used to determine the inverse operation process corresponding to the operation process based on the operation process and the result to be verified; execute the inverse operation process; and verify the correctness of the business LLM executing the operation process based on the result of executing the inverse operation process.

[0145] Optionally, the verification module 404 is specifically used to generate a preset type of computer program code for executing the calculation process as verification code; call the calculator tool corresponding to the preset type of computer program code, and input the verification code into the calculator tool to obtain the calculation result output by the calculator tool after executing the calculation process as a comparison result; and verify the correctness of the business LLM executing the calculation process based on the result to be verified and the comparison result.

[0146] Optionally, the device further includes:

[0147] The training module 405 is used to acquire sample output results obtained based on sample input data, and sample inference processes for obtaining the sample output results; and to determine the labeled operation processes and the labeled verification processes for verifying each labeled operation process in the sample inference process; input the sample output results and the sample inference process into a verification LLM to be trained; extract the sample operation processes contained in the sample inference process through the verification LLM to be trained; verify the correctness of the execution of each extracted sample operation process based on the sample operation results corresponding to the sample operation process contained in the sample inference process through the verification LLM to be trained, and output the verification process as a sample verification process; and fine-tune at least some model parameters in the verification LLM to be trained based on the first difference between the sample operation process and the labeled operation process, and the second difference between the sample verification process and the labeled verification process.

[0148] Optionally, the input data includes: the computational logic of the special operators corresponding to the operator names contained in the knowledge high-order program HOP, and the input parameters of the special operators;

[0149] The computational logic of the special operator is defined through natural language, and the HOP is computer program code of a preset type;

[0150] The HOP is a computer program code execution environment of the preset type that runs based on business data. When the execution reaches a statement in the HOP that contains the operator name, the calculation logic of the special operator is input into the business LLM. The business data and / or the intermediate results obtained from the current execution of the HOP are used as input parameters of the special operator into the business LLM. The business LLM then executes the calculation logic of the special operator according to the input parameters and the preset knowledge base to determine the execution result of the operator, which is then used as the output result of the business LLM.

[0151] Optionally, the input data is the business LLM that the runtime environment inputs each time it runs to a statement containing the operator name of a special operator in the HOP;

[0152] The verification module 404 is further configured to, when the verification of the correctness of each operation process included in the reasoning process passes, return the verification result of the output result to the business LLM, so that the business LLM returns the output result as the execution result of the special operator to the runtime environment, so that the runtime environment continues to execute the statement according to the execution result of the special operator.

[0153] This specification also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can be used to perform the verification method for the LLM output results provided above.

[0154] based on Figure 1 The verification method for the LLM output results shown in this specification is also provided in the embodiments. Figure 5 The diagram shows the structure of the electronic device. Figure 5 At the hardware level, the electronic device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to implement the aforementioned method for verifying the LLM output results.

[0155] The above description is merely an embodiment of this specification and is not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this specification.

Claims

1. A method for verifying the output of a large language model (LLM), the method comprising: Obtain the output results obtained by the business LLM based on the input data, and the reasoning process by which the business LLM obtains the output results; The output and the reasoning process are input into a pre-trained verification LLM; The computational processes included in the inference process are extracted through the verification LLM; Through the verification LLM, for each extracted operation process, the operation result obtained by the business LLM in executing the operation process is taken as the verification result, and the correctness of the business LLM in executing the operation process is verified based on the verification result. The input data includes: the calculation logic of the special operators corresponding to the operator names contained in the knowledge high-order program HOP, and the input parameters of the special operators; The computational logic of the special operator is defined through natural language, and the HOP is computer program code of a preset type; The HOP is a computer program code execution environment of the preset type that runs based on business data. When the execution reaches a statement in the HOP that contains the operator name, the calculation logic of the special operator is input into the business LLM. The business data and / or the intermediate results obtained from the current execution of the HOP are used as input parameters of the special operator into the business LLM. The business LLM then executes the calculation logic of the special operator according to the input parameters and the preset knowledge base to determine the execution result of the operator, which is then used as the output result of the business LLM.

2. The method as described in claim 1, wherein the operation process includes at least one of a logical operation process and an arithmetic operation process.

3. The method as described in claim 2, wherein verifying the correctness of the business LLM performing the calculation process based on the verification result to be verified, specifically includes: Based on the calculation process and the result to be verified, determine the inverse calculation process corresponding to the calculation process; Perform the inverse operation process; Based on the result of the inverse operation, the correctness of the business LLM performing the operation is verified.

4. The method as described in claim 2, wherein verifying the correctness of the business LLM performing the calculation process based on the verification result to be verified, specifically includes: Generate a pre-defined type of computer program code to perform the operation, as a verification code; Call the calculator tool corresponding to the computer program code of the preset type, input the verification code into the calculator tool, and obtain the calculation result output by the calculator tool after performing the calculation process, which is used as the comparison result; Based on the results to be verified and the comparison results, the correctness of the operation process performed by the business LLM is verified.

5. The method as described in any one of claims 1 to 4, wherein pre-training the verification LLM specifically includes: Obtain the sample output result based on the sample input data, and the sample inference process for obtaining the sample output result; and determine the labeled operation process and the label verification process for verifying each labeled operation process in the sample inference process. The sample output results and the sample reasoning process are input into the verification LLM to be trained; The sample operation process included in the sample inference process is extracted through the verification LLM to be trained; Using the verification LLM to be trained, for each extracted sample operation process, the correctness of the execution of the sample operation process is verified based on the sample operation result corresponding to the sample operation process contained in the sample inference process, and the verification process is output as the sample verification process. Based on the first difference between the sample operation process and the annotation operation process, and the second difference between the sample verification process and the annotation verification process, at least some model parameters in the verification LLM to be trained are fine-tuned.

6. The method as described in claim 1, wherein the input data is the business LLM input by the runtime environment each time it runs to a statement containing the operator name of a special operator in the HOP; The method further includes: When the correctness of each operation process included in the reasoning process is verified and passes, the verification result of the output result is returned to the business LLM, so that the business LLM returns the output result as the execution result of the special operator to the runtime environment, so that the runtime environment can continue to execute the statement according to the execution result of the special operator.

7. A verification device for the output results of a large language model (LLM), the device comprising: The acquisition module is used to acquire the output results obtained by the business LLM based on the input data, as well as the reasoning process by which the business LLM obtains the output results; The input module is used to input the output results and the reasoning process into a pre-trained verification LLM; An extraction module is used to extract the computational processes included in the inference process through the verification LLM; The verification module is used to verify the correctness of the business LLM's execution of the operation process for each extracted operation process through the verification LLM. The operation result obtained by the business LLM in executing the operation process is taken as the verification result. The input data includes: the calculation logic of the special operators corresponding to the operator names contained in the knowledge high-order program HOP, and the input parameters of the special operators; The computational logic of the special operator is defined through natural language, and the HOP is computer program code of a preset type; The HOP is a computer program code execution environment of the preset type that runs based on business data. When the execution reaches a statement in the HOP that contains the operator name, the calculation logic of the special operator is input into the business LLM. The business data and / or the intermediate results obtained from the current execution of the HOP are used as input parameters of the special operator into the business LLM. The business LLM then executes the calculation logic of the special operator according to the input parameters and the preset knowledge base to determine the execution result of the operator, which is then used as the output result of the business LLM.

8. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in any one of claims 1-6.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in any one of claims 1-6.

Images