A method, apparatus, storage medium, and device for executing high-level knowledge programs.

Abstract

This specification discloses an execution method for a knowledge-based high-order program (HOP). This method transforms the task to be executed into a corresponding HOP, uses the semantic precision of computer program code to constrain the overall logical accuracy of the task execution process, and expresses specific steps that require knowledge to execute through special operators defined by natural language. The LLM executes the special operator based on a knowledge base containing the knowledge, and the accuracy of the LLM execution of the special operator is ensured by verifying the correctness of the execution result of the special operator, thereby improving the accuracy and reliability of the entire task execution.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a method, apparatus, storage medium, and device for executing high-level knowledge programs. Background Technology

[0002] With the development of artificial intelligence (AI) technology, large language models (LLM) have been widely applied in various fields.

[0003] However, LLM may suffer from problems such as lack of knowledge, knowledge errors, or outdated knowledge due to limitations in training data. These problems may cause LLM to experience "illusions" when performing tasks, thereby reducing the accuracy and reliability of LLM in performing tasks.

[0004] Therefore, how to perform tasks accurately and reliably based on LLM has become an urgent problem to be solved. Summary of the Invention

[0005] This specification provides an embodiment of a method, apparatus, storage medium, and electronic device for executing a knowledge high-order program (HOP), in order to partially solve the problems existing in the prior art.

[0006] The embodiments in this specification adopt the following technical solutions:

[0007] This specification provides a method for executing a knowledge-high-order program (HOP), wherein the HOP is computer program code of a preset type, the HOP contains operator names, and the computational logic of the operator corresponding to the operator name is defined through natural language; the method includes:

[0008] The runtime environment for running the preset type of computer program code is configured to load business data.

[0009] Based on the business data, the HOP is run through the operating environment;

[0010] When the execution reaches a statement in the HOP that contains the operator name, the pre-trained Large Language Model (LLM) is invoked;

[0011] The business data and / or the intermediate results obtained from the current operation of the HOP are used as input parameters of the operator corresponding to the operator name to the LLM, so that the LLM executes the calculation logic of the operator corresponding to the operator name according to the input parameters and the preset knowledge base to determine the execution result of the operator;

[0012] The execution result is checked for correctness, and if the correctness check passes, the statement is executed again based on the execution result of the operator.

[0013] This specification provides an execution device for a knowledge-high-order program (HOP), wherein the HOP is computer program code of a preset type, the HOP contains operator names, and the computational logic of the operator corresponding to the operator name is defined through natural language; the device includes:

[0014] The loading module is used to run the computer program code of the preset type in the runtime environment and load business data;

[0015] The runtime module is used to run the HOP in the runtime environment based on the business data.

[0016] The calling module is used to invoke the pre-trained Large Language Model (LLM) when the execution reaches a statement containing the operator name in the HOP;

[0017] The operator execution module is used to input the business data and / or the intermediate results obtained by the current running of the HOP into the LLM as input parameters of the operator corresponding to the operator name, so that the LLM executes the calculation logic of the operator corresponding to the operator name according to the input parameters and the preset knowledge base, so as to determine the execution result of the operator;

[0018] The verification module is used to verify the correctness of the execution result;

[0019] The running module is also used to continue executing the statement based on the execution result of the operator when the correctness check passes.

[0020] This specification provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the execution method of the aforementioned high-order knowledge program HOP.

[0021] This specification provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the above-mentioned method for executing the high-order program HOP.

[0022] The above-described at least one technical solution adopted in the embodiments of this specification can achieve the following beneficial effects:

[0023] This specification discloses an embodiment of a knowledge-based high-order program (HOP) execution method. This method runs an HOP written in the same type of computer program code within a pre-defined runtime environment. When execution reaches a statement in the HOP containing a special operator name, a pre-trained LLM is invoked to execute the computational logic defined in natural language for that special operator. This determines the execution result of the special operator and verifies its correctness. If the correctness verification passes, the statement continues execution based on the result. This method transforms the task to be executed into a corresponding HOP, leveraging the semantic precision of the computer program code to constrain the overall logical accuracy of the task execution process. Specific steps requiring knowledge are expressed using special operators defined in natural language. The LLM executes these special operators based on a knowledge base containing that knowledge, and the accuracy of the LLM's execution of these special operators is ensured by verifying their correctness, thereby improving the accuracy and reliability of the entire task execution. Attached Figure Description

[0024] The accompanying drawings, which are included to provide a further understanding of this specification and form part of this specification, illustrate exemplary embodiments and are used to explain this specification, but do not constitute an undue limitation thereof. In the drawings:

[0025] Figure 1 A flowchart illustrating an HOP execution method provided in an embodiment of this specification;

[0026] Figure 2 A flowchart illustrating the HOP generation method provided in the embodiments of this specification;

[0027] Figure 3 A schematic diagram of an execution device for a HOP provided in an embodiment of this specification;

[0028] Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this specification. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0030] The technical solutions provided in the various embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0031] Figure 1 A flowchart illustrating the execution method of a knowledge-advanced program (HOP) provided in this specification includes the following steps:

[0032] S100: Run the computer program code of the preset type in the runtime environment and load business data.

[0033] In the embodiments described in this specification, the following are employed: Figure 1 The device used to execute the knowledge higher-order program (HOP) in the method shown can be any electronic device, such as a computer, server, or server cluster consisting of multiple servers. For ease of description, the following explanation uses a server as an example.

[0034] The HOPs mentioned in this specification are computer program code of a preset type, such as Python. Each HOP contains an operator name, which also conforms to the function name format requirements of this preset type of computational program code. However, the computational logic of the operator corresponding to that operator name is defined through natural language; this operator will be referred to as a special operator below.

[0035] The server can first obtain the HOP and run the runtime environment of the computer program code of that preset type, and then load the business data required by the HOP. For example, when the HOP is Python code, the server can launch the Python runtime environment and load the business data.

[0036] The HOP is generated by LLM based on the business logic process corresponding to a certain business, which is described in natural language. The generation process of the HOP will be explained later.

[0037] S102: Based on the business data, run the HOP through the operating environment.

[0038] In the embodiments of this specification, the framework code of the HOP is written by the aforementioned preset type of computer program code. Even if the code contains special operators, the operator names of the special operators conform to the function name format requirements of the preset type of computer program code. Therefore, in terms of form, the HOP is essentially the preset type of computer program code. The server can run the HOP based on the loaded business data, using the business data as the input parameter of the entire HOP, and run the HOP through the runtime environment of the preset type of computer program code.

[0039] S104: When the execution reaches the statement in the HOP that contains the operator name, the pre-trained large language model LLM is invoked.

[0040] When the server runs a statement containing the operator name of the specific operator in the HOP through this runtime environment, it can invoke the pre-trained LLM.

[0041] S106: Input the business data and / or the intermediate results obtained from the current operation of the HOP into the LLM as input parameters of the operator corresponding to the operator name, so that the LLM executes the calculation logic of the operator corresponding to the operator name according to the input parameters and the preset knowledge base, so as to determine the execution result of the operator.

[0042] Since the computational logic of this special operator is defined by natural language, the runtime environment of the computer program code of this preset type cannot execute the special operator normally. Therefore, the server can use the business data and the intermediate results obtained by running the HOP as input parameters of the special operator into the LLM, so that the LLM can execute the computational logic of the special operator according to the input parameters and the knowledge in the preset knowledge base, and obtain the execution result of the special operator.

[0043] S108: Perform a correctness check on the execution result.

[0044] Since the computational logic of special operators is defined in natural language and needs to be executed by LLM, in order to avoid the LLM executing the special operators defined in natural language inaccurately, the execution result can be checked for correctness to determine whether the LLM has correctly executed the computational logic of the special operator defined in natural language.

[0045] S110: When the correctness check passes, continue executing the statement based on the execution result of the operator.

[0046] If the correctness check passes, the LLM returns the execution result of the special operator to the execution environment of the computer program code of the preset type, which then continues to execute the statement until the entire HOP has been executed.

[0047] Among them, the above Figure 2 The process of generating the HOP shown can be as follows: Figure 2 As shown. Figure 2 The flowchart of the HOP generation method provided in the embodiments of this specification specifically includes the following steps:

[0048] S200: Obtain the business logic process described in natural language.

[0049] In the embodiments described in this specification, the following are employed: Figure 2 The device used to generate the code, as shown, can be any electronic device, such as a computer, server, or server cluster consisting of multiple servers. This device can be used with... Figure 1The method shown can be performed on the same device or different devices. For ease of description, the following explanation will use the same server as an example.

[0050] To generate code for executing a specific business function, the server first obtains the business logic process for that function. Specifically, the business logic process described in this specification includes Standard Operating Procedures (SOPs). An SOP is a standard operating procedure for that business function, described in natural language. It is a document that details the work, workflow, and steps involved in that business function. It provides the organization's internal standard process specifications for that business function, for reference and adherence by internal personnel. In addition to the standardized process for that business function, the SOP also includes clearly defined responsibilities and authorities for each role within the business function, emergency measures, quality standards and assessments, document management, and other related content.

[0051] The following embodiments of this manual use the above-mentioned business as a risk control business as an example to illustrate the code generation method provided in this manual.

[0052] When the aforementioned business involves risk control based on recorded risk logs, the business logic process of this risk control business (i.e., the risk control logic process) can be described in the following natural language:

[0053] The first step is to analyze the cmd field to determine whether the cmd field has been granted executable permissions. If the cmd command line has not been granted executable permissions, the conclusion is that no action is required.

[0054] The second step is to determine whether the installation package is for commonly used software. If it is, the conclusion is that no action is required.

[0055] The third step is to determine if the cmd field has been manually reviewed by security operations personnel within the past N days and has been determined to require no action. For example, if the chmod command line is determined to require no action, then the risk log does not need to be processed.

[0056] The fourth step is to determine, based on the results of the above three steps, whether the risk log requires no action or generates an attack alert.

[0057] The aforementioned risk log is a risk log of other businesses recorded by the user when performing other businesses, and the aforementioned risk control business is the business that needs to perform risk control on the other businesses based on the risk log.

[0058] S202: Input the business logic process into a pre-trained large language model (LLM).

[0059] After obtaining the aforementioned business logic process, the server can input this naturally described business logic process into a pre-trained LLM. This LLM can be pre-deployed on the server, or it can be deployed on other devices. If deployed on other devices, the server can send the business logic process to those devices, allowing them to input the received business logic process into the LLM. The following explanation uses the deployment of the LLM on the server as an example.

[0060] To avoid the LLM generating code that is inconsistent with the actual business logic process due to "illusion," and to improve the accuracy and reliability of the LLM generated code, in the embodiments of this specification, the server directly outputs the actual business logic process obtained in step S200 to the LLM, so that the LLM generates code for executing the business based on the business logic process.

[0061] S204: Identify the steps included in the business logic process and the logical relationships between the steps through the LLM.

[0062] Generally, a business logic process consists of several steps arranged according to certain logical relationships. In the example above, the first step is executed. If the result of the first step is negative, no action is needed. If the result is positive, the second step is executed. If the result of the second step is positive, no action is needed. Otherwise, the third step is executed. If the result of the third step is positive, no action is needed. Otherwise, an attack alert is generated. Therefore, the logical relationships between different steps in a business logic process constitute its basic framework. Specifically, the execution of each step requires the knowledge corresponding to the business logic process, including the concepts, attributes, and relationships between the entities required in the process. For example, in the first step of the example above, it is necessary to determine whether the `cmd` field has been granted executable permissions. The knowledge required to execute this first step includes at least: the concept of the `cmd` field, the concept of executable permissions, and the relationship between the `cmd` field and executable permissions (i.e., which `cmd` fields indicate that executable permissions have been granted, and which indicate that executable permissions have not been granted).

[0063] Therefore, it can be seen that a business logic process is essentially composed of the logical relationships between each step and the knowledge required to execute each step. Thus, in order to generate code to execute the business logic process, LLM needs to identify the steps contained in the business logic process and the logical relationships between each step based on the business logic process described in natural language.

[0064] Specifically, since identifying the steps involved in the business logic process and the logical relationships between them may require a knowledge base corresponding to that business logic process, in this embodiment, the knowledge base corresponding to the business logic process can be pre-injected into the pre-trained LLM. This knowledge base may include at least one of the following:

[0065] The knowledge graph corresponding to this business logic process includes the conceptual descriptions of each entity required in the business logic process (used to describe what the entity is, such as the concept of the cmd command line in the example above), the attributes of each entity (used to describe all possible attributes of the entity in this business logic), and the relationships between the entities.

[0066] The business logic process requires the calling interfaces of tools, including APIs or SDKs. For example, in the second step of the above example, it is necessary to determine whether the installation package is a common software installation package. This requires calling a tool to obtain a list of common software installation packages, which requires calling the tool through its calling interface.

[0067] After injecting the knowledge base containing the knowledge graph corresponding to the business logic process and / or the calling interface of the required tools into the pre-trained LLM, when the server inputs the business logic process described in natural language into the LLM, the LLM can, based on its own reasoning ability, identify the semantics of the business logic process described in natural language, map the concepts in the business logic process to entities in the knowledge graph, and then, based on the semantics, break down the entire business logic process into several steps, and finally reason out the logical relationships between each step.

[0068] S206: Based on the logical relationship, generate computer program code of a preset type to represent the logical relationship between the steps, as framework code.

[0069] Because natural language inherently possesses a certain degree of ambiguity and vagueness—a flaw inherent in any natural language—directly using natural language to express the logical relationships between steps in a business logic process is highly likely to lead to logical confusion in those steps. In particular, when others other than the person who wrote the business logic process read it, their understanding may be inaccurate due to the ambiguity and vagueness of natural language, thus affecting the business itself.

[0070] As a formal and precise language, computer programming languages ​​have clear semantics and strict logical structures, which can effectively avoid the above-mentioned problems of natural languages. Therefore, in the embodiments of this specification, for the logical relationships between the steps in the business logic process, the server can generate computer program code of a preset type to represent the logical relationships between the steps based on the logical relationships between the steps identified in step S204 through the LLM, as the framework code.

[0071] The computer program code of this preset type can be any type of computer program code, including but not limited to Python code. The following explanation uses Python code as an example only.

[0072] S208: For each step, generate at least one operator for executing the step based on the natural language used to describe the step in the business logic process and the preset knowledge base corresponding to the business logic process.

[0073] In the embodiments described in this specification, the execution order of steps S206 and S208 is not important. Specifically, the server can execute step S106 and step S208 simultaneously through the LLM.

[0074] Since natural language is more natural and has better comprehensibility than computer programming languages, the knowledge required for each step in the business logic process is represented by an operator defined in natural language in the embodiments of this specification.

[0075] In this specification, the computational logic of the operators described in the embodiments is defined using natural language, but the operator names still conform to the function name format requirements of the aforementioned preset type of computer program code. Therefore, the operators defined using natural language in this specification are essentially user-defined functions in the preset type of computer program code, and these functions are high-level abstract expressions of the logic that implements the step.

[0076] Specifically, for each step identified from the business logic process, LLM can determine the knowledge required to execute that step based on the semantics of the natural language used to describe that step within the business logic process, and the knowledge base corresponding to that business logic process. This knowledge includes one or more entities in the knowledge graph, one or more attributes of that entity, the relationships between that entity and other entities, and / or the calling interface of the tools required for that step. Once the knowledge required to execute the step is determined, at least one operator for executing that step can be generated. This operator is defined in natural language, and its name conforms to the function name format requirements of the aforementioned preset type of computer program code.

[0077] Continuing with the previous example, the first step was to determine whether the cmd field had executable permissions. Therefore, LLM can generate an operator name hop_judge that meets the requirements of Python code function names, and define the operator hop_judge as follows:

[0078] def hop_judge

[0079] Determine if {cmd} has been granted executable permissions.

[0080] Therefore, the operator hop_judge is defined entirely using natural language, specifically the statement "determine whether {cmd} has executable permissions," making it more understandable. The operators described above, which use natural language to define their computational logic, are special operators in HOP.

[0081] The above is just an example of using one operator to express one step. Those skilled in the art should understand that in practical applications, for more complex steps, more operators (such as two or more operators) can be used to express the step.

[0082] S210: Based on the framework code and the operator name generated for each step, generate a higher-order knowledge program for executing the business logic process.

[0083] As explained above, a business logic process is essentially composed of the logical relationships between each step and the knowledge required to execute each step. The logical relationships have been expressed by the generated framework code in step S206, and the knowledge required to execute each step has been expressed by operators defined in natural language and whose operator names still conform to the preset type of computer program code function name format requirements in step S208. Therefore, LLM can combine the framework code and the operators corresponding to each step to generate a knowledge high-order program (HOP) for executing the business logic process.

[0084] Continuing with the previous example, LLM combines the operators corresponding to each step into the framework code, generating the following HOP for executing the above risk control logic process:

[0085] def evaluate(cmd):

[0086] if hop_judge_a("Check if (cmd) has been granted executable permissions") #Higher-order condition

[0087] pkg = hop_get("Get the installation package used by (cmd)")

[0088] If not pkg in hop_knowledge_retrieve("common installation packages") #Advanced knowledge concept matching

[0089] If not hop_judge_b("Check history, (cmd) whether it was manually judged as requiring no action in history")

[0090] Return "Attack Alert"

[0091] Return "No action required"

[0092] In the HOP above, the content within parentheses ("") defines the corresponding operators using natural language. These operators include hop_judge_a, hop_get, hop_knowledge_retrieve, and hop_judge_b. The entire risk control logic process, as expressed in the Python framework above, is as follows: First, the operator hop_judge_a is executed to determine if (cmd) has executable permissions. If yes, the operator hop_get is executed to retrieve the installation package used by (cmd) and assign it to pkg. Otherwise, "No action required" is output. After retrieving the installation package used by (cmd) and assigning it to pkg, the operator hop_knowledge_retrieve is executed to retrieve commonly used installation packages and determine if pkg is not among them. If yes, the operator hop_judge_b is executed. Otherwise, "No action required" is output. Finally, the operator hop_judge_b is executed to determine if (cmd) has not been manually judged as requiring no action in the history. If so, "Attack Alert" is output; otherwise, "No action required" is output.

[0093] As can be seen, the HOP represented by Python code above is completely consistent with the original risk control logic process described in natural language. Therefore, even if the personnel in the department performing risk control operations change, the HOP can still be read by other personnel (other than those who wrote the original risk control logic process) without causing logical confusion in each step due to the ambiguity and vagueness inherent in natural language. Furthermore, although the special operators in the HOP code are defined in natural language, the operator names are still function names in Python code format. Therefore, the HOP code itself is composed of Python framework code referencing various Python-formatted operator names. Formally, it is still Python code. Thus, this HOP code is not only human-readable but also usable by... Figure 1 The method shown is executed directly in the Python execution environment to implement the risk control logic process described above.

[0094] In the embodiments of this specification, in order for the LLM to accurately convert the business logic process described in the input natural language into the above-mentioned HOP, it is not enough to simply inject the knowledge base corresponding to the business logic process into the LLM. It is also necessary to fine-tune and train the LLM in advance under the business scenario corresponding to the business logic process.

[0095] Specifically, when fine-tuning the LLM, a sample logical process described in natural language can be obtained first. This sample logical process also contains several steps, referred to below as the sample step. This sample logical process and the aforementioned business logical process are business logical processes within the same business scenario; that is, the knowledge base corresponding to the sample logical process is exactly the same as the knowledge base corresponding to the aforementioned business logical process.

[0096] After obtaining the sample logic process, it can also be done through... Figure 2 Steps S202-S210, as shown, involve inputting the sample logic process into the LLM to be trained. The LLM identifies the sample steps within the sample logic process and the logical relationships between them. Based on these relationships, a preset type of computer program code representing the logical relationships between the sample steps is generated as the framework code to be optimized. For each sample step, based on the natural language used to describe that step in the sample logic process and the preset knowledge base corresponding to the sample logic process, at least one operator is generated to execute that step as an operator to be optimized. The computational logic of this operator is also defined using natural language, and the operator name conforms to the function name format of the preset type of computer program code. Finally, based on the framework code to be optimized and the operator names of the operators generated for each sample step, an optimized HOP is generated to execute the sample logic process. This process is similar to... Figure 2 The steps S202 to S210 shown are exactly the same, so they will not be repeated here.

[0097] After the LLM to be trained generates the HOP to be optimized, the optimized HOP after adjustments can be determined. Since the HOP to be optimized is also in the form of the aforementioned preset type of computer program code (such as Python code), it can be manually adjusted to ensure that the logical relationships between the sample steps expressed in the optimized framework code fully conform to the original sample logic process, and that the natural language corresponding to the operators to be optimized, defined by natural language, is more closely aligned with the knowledge in the knowledge base corresponding to the sample logic process, and that the computational logic of the operators to be optimized is clearer and more accurate. Of course, the HOP to be optimized can also be adjusted using other trained LLMs; this embodiment does not limit this approach.

[0098] After obtaining the optimized HOP, supervised fine-tuning (SFT) training can be performed on the LLM to be trained based on the optimized HOP. That is, the optimized HOP is used as the annotation corresponding to the sample logic process, and this serves as a supervision signal to adjust the model parameters in the LLM to be trained, so that the LLM can adapt to generating various business logic processes in the business scenario. The device used to train the LLM to be trained is the same as the aforementioned execution... Figure 2 The device used to generate the code shown can be the same device or different devices. Furthermore, because this embodiment aims to preserve the original inference capabilities of the LLM as much as possible, adapting it only to this business scenario, when performing SFT on the LLM, not all model parameters are adjusted; only a portion of the model parameters are adjusted. The adjusted model parameters may include only the LoRA model parameters within the LLM.

[0099] After obtaining the LLM through the SFT training described above, the server can then... Figure 2 The method shown generates the HOP corresponding to the business logic process in this business scenario. However, when using... Figure 1 When the method shown is used to execute the HOP, since the HOP not only contains computer program code of the above-mentioned preset type (such as Python), but also contains special operators defined by natural language, the LLM obtained by training above is also required to assist in the execution of the HOP.

[0100] When the above business scenario is a risk control business scenario, that is, when the above business logic process is a risk control logic process under the risk control business scenario, the loaded business data can be a pre-recorded risk log. This risk log is the risk log of other businesses recorded by the user when performing other businesses. The risk control business is the business that needs to perform risk control on the other businesses based on this risk log.

[0101] pass Figure 1 After step S100, as shown, launches the runtime environment of the preset type of computer program code and loads the business data, the program can be run based on the business data through the runtime environment. Figure 2 The generated HOP.

[0102] When the execution reaches a special operator defined in natural language within the HOP, the aforementioned LLM needs to be invoked. Since the framework of this HOP is still predefined type computer program code, the runtime environment using this predefined type of computer program code can directly run the HOP. As for the special operators defined in natural language within the HOP, these special operators are essentially user-defined functions within the predefined type of computer program code. The HOP references these function names, meaning it executes the function corresponding to that name, obtains the execution result, and then continues executing subsequent code based on that result. Therefore, when the execution reaches the operator name of a special operator defined in natural language within the HOP, the predefined computational logic of that operator needs to be invoked. When the runtime environment parses the computational logic of this operator, if it determines that the computational logic is defined in natural language, the runtime environment can directly invoke the aforementioned LLM.

[0103] After calling the LLM mentioned above, it can be accessed through Figure 1 In step S106, the business data and / or the intermediate results obtained from the current running of the HOP are used as input parameters to the LLM, so that the LLM executes the calculation logic of the operator according to the input parameters and the knowledge base corresponding to the above business logic process to determine the execution result of the operator.

[0104] Continuing with the previous example, for the operator hop_judge_a in the HOP above, its natural language-defined computational logic is "to determine whether (cmd) has been granted executable permissions." Business data, i.e., pre-recorded risk logs, can be used as the input parameter of this operator. The computational logic defined in the natural language of this operator, along with the input parameter, should be input into the LLM. A prompt message can also be input to the LLM, guiding it to first identify the cmd command line (i.e., an entity in the knowledge graph) in the input parameter based on the knowledge base corresponding to the risk control logic process. Then, based on the knowledge base and the identified cmd command line, it determines the attributes of the cmd command line in the input parameter (i.e., the attributes of the entity in the knowledge graph). Based on these attributes, it determines whether the cmd command line has been granted executable permissions (i.e., the relationship between different entities in the knowledge graph). The judgment result is used as the output parameter of the operator hop_judge_a, returning to the Python runtime environment, allowing the Python runtime environment to continue executing subsequent code in the HOP.

[0105] For example, for the operator hop_knowledge_retrieve in the above HOP, its computational logic defined in natural language is "common installation packages". The computational logic defined in natural language for this operator can be input into the LLM. Prompt information can also be input into the LLM, so that the LLM, guided by the prompt information, first determines the tool for obtaining the list of common installation packages based on the knowledge base corresponding to the risk control logic process, then calls the tool through the calling interface to obtain the common installation packages, and finally uses the obtained list of common installation packages as the output parameter of the operator hop_knowledge_retrieve and returns it to the Python runtime environment, so that the Python runtime environment continues to execute the subsequent code in the HOP to determine whether the intermediate result pkg of the current HOP is not included in the list of common installation packages.

[0106] Those skilled in the art should understand that the above description uses the business logic process as an example of the risk control logic process. In fact, the HOP generation method provided in the embodiments of this specification can generate HOPs corresponding to business logic processes in any business scenario and run them accordingly. Furthermore, the above-mentioned preset type of computer program code is only illustrated using Python code as an example; the preset type of computer program code described in the embodiments of this specification can also be other types of computer program code.

[0107] exist Figure 1 In step S108, since the computational logic of the special operator is defined by natural language and the LLM may experience "illusions," in order to ensure the accuracy of the LLM's execution of the special operator, this embodiment of the specification also needs to perform a correctness check on the execution result obtained by the LLM in executing the special operator. If the correctness check passes, the LLM can return the execution result to the runtime environment of the computer program code of the aforementioned preset type, so that the runtime environment can continue to execute the HOP until the entire HOP is executed. If the correctness check fails, the LLM can re-execute the computational logic of the special operator to redetermine the execution result of the special operator until the obtained execution result passes the correctness check, or the number of times the LLM executes the computational logic of the special operator reaches the preset number.

[0108] If the LLM executes the computational logic of a special operator a preset number of times, and the execution result of the last determined special operator still fails the correctness check, an assistance execution request can be generated and output by the LLM. This assistance execution request is used to request human assistance in obtaining the execution result of the special operator. This assistance execution request can be sent to a user device, whereby the user can input the execution result of the special operator according to the computational logic and input parameters carried in the assistance execution request. The user device then returns the user-input execution result to the LLM. The LLM receives the execution result returned by the user device and considers the received execution result as the execution result that has passed the correctness check. Furthermore, based on the computational logic of the special operator and the received user-input execution result, supervised fine-tuning training can be performed on at least a portion of the model parameters in the LLM, such as supervised fine-tuning training only on the LoRA model parameters in the LLM.

[0109] When performing correctness checks on the execution results of the special operators obtained from LLM, at least one of the following three aspects can be checked:

[0110] Firstly, the correctness of the execution result obtained by the LLM can be verified according to preset result verification rules. Specifically, this first aspect may include: using another LLM, executing the special operator based on its calculation logic and input parameters, and the knowledge base corresponding to the aforementioned business logic process, to obtain an execution result; if the execution result obtained by the LLM in step S106 is the same as the execution result obtained by the other LLM, then the correctness verification of the execution result obtained by the LLM is determined to be passed; otherwise, the correctness verification fails; and / or, using the LLM, re-executing the calculation logic of the special operator; if the execution result obtained by the LLM in step S106 is the same as the execution result obtained by re-executing the special operator, then the correctness verification of the execution result obtained by the LLM is determined to be passed; otherwise, the correctness verification fails; and / or, determining whether the execution result obtained by the LLM in step S106 falls within a preset range for the special operator; if it falls within this range, then the correctness verification of the execution result obtained by the LLM is determined to be passed; otherwise, the correctness verification fails.

[0111] Of course, the correctness verification method in the first aspect mentioned above may also include other verification methods, which will not be listed in this specification.

[0112] Secondly, the LLM can determine the calculation process by which it obtains the execution result, and perform a correctness check on the calculation process according to preset calculation process verification rules. Specifically, this second aspect may include: when the execution result of the special operator is a numerical result, verifying the correctness of the execution result obtained by the LLM in step S106 using mathematical inverse operations based on the calculation process by which the LLM obtains the execution result; and / or, through the LLM, determining the semantic key points in the calculation logic based on the semantics of the calculation logic defined by the special operator in natural language, and judging whether the calculation process by which the LLM obtains the execution result contains all the semantic key points; if not all are contained, the correctness check fails.

[0113] Of course, the correctness verification method in the second aspect mentioned above may also include other verification methods, which will not be listed in this specification.

[0114] Thirdly, the input and output parameter formats when the LLM executes the computational logic of the special operator are determined, and the correctness of these formats is verified according to preset format validation rules. Specifically, the input and output parameter formats of the special operator can be preset using JSON Schema. Based on the preset input parameter format using JSON Schema, the correctness of the format of the input parameters received by the LLM for the special operator is determined. Similarly, based on the preset output parameter format using JSON Schema, the correctness of the format of the execution result obtained by the LLM for the special operator is determined.

[0115] To ensure the accuracy of LLM's execution of this special operator, the correctness checks of the above three aspects can be performed simultaneously, and more than two of the above-mentioned checks of the first and second aspects can be selected.

[0116] The above is an example of an HOP execution method provided in this specification. Based on the same idea, this specification also provides corresponding devices, storage media and electronic devices.

[0117] Figure 3 This is a schematic diagram of an execution device for an HOP provided in an embodiment of this specification. The HOP is computer program code of a preset type, and the HOP contains operator names. The computational logic of the operator corresponding to the operator name is defined through natural language. The device includes:

[0118] The loading module 301 is used to run the computer program code of the preset type in the runtime environment and load business data;

[0119] The running module 302 is used to run the HOP in the running environment based on the business data;

[0120] Module 303 is invoked to invoke a pre-trained large language model (LLM) when the execution reaches a statement in the HOP that contains the operator name.

[0121] Operator execution module 304 is used to input the business data and / or the intermediate results obtained by running the HOP as input parameters of the operator corresponding to the operator name into the LLM, so that the LLM executes the calculation logic of the operator corresponding to the operator name according to the input parameters and the preset knowledge base to determine the execution result of the operator;

[0122] The verification module 305 is used to verify the correctness of the execution result;

[0123] The running module 302 is further configured to continue executing the statement based on the execution result of the operator when the correctness check passes.

[0124] Optionally, the operator name conforms to the function name format of the computer program code of the preset type;

[0125] The calling module 303 is specifically used to call the calculation logic of the operator corresponding to the operator name when the execution reaches the statement containing the operator name in the HOP; and to call the LLM when the calculation logic is defined by natural language through the execution environment.

[0126] Optionally, the operator execution module 304 is further configured to, when the correctness check fails, re-execute the calculation logic of the operator through the LLM to redetermine the execution result of the operator.

[0127] Optionally, the operator execution module 304 is further configured to generate and output an assistance execution request when the number of times the LLM executes the computational logic of the operator reaches a preset number, and the execution result of the last determined operator still fails the correctness check; and receive the execution result of the operator input by the user equipment in response to the assistance execution request as the execution result that has passed the correctness check.

[0128] Optionally, the verification module 305 is specifically used to: verify the correctness of the execution result according to a preset result verification rule; and / or, determine the calculation process by which the LLM obtains the execution result, and verify the correctness of the calculation process according to a preset calculation process verification rule; and / or, determine the input parameter format and output parameter format when the LLM executes the calculation logic of the operator, and verify the correctness of the input parameter format and output parameter format according to a preset format verification rule.

[0129] Optionally, the HOP is generated by the LLM based on business logic processes described in natural language;

[0130] The knowledge base is the knowledge base corresponding to the business logic process.

[0131] Optionally, the business logic process includes a risk control logic process;

[0132] The business data includes pre-recorded risk logs;

[0133] The preset type of computer program code includes Python.

[0134] This specification also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can be used to perform the HOP execution method provided above.

[0135] based on Figure 1 The implementation method of the HOP shown in this specification is further provided in the embodiments. Figure 4 The diagram shows the structure of the electronic device. Figure 4 At the hardware level, the electronic device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to implement the HOP execution method described above.

[0136] The above description is merely an embodiment of this specification and is not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this specification.

Claims

1. A method for executing a knowledge-high-order program (HOP), wherein the HOP is computer program code of a preset type, the HOP contains operator names, and the computational logic of the operator corresponding to the operator name is defined through natural language; the method includes: The runtime environment for running the preset type of computer program code is configured to load business data. Based on the business data, the HOP is run through the operating environment; When the execution reaches a statement in the HOP that contains the operator name, the pre-trained Large Language Model (LLM) is invoked; The business data and / or the intermediate results obtained from the current operation of the HOP are used as input parameters of the operator corresponding to the operator name to the LLM, so that the LLM executes the calculation logic of the operator corresponding to the operator name according to the input parameters and the preset knowledge base to determine the execution result of the operator; The execution result is checked for correctness, and if the correctness check passes, the statement is executed again based on the execution result of the operator.

2. The method as described in claim 1, wherein the operator name conforms to the function name format of the preset type of computer program code; When the execution reaches a statement in the HOP that contains the operator name, the pre-trained LLM is invoked, specifically including: When the execution reaches a statement in the HOP that contains the operator name, the calculation logic of the operator corresponding to the operator name is invoked; When the computational logic is defined in natural language by parsing the runtime environment, the LLM is invoked.

3. The method of claim 1, further comprising: If the correctness check fails, the operator's calculation logic is re-executed through the LLM to redetermine the operator's execution result.

4. The method of claim 3, further comprising: When the number of times the LLM executes the computation logic of the operator reaches a preset number, and the execution result of the last determined operator still fails the correctness check, an assistance execution request is generated and output. The execution result of the operator input by the user equipment in response to the assistance execution request is received as the execution result that has passed the correctness check.

5. The method as described in claim 1, wherein the correctness verification of the execution result specifically includes: The correctness of the execution result is verified according to the preset result verification rules; and / or The calculation process by which the LLM obtains the execution result is determined, and the correctness of the calculation process is verified according to the preset calculation process verification rules. and / or The input and output parameter formats are determined when the LLM executes the computational logic of the operator, and the correctness of the input and output parameter formats is verified according to the preset format verification rules.

6. The method as described in any one of claims 1 to 5, wherein the HOP is generated by the LLM based on a business logic process described in natural language; The knowledge base is the knowledge base corresponding to the business logic process.

7. The method as described in claim 6, wherein the business logic process includes a risk control logic process; The business data includes pre-recorded risk logs; The preset type of computer program code includes Python.

8. An execution device for a knowledge high-order program HOP, wherein the HOP is computer program code of a preset type, the HOP contains operator names, and the computational logic of the operator corresponding to the operator name is defined by natural language; the device comprises: The loading module is used to run the computer program code of the preset type in the runtime environment and load business data; The runtime module is used to run the HOP in the runtime environment based on the business data. The calling module is used to invoke the pre-trained Large Language Model (LLM) when the execution reaches a statement containing the operator name in the HOP; The operator execution module is used to input the business data and / or the intermediate results obtained by the current running of the HOP into the LLM as input parameters of the operator corresponding to the operator name, so that the LLM executes the calculation logic of the operator corresponding to the operator name according to the input parameters and the preset knowledge base, so as to determine the execution result of the operator; A verification module is used to verify the correctness of the execution result; The running module is also used to continue executing the statement based on the execution result of the operator when the correctness check passes.

9. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in any one of claims 1-7.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the method described in any one of claims 1-7.

Images