Execution method and device of knowledge high-order program, storage medium and equipment

Abstract

The embodiment of the invention discloses a knowledge high-order program HOP execution method, which is characterized in that a task needing to be executed is changed into a corresponding HOP, the overall logic accuracy of the task execution process is constrained by means of semantic accuracy of computer program codes, and specific steps which can only be executed by adopting knowledge can be executed in the task execution process. If the task is executed, the task is expressed by a special operator defined by a natural language, the special operator is executed by the LLM based on a knowledge base containing the knowledge, and the accuracy of executing the special operator by the LLM is ensured by performing correctness verification on the execution result of the special operator, so that the accuracy and reliability of executing the whole task are improved.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a method, apparatus, storage medium, and device for executing a high-level knowledge program. Background Art

[0002] With the development of artificial intelligence (AI) technology, large language models (LLM) have been widely used in various fields.

[0003] However, due to the limitations of training data, LLM may suffer from problems such as lack of knowledge, knowledge errors, or outdated knowledge. These problems may cause LLM to experience "hallucinations" when performing tasks, thereby reducing the accuracy and reliability of LLM's task execution.

[0004] Therefore, how to accurately and reliably execute tasks based on LLM has become an urgent problem to be solved. Summary of the Invention

[0005] The embodiments of this specification provide a method, device, storage medium, and electronic device for executing a high-level knowledge program (HOP) to partially solve the problems existing in the above-mentioned prior art.

[0006] The embodiments of this specification adopt the following technical solutions: This specification provides a method for executing a high-level knowledge program (HOP). The HOP is a computer program code of a preset type. The HOP includes operator names, and the calculation logic of the operators corresponding to the operator names is defined in natural language. The method includes: Running the operating environment of the preset type of computer program code and loading business data; Based on the business data, running the HOP through the operating environment; When running to the statement containing the operator name in the HOP, calling the pre-trained large language model LLM; Input the business data and / or the intermediate result obtained by currently running the HOP as input parameters of the operator corresponding to the operator name into the LLM, so that the LLM executes the calculation logic of the operator corresponding to the operator name according to the input parameters and a preset knowledge base to determine the execution result of the operator; The correctness of the execution result is checked, and when the correctness check passes, the statement is continued to be executed according to the execution result of the operator.

[0007] This specification provides an execution device for a high-level knowledge program (HOP). The HOP is a computer program code of a preset type. The HOP includes an operator name, and the calculation logic of the operator corresponding to the operator name is defined in natural language. The device includes: A loading module, configured to run an operating environment of the preset type of computer program code and load business data; An operating module, configured to operate the HOP through the operating environment based on the service data; A calling module, configured to call a pre-trained large language model LLM when running to a statement containing the operator name in the HOP; An operator execution module is configured to input the business data and / or the intermediate result obtained by currently running the HOP as input parameters of the operator corresponding to the operator name into the LLM, so that the LLM executes the calculation logic of the operator corresponding to the operator name according to the input parameters and a preset knowledge base to determine the execution result of the operator; A verification module, used to verify the correctness of the execution result; The running module is further configured to continue executing the statement according to the execution result of the operator when the correctness check passes.

[0008] This specification provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the execution method of the above-mentioned knowledge high-order program HOP.

[0009] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the execution method of the above-mentioned high-level knowledge program HOP is implemented.

[0010] At least one of the above technical solutions adopted in the embodiments of this specification can achieve the following beneficial effects: The embodiment of this specification discloses a method for executing a knowledge high-level program (HOP). The method runs the HOP written in the same preset type of computer program code through the running environment of the preset type of computer program code. When the statement containing the operator name of a special operator in the HOP is run, the pre-trained LLM is called to execute the calculation logic defined by the natural language of the special operator to determine the execution result of the special operator, and the correctness of the execution result is checked. After passing the correctness check, the statement is continued to be executed according to the execution result. The method constrains the overall logical accuracy of the task execution process by converting the task to be executed into the corresponding HOP, and for the specific steps that require knowledge to be executed, they are expressed by special operators defined in natural language, and the special operator is executed by the LLM based on the knowledge base containing the knowledge, and the accuracy of the LLM's execution of the special operator is guaranteed by checking the correctness of the execution result of the special operator, thereby improving the accuracy and reliability of the execution of the entire task. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The exemplary embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation of this specification. In the drawings: Figure 1 A flow chart of a HOP execution method provided in an embodiment of this specification; Figure 2 A flow chart of a method for generating a HOP according to an embodiment of this specification; Figure 3 A schematic diagram of a HOP execution device provided in an embodiment of this specification; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this specification. DETAILED DESCRIPTION

[0012] To make the objectives, technical solutions, and advantages of this specification more clear, the following will clearly and completely describe the technical solutions of this specification in conjunction with the specific embodiments of this specification and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this specification.

[0013] The technical solutions provided by the embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0014] Figure 1The flowchart of a method for executing a high-level knowledge program (HOP) provided in an embodiment of this specification specifically includes the following steps: S100: Running the operating environment of the preset type of computer program code and loading business data.

[0015] In the embodiments of this specification, Figure 1 The device that executes the HOP in the method shown can be any electronic device, such as a computer, a server, or a server cluster composed of multiple servers, etc. For ease of description, the following description only takes the server as an example.

[0016] The HOP described in this specification refers to a predefined type of computer program code, such as Python. The HOP includes operator names that conform to the function name format requirements of the predefined type of computer program code. However, the computational logic of the operator corresponding to the operator name is defined in natural language. These operators are referred to as special operators below.

[0017] The server can first obtain the HOP and run the operating environment of the preset type of computer program code, and then load the business data required by the HOP. For example, when the HOP is Python code, the server can pull up the Python operating environment and load the business data.

[0018] The HOP is generated by the LLM based on a business logic process corresponding to a certain business. The business logic process is described in natural language. The generation process of the HOP will be described later.

[0019] S102: Based on the service data, run the HOP through the running environment.

[0020] In the embodiments of this specification, the framework code of the HOP is written by the above-mentioned preset type of computer program code. Even if the code contains special operators, the operator name of the special operator also conforms to the function name format requirements of the preset type of computer program code. Therefore, from a formal point of view, the HOP is essentially the preset type of computer code. The server can run the HOP based on the loaded business data, using the business data as the input parameter of the entire HOP, and by pulling up the running environment of the preset type of computer program code.

[0021] S104: When the statement containing the operator name is run in the HOP, the pre-trained large language model LLM is called.

[0022] When the server runs through the execution environment and reaches a statement in the HOP that contains the operator name of the special operator, the pre-trained LLM can be called.

[0023] S106: Input the business data and / or the intermediate result obtained by currently running the HOP into the LLM as the input parameters of the operator corresponding to the operator name, so that the LLM executes the calculation logic of the operator corresponding to the operator name according to the input parameters and the preset knowledge base to determine the execution result of the operator.

[0024] Since the calculation logic of the special operator is defined by natural language, the running environment of the preset type of computer program code cannot execute the special operator normally. Therefore, the server can input the business data and the intermediate results obtained by currently running the HOP as the input parameters of the special operator into the LLM, so that the LLM can execute the calculation logic of the special operator according to the input parameters and the knowledge in the preset knowledge base to obtain the execution result of the special operator.

[0025] S108: Check the correctness of the execution result.

[0026] Since the calculation logic of the special operator is defined in natural language and needs to be executed by LLM, in order to avoid inaccurate execution of the special operator defined in natural language by LLM, the correctness of the execution result can be checked to determine whether LLM correctly executes the calculation logic of the special operator defined in natural language.

[0027] S110: When the correctness check passes, continue to execute the statement according to the execution result of the operator.

[0028] If the correctness check is passed, the LLM returns the execution result of the special operator to the execution environment of the preset type of computer program code, and the execution environment continues to execute the statement until the HOP is completely executed.

[0029] Among them, the above Figure 2 The generation process of the HOP shown can be as follows Figure 2 shown. Figure 2 The flow chart of the HOP generation method provided in the embodiment of this specification specifically includes the following steps: S200: Acquire a business logic process described in a natural language.

[0030] In the embodiments of this specification, Figure 2 The device for generating codes by the method shown can be any electronic device, such as a computer, a server, or a server cluster consisting of multiple servers. Figure 1 The devices for executing HOP in the method shown may be the same or different. For the convenience of description, the following description is still based on the same server as an example.

[0031] In order to generate code for executing a certain business, the server can first obtain the business logic process of the business. Specifically, the business logic process described in this specification includes a standard operating procedure (SOP). The SOP is a standard operating procedure for the business described in natural language. It is a document that details the work, workflow, and steps in the business. It provides standard process specifications within the organization of the business for reference and operation by internal personnel of the organization. In addition to containing the standardized process of the business, the SOP also includes clear responsibilities and authorities of each role in the business, emergency measures, quality standards and assessments, document management, and other content.

[0032] In the embodiments of this specification, the code generation method provided in this specification is described below by taking the above-mentioned business as the risk control business as an example.

[0033] When the above-mentioned business is a risk control business based on the recorded risk log, the business logic process of the risk control business (that is, the risk control logic process) can be described in the following natural language: The first step is to analyze the cmd field to determine whether the cmd field grants the file executable permission. If the cmd command line does not grant the file executable permission, the conclusion is that no action is required. The second step is to determine whether the installation package is a commonly used software installation package. If it is a commonly used installation software, the conclusion is that no action is required; The third step is to determine whether the cmd field has been manually analyzed and judged by security operations personnel within the past N days and determined to require no action. For example, if the chmod command line is judged to require no action, then no action is required for this risk log. The fourth step is to determine whether the risk log does not need to be handled or generates an attack alarm based on the results of the above three steps.

[0034] The risk log is a risk log of other businesses recorded by the user when performing other businesses, and the risk control business is a business that requires risk control of the other businesses based on the risk log.

[0035] S202: Input the business logic process into a pre-trained large language model LLM.

[0036] After the server obtains the business logic process described above, it can input the business logic process described in natural language into a pre-trained LLM. This LLM can be pre-deployed on the server, or it can be deployed on other devices. If deployed on other devices, the server can send the business logic process to the other devices, allowing them to input the received business logic process into the LLM. The following description uses the LLM deployed on the server as an example.

[0037] In order to avoid the LLM from generating "hallucinations" and code that is inconsistent with the actual business logic process, and to improve the accuracy and reliability of the LLM generated code, in the embodiment of this specification, the server directly outputs the actual business logic process obtained in step S200 to the LLM, so that the LLM generates code for executing the business based on the business logic process.

[0038] S204: Identify the steps included in the business logic process and the logical relationships between the steps through the LLM.

[0039] Generally, a business logic process consists of several steps arranged according to certain logical relationships. For example, in the example above, the first step is executed first. If the first step evaluates to "no," the result is output directly, indicating no action is required. If the result is "yes," the second step is executed. If the second step evaluates to "yes," the result is output directly, indicating no action is required. Otherwise, the third step is executed. If the third step evaluates to "yes," the result is output directly, indicating no action is required. Otherwise, an attack alert is generated. Therefore, the logical relationships between the different steps in a business logic process constitute the basic framework of the process. Specifically, the execution of each step requires the use of knowledge specific to the business logic process, including the concepts and attributes of the entities required in the process, as well as the relationships between them. For example, in the example above, the first step requires determining whether the "cmd" field grants executable permissions to a file. The knowledge required to execute this first step includes at least the concept of the "cmd" field, the concept of file executable permissions, and the relationship between the "cmd" field and file executable permissions (i.e., which "cmd" field indicates that the file has been granted executable permissions, and which "cmd" field indicates that the file has not been granted executable permissions).

[0040] It can be seen from this that a business logic process is essentially composed of the logical relationship between each step and the knowledge required to execute each step. Therefore, to generate code for executing the business logic process, LLM needs to identify the steps contained in the business logic process and the logical relationship between the steps based on the business logic process described in natural language.

[0041] Specifically, since identifying the steps and logical relationships between the steps in a business logic process may also require a knowledge base corresponding to the business logic process, in the embodiments of this specification, the knowledge base corresponding to the business logic process may be pre-injected into the pre-trained LLM. The knowledge base may include at least one of the following: The knowledge graph corresponding to the business logic process includes the concept description of each entity required in the business logic process (used to describe what the entity is, such as the concept of cmd command line in the above example), the attributes of each entity (used to describe all possible attributes of the entity in the business logic), and the relationship between each entity; The calling interface of the tool required in the business logic process includes API or SDK, etc. For example, in the second step of the above example, it is necessary to determine whether the installation package is an installation package of commonly used software. In this case, it is necessary to call the tool to obtain a list of installation packages of commonly used software. This requires calling the tool through the tool's calling interface.

[0042] After injecting the above-mentioned knowledge base containing the knowledge graph corresponding to the business logic process and / or the calling interface of the required tools into the pre-trained LLM, when the server inputs the business logic process described in natural language to the LLM, the LLM can identify the semantics of the business logic process described in the natural language based on its own reasoning ability, and map the concepts in the business logic process into entities in the knowledge graph. Then, based on the semantics, the entire business logic process is split into several steps, and finally the logical relationship between the steps is inferred.

[0043] S206: Based on the logical relationship, generate a preset type of computer program code for representing the logical relationship between the steps as a framework code.

[0044] Because natural language itself has a certain degree of ambiguity and ambiguity, which is an inherent defect of any natural language, directly using natural language to express the logical relationship between the steps in the business logic process is very likely to lead to logical confusion among the steps. In particular, when people other than the person who wrote the business logic process read the business logic process, the ambiguity and ambiguity of natural language may lead to inaccurate understanding, which will affect the business itself.

[0045] Computer programming language, as a formalized and precise language, has clear semantics and strict logical structure, and can effectively avoid the above-mentioned problems of natural language. Therefore, in the embodiment of this specification, for the logical relationship between each step in the business logic process, the server can generate a preset type of computer program code for representing the logical relationship between each step as a framework code through LLM based on the logical relationship between each step identified in step S204.

[0046] The preset type of computer program code may be any type of computer program code, including but not limited to Python code. The following description will only take Python code as an example.

[0047] S208: For each step, generate at least one operator for executing the step according to the natural language used to describe the step in the business logic process and a preset knowledge base corresponding to the business logic process.

[0048] In the embodiment of this specification, the execution order of step S206 and step S208 may not be particular. Specifically, the server may execute step S208 through the LLM while executing step S106 through the LLM.

[0049] As for the knowledge required for each step in the business logic process, natural language is more natural and more understandable than computer programming language. Therefore, in the embodiments of this specification, an operator defined in natural language is used to represent a step.

[0050] The computational logic of the operators described in the embodiments of this specification is defined in natural language, but the operator names of these operators still conform to the function name format requirements of the aforementioned predefined type of computer program code. Therefore, the operators defined in natural language in this specification are essentially custom functions within the predefined type of computer program code, which are high-level abstract expressions of the logic that implements the steps.

[0051] Specifically, for each step identified in the business logic process, the LLM can determine the knowledge required to execute the step, such as one or more entities in the knowledge graph, one or more attributes of the entity, the relationship between the entity and other entities, and / or the calling interface of the tool required for the step, based on the semantics of the natural language used to describe the step in the identified business logic process and the knowledge base corresponding to the business logic process. After determining the knowledge required to execute the step, at least one operator for executing the step can be generated based on the knowledge. The operator is defined in natural language and its name conforms to the function name format requirements of the aforementioned preset type of computer program code.

[0052] Continuing with the previous example, the first step is to determine whether the cmd field grants the file executable permission. Therefore, LLM can generate an operator name hop_judge that meets the Python code function name requirements and define the operator hop_judge as follows: def hop_judge Determine whether {cmd} grants file executable permissions As can be seen, the definition of the operator hop_judge is completely defined using natural language, "Judge whether {cmd} grants file executable permissions," making the operator more understandable. Operators that use natural language to define computational logic are special operators in HOP.

[0053] The above is just an example of expressing a step with one operator. Those skilled in the art should understand that in actual applications, for more complex steps, more operators (such as more than two operators) can be used to express the step.

[0054] S210: Generate a knowledge high-level program for executing the business logic process according to the framework code and the operator name of the operator generated for each step.

[0055] As explained above, a business logic process is essentially composed of the logical relationships between the steps and the knowledge required to execute each step. The logical relationships have been expressed through the generated framework code in step S206, and the knowledge required to execute each step has been expressed through the operators defined in natural language and whose operator names still conform to the preset type of computer program code function name format requirements in step S208. Therefore, LLM can combine the framework code and the operators corresponding to each step to generate a knowledge high-level program (HOP) for executing the business logic process.

[0056] Continuing with the above example, LLM combines the operators corresponding to each step into the framework code to generate the following HOP for executing the above risk control logic process: def evaluate(cmd): if hop_judge_a("Judge (cmd) whether to grant file executable permissions") #High-level judgment pkg = hop_get("Get the installation package used by (cmd)") if not pkg in hop_knowledge_retrieve("Common Installation Package") #High-level knowledge concept matching if not hop_judge_b("Check the history to see if (cmd) has been manually judged as not requiring action") return "Attack Alarm" return "No need to dispose" In the HOP above, the content enclosed by ("") is the natural language definition of the corresponding operator. These operators include hop_judge_a, hop_get, hop_knowledge_retrieve, and hop_judge_b. The framework code expressed in Python above shows the entire risk control logic process as follows: first, execute the hop_judge_a operator to determine whether (cmd) has file executable permissions. If so, execute the hop_get operator, obtain the installation package used by (cmd), and assign it to pkg. If not, directly output "No action required." After obtaining the installation package used by (cmd) and assigning it to pkg, execute the hop_knowledge_retrieve operator to obtain the commonly used installation package and determine whether pkg is not among the commonly used installation packages. If so, execute the hop_judge_b operator. If not, directly output "No action required." Execute the hop_judge_b operator to determine whether (cmd) has not been manually determined to require action in the past. If so, output "Attack Alarm"; otherwise, output "No action required."

[0057] It can be seen that the HOP represented by the Python code is completely consistent with the original intention of the risk control logic process described in natural language. Therefore, through the HOP, even if there is a change in the personnel in the department that performs the risk control business, when the HOP is read by other personnel (other than the personnel who wrote the original risk control logic process), the logical confusion of each step caused by the ambiguity and ambiguity of the natural language itself can be avoided. Moreover, although the special operators in the above HOP code are defined by natural language, the operator names are still function names in the Python code format. Therefore, the HOP code itself is composed of Python framework code referencing various Python-formatted operator names. From a formal point of view, it is still Python code. Therefore, the HOP code is not only readable by humans, but can also be used Figure 1 The method shown is directly executed in the Python execution environment to implement the above risk control logic process.

[0058] In the embodiments of this specification, in order for the LLM to accurately convert the business logic process described in the input natural language into the above-mentioned HOP, it is not enough to simply inject the knowledge base corresponding to the business logic process into the LLM. The LLM also needs to be fine-tuned and trained in the business scenario corresponding to the business logic process in advance.

[0059] Specifically, when fine-tuning the LLM, you can first obtain a sample logical process described in natural language. This sample logical process also includes several steps, referred to as sample steps below. This sample logical process and the aforementioned business logic process are business logic processes in the same business scenario. That is, the knowledge base corresponding to the sample logical process is exactly the same as the knowledge base corresponding to the aforementioned business logic process.

[0060] After obtaining the sample logic process, you can also Figure 2 In the steps S202 to S210 shown, the sample logic process is input into the LLM to be trained, and the sample steps contained in the sample logic process and the logical relationship between the sample steps are identified by the LLM to be trained. According to the logical relationship between the sample steps, a preset type of computer program code for representing the logical relationship between the sample steps is generated as the framework code to be optimized. Moreover, for each sample step, according to the natural language used to describe the sample step in the sample logic process and the knowledge base corresponding to the preset sample logic process, at least one operator for executing the sample step is generated as the operator to be optimized. The calculation logic of the operator to be optimized is also defined in natural language, and the operator name of the operator to be optimized also conforms to the function name format of the above-mentioned preset type of computer program code. Finally, according to the framework code to be optimized and the operator name of the operator to be optimized generated for each sample step, a HOP to be optimized for executing the sample logic process is generated. This process is the same as Figure 2 Steps S202 to S210 are identical and will not be described again here.

[0061] After the LLM to be trained generates the HOP to be optimized, the optimized HOP after adjusting the HOP to be optimized can be determined. Among them, since the HOP to be optimized is also the above-mentioned preset type of computer program code (such as Python code) in form, the HOP to be optimized can be manually adjusted so that the logical relationship between the sample steps expressed by the framework code to be optimized is completely consistent with the original sample logical process, and the natural language corresponding to the operator to be optimized defined by natural language contained therein is more in line with the knowledge in the knowledge base corresponding to the sample logical process, and the calculation logic of the operator to be optimized expressed is clearer and more accurate. Of course, the HOP to be optimized can also be adjusted by other trained LLMs, and this embodiment of the specification does not limit this.

[0062] After obtaining the optimized HOP, the supervised fine-tuning training (SFT) of the LLM to be trained can be performed based on the optimized HOP, that is, the optimized HOP is used as the label corresponding to the sample logic process, and used as the supervision signal to adjust the model parameters of the LLM to be trained, so that the model parameters of the LLM to be trained are adjusted to make the LLM adapt to generate various business logic processes in the business scenario. Figure 2 The device used for code generation can be the same device or a different device. Furthermore, since the embodiments of this specification require that the original reasoning capabilities of the LLM be maintained as much as possible and adapted only to the business scenario, when performing SFT on the LLM, not all model parameters of the LLM are adjusted; instead, only some of the model parameters within the LLM are adjusted. The adjusted model parameters may only include the LoRA model parameters within the LLM.

[0063] After the LLM is obtained through the above SFT training, the server can Figure 2 The method shown generates the HOP corresponding to the business logic process in this business scenario. Figure 1 When the method shown executes the HOP, since the HOP not only includes the above-mentioned preset type of computer program code (such as Python), but also includes special operators defined by natural language, the LLM obtained by the above-mentioned training is also required to assist in the execution when executing the HOP.

[0064] When the above-mentioned business scenario is a risk control business scenario, that is, the above-mentioned business logic process is a risk control logic process under the risk control business scenario, the loaded business data can be a pre-recorded risk log, which is a risk log of other businesses recorded by the user when performing other businesses. The risk control business is a business that needs to perform risk control on the other businesses based on the risk log.

[0065] pass Figure 1 After the step S100 shown pulls up the operating environment of the preset type of computer program code and loads the business data, the business data can be used to run the program through the operating environment. Figure 2 Generated HOP.

[0066] When running to the special operator defined by natural language in the HOP, it is necessary to call the above-mentioned LLM. Among them, since the framework of the HOP is still a preset type of computer program code, the operating environment using the preset type of computer program code can directly run the HOP. As for the special operator defined by natural language contained in the HOP, since the special operator is essentially a custom function in the preset type of computer program code, the HOP references the function name, which means executing the function corresponding to the function name, obtaining the execution result, and then continuing to execute the subsequent code according to the execution result. Therefore, when running to the operator name of the special operator defined by natural language in the HOP, it is necessary to call the pre-defined calculation logic of the operator. When the calculation logic of the operator is parsed by the operating environment, if it is parsed that the calculation logic is defined by natural language, the operating environment can directly call the above-mentioned LLM.

[0067] After calling the above LLM, you can Figure 1 In step S106 shown, the business data and / or the intermediate results obtained by currently running the HOP are input into the LLM as input parameters of the operator, so that the LLM executes the calculation logic of the operator according to the input parameters and the knowledge base corresponding to the above-mentioned business logic process to determine the execution result of the operator.

[0068] Continuing with the above example, for the operator hop_judge_a in the above HOP, its natural language defined calculation logic is "Judge whether (cmd) grants file executable permissions." In this case, business data, that is, the pre-recorded risk log, can be used as the operator's input parameter. The operator's natural language defined calculation logic and the input parameter are both input into the LLM. Prompt information can also be input into the LLM. Under the guidance of the prompt information, the LLM first identifies the cmd command line in the input parameter (i.e., the entity in the knowledge graph) based on the knowledge base corresponding to the risk control logic process. Then, based on the knowledge base and the identified cmd command line, the LLM determines the attributes of the cmd command line in the input parameter (i.e., the attributes of the entity in the knowledge graph). Based on the attributes, the LLM determines whether the cmd command line grants file executable permissions (i.e., the relationship between different entities in the knowledge graph). The judgment result is used as the output parameter of the operator hop_judge_a and returned to the Python runtime environment, allowing the Python runtime environment to continue executing subsequent code in the HOP.

[0069] For another example, for the operator hop_knowledge_retrieve in the above HOP, the calculation logic defined in natural language is "common installation packages". The calculation logic defined in natural language of the operator can be input into LLM, and prompt information can be input to LLM, so that the LLM, under the guidance of the prompt information, first determines the tool for obtaining the list of commonly used installation packages based on the knowledge base corresponding to the risk control logic process, and then calls the tool through the calling interface of the tool to obtain the commonly used installation packages through the called tool. Finally, the obtained list of commonly used installation packages is used as the output parameter of the operator hop_knowledge_retrieve and returned to the Python runtime environment, so that the Python runtime environment continues to execute the subsequent code in the HOP to determine whether the intermediate result pkg of the current running of the HOP is not included in the list of commonly used installation packages.

[0070] Those skilled in the art will appreciate that the above description uses the risk control logic process as an example. In reality, the HOP generation methods provided in the embodiments of this specification can be used to generate and execute HOPs corresponding to business logic processes in any business scenario. Furthermore, the aforementioned preset computer program code is described using Python code as an example; the preset computer program code described in the embodiments of this specification can also be other types of computer program code.

[0071] exist Figure 1 In step S108 shown, since the calculation logic of the special operator is defined through natural language and the LLM may have "hallucinations", in order to ensure the accuracy of the LLM's execution of the special operator, the embodiment of this specification also needs to perform a correctness check on the execution result obtained by the LLM executing the special operator. If the correctness check passes, the LLM can return the execution result to the operating environment of the above-mentioned preset type of computer program code, so that the operating environment continues to execute the HOP until the entire HOP is executed. If the correctness check fails, the calculation logic of the special operator can be re-executed by the LLM to re-determine the execution result of the special operator until the execution result passes the correctness check, or the number of times the LLM executes the calculation logic of the special operator reaches a preset number.

[0072] If the number of times that the LLM executes the calculation logic of the special operator reaches a preset number, and the execution result of the special operator determined for the last time still fails to pass the correctness check, an assistance execution request can be generated by the LLM and output. The assistance execution request is used to request manual assistance to obtain the execution result of the special operator. The assistance execution request can be sent to a user device, and the user can input the execution result of the special operator to the user device according to the calculation logic and input parameters of the special operator carried in the assistance execution request, and the user device returns the execution result input by the user to the LLM. The LLM receives the execution result returned by the user device and uses the received execution result as an execution result that has passed the correctness check. It is also possible to perform supervised fine-tuning training on at least part of the model parameters in the LLM based on the calculation logic of the special operator and the execution result received from the user, such as only performing supervised fine-tuning training on the model parameters of LoRA in the LLM.

[0073] When performing the correctness check on the execution result of the special operator obtained by the LLM, the correctness check may be performed on at least one of the following three aspects: In a first aspect, the execution result obtained by the LLM can be verified for correctness according to a preset result verification rule. This first aspect may specifically include: using another LLM to execute the special operator according to the calculation logic and input parameters of the special operator and the knowledge base corresponding to the above-mentioned business logic process to obtain an execution result; if the execution result obtained by the LLM in step S106 is the same as the execution result obtained by the other LLM, then determining that the correctness verification of the execution result obtained by the LLM itself has passed; otherwise, the correctness verification has failed; and / or using the LLM to re-execute the calculation logic of the special operator; if the execution result obtained by the LLM in step S106 is the same as the execution result obtained by re-executing the special operator, then determining that the correctness verification of the execution result obtained by the LLM itself has passed; otherwise, the correctness verification has failed; and / or determining whether the execution result obtained by the LLM in step S106 falls within a preset range for the special operator; if so, determining that the correctness verification of the execution result obtained by the LLM itself has passed; otherwise, the correctness verification has failed.

[0074] Of course, the correctness verification method of the first aspect mentioned above may also include other verification methods, which will not be described one by one in this specification.

[0075] Secondly, the LLM can determine the computational process by which it obtains the execution result and perform a correctness check on the computational process according to preset computational process verification rules. This second aspect may specifically include: when the execution result of the special operator is a numerical value, verifying the correctness of the execution result obtained by the LLM in step S106 using a mathematical inverse operation based on the computational process by which the LLM obtains the execution result; and / or, using the LLM, based on the semantics of the computational logic defined by the special operator in natural language, determining the semantic key points in the computational logic, and determining whether the computational process by which the LLM obtains the execution result includes all semantic key points. If not, the correctness check fails.

[0076] Of course, the correctness verification method of the second aspect mentioned above may also include other verification methods, which will not be described one by one in this specification.

[0077] Third, determine the input and output formats when the LLM executes the calculation logic of the special operator, and perform correctness verification on the input and output formats according to preset format verification rules. Specifically, the input and output formats of the special operator can be preset using JSONSchema, and based on the input format preset by JSON Schema, determine whether the format of the input parameters of the special operator received by the LLM is correct. Based on the output format preset by JSON Schema, determine whether the format of the execution result of the special operator obtained by the LLM is correct.

[0078] In order to ensure the accuracy of LLM in executing the special operator, the correctness checks of the above three aspects can be performed simultaneously, and more than two verification methods can be selected for the verification methods of the above first aspect and the second aspect.

[0079] The above is a method for executing HOP provided in an embodiment of this specification. Based on the same idea, this specification also provides corresponding devices, storage media, and electronic devices.

[0080] Figure 3 This is a schematic diagram of an execution device for a HOP provided in an embodiment of this specification. The HOP is a computer program code of a preset type. The HOP includes an operator name, and the calculation logic of the operator corresponding to the operator name is defined in natural language. The device includes: A loading module 301 is configured to run an operating environment of the preset type of computer program code and load business data; An operating module 302 is configured to operate the HOP through the operating environment based on the service data; A calling module 303 is configured to call a pre-trained large language model LLM when running to a statement containing the operator name in the HOP; An operator execution module 304 is configured to input the business data and / or the intermediate result obtained by currently running the HOP as input parameters of the operator corresponding to the operator name into the LLM, so that the LLM executes the calculation logic of the operator corresponding to the operator name based on the input parameters and a preset knowledge base to determine the execution result of the operator; Verification module 305, used to verify the correctness of the execution result; The execution module 302 is further configured to continue executing the statement according to the execution result of the operator when the correctness check passes.

[0081] Optionally, the operator name conforms to the function name format of the preset type of computer program code; The calling module 303 is specifically used to call the calculation logic of the operator corresponding to the operator name when running to the statement containing the operator name in the HOP; when parsing the calculation logic through the running environment and finding that it is defined by natural language, calling the LLM.

[0082] Optionally, the operator execution module 304 is further configured to, when the correctness check fails, re-execute the calculation logic of the operator through the LLM to re-determine the execution result of the operator.

[0083] Optionally, the operator execution module 304 is also used to generate and output an assisted execution request when the LLM executes the calculation logic of the operator for a preset number of times and the execution result of the operator determined for the last time still fails to pass the correctness check; and receive the execution result of the operator input by the user device in response to the assisted execution request as an execution result that has passed the correctness check.

[0084] Optionally, the verification module 305 is specifically used to verify the correctness of the execution result according to a preset result verification rule; and / or, determine the calculation process of the LLM to obtain the execution result, and verify the correctness of the calculation process according to a preset calculation process verification rule; and / or, determine the input parameter format and output parameter format when the LLM executes the calculation logic of the operator, and verify the correctness of the input parameter format and output parameter format according to a preset format verification rule.

[0085] Optionally, the HOP is generated by the LLM according to a business logic process described in a natural language; The knowledge base is the knowledge base corresponding to the business logic process.

[0086] Optionally, the business logic process includes a risk control logic process; The business data includes a pre-recorded risk log; The preset type of computer program code includes Python.

[0087] This specification also provides a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, it can be used to execute the HOP execution method provided above.

[0088] based on Figure 1 The execution method of the HOP shown in the embodiment of this specification also provides Figure 4 The structural diagram of the electronic device shown in FIG. Figure 4 At the hardware level, the electronic device includes a processor, an internal bus, a network interface, memory, and non-volatile storage, and may also include other hardware required for its operations. The processor reads the corresponding computer program from the non-volatile storage into the memory and then runs it, thereby implementing the above-mentioned HOP execution method.

[0089] The foregoing is merely an example of the present invention and is not intended to limit the present invention. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be included within the scope of the claims of the present invention.

Claims

1. A method for executing a high-level knowledge program (HOP), wherein the HOP is a computer program code of a preset type, the HOP includes an operator name, and the calculation logic of the operator corresponding to the operator name is defined in natural language; the method comprises: Running the operating environment of the preset type of computer program code and loading business data; Based on the business data, running the HOP through the operating environment; When running to the statement containing the operator name in the HOP, calling the pre-trained large language model LLM; Input the business data and / or the intermediate result obtained by currently running the HOP as input parameters of the operator corresponding to the operator name into the LLM, so that the LLM executes the calculation logic of the operator corresponding to the operator name according to the input parameters and a preset knowledge base to determine the execution result of the operator; The correctness of the execution result is checked, and when the correctness check passes, the statement is continued to be executed according to the execution result of the operator.

2. The method according to claim 1, wherein the operator name conforms to the function name format of the computer program code of the preset type; When the statement containing the operator name in the HOP is run, the pre-trained LLM is called, specifically including: When running to the statement containing the operator name in the HOP, the calculation logic of the operator corresponding to the operator name is called; When it is found through the operating environment that the computing logic is defined in a natural language, the LLM is called.

3. The method of claim 1, further comprising: When the correctness check fails, the calculation logic of the operator is re-executed through the LLM to re-determine the execution result of the operator.

4. The method of claim 3, further comprising: When the number of times the LLM executes the calculation logic of the operator reaches a preset number, and the last determined execution result of the operator still fails the correctness check, an assistance execution request is generated and output; An execution result of the operator input by the user equipment in response to the execution assistance request is received as an execution result that has passed the correctness check.

5. The method according to claim 1, wherein the correctness check of the execution result specifically comprises: Verify the correctness of the execution result according to the preset result verification rules; and / or Determining the calculation process by which the LLM obtains the execution result, and verifying the correctness of the calculation process according to a preset calculation process verification rule; and / or Determine the input parameter format and output parameter format when the LLM executes the calculation logic of the operator, and perform correctness verification on the input parameter format and output parameter format according to preset format verification rules.

6. The method according to any one of claims 1 to 5, wherein the HOP is generated by the LLM according to a business logic process described in a natural language; The knowledge base is the knowledge base corresponding to the business logic process.

7. The method according to claim 6, wherein the business logic process includes a risk control logic process; The business data includes a pre-recorded risk log; The preset type of computer program code includes Python.

8. A device for executing a high-level knowledge program (HOP), wherein the HOP is a computer program code of a predetermined type, the HOP includes operator names, and the computational logic of the operators corresponding to the operator names is defined in natural language; the device comprises: A loading module, configured to run an operating environment of the preset type of computer program code and load business data; An operating module, configured to operate the HOP through the operating environment based on the service data; A calling module, configured to call a pre-trained large language model LLM when running to a statement containing the operator name in the HOP; An operator execution module is configured to input the business data and / or the intermediate result obtained by currently running the HOP as input parameters of the operator corresponding to the operator name into the LLM, so that the LLM executes the calculation logic of the operator corresponding to the operator name according to the input parameters and a preset knowledge base to determine the execution result of the operator; A verification module, used to verify the correctness of the execution result; The running module is further configured to continue executing the statement according to the execution result of the operator when the correctness check passes.

9. A computer-readable storage medium storing a computer program, wherein the computer program implements the method according to any one of claims 1 to 7 when executed by a processor.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the program.

Images