Agent control behavior tree program automatic generation method and device based on multi-modal guidance

Abstract

The application discloses a kind of multi-modal guide-based intelligent agent control behavior tree program automatic generation method and device, the method steps include: step S1. obtaining the natural language description describing the task execution process to be solved and corresponding positive and negative example regulation;Step S2. according to the generated natural language description, task execution framework is generated using large language model;Step S3. according to the generated task execution framework, search target behavior tree, if it is determined that the execution framework is not feasible, step S4 is entered;Step S4. repair the current execution framework determined to be not feasible, first repair the control flow node of execution framework, if repair is not successful, repair the structure of execution framework.The application can generate intelligent agent control behavior tree program in a more efficient, higher accuracy, user-friendly manner, while improving the efficiency and accuracy of the generated behavior tree program.

Description

Technical Field

[0001] This invention relates to the field of intelligent agent control technology, and in particular to a method and apparatus for automatically generating intelligent agent control behavior tree programs based on multimodal guidance. Background Technology

[0002] Behavior trees, characterized by their structure, modularity, and reactivity, are widely used in intelligent agent control fields such as robot control and autonomous vehicles. However, the highly structured nature of behavior trees makes constructing behavior tree programs that meet requirements challenging. To address this issue, an effective approach is automatic program synthesis, which searches the program space to find behavior tree programs that satisfy the given specifications. Program synthesis is an automatic program generation technique that searches the program space and verifies whether candidate programs conform to a given specification. Its purpose is to automatically generate programs in a domain-specific language that satisfy the specifications. Among these techniques, Example-Guided Synthesis (PBE) is a key technology in program synthesis. It uses examples related to the task as specifications for program synthesis, ensuring that the target program satisfies all example specifications.

[0003] Existing methods for generating behavior tree programs mainly include: behavior tree generation through planning learning, behavior tree generation through genetic algorithms, behavior tree generation based on formal verification techniques, and behavior tree generation through fine-tuning lightweight large language models. However, these methods typically rely on user-provided challenging task-related information as guiding information for behavior tree program synthesis. Planning learning for behavior tree generation usually involves using node information from predefined condition node libraries and action node libraries to construct a behavior tree based on the preconditions and postconditions of action nodes. This allows the agent to transition from an initial state to a target state, which is considered the achievement of the goal. However, this method relies on predefined action node libraries, requiring designers to precisely design each action node and its preconditions and postconditions, resulting in complexity and low efficiency. Genetic algorithms for behavior tree generation rely on the design of fitness functions, continuously generating new programs through crossover, mutation, and selection. The fitness function serves as a heuristic for the search, terminating when a satisfactory solution is found or the fitness function no longer improves. Behavior tree generation methods based on formal verification techniques generate behavior tree programs that satisfy a given Linear Temporal Logic (LTL) requirement. They then use the CSP model verification method to verify whether the behavior tree program satisfies the given LTL requirement, and continuously search the behavior tree program using Monte Carlo Tree Search (MCTS). This type of method relies on the design of the LTL formula, is relatively complex to implement, and requires a long synthesis process. While fine-tuning a lightweight large language model can effectively improve the accuracy of behavior tree programs, it cannot completely avoid the illusion problem inherent in large language models.

[0004] In summary, there is an urgent need to provide a method for automatically generating behavior tree programs for intelligent agents, in order to achieve more efficient, accurate, and user-friendly behavior tree program generation. In particular, for various complex tasks in the real world, it is challenging to provide the necessary inputs for these tasks when using existing behavior tree program generation methods. Furthermore, the increased complexity of the tasks also leads to longer search times when solving these tasks, resulting in serious efficiency problems. Summary of the Invention

[0005] The technical problem to be solved by the present invention is as follows: In view of the above-mentioned problems existing in the prior art, the present invention provides a method and apparatus for automatically generating intelligent agent control behavior tree programs based on multimodal guidance, which is simple to implement, low in cost, efficient and accurate in generation, and can automatically generate intelligent agent control behavior tree programs in a more user-friendly way, while improving the efficiency and accuracy of generating behavior tree programs.

[0006] To solve the above-mentioned technical problems, the technical solution proposed by this invention is as follows: A method for automatically generating intelligent agent control behavior tree programs based on multimodal guidance, comprising the following steps: Step S1. Task Information Input: Obtain a natural language description of the execution process of the task to be solved and the corresponding positive and negative example specifications; Step S2. Task framework generation: Based on the natural language description generated in step S1, a task execution framework containing only behavior tree control flow nodes and structure is generated using a large language model; Step S3. Behavior Tree Search: Search for the target behavior tree based on the task execution framework generated in step S2. If the target behavior tree program cannot be found through the search, it is determined to be an infeasible execution framework and proceeds to step S4. Step S4. Framework Repair: Repair the execution framework that is currently determined to be infeasible. First, repair the control flow nodes of the execution framework. Prioritize searching the control flow nodes in the execution framework. By adjusting the assignment of at least one control flow node, a new execution framework is formed. Search the new execution framework to determine whether the repair is successful. If the repair is unsuccessful, repair the structure of the execution framework. By obtaining all failure kernels in the execution framework that do not meet the positive examples, construct a behavior tree to obtain the failure kernel behavior tree. Merge the failure kernel behavior trees that do not meet the positive examples with the subtrees in the execution framework to form a new behavior tree to replace the corresponding subtrees and form a new behavior tree part of the program. Continue until a part of the program meets all the positive examples to complete the repair of the execution framework structure.

[0007] Furthermore, in step S1, positive examples describe semantic sequences that the behavior tree can generate, and positive examples are behavioral sequences that the target behavior tree can control the agent to generate; negative examples describe semantic sequences that the behavior tree is prohibited from generating, and negative examples are behavioral sequences generated by the target behavior tree.

[0008] Further, in step S3, the target behavior tree is searched using a top-down enumeration search method, including the following steps: Step S311. Based on the type of the hole node, select information of the same type from the predefined node library and fill it in; Step S312. Use the depth-first search method to search downwards from the root node of the partial program syntax tree. If the current position is a hole node during the search, fill in the node information of the same type as the current hole node according to the type of the current node to obtain a set of different partial programs. The partial programs are programs that contain hole nodes. The types of the nodes include condition nodes and action nodes.

[0009] Furthermore, step S3 also includes pruning during the search process using an abstract-based pruning method to remove infeasible parts of the program and their corresponding search space.

[0010] Furthermore, the step of pruning using an abstract-based pruning method includes: Step S321. Upper abstraction: The upper abstraction determines whether the current part of the program can be obtained by enumeration search to satisfy all positive example requirements, that is, whether there is a subsequent program that satisfies the positive example reduction. If the upper abstraction determines that the current part of the program is not feasible, that is, the behavior tree that satisfies all positive examples cannot be obtained by enumeration search, then the current part of the program is pruned. The part of the program is the program containing hole nodes. Step S322. Lower Abstraction: The lower abstraction determines whether the current part of the program can be obtained by enumeration search to find a program that satisfies all the counterexample requirements, that is, whether there is a subsequent program that satisfies the counterexample reduction. If the lower abstraction determines that the program is not feasible, that is, all subsequent programs of the program will produce a certain counterexample, then the current part of the program is pruned.

[0011] Furthermore, in step S4, the step of repairing the control flow node of the execution framework includes: Step S411. Perform a top-down depth-first search for the control flow nodes in the execution framework; Step S412. Adjust the assignments of m control flow nodes in any execution frame to obtain a new set of execution frames, where m≤n, and n is the total number of control flow nodes in the execution frame; Step S413. Search for each new execution framework and determine whether the repair is successful. If the execution framework is successfully repaired, generate the target program; otherwise, proceed to repair the structure of the execution framework.

[0012] Furthermore, in step S4, the step of repairing the structure of the execution framework includes: Step S421. Divide all positive examples into two parts: a set of positive examples that are satisfied by the upper abstraction of the current execution framework and a set of positive examples that are not satisfied by the upper abstraction. Step S422. For each positive example that is not satisfied on the execution framework, find the positive example that is most similar to the current negative example from the set of positive examples that are satisfied on the upper abstract, that is, the positive example whose sum of the lengths of the same prefix and suffix is ​​the largest. Step S423. Based on the most similar positive example found, remove the common prefixes and suffixes of the current non-positive example to obtain the difference between the two, and obtain the failure kernel of the current non-positive example; Step S424. Construct a behavior tree based on the failure kernels that do not currently satisfy the positive example, and use the sequence node as the node information of the failure kernel corresponding to the control flow node and leaf node in the behavior tree. After assigning the information of the failure kernel to the leaf node in turn, the failure kernel behavior tree is obtained. Step S425. Search each control flow node and leaf node in the execution framework, and use a depth-first search strategy to try to connect the failed kernel behavior tree with the subtree formed by the current node and its corresponding child node, merge them into a behavior tree, and replace the subtree formed by the current node and its corresponding child node with the merged behavior tree to obtain a behavior tree part of the program with a new structure. Step S426. Determine whether the partial program obtained from the behavior tree in step S425 can satisfy the current non-satisfied positive examples. If it can, repeat steps S422 to S426 to repair the remaining non-satisfied positive examples until the partial program satisfies all positive examples and no negative examples appear. Search for the target program in the final partial program.

[0013] An automatic generation device for intelligent agent control behavior tree programs based on multimodal guidance, comprising: The task information input module is used to obtain a natural language description of the execution process of the task to be solved and the corresponding positive and negative example reductions. The positive examples are the behavior sequences that the target behavior tree can control the agent to generate, and the negative examples are the behavior sequences generated by the target behavior tree. The task framework generation module is used to generate a task execution framework that contains only behavior tree control flow nodes and structure based on the natural language description generated by the task information input module and a large language model. The behavior tree search module is used to search for the target behavior tree based on the task execution framework generated by the task framework generation module. If the target behavior tree program cannot be found through the search, it is determined to be an infeasible execution framework and the process proceeds to step S4. The framework repair module is used to repair execution frameworks that are currently deemed infeasible. First, it repairs the control flow nodes of the execution framework. It prioritizes searching the control flow nodes in the execution framework and forms a new execution framework by adjusting the assignment of at least one control flow node. It then searches the new execution framework to determine if the repair was successful. If the repair is unsuccessful, it repairs the structure of the execution framework. It constructs a behavior tree by obtaining all failure kernels in the execution framework that do not meet the positive examples, and then merges the behavior trees of each failure kernel that does not meet the positive examples with the subtrees in the execution framework to form a new behavior tree to replace the corresponding subtrees and form a new behavior tree part of the program. This process continues until a part of the program meets all the positive examples, thus completing the repair of the execution framework structure.

[0014] A computer device includes a processor and a memory, the memory being used to store a computer program, and the processor being used to execute the computer program to perform the method described above.

[0015] A computer-readable storage medium storing a computer program that, when executed, implements the method described above.

[0016] Compared with the prior art, the advantages of the present invention are as follows: 1. This invention combines natural language description and positive / negative examples. First, it generates a task execution framework based on a general large language model prompted by the natural language description of the task. Then, it uses example-guided synthesis technology to search the program space, abstracting the properties of the behavior tree in a more user-friendly way. When a target behavior tree program that satisfies the positive / negative example specification cannot be obtained by searching the execution framework, a feasible execution framework is obtained by sequentially repairing the control flow nodes and the structure of the execution framework. Finally, the target program is obtained through searching. This invention can efficiently implement a more user-friendly behavior tree synthesis method by combining the natural language description of the task and positive / negative examples, while improving search efficiency and reducing the difficulty of generating behavior tree programs.

[0017] 2. In the process of repairing the structure of the execution framework, this invention constructs a repair subtree by locating the failed kernels that do not meet the positive examples in some programs. The behavior trees of each failed kernel that does not meet the positive examples are merged with the subtrees in the execution framework to form a new behavior tree to replace the corresponding subtrees and form a new behavior tree part of the program. This makes the generated program able to meet all failed positive examples. It can gradually repair the errors in the execution framework according to the characteristics of the behavior tree execution framework and the relationship between positive and negative examples, effectively improving the accuracy of the comprehensive success of the behavior tree program. It can also expand the structure of the execution framework, thereby expanding the program search space and increasing the probability of generating the target behavior tree program. Attached Figure Description

[0018] Figure 1This is a schematic diagram illustrating the implementation process of the method for automatically generating intelligent agent control behavior tree programs based on multimodal guidance in this embodiment.

[0019] Figure 2 This is a schematic diagram of task information input in a specific application embodiment of the present invention.

[0020] Figure 3 This is a schematic diagram of the target behavior tree program for a task in a specific application embodiment of the present invention.

[0021] Figure 4 This is a schematic diagram illustrating the implementation process of repairing the execution framework structure in a specific application embodiment of the present invention. Detailed Implementation

[0022] The present invention will be further described below with reference to the accompanying drawings and specific preferred embodiments, but this does not limit the scope of protection of the present invention.

[0023] Traditional behavior tree generation methods typically rely on user-provided challenging task-related information as guiding information for behavior tree synthesis. However, program enumeration search requires searching the program space for all candidate programs to satisfy a given specification, which is inefficient and time-consuming, and generating the execution framework is also a challenging task. Natural language descriptions of the task are more designer-friendly and less difficult, effectively guiding the generation of the target behavior tree program. Existing techniques for generating behavior tree programs from natural language descriptions mostly utilize fine-tuning of large language models, but the illusion problem remains unresolved. Furthermore, fixing infeasible frameworks is a key challenge in program synthesis.

[0024] This invention significantly improves search efficiency by leveraging the properties of behavior trees abstracted from behavior sequences, using positive and negative examples to guide the synthesis process, and combining this with partial program feasibility verification. Furthermore, by abstracting the behavior sequences generated by the behavior tree into examples, highly structured behavior tree programs can be simplified into more flattened behavior sequences, effectively reducing the difficulty of generating behavior tree programs. Simultaneously, by combining the natural language description of the task with positive and negative examples, a task execution framework is generated using the task's natural language description, followed by target behavior tree search. Infeasible behavior tree execution frameworks are repaired. First, the control flow nodes of the execution framework are repaired by modifying the assignments of these nodes to correct control flow information errors. If a feasible execution framework cannot be obtained, the structure of the execution framework is repaired based on the failure kernels that do not satisfy the positive examples, ensuring that the resulting program satisfies the failed positive example. This allows for the gradual repair of errors in the execution framework based on its characteristics and the relationship between positive and negative examples, effectively improving the accuracy of behavior tree program synthesis, search efficiency, and success rate. Thus, by combining the natural language description of the task with positive and negative examples, behavior tree programs that satisfy the execution behavior can be generated efficiently.

[0025] like Figure 1 As shown, the steps of the method for automatically generating a behavior tree program for intelligent agent control based on multimodal guidance in this embodiment include: Step S1. Task Information Input: Obtain a natural language description of the execution process of the task to be solved and the corresponding positive and negative example specifications.

[0026] In this embodiment, a natural language description of the task execution process and corresponding positive and negative example specifications are provided based on the task to be solved. Positive examples describe the semantic sequences that the behavior tree must be able to generate, and these positive examples are behavioral sequences that the target behavior tree can control the agent to generate. Negative examples describe the semantic sequences that the behavior tree must prohibit from generating, and these negative examples are behavioral sequences generated by the target behavior tree. This embodiment effectively synthesizes behavior tree programs that satisfy the positive and negative example specifications by applying the Example Guided Synthesis (PBE) technique to the behavior tree program synthesis task.

[0027] This embodiment combines a natural language description of the task with positive and negative examples to form a multimodal specification, guiding the behavior tree synthesis process. The semantics of the behavior tree can be abstracted using behavior sequences. For example, the input information for an automatic charging task like Power is as follows: Figure 2 The description includes a natural language description of the task execution process ("When the robot is in a low battery state, it should check if it is at a charging station or move to a charging station and eventually start charging; otherwise, it continues to perform other tasks") and positive and negative examples, where positive examples are behaviors that the agent must be able to perform, such as positive examples.<batteryLow,goStation,Charge> This indicates that the robot should move to a charging station and recharge when it is low on battery; a counterexample would be to prohibit the robot from performing certain actions at any time, such as the counterexample. <shutoff>This indicates that the task requires the robot to remain powered on at all times. The two input items mentioned above (natural language description and positive / negative examples) can be determined simply by obtaining information related to task execution, thus reducing the difficulty of providing input.

[0028] Step S2. Task framework generation: Based on the natural language description generated in step S1, a task execution framework containing only behavior tree control flow nodes and structures is generated using a large language model.

[0029] In this embodiment, based on the task-related natural language description provided in step S1, a general large language model is used to understand the task execution process, generating a task execution framework that contains only behavior tree control flow nodes (fallback nodes, sequence nodes) and structural information. That is, the execution framework generated by the large language model is a behavior tree program with structural information, containing only control flow nodes and unknown leaf nodes. Based on this execution framework, searching and filling are performed to obtain the target behavior tree program. Specifically, few-shot prompting and language pattern techniques can be combined to improve the accuracy of the large language model's task understanding and the correctness of the execution framework.

[0030] Step S3. Behavior Tree Search: Search for the target behavior tree based on the task execution framework generated in step S2. If the target behavior tree program cannot be found through the search, it is determined to be an infeasible execution framework and proceeds to step S4.

[0031] In this embodiment, the unknown information (hole nodes) contained in the execution framework is continuously filled into the program space corresponding to the search execution framework, and finally the target behavior tree program is obtained.

[0032] In this embodiment, a top-down enumeration search method is specifically used to search the target behavior tree, and the steps include: Step S311. Based on the type of the hole node, select information of the same type from the predefined node library and fill it in; Step S312. Use the depth-first search method to search downwards from the root node of the partial program syntax tree. When the search encounters a hole node, fill in the node information of the same type as the hole node, i.e., condition node information or action node information, according to the type of the current hole node (condition or action), to obtain a set of different partial programs. The partial programs are programs that contain hole nodes.

[0033] This embodiment also includes pruning based on abstraction during the search process to remove infeasible parts of the program and their corresponding search space. Since feasibility cannot be directly determined based on partial programs, this embodiment uses the abstraction of partial programs to determine their feasibility. By using pruning techniques, the enumeration search can be accelerated, further improving the efficiency and success rate of behavior tree program search.

[0034] In this embodiment, the steps of pruning using an abstract-based pruning method include: Step S321. Upper abstraction: The upper abstraction determines whether the current part of the program can be obtained by enumeration search to satisfy all positive example requirements, that is, whether there is a subsequent program that satisfies the positive example reduction. If the upper abstraction determines that the current part of the program is not feasible, that is, the behavior tree that satisfies all positive examples cannot be obtained by enumeration search, then the current part of the program is pruned. The part of the program is the program containing hole nodes. Step S322. Lower Abstraction: The lower abstraction determines whether the current part of the program can be obtained by enumeration search to find a program that satisfies all the counterexample requirements, that is, whether there is a subsequent program that satisfies the counterexample reduction. If the lower abstraction determines that the program is not feasible, that is, all subsequent programs of the program will produce a certain counterexample, then the current part of the program is pruned.

[0035] This embodiment accelerates the enumeration search process by employing pruning techniques based on abstraction. The pruning process includes upper and lower abstraction of a portion of the program. The upper abstraction of the portion of the program determines whether a behavior tree program that satisfies all positive examples can be generated through enumeration search based on the type of hole nodes in the portion of the program. If not, it can be safely pruned. The lower abstraction extracts the sequences that have already been generated in the portion of the program and determines whether all subsequent programs of that portion of the program will generate these sequences. If a negative example has appeared, it can be safely pruned.

[0036] Specifically, the upper abstraction of a partial program is all the properties of all possible subsequent programs of that partial program. Combining the semantics and reduction characteristics of behavior trees, this embodiment uses the upper abstraction to determine whether the partial program can obtain a program that satisfies all positive example requirements through enumeration search, that is, whether there is a subsequent program that satisfies the positive example reduction. If the upper abstraction determines that the partial program is infeasible, that is, it is impossible to obtain a behavior tree that satisfies all positive examples through enumeration search, then the partial program can be safely pruned. The lower abstraction of a partial program is the common property of all possible subsequent programs of that partial program. Combining the semantics and reduction characteristics of behavior trees, this embodiment uses the lower abstraction to determine whether the partial program can obtain a program that satisfies all negative example requirements through enumeration search, that is, whether there is a subsequent program that satisfies the negative example reduction. If the lower abstraction determines that the partial program is infeasible, that is, all subsequent programs of the partial program will produce a certain negative example, then the partial program can be safely pruned.

[0037] In specific application embodiments Figure 2 The target behavior tree program obtained from the task shown is as follows: Figure 3 As shown, behavior tree and Figure 2 The execution process of the tasks is consistent. The Sequence and Fallback nodes are the control flow nodes of the behavior tree. As non-leaf nodes, the Sequence node controls the execution of its child nodes from left to right. It can only control the execution of the right subtree when the left subtree returns to a success state. Once a child node returns to a running or failed state, the Sequence node immediately returns to that state. The Fallback node controls the execution of its child nodes from left to right. It can only control the execution of the right subtree when the left subtree returns to a failure state. Once a child node returns to a running or successful state, the Fallback node immediately returns to that state. The gray elliptical dashed box nodes are the condition nodes of the behavior tree, determining the agent's state information at a certain moment, such as `batteryLow` determining whether the agent is in a low battery state. The blue rectangular nodes represent the action nodes of the behavior tree, controlling the agent to execute predefined actions or functions, such as `Charge` controlling the agent to charge.

[0038] Step S4. Framework Repair: Repair the execution framework that is currently determined to be infeasible. First, repair the control flow nodes of the execution framework. Prioritize searching the control flow nodes in the execution framework. By adjusting the assignment of at least one control flow node, a new execution framework is formed. Search the new execution framework to determine whether the repair is successful. If the repair is unsuccessful, repair the structure of the execution framework. By obtaining all failure kernels in the execution framework that do not meet the positive examples, construct a behavior tree to obtain the failure kernel behavior tree. Merge the behavior trees of each failure kernel that does not meet the positive examples with the subtrees in the execution framework to form a new behavior tree to replace the corresponding subtrees and form a new behavior tree part of the program. Continue until a part of the program meets all the positive examples to complete the repair of the execution framework structure.

[0039] Because general-purpose large language models are difficult to train specifically for behavior tree-related content and are susceptible to the illusion of large language models, their accuracy in generating behavior tree execution frameworks is limited, which may lead to errors in the execution framework. This embodiment repairs infeasible execution frameworks by first repairing the control flow nodes of the execution framework. If the execution framework repair is successful, the target program is generated. If the repair fails, the structure of the execution framework is repaired. This can gradually correct the errors in the execution framework and improve the success rate of synthesizing the target program.

[0040] In this embodiment, the steps for repairing the control flow node of the execution framework include: Step S411. Perform a top-down depth-first search for the control flow nodes in the execution framework; Step S412. Adjust the assignments of m control flow nodes in any execution frame to obtain a new set of execution frames, where m≤n, and n is the total number of control flow nodes in the execution frame; Step S413. Search for each new execution framework and determine whether the repair is successful. If the execution framework is successfully repaired, generate the target program; otherwise, proceed to repair the structure of the execution framework.

[0041] Specifically, during the repair process of the control flow nodes of the execution framework, the control flow nodes in the execution framework are first searched from top to bottom in a depth-first manner. If there are n control flow nodes in the execution framework, the assignment of any m (m≤n) control flow nodes is changed to obtain a set of execution frameworks. Then, these execution frameworks are searched. For example, the enumeration search and pruning techniques in step S3 can be used for the search. If the execution framework is successfully repaired, the target program is generated. If the repair is unsuccessful, the structure of the execution framework is further repaired.

[0042] In example-guided synthesis, negative example-guided synthesis is a key technique for improving success rate and efficiency. This embodiment, in the process of repairing infeasible behavior tree execution frameworks, first corrects control flow errors by modifying the assignments of control flow nodes in the execution framework. If a feasible execution framework cannot be obtained, the failure kernel of the failed positive example is located based on the abstract set of satisfied positive examples on the execution framework. The structure of the execution framework is then repaired based on this failure kernel, ensuring that the resulting program satisfies the failed positive example. This achieves efficient repair and avoids overfitting, further improving the success rate of behavior tree program generation.

[0043] In this embodiment, the steps for repairing the structure of the execution framework include: Step S421. Divide all positive examples into two parts: a set of positive examples that are satisfied by the upper abstraction of the current execution framework and a set of positive examples that are not satisfied by the upper abstraction. Step S422. For each positive example that is not satisfied on the execution framework, find the positive example that is most similar to the current negative example from the set of positive examples that are satisfied on the upper abstract, that is, the positive example whose sum of the lengths of the same prefix and suffix is ​​the largest. Step S423. Based on the most similar positive example found, remove the common prefixes and suffixes of the current non-positive example to obtain the difference between the two, and obtain the failure kernel of the current non-positive example; Step S424. Construct a behavior tree based on the failure kernels that do not currently satisfy the positive example, and use the sequence node as the node information of the failure kernel corresponding to the control flow node and leaf node in the behavior tree. After assigning the information of the failure kernel to the leaf node in sequence, the failure kernel behavior tree is obtained. Step S425. Search each control flow node and leaf node in the execution framework, and use a depth-first search strategy to try to connect the failed kernel behavior tree with the subtree formed by the current node and its corresponding child node, merge them into a behavior tree, and replace the subtree formed by the current node and its corresponding child node with the merged behavior tree to obtain a behavior tree part of the program with a new structure. Step S426. Determine whether the partial program obtained from the behavior tree in step S425 can satisfy the current non-satisfied positive examples. If it can, repeat steps S422 to S426 to repair the remaining non-satisfied positive examples until the partial program satisfies all positive examples and no negative examples appear. Search for the target program in the final partial program.

[0044] Specifically, firstly, all positive examples are divided into two parts: a set of positive examples that satisfy the upper abstraction of the execution framework and a set of positive examples that do not satisfy the upper abstraction. For each positive example that does not satisfy the upper abstraction of the execution framework, the most similar positive example is found from the set of positive examples that satisfy the upper abstraction, i.e., the sum of the lengths of the common prefixes and suffixes of the two is maximized. Then, based on the most similar satisfied positive example, the common prefixes and suffixes of the current unsatisfied positive example are removed to obtain the difference between the two, i.e., the failure kernel of the current unsatisfied positive example. A behavior tree is constructed based on the failure kernel of the current unsatisfied positive example. The control flow nodes of this behavior tree are sequence nodes, and the leaf nodes correspond to the node information of the failure kernel. The information of the failure kernel is assigned to the leaf nodes in sequence. This behavior tree is the failure kernel behavior tree. Then, the search execution framework is... For each control flow node and leaf node, a depth-first search strategy is used to attempt to connect the failed core behavior tree with the subtree formed by the current node and its child nodes using fallback nodes, merging them into a behavior tree. This behavior tree then replaces the subtree formed by the current node and its child nodes, resulting in a behavior tree partial program with a new structure. Then, the abstraction technique in step S32 is used to determine whether the newly generated behavior tree partial program can satisfy the current non-satisfied positive examples. If it satisfies the positive example, the above steps are continued to repair other non-satisfied positive examples until the partial program satisfies all positive examples and no negative examples appear. Finally, for the partial program that satisfies all positive examples and no negative examples appear after repair, the enumeration search and abstract pruning technique in step S3 is used to continue searching for the target program.

[0045] In a specific application embodiment, the process of repairing the structure of the execution framework and the final behavior tree are as follows: Figure 4 As shown, the set of positive examples is divided into a set of positive examples that are abstractly satisfied and a set of positive examples that are not satisfied in the current part of the program. A failed positive example is selected from the set of failed positive examples, and the reason why the failed positive example is not satisfied is learned, i.e., the failure kernel. The failure kernel is constructed into a subtree, where the control flow nodes are all Sequence nodes, the leaf nodes are composed of the failure kernel, and "□" represents a hole node. Figure (a) shows the initial infeasible sketch, and (b) shows the partial behavior tree program obtained by merging the failure kernel subtree with the v1 node in (a) in step S425. The content in the blue dashed box is the part obtained by merging the failure kernel subtree and the v1 node using Fallback. (c) is a partial behavior tree obtained by merging the failed kernel tree with the v3 node in (a) in step S425. This partial behavior tree is infeasible, and the search for other nodes will continue to be performed for merging. (d) is a partial behavior tree obtained by merging the failed kernel tree with the v5 node in (a) in step S425. This partial behavior tree is feasible, and the search for the target program will continue according to the enumeration search and abstract pruning techniques in step S3.

[0046] Furthermore, if steps S41 and S42 fail to repair the failed execution framework, the large language model can be invoked again, providing the current failed execution framework to the large language model and requesting the generation of a new execution framework.

[0047] This embodiment also provides an automatic generation device for intelligent agent control behavior tree programs based on multimodal guidance, including: The task information input module is used to obtain a natural language description of the execution process of the task to be solved and the corresponding positive and negative example reductions. The positive examples are the behavior sequences that the target behavior tree can control the agent to generate, and the negative examples are the behavior sequences generated by the target behavior tree. The task framework generation module is used to generate a task execution framework that contains only behavior tree control flow nodes and structure based on the natural language description generated by the task information input module and a large language model. The behavior tree search module is used to search for the target behavior tree based on the task execution framework generated by the task framework generation module. If the target behavior tree program cannot be found through the search, it is determined to be an infeasible execution framework and the process proceeds to step S4. The framework repair module is used to repair execution frameworks that are currently deemed infeasible. It first repairs the control flow nodes of the execution framework, prioritizing the search of control flow nodes in the execution framework. By adjusting the assignment of at least one control flow node, a new execution framework is formed. The new execution framework is then searched to determine whether the repair was successful. If the repair is unsuccessful, the structure of the execution framework is repaired. By obtaining all failure kernels in the execution framework that do not meet the positive examples, a behavior tree is constructed to obtain the failure kernel behavior tree. The behavior trees of each failure kernel that does not meet the positive examples are merged with the subtrees in the execution framework to form a new behavior tree to replace the corresponding subtrees and form a new behavior tree part of the program. This process continues until a part of the program meets all the positive examples, thus completing the repair of the execution framework structure.

[0048] The automatic generation device for intelligent agent control behavior tree program based on multimodal guidance in this embodiment corresponds one-to-one with the above-mentioned automatic generation method for intelligent agent control behavior tree program based on multimodal guidance, and will not be described in detail here.

[0049] This embodiment further provides a computer device, including a processor and a memory, the memory for storing a computer program and the processor for executing the computer program to perform the method as described above.

[0050] It is understood that the method described in this embodiment can be executed by a single device, such as a computer or server, or it can be applied to a distributed scenario where multiple devices cooperate to complete the task. In a distributed scenario, one of the multiple devices may execute only one or more steps of the method described in this embodiment, and the multiple devices interact to complete the method. The processor can be implemented using a general-purpose CPU, microprocessor, application-specific integrated circuit, or one or more integrated circuits, and is used to execute relevant programs to implement the method described in this embodiment. The memory can be implemented using read-only memory (ROM), random access memory (RAM), static storage devices, and dynamic storage devices. The memory can store the operating system and other applications. When the method described in this embodiment is implemented through software or firmware, the relevant program code is stored in the memory and called and executed by the processor.

[0051] This embodiment further provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described above.

[0052] Those skilled in the art will understand that the above embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0053] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the invention. Therefore, any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention should fall within the protection scope of the present invention.< / shutoff>

Claims

1. A method for automatically generating an intelligent agent control behavior tree program based on multimodal guidance, characterized by the following steps: include: Step S1. Task Information Input: Obtain a natural language description of the execution process of the task to be solved and the corresponding positive and negative example specifications; Step S2. Task framework generation: Based on the natural language description generated in step S1, a task execution framework containing only behavior tree control flow nodes and structure is generated using a large language model; Step S3. Behavior Tree Search: Search for the target behavior tree based on the task execution framework generated in step S2. If the target behavior tree program cannot be found through the search, it is determined to be an infeasible execution framework and proceeds to step S4. Step S4. Framework Repair: Repair the execution framework that is currently determined to be infeasible. First, repair the control flow nodes of the execution framework. Prioritize searching the control flow nodes in the execution framework. By adjusting the assignment of at least one control flow node, a new execution framework is formed. Search the new execution framework to determine whether the repair is successful. If the repair is unsuccessful, repair the structure of the execution framework. By obtaining all failure kernels in the execution framework that do not meet the positive examples, construct a behavior tree to obtain the failure kernel behavior tree. Merge the failure kernel behavior trees that do not meet the positive examples with the subtrees in the execution framework to form a new behavior tree to replace the corresponding subtrees and form a new behavior tree part of the program. Continue until a part of the program meets all the positive examples to complete the repair of the execution framework structure.

2. The method for automatically generating an intelligent agent control behavior tree program based on multimodal guidance according to claim 1, characterized in that, In step S1, positive examples describe semantic sequences that the behavior tree can generate, and positive examples are behavioral sequences that the target behavior tree can control the agent to generate. Negative examples describe semantic sequences that the behavior tree is prohibited from generating, and negative examples are behavioral sequences generated by the target behavior tree.

3. The method for automatically generating an intelligent agent control behavior tree program based on multimodal guidance according to claim 1, characterized in that, In step S3, the target behavior tree is searched using a top-down enumeration search method. The steps include: Step S311. Based on the type of the hole node, select information of the same type from the predefined node library and fill it in; Step S312. Use the depth-first search method to search downwards from the root node of the partial program syntax tree. If the current position is a hole node during the search, fill in the node information of the same type as the current hole node according to the type of the current node to obtain a set of different partial programs. The partial programs are programs that contain hole nodes. The types of the nodes include condition nodes and action nodes.

4. The method for automatically generating an intelligent agent control behavior tree program based on multimodal guidance according to claim 1, characterized in that, Step S3 also includes pruning during the search process using an abstract-based pruning method to remove infeasible parts of the program and their corresponding search space.

5. The method for automatically generating an intelligent agent control behavior tree program based on multimodal guidance according to claim 4, characterized in that, The steps for pruning using an abstract-based pruning method include: Step S321. Upper abstraction: The upper abstraction determines whether the current part of the program can be obtained by enumeration search to satisfy all positive example requirements, that is, whether there is a subsequent program that satisfies the positive example reduction. If the upper abstraction determines that the current part of the program is not feasible, that is, the behavior tree that satisfies all positive examples cannot be obtained by enumeration search, then the current part of the program is pruned. The part of the program is the program containing hole nodes. Step S322. Lower Abstraction: The lower abstraction determines whether the current part of the program can be obtained by enumeration search to find a program that satisfies all the counterexample requirements, that is, whether there is a subsequent program that satisfies the counterexample reduction. If the lower abstraction determines that the program is not feasible, that is, all subsequent programs of the program will produce a certain counterexample, then the current part of the program is pruned.

6. The method for automatically generating an intelligent agent control behavior tree program based on multimodal guidance according to any one of claims 1 to 5, characterized in that, Step S4, which involves repairing the control flow nodes of the execution framework, includes: Step S411. Perform a top-down depth-first search for the control flow nodes in the execution framework; Step S412. Adjust the assignments of m control flow nodes in any execution frame to obtain a new set of execution frames, where m≤n, and n is the total number of control flow nodes in the execution frame; Step S413. Search for each new execution framework and determine whether the repair is successful. If the execution framework is successfully repaired, generate the target program; otherwise, proceed to repair the structure of the execution framework.

7. The method for automatically generating an intelligent agent control behavior tree program based on multimodal guidance according to any one of claims 1 to 5, characterized in that, Step S4, the step of repairing the structure of the execution framework, includes: Step S421. Divide all positive examples into two parts: a set of positive examples that are satisfied by the upper abstraction of the current execution framework and a set of positive examples that are not satisfied by the upper abstraction. Step S422. For each positive example that is not satisfied on the execution framework, find the positive example that is most similar to the current negative example from the set of positive examples that are satisfied on the upper abstract, that is, the positive example whose sum of the lengths of the same prefix and suffix is ​​the largest. Step S423. Based on the most similar positive example found, remove the same prefixes and suffixes of the current non-positive example to obtain the difference between the two, and obtain the failure kernel of the current non-positive example; Step S424. Construct a behavior tree based on the failure kernels that do not currently satisfy the positive example, and use the sequence node as the node information of the failure kernel corresponding to the control flow node and leaf node in the behavior tree. After assigning the information of the failure kernel to the leaf node in turn, the failure kernel behavior tree is obtained. Step S425. Search each control flow node and leaf node in the execution framework, and use a depth-first search strategy to try to connect the failed kernel behavior tree with the subtree formed by the current node and its corresponding child node, merge them into a behavior tree, and replace the subtree formed by the current node and its corresponding child node with the merged behavior tree to obtain a behavior tree part of the program with a new structure. Step S426. Determine whether the partial program obtained from the behavior tree in step S425 can satisfy the current non-satisfied positive examples. If it can, repeat steps S422 to S426 to repair the remaining non-satisfied positive examples until the partial program satisfies all positive examples and no negative examples appear. Search for the target program in the final partial program.

8. A device for automatically generating intelligent agent control behavior tree programs based on multimodal guidance, characterized in that, include: The task information input module is used to obtain a natural language description of the execution process of the task to be solved and the corresponding positive and negative example reductions. The positive examples are the behavior sequences that the target behavior tree can control the agent to generate, and the negative examples are the behavior sequences generated by the target behavior tree. The task framework generation module is used to generate a task execution framework that contains only behavior tree control flow nodes and structure based on the natural language description generated by the task information input module and a large language model. The behavior tree search module is used to search for the target behavior tree based on the task execution framework generated by the task framework generation module. If the target behavior tree program cannot be found through the search, it is determined to be an infeasible execution framework and the process proceeds to step S4. The framework repair module is used to repair execution frameworks that are currently deemed infeasible. First, it repairs the control flow nodes of the execution framework. It prioritizes searching the control flow nodes in the execution framework and forms a new execution framework by adjusting the assignment of at least one control flow node. It then searches the new execution framework to determine if the repair was successful. If the repair is unsuccessful, it repairs the structure of the execution framework. It constructs a behavior tree by obtaining all failure kernels in the execution framework that do not meet the positive examples, and then merges the behavior trees of each failure kernel that does not meet the positive examples with the subtrees in the execution framework to form a new behavior tree to replace the corresponding subtrees and form a new behavior tree part of the program. This process continues until a part of the program meets all the positive examples, thus completing the repair of the execution framework structure.

9. A computer device comprising a processor and a memory, the memory being used to store a computer program, characterized in that, The processor is used to execute the computer program to perform the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed, it implements the method as described in any one of claims 1 to 7.

Images