Robot control method and device, equipment and medium

Abstract

The invention provides a robot control method and device, equipment and a medium. The method comprises the steps that a control instruction of a robot and sensing data of the robot in any mode in the environment where the robot is located are obtained; the perception data are input into the behavior tree generation large model after supervision and fine tuning, scene description is carried out on the environment where the robot is located, and a scene description text is obtained; inputting the control instruction and the scene description text into a behavior tree generation large model, decomposing and pre-planning the control instruction in combination with the scene description text, generating a plurality of sub-description texts, and obtaining a target description text based on the plurality of sub-description texts; the target description text is input into a behavior tree generation large model for behavior tree generation, a target behavior tree containing a plurality of action nodes is obtained, and each action node corresponds to at least one sub description text; and the robot is controlled to execute the corresponding action node based on the target behavior tree, so that the working efficiency of the robot and the task completion quality can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of robot control technology, and in particular to a robot control method, device, equipment and medium. Background Art

[0002] With the rapid development of science and technology, robots have become an indispensable part of modern industry and daily life. The application scope of robots is also becoming increasingly wide, ranging from simple automated production lines to complex smart home management, unmanned driving, exploration of unknown environments, disaster rescue, medical surgery and other mission scenarios.

[0003] However, with the continuous expansion of application fields, higher requirements are placed on the ability of robots to autonomously execute and complete complex tasks. In related technologies, robots can execute simple control instructions, but due to the robot's one-sided understanding of control instructions, there are obvious limitations in processing complex control instructions, thereby reducing the robot's work efficiency and the quality of task completion. Summary of the invention

[0004] The main purpose of the embodiments of the present disclosure is to provide a robot control method, device, equipment and medium, which can improve the robot's work efficiency and the quality of task completion.

[0005] To achieve the above objectives, a first aspect of an embodiment of the present disclosure provides a robot control method, comprising:

[0006] Obtain control instructions for the robot and perception data of the robot in any mode of the environment;

[0007] Inputting the perception data into the behavior tree generation model after supervised fine-tuning, performing a scene description on the environment where the robot is located, and obtaining a scene description text;

[0008] Inputting the control instruction and the scenario description text into the behavior tree generation model, decomposing and pre-planning the control instruction in combination with the scenario description text, generating a plurality of sub-description texts, and obtaining a target description text based on the plurality of sub-description texts;

[0009] Inputting the target description text into the behavior tree generation model to generate a behavior tree, thereby obtaining a target behavior tree including a plurality of action nodes, wherein each action node corresponds to at least one of the sub-description texts;

[0010] The robot is controlled to execute the corresponding action node based on the target behavior tree.

[0011] In some embodiments, the control instruction and the scenario description text are input into the behavior tree generation model, and the control instruction is decomposed and pre-planned in combination with the scenario description text to generate multiple sub-description texts, including:

[0012] Construct multiple levels of thinking prompt texts based on the preset thinking prompt framework;

[0013] The control instructions, the scenario description text, and the multiple levels of thought prompt texts are input into the behavior tree generation model in sequence respectively, and the behavior tree generation model is guided by the multiple levels of thought prompt texts to decompose and pre-plan the control instructions in combination with the scenario description text to generate multiple sub-description texts.

[0014] In some embodiments, the multiple levels of thought prompt texts include thought decomposition prompt texts, thought generation prompt texts, and state assessment prompt texts;

[0015] The control instruction, the scenario description text and the thought prompt texts of multiple levels are input into the behavior tree generation model in sequence, and the behavior tree generation model is guided by the thought prompt texts of multiple levels to decompose and pre-plan the control instruction in combination with the scenario description text to generate multiple sub-description texts, including:

[0016] Inputting the control instruction, the scenario description text and the thought decomposition prompt text into the behavior tree generation model, guiding the behavior tree generation model to perform thought decomposition on the control instruction in combination with the scenario description text through the thought decomposition prompt text, and generating a plurality of thought step information;

[0017] Inputting the plurality of thought step information and the thought generation prompt text into the behavior tree generation model, guiding the behavior tree generation model to generate thoughts for the plurality of thought step information through the thought generation prompt text, and generating a plurality of corresponding candidate execution actions;

[0018] Inputting the plurality of candidate execution actions and the state evaluation prompt text into the behavior tree generation macromodel, guiding the behavior tree generation macromodel to generate thoughts for the plurality of candidate execution actions through the state evaluation prompt text, and generating a plurality of corresponding action scores;

[0019] Based on the plurality of action scores, the corresponding plurality of candidate execution actions are screened to determine a plurality of target execution actions that meet preset requirements, and corresponding multiple sub-description texts are generated based on the plurality of target execution actions.

[0020] In some embodiments, the step of screening the corresponding candidate execution actions based on the plurality of action scores to determine a plurality of target execution actions that meet preset requirements includes:

[0021] Determine a start node and a target node from a plurality of candidate execution actions;

[0022] By using a preset heuristic search algorithm, the shortest path between the start node and the target node is determined according to the plurality of action scores, and each of the candidate execution actions in the shortest path is determined as a target execution action.

[0023] In some embodiments, the inputting of the plurality of thought step information and the thought generation prompt text into the behavior tree generation macro model, guiding the behavior tree generation macro model to generate thoughts for the plurality of thought step information through the thought generation prompt text, and generating a plurality of corresponding candidate execution actions, includes:

[0024] Inputting a plurality of the thought step information and the thought generation prompt text into the behavior tree generation model, and calling an action library matching the robot, wherein the action library includes a plurality of executable actions of the robot and an action description corresponding to each of the executable actions;

[0025] The behavior tree generation large model is guided by the thought generation prompt text to generate thoughts for the plurality of thought step information, and based on the plurality of executable actions and the corresponding action descriptions, a plurality of candidate execution actions matching at least one executable action are generated.

[0026] In some embodiments, the control instruction and the scenario description text are input into the behavior tree generation model, and the control instruction is decomposed and pre-planned in combination with the scenario description text to generate multiple sub-description texts, including:

[0027] Inputting the control instruction and the scene description text into the behavior tree generation model, performing word segmentation mapping on the control instruction to obtain a first text feature, encoding the perception data to obtain a second text feature, and fusing the first text feature and the second text feature to obtain a fusion feature;

[0028] The fusion features are decomposed and pre-planned based on the behavior tree generation model to generate multiple sub-description texts.

[0029] In some embodiments, encoding the perception data to obtain a second text feature, and fusing the first text feature and the second text feature to obtain a fused feature, comprises:

[0030] Encoding the perception data in a corresponding mode to obtain corresponding perception features;

[0031] Mapping the perceptual feature to a preset language embedding space to obtain a corresponding second text feature;

[0032] The first text feature is aligned with the second text feature, and after the alignment, the first text feature and the second text feature are fused to obtain a fused feature.

[0033] In some embodiments, the sensory data includes visual data and audio data, and the visual data and the audio data are of different modalities;

[0034] The encoding of the perception data to obtain a second text feature, and fusing the first text feature and the second text feature to obtain a fused feature, include:

[0035] Encoding the visual data to obtain a first subtext feature, and encoding the audio data to obtain a second subtext feature;

[0036] The first text feature, the first sub-text feature and the second sub-text feature are fused to obtain a fused feature.

[0037] In some embodiments, the behavior tree generation large model is trained by the following steps, including:

[0038] Obtain a preset behavior tree data set, wherein the behavior tree data set includes a plurality of sample behavior tree data, each of the sample behavior tree data includes a sample description text and a corresponding label behavior tree;

[0039] Input each of the sample description texts into the initial behavior tree generation model, generate a behavior tree for each of the sample description texts, and obtain a corresponding sample behavior tree;

[0040] The initial behavior tree generation large model is supervised and fine-tuned based on the sample behavior tree and the corresponding label behavior tree to obtain the behavior tree generation large model after supervised and fine-tuned.

[0041] In some embodiments, the sample description text is obtained by the following steps, including:

[0042] Get multiple preset label behavior trees;

[0043] Construct a description text generation prompt text of the label behavior tree, and input the description text generation prompt text and each of the label behavior trees into a preset description text generation model to generate sample description texts corresponding to each of the label behavior trees.

[0044] In some embodiments, the description text generation prompt text of the label behavior tree is constructed, and the description text generation prompt text and each label behavior tree are input into a preset description text generation model to generate sample description text corresponding to each label behavior tree, including:

[0045] Selecting multiple target behavior trees containing different behavior tree elements from the multiple tagged behavior trees;

[0046] Constructing a corresponding standard description text for each target behavior tree, and performing supervised fine-tuning on a preset description text generation model based on each target behavior tree and the corresponding standard description text, to obtain the description text generation model after supervised fine-tuning;

[0047] Construct a description text generation prompt text of the label behavior tree, and input the description text generation prompt text and each of the label behavior trees into the description text generation large model after supervised fine-tuning to generate sample description texts corresponding to each of the label behavior trees.

[0048] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present disclosure provides a control device for a robot, comprising:

[0049] A data acquisition module, used to acquire control instructions of the robot and perception data of any mode in the environment where the robot is located;

[0050] A scene description module, used to input the perception data into the behavior tree generation model after supervised fine-tuning, perform scene description on the environment where the robot is located, and obtain a scene description text;

[0051] An instruction decomposition module, used for inputting the control instruction and the scenario description text into the behavior tree generation model, decomposing and pre-planning the control instruction in combination with the scenario description text, generating a plurality of sub-description texts, and obtaining a target description text based on the plurality of sub-description texts;

[0052] a behavior tree generation module, configured to input the target description text into the behavior tree generation macromodel to generate a behavior tree, and obtain a target behavior tree including a plurality of action nodes, wherein each of the action nodes corresponds to at least one of the sub-description texts;

[0053] A control module is used to control the robot to execute the corresponding action node based on the target behavior tree.

[0054] To achieve the above-mentioned purpose, the third aspect of the embodiments of the present disclosure proposes an electronic device, which includes a memory and a processor, the memory stores a computer program, and the processor implements the robot control method described in the first aspect embodiment when executing the computer program.

[0055] To achieve the above-mentioned purpose, the fourth aspect of the embodiment of the present disclosure proposes a storage medium, which is a computer-readable storage medium, and the storage medium stores a computer program. When the computer program is executed by a processor, it implements the robot control method described in the above-mentioned first aspect embodiment.

[0056] The robot control method, device, equipment and medium proposed in the embodiments of the present disclosure can be applied to the robot control device. By executing the robot control method, after obtaining the robot control instruction and the perception data in any mode of the robot environment, the perception data can be input into the behavior tree generation model after supervision and fine-tuning, and the robot environment is described in the scene to obtain the scene description text, thereby enhancing the understanding of the robot's surrounding environment. Then, the control instruction and the scene description text are input into the behavior tree generation model, and the control instruction is decomposed and pre-planned in combination with the scene description text to generate multiple sub-description texts, and the target description text is obtained based on the multiple sub-description texts, so that the complex instruction can be decomposed into multiple executable tasks, thereby enhancing the understanding of the instruction. Then, the target description text is input into the behavior tree generation model to generate a behavior tree, and a target behavior tree containing multiple action nodes is obtained, wherein each action node corresponds to at least one sub-description text. Finally, based on the target behavior tree, the robot is controlled to execute the corresponding action node, which can improve the robot's work efficiency and the quality of task completion. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 is a flowchart of a robot control method provided by an embodiment of the present disclosure;

[0058] Figure 2 is a schematic diagram of perception data in a visual mode provided by an embodiment of the present disclosure;

[0059] Figure 3 yes Figure 2 Step S103 further includes a flow chart;

[0060] Figure 4 yes Figure 3 Step S202 further includes a flow chart;

[0061] Figure 5 yes Figure 4 Step S304 further includes a flow chart;

[0062] Figure 6 yes Figure 4 Step S302 further includes a flow chart;

[0063] Figure 7 yes Figure 2 Another flowchart diagram further comprising step S103;

[0064] Figure 8 yes Figure 7 Step S601 further includes a flow chart;

[0065] Fig. 9 yes Figure 7 Another flowchart diagram further comprising step S601;

[0066] Fig.10 is a complete schematic diagram of the data processing process provided in the embodiments of the present disclosure;

[0067] Fig.11 It is a flowchart of a large model training process for generating a behavior tree according to an embodiment of the present disclosure;

[0068] Fig.12 is a flowchart of a sample description text generation process provided by an embodiment of the present disclosure;

[0069] Fig.13 yes Fig.12 Step S1002 further includes a flow chart;

[0070] Fig.14 is a schematic diagram of functional modules of a control device for a robot provided by an embodiment of the present disclosure;

[0071] Fig.15 It is a schematic diagram of the hardware structure of the electronic device provided by the embodiment of the present disclosure. DETAILED DESCRIPTION

[0072] In order to make the purpose, technical solution and advantages of the present disclosure more clear, the present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present disclosure and are not used to limit the present disclosure.

[0073] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present disclosure belongs. The terms used herein are only for the purpose of describing the embodiments of the present disclosure and are not intended to limit the present disclosure.

[0074] First, some nouns involved in this disclosure are analyzed:

[0075] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0076] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0077] Behavior Trees (BTs) organize behaviors through a tree structure, providing a modular, flexible and scalable method to define complex behaviors. A behavior tree consists of multiple nodes, each of which represents a behavior or decision. These nodes are connected together in a specific way to form a tree structure. In a behavior tree, the root node is the starting point, and the behavior of the system is determined by traversing the child nodes.

[0078] Tree of Thoughts (ToT) is a framework for solving problems using large language models, by breaking down problems, generating thinking steps, evaluating states, and using search algorithms to explore solutions. It mimics the human decision-making process and is versatile and modular, but is also limited by the capabilities of language models and search strategies.

[0079] In the related art, the robot is able to execute simple control instructions. However, due to the robot's partial understanding of the control instructions, it has obvious limitations when processing complex control instructions, thereby reducing the robot's work efficiency and the quality of task completion.

[0080] Based on this, the embodiments of the present disclosure provide a robot control method, device, equipment and medium, which can improve the robot's work efficiency and the quality of task completion.

[0081] The control method of the robot in the embodiment of the present disclosure can be explained through the following embodiments.

[0082] The embodiments of the present disclosure can acquire and process relevant data based on artificial intelligence technology.

[0083] The robot control method provided by the embodiments of the present disclosure can be applied to the terminal, can also be applied to the server side, and can also be software running in the terminal or the server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the robot control method, etc., but is not limited to the above forms.

[0084] It should be noted that when the embodiments of the present disclosure are applied in a terminal, a server or software, the target behavior tree can be generated by executing the control method of the robot and then sent to the robot so that the robot executes the corresponding action nodes on the target behavior tree; or, the terminal can be the robot body, and the robot can execute the corresponding action nodes on the target behavior tree after generating the target behavior tree by executing the control method of the robot.

[0085] The present disclosure can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present disclosure can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present disclosure can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0086] It should be noted that in various specific embodiments of the present disclosure, when obtaining control instructions and perception data for related processing, the user's permission or consent will be obtained first. Moreover, the collection, use and processing of these data will comply with relevant laws, regulations and standards. In addition, when the embodiment of the present disclosure needs to obtain control instructions and perception data, the user's separate permission or consent can be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary control instructions and perception data for the normal operation of the embodiment of the present disclosure can be obtained.

[0087] See also Figure 1 , Figure 1 is a flow chart of a robot control method provided by an embodiment of the present disclosure, Figure 1 The method may include but is not limited to steps S101 to S105.

[0088] Step S101, obtaining control instructions of the robot and perception data of any mode in the environment where the robot is located;

[0089] Step S102, input the perception data into the behavior tree generation model after supervised fine-tuning, perform a scene description of the robot's environment, and obtain a scene description text;

[0090] Step S103, inputting the control instruction and the scenario description text into the behavior tree generation model, decomposing and pre-planning the control instruction in combination with the scenario description text, generating multiple sub-description texts, and obtaining the target description text based on the multiple sub-description texts;

[0091] Step S104, inputting the target description text into the behavior tree generation model to generate a behavior tree, and obtaining a target behavior tree containing multiple action nodes;

[0092] Wherein, each action node corresponds to at least one sub-description text;

[0093] Step S105, controlling the robot to execute corresponding action nodes based on the target behavior tree.

[0094] For the above step S101, a robot refers to a mechanical device or system that can autonomously perform operations or respond to instructions. Exemplarily, the robot can be equipped with various sensors and actuators for sensing the environment, receiving instructions, and performing corresponding actions. Control instructions refer to instructions for operating and controlling the robot. Control instructions can be simple, such as asking the robot to move forward or backward, or complex, such as asking the robot to perform a series of complex actions in a specific environment to complete a task. Control instructions can be generated by a human operator or system and sent to the robot to guide its behavior. For example, the robot can be a household robot, and the control instruction can be "bring me a bottle of Coke."

[0095] Perception data is information about the robot's surroundings collected by various sensors. These sensors may include but are not limited to cameras, lidars, infrared sensors, sonars, temperature sensors, humidity sensors, pressure sensors, microphones, etc. They can detect and measure different physical quantities or phenomena in the environment. Therefore, perception data has multiple modalities, that is, different data types or information representation methods. For example, a camera can capture visual images (visual modality), a lidar can generate point cloud data of the environment (three-dimensional spatial modality), and a microphone sensor can detect audio data in the robot's surroundings (audio modality).

[0096] Exemplarily, there are multiple ways to acquire sensory data. For example, sensory data is data collected by sensor equipment deployed in the robot's environment, and after obtaining the robot's control instructions, the collected sensory data is transmitted to the system in real time; or, sensory data can also be data collected by sensors deployed on the robot. Furthermore, in the embodiments of the present disclosure, after obtaining the robot's control instructions, obtaining sensory data of any modality in the environment through sensors set on the robot is used as an example, which does not represent a limitation of the embodiments of the present disclosure.

[0097] Regarding the above step S102, a behavior tree is a data structure used to represent the robot's behavior decision-making process. It usually contains a series of nodes, which represent actions or decision points that the robot can perform. In the embodiment of the present disclosure, a behavior tree generation model is used to automatically generate these behavior trees. It should be noted that the behavior tree generation model has undergone supervised fine-tuning, which means that it has been learned on a large amount of training data to optimize its ability to generate behavior trees.

[0098] In the disclosed embodiment, the perception data is input into the behavior tree generation model, which can use the perception data to generate a scene description text for the robot's environment. This text is a concise and structured representation that captures the most important features of the environment, such as the position, size, shape, color, etc. of objects, as well as the positional relationships between them.

[0099] Furthermore, corresponding scene description prompt text can be constructed for the perception data and input into the behavior tree generation model together with the perception data. For example, see Figure 2 , Figure 2 A schematic diagram of perception data in a visual mode provided in an embodiment of the present disclosure. If the perception data is visual perception data in a visual mode, a scene description prompt text may be constructed as "You will obtain a perception picture representing the current environment information obtained in real time by the robot's sensor. Your goal is to generate an environmental description of the current robot's environment based on the picture. The description includes the current environment attributes contained in the picture, the semantic information of objects in the environment, the state of the objects, the attributes that the objects may have, and the spatial relationship between the objects." Finally, the behavior tree generation model can perform a scene description on the visual perception data based on the prompt of the scene description prompt text, and the obtained scene description text includes "This picture shows a modern kitchen. The kitchen is simple and practical, with wooden cabinets and black cabinets. The white countertops give it a warm and stylish feel. The walls are painted pure white, reflecting plenty of light and making the entire space appear more spacious and bright. The floor is paved with flawless white tiles, which are easy to clean and maintain. On one side of the kitchen, there is a storage system composed of a series of high and low cabinets. The high cabinets have open shelves on the top for convenient storage of commonly used cooking spices or decorations; below are closed drawers and cabinets for storing tableware and other kitchen supplies. An integrated stove and sink are installed on the other wall to facilitate daily cooking operations. A large refrigerator is placed next to it, providing enough refrigeration space to store food and drinks. In addition, a small work area is set up in the corner, equipped with a sink and a cutting board. This can be used as a place to wash ingredients, or to temporarily place freshly cut vegetables or food being prepared."

[0100] With respect to the above step S103, in the embodiment of the present disclosure, the control instructions and the scenario description text are input into the behavior tree generation model, and the control instructions are decomposed and pre-planned in combination with the scenario description text. This process not only enhances the robot's ability to understand the instructions, but also ensures that the robot can perform tasks in the correct order and conditions, thereby improving work efficiency and the quality of task completion.

[0101] Decomposition refers to that the behavior tree generation model decomposes complex control instructions into multiple smaller, executable subtasks, which can be simple actions such as "move to a certain position", "grab an object", etc.; pre-planning refers to that the behavior tree generation model sorts and plans these subtasks according to the information in the scene description text to ensure that they can be executed in the correct order and conditions, including considering obstacles, object accessibility, time constraints and other factors, which are not specifically limited in the embodiments of the present disclosure.

[0102] During the decomposition and pre-planning process, the behavior tree generation model will generate multiple sub-description texts, each of which corresponds to a subtask. Then, the behavior tree generation model will generate a target description text based on these sub-description texts. The target description text is a complete description of the entire control instruction, which includes all subtasks and their order and conditions.

[0103] With respect to the above step S104, in the embodiment of the present disclosure, the target description text is input into the behavior tree generation model to generate the behavior tree. The behavior tree generation model takes the target description text as input and generates a target behavior tree. The target behavior tree is a structured representation, which contains multiple action nodes, each of which corresponds to an action or decision point that the robot needs to perform. Among them, the action node is the basic unit of the behavior tree, which represents the specific actions that the robot can perform, such as "moving to a certain position", "grabbing an object", etc. Furthermore, in the action node, a condition node can also be included, and the condition node will determine which subtree or action to execute according to specific conditions.

[0104] It should be noted that the relationship between multiple action nodes can be sequential or concurrent, that is, the action nodes in the target behavior tree can be executed sequentially or concurrently, depending on the task requirements and the capabilities of the robot. Each action node corresponds to at least one sub-description text, that is, it can correspond to one of the sub-description texts and execute one action through one action node, or it can correspond to multiple sub-description texts and execute multiple actions through one action node, thus ensuring that the robot can accurately understand and execute each action.

[0105] With respect to the above step S105, after generating the target behavior tree, the embodiment of the present disclosure can control the robot to execute. The robot's control system can interpret the target behavior tree and control the robot's behavior according to the action nodes therein, wherein the control system includes the robot's motion controller, sensor interface, actuator interface and other components, which work together to ensure that the robot can accurately execute the action nodes in the target behavior tree. Through this process, the robot can accurately understand and execute complex task instructions, thereby improving work efficiency and the quality of task completion.

[0106] It should be noted that in the target behavior tree, if the action nodes are arranged in a certain order, the robot will execute each action node in this order. If the target behavior tree contains conditional nodes, the robot will make conditional judgments when executing these nodes. Based on the judgment results, the robot can choose to execute action nodes in a subtree or skip certain action nodes. If the target behavior tree includes action nodes with concurrent relationships, the target behavior tree also allows action nodes to be executed concurrently, so that the robot can execute multiple actions at the same time to improve efficiency.

[0107] For example, Figure 2 When the control instruction is "bring me a bottle of Coke", if the robot only executes according to this control instruction, Figure 2 In the environment shown, there is no Coke, so the prior art can only control the robot to search and detect Coke. If there is no Coke on the surface of the entire room, the final task will fail. On the contrary, in the disclosed embodiment, by introducing multimodal perception data, and by combining the perception data to decompose and pre-plan the control instructions, the target description text obtained is a better task description, so that the disclosed embodiment can combine the scene analysis results and the optimal task description to prompt the multimodal large model, and generate a more reasonable target behavior tree. The behavior tree will pass through multiple action nodes, including searching whether there is Coke in the environment, if not, moving to the vicinity of the refrigerator and searching whether there is Coke in the refrigerator after opening the refrigerator, and if there is Coke in the refrigerator, taking the Coke in the refrigerator to the user, etc., which can control the robot to check the refrigerator and search for Coke, and there is indeed a possibility that there is Coke in the refrigerator. At this time, the task will be completed quickly, thereby improving the robot's work efficiency and the quality of task completion. It should be noted that the action nodes in this embodiment are only exemplary and do not represent limitations on the disclosed embodiments.

[0108] In addition, it should be noted that in the embodiments of the present disclosure, during the process of the robot executing an action node, the robot may receive feedback from the environment through its sensors and use it to adjust the robot's behavior to ensure that the task can be successfully completed. For example, if the robot encounters an obstacle while performing an action, it may adjust its motion trajectory based on the feedback from the sensors.

[0109] When the robot successfully executes all action nodes in the target behavior tree, the task indicated by the control instruction is considered completed. At this time, the robot can update its internal state to reflect the completion of the task. At the same time, the robot can also send a task completion signal to the external system (such as the task planner, the overall control system, etc.).

[0110] In summary, the disclosed embodiment, through steps S101 to S105, executes the control method of the robot. After obtaining the control instruction of the robot and the perception data in any mode in the environment where the robot is located, the perception data can be input into the behavior tree generation model after supervised fine-tuning, and the scene description of the robot's environment is performed to obtain the scene description text, thereby enhancing the understanding of the robot's surrounding environment. Subsequently, the control instruction and the scene description text are input into the behavior tree generation model, and the control instruction is decomposed and pre-planned in combination with the scene description text to generate multiple sub-description texts, and a target description text is obtained based on the multiple sub-description texts, so that the complex instruction can be decomposed into multiple executable tasks, thereby enhancing the understanding of the instruction. Then, the target description text is input into the behavior tree generation model to generate a behavior tree, and a target behavior tree containing multiple action nodes is obtained, wherein each action node corresponds to at least one sub-description text. Finally, based on the target behavior tree, the robot is controlled to execute the corresponding action node, which can improve the robot's work efficiency and the quality of task completion.

[0111] See also Figure 3 , Figure 3 yes Figure 2 In some embodiments, step S103 may further include steps S201 to S202:

[0112] Step S201, constructing multiple levels of thinking prompt texts based on a preset thinking prompt framework;

[0113] Step S202, respectively input the control instructions, the scene description text and the multiple levels of thought prompt text into the behavior tree generation model in sequence, and guide the behavior tree generation model to decompose and pre-plan the control instructions in combination with the scene description text through the multiple levels of thought prompt text to generate multiple sub-description texts.

[0114] In the above steps, the thinking prompt framework is a framework pre-designed in the embodiment of the present disclosure, which is used to guide the behavior tree generation model to perform complex task decomposition and planning. It contains multiple levels, each of which represents a different depth of thinking or granularity of task decomposition. These levels can be from high to low, from abstract to concrete, etc., so as to gradually guide the behavior tree generation model to decompose complex control instructions into executable subtasks. In the embodiment of the present disclosure, under the thinking prompt framework, multiple levels of thinking prompt texts are constructed. These texts are presented in the form of natural language, and each level has a corresponding text description, which is used to guide the behavior tree generation model to think at this level, thereby providing a structured thinking path for the behavior tree generation model to help it better understand and process complex control instructions.

[0115] Specifically, the disclosed embodiment inputs control instructions, scenario description texts, and previously constructed multiple levels of thought prompt texts into the behavior tree generation model. Through multiple levels of thought prompt texts, the behavior tree generation model is guided to think and plan according to the preset thinking framework. These texts not only provide clues for task decomposition, but also help the behavior tree generation model to consider the environmental information in the scenario description text during the decomposition process, thereby generating sub-description texts that are more in line with the actual environment and task requirements.

[0116] Finally, under the guidance of the behavior tree generation model, the control instruction is gradually decomposed into multiple executable subtasks or actions, and corresponding multiple sub-description texts are obtained. Therefore, by introducing multiple levels of thinking prompt texts, the embodiment of the present disclosure can improve the decomposition and planning capabilities of the behavior tree generation model for complex control instructions, so that it can generate more accurate and executable sub-description texts, and then build a more efficient and reliable target behavior tree.

[0117] In some embodiments, the thinking tree prompt framework in the embodiments of the present disclosure is mainly composed of four modules: thinking decomposition, thinking generation, state evaluation and search algorithm. Therefore, multiple levels of thinking prompt texts include thinking decomposition prompt texts, thinking generation prompt texts and state evaluation prompt texts. Among them, the thinking prompt algorithm is used to prompt the large model to perform thinking decomposition, the thinking generation prompt text is used to guide the large model to perform thinking generation, and the state evaluation prompt text is used to guide the large model to perform state evaluation on candidate actions. Furthermore, multiple levels of thinking prompt texts can also include action description prompt texts, which are used to prompt the large model to use clear and logical natural language to output description texts according to the process in the actions screened after scoring.

[0118] Based on this, see Figure 4 , Figure 4 yes Figure 3 In some embodiments, step S202 may further include steps S301 to S304:

[0119] Step S301, inputting the control instruction, the scenario description text and the thought decomposition prompt text into the behavior tree generation model, guiding the behavior tree generation model to perform thought decomposition on the control instruction in combination with the scenario description text through the thought decomposition prompt text, and generating a plurality of thought step information;

[0120] Step S302, inputting the multiple thought step information and thought generation prompt text into the behavior tree generation model, guiding the behavior tree generation model to generate thoughts for the multiple thought step information through the thought generation prompt text, and generating corresponding multiple candidate execution actions;

[0121] Step S303, inputting a plurality of candidate execution actions and state evaluation prompt texts into the behavior tree generation model, guiding the behavior tree generation model to generate thoughts for the plurality of candidate execution actions through the state evaluation prompt texts, and generating a plurality of corresponding action scores;

[0122] Step S304, screening from the corresponding multiple candidate execution actions based on the multiple action scores, determining multiple target execution actions that meet preset requirements, and generating corresponding multiple sub-description texts based on the multiple target execution actions.

[0123] In the above steps, the control instructions, scene description text, and thought decomposition prompt text are first input into the behavior tree generation model. The behavior tree generation model decomposes the control instructions according to the guidance of the thought decomposition prompt text and the scene description text, and uses natural language processing technology to understand the instructions and scene descriptions, and generates a series of thought step information according to the logical structure in the prompt text. Exemplarily, the thought decomposition prompt text can be "I need you to help me {task_instruction}, let's think step by step, and save each step in a json file, where step is the number of steps and name is the name of each step", so as to prompt the behavior tree generation model to perform thought decomposition, where the parameter task_instruction represents the input control instruction, and further, task_instruction can be the information after the control instruction and the scene description text are fused. Through the thought decomposition prompt text, the control instruction can be combined with the scene description text to perform thought decomposition and generate multiple thought step information.

[0124] Subsequently, based on the decomposition of thoughts, the disclosed embodiment inputs multiple thought step information and thought generation prompt texts into the behavior tree generation model. The behavior tree generation model further processes each thought step information according to the guidance of the thought generation prompt text, and generates corresponding candidate execution actions. The candidate execution actions are specific commands that the robot can directly execute. Exemplarily, the thought generation prompt text can be "What are the {robot_action} actions corresponding to the {step_name} step? Output in array form to ensure that the output can be parsed by json.loads", so as to guide the behavior tree generation model to generate thoughts, wherein the step_name in the prompt text corresponds to the step name obtained by the thought decomposition, and robot_action is the candidate execution action that needs to be output, such as "move from point A to point B", "pick up an object", etc.

[0125] Next, after generating multiple candidate execution actions, the embodiment of the present disclosure inputs the candidate execution actions and the state evaluation prompt text into the behavior tree generation model, and the behavior tree generation model performs a detailed evaluation of each execution action according to the guidance of the state evaluation prompt text. Exemplarily, the state evaluation prompt text may be "analyze the requirements of the {step_name} step for the {robot_action} action, score these actions, the total score is 10 points, and the score is directly output", and finally, the score of each candidate execution action is output as the action score.

[0126] Finally, after obtaining multiple action scores, the disclosed embodiment screens the execution actions according to these scores, determines which actions are feasible, efficient or safe, determines the target execution actions that meet the preset requirements, and generates corresponding multiple sub-description texts based on these target execution actions. Ultimately, the behavior tree generation model can use the thought decomposition prompt text, thought generation prompt text and state evaluation prompt text to guide the decomposition, action generation and evaluation process of the control instructions, thereby generating a series of target actions that meet the preset requirements and corresponding sub-description texts, which will serve as input information for the behavior tree generation to further guide the robot's behavior planning and execution.

[0127] Furthermore, after determining the target execution action, the target execution action and the action description prompt text can be input into the behavior tree generation model in the embodiment of the disclosure. The behavior tree generation model generates detailed natural language text for each target execution action according to the guidance of the action description prompt text, and obtains multiple sub-description texts corresponding to multiple target execution actions. Exemplarily, the action description prompt text can be "read the {tot_json} file, use clear and logical natural language, and output the task description of the control instruction {task_instruction} according to the process in the file", where the {tot_json} file is the text stored after the target execution node is determined. Finally, the embodiment of the disclosure will obtain the optimal description of the control instruction.

[0128] See also Figure 5 , Figure 5 yes Figure 4 In some embodiments, step S304 may further include steps S401 to S402:

[0129] Step S401, determining a start node and a target node from a plurality of candidate execution actions;

[0130] Step S402, determining the shortest path between the start node and the target node according to the plurality of action scores through a preset heuristic search algorithm, and determining each of the candidate execution actions in the shortest path as a target execution action.

[0131] In the above steps, we first need to determine the start node and target node from multiple candidate execution actions. The start node is the current state of the robot or the first action to be performed, representing the starting point of the robot to start executing the action sequence, while the target node is the final state that the robot needs to reach or the final action to be completed, representing the end point of the robot's execution of the action sequence.

[0132] Exemplarily, the embodiments of the present disclosure may select a suitable heuristic search algorithm as the preset heuristic search algorithm. For example, the A* algorithm may be used as the preset heuristic search algorithm to determine the shortest path between the start node and the target node.

[0133] In the process of using the A* algorithm to determine the shortest path, you first need to create an open list to record all nodes to be processed, and a closed list to record the nodes that have been processed, then add the starting node to the open list and set its cost to 0. Then enter the loop processing process, select the node with the lowest cost from the open list as the current node, if the current node is the target node, the algorithm ends, and the shortest path can be found by backtracking, and the current node is removed from the open list and added to the closed list. Then expand the node and traverse all the neighboring nodes of the current node, that is, the candidate execution action. If the neighboring node is already in the closed list, skip the node. If the neighboring node is no longer in the open list, calculate its cost and add it to the open list, and record its parent node. If the neighboring node is already in the open list, but the new path cost is lower, update its cost and parent node information.

[0134] Subsequently, the cost calculation is performed. The cost function F=G+H of the A* algorithm in the disclosed embodiment, where G represents the actual cost from the starting point to the current node, and H represents the estimated cost from the current node to the target node. In the disclosed embodiment, G can represent the total time or total energy consumption of the robot to reach the current node by executing the action sequence, and H can be calculated using heuristic functions such as Manhattan distance, diagonal distance, or Euclidean distance, depending on the robot's movement mode and environmental characteristics. When the target node is found, the shortest path from the starting node to the target node can be constructed by backtracking the parent node information.

[0135] Through the above steps, the disclosed embodiment can effectively determine the shortest path between the starting node and the target node using the A* algorithm, and determine each candidate execution action in the shortest path as the target execution action, which will serve as the basis for the robot execution action sequence in the subsequent steps. Finally, the shortest path process is written into the json file, where step_name is the step name, and robot_action is the robot action selected for this step, that is, the target execution action.

[0136] It should be noted that the above embodiment shows a scenario in which the A* algorithm is used as a preset heuristic search algorithm to determine the shortest path between a starting node and a target node. In addition, the embodiment of the present disclosure may also use other heuristic search algorithms, such as genetic algorithms, simulated annealing algorithms, ant colony algorithms, greedy algorithms, etc., and the embodiment of the present disclosure does not make any specific limitations on this.

[0137] Based on this, see Figure 6 , Figure 6 yes Figure 4 In some embodiments, step S302 may further include steps S501 to S502:

[0138] Step S501, inputting multiple thought step information and thought generation prompt text into the behavior tree generation model, and calling the action library matching the robot;

[0139] The action library includes multiple executable actions of the robot and the action descriptions corresponding to each executable action;

[0140] Step S502, guide the behavior tree generation large model to generate thoughts for multiple thought step information through thought generation prompt text, and generate multiple candidate execution actions matching at least one executable action based on multiple executable actions and corresponding action descriptions.

[0141] In some embodiments, the disclosed embodiments may also pre-build a robot action library to prompt the behavior tree generation model in a one-shot manner. Specifically, in order to ensure that the action nodes contained in the behavior tree generated by the behavior tree generation model are actions that the current robot can perform, and to ensure that each action is executed in place, the disclosed embodiments save the robot's action information in a json file format and build the robot's action library. The action library contains multiple executable actions of the robot, such as "action name", and "action description" corresponding to each executable action. Furthermore, the action library may also contain several types of information such as "robot type" and "action parameters", which are not specifically limited by the disclosed embodiments.

[0142] Then, the embodiments of the present disclosure use the json file of the above action library as a parameter of the system-level prompt word, and prompt the behavior tree generation large model in a one-shot manner. The prompt word starts with "You will obtain a robot action library in json format, which contains the executable actions of the robot and action descriptions. If the user asks you to generate a behavior tree based on the task description, please refer to the robot action library to ensure that the action nodes on the behavior tree can match the action library." and then appends the tag and parameter information "<robot_action_lib>${robot_action_lib}< / robot_action_lib>", where the parameter robot_acition_lib will be assigned the global path of the action library through code. This step will guide the behavior tree generation large model to perform a feasibility check on the action nodes in the behavior tree when generating the behavior tree, ensuring that the behavior tree is executable.

[0143] In the above steps, after introducing the action library, the thinking generation prompt text in the embodiments of the present disclosure can be "Refer to the robot's action library {robot_action_lib}, what are the {robot_action} actions corresponding to the {step_name} step. Output in array form, ensuring that the output can be parsed by json.loads" to guide the behavior tree generation large model to generate thinking for multiple thinking step information through the thinking generation prompt text, and generate multiple candidate execution actions that match at least one executable action based on multiple executable actions and corresponding action descriptions, thereby ensuring that the generated candidate execution actions can be executed by the robot. Among them, robot_action_lib in the prompt text corresponds to the action library constructed in the above embodiments, step_name corresponds to the step name obtained by thinking decomposition, and robot_action is the candidate execution action to be output, such as "Move from point A to point B", "Pick up an object", etc.

[0144] Please refer to Figure 7 , Figure 7 is Figure 2 Another process schematic diagram further included in step S103 in. In some embodiments, step S103 may further include steps S601 to S602:

[0145] Step S601, input the control instruction and the scene description text into the behavior tree generation large model, perform word segmentation mapping on the control instruction to obtain the first text feature, encode the perception data to obtain the second text feature, and fuse the first text feature and the second text feature to obtain the fusion feature;

[0146] Step S602, based on the behavior tree generation large model, decompose and pre-plan the fusion feature to generate multiple sub-description texts.

[0147] In the above steps, when the behavior tree generation model receives the control instruction, it performs word segmentation on the control instruction text, that is, decomposing long sentences or paragraphs into independent words or phrases, which are then mapped to a vector space that the model can understand, forming a first text feature. The behavior tree generation model in the embodiment of the present disclosure configures corresponding encoders for perception data of different modalities. When the behavior tree generation model receives the perception data, the perception data can be encoded by the encoder of the corresponding modality and converted into a vector form that the model can understand, forming a second text feature.

[0148] Next, the behavior tree generation model fuses the first text feature and the second text feature to form a fused feature, thereby converting the control command and perception data into a format that the model can process. Through feature fusion, the model can simultaneously consider the intention of the command and the context of the environment, providing a basis for subsequent task decomposition and pre-planning.

[0149] Finally, the disclosed embodiment utilizes the behavior tree to generate a large model to parse and pre-plan the fusion features, breaking them down into multiple executable subtasks or actions, thereby converting complex control instructions and environmental information into a series of clear, executable subtasks, and formulating a detailed execution plan for each subtask, which helps to improve the robot's work efficiency and the quality of task completion, so that when inputting control instructions, the perception data can be combined to understand the current environment, so as to enhance the description of the task that conforms to the current scenario.

[0150] See also Figure 8 , Figure 8 yes Figure 7 In some embodiments, step S601 may further include steps S701 to S703:

[0151] Step S701, encoding the perception data in a corresponding mode to obtain corresponding perception features;

[0152] Step S702, mapping the perceptual feature to a preset language embedding space to obtain a corresponding second text feature;

[0153] Step S703: align the first text feature with the second text feature, and fuse the first text feature and the second text feature after the alignment to obtain a fused feature.

[0154] In the above steps, the behavior tree generation model configures corresponding encoders for perception data of different modes. When the behavior tree generation model receives the perception data, it can first encode the perception data in the corresponding mode to obtain the corresponding perception features. The preset language embedding space is the language embedding space in the word segmentation mapping process of the first text feature. Since the behavior tree generation model performs word segmentation mapping on the control instruction to obtain the first text feature, each vector represents the meaning of a word or phrase. Therefore, in order to map the perception feature to this space so that similar words or phrases are close to each other in the vector space, the perception feature needs to be mapped to the preset language embedding space to obtain the corresponding second text feature.

[0155] Exemplarily, in the disclosed embodiment, a multi-layer perceptron mapper (MLP Projector) can be introduced. When the behavior tree generates a large model to encode the perception data in the corresponding modality and obtain the corresponding perception features, the perception features can be input into the multi-layer perceptron mapper to map these features into the language embedding space to obtain the corresponding second text features.

[0156] It should be noted that after obtaining the second text feature, the length of the second text feature is not necessarily the same as that of the first text feature, but since the first text feature and the second text feature are now represented in the same vector space, the correspondence between them can be found by calculating the similarity or distance between them. This process is called alignment. The aligned features can be merged by splicing, weighted summation or other more complex fusion strategies to form a unified feature representation, thereby obtaining a fused feature, so that the fused feature contains both the semantic information of the control instruction and the perceptual information of the environment, so as to achieve the combination of perceptual data and understanding of the current environment when inputting the control instruction, so as to enhance the description of the task that conforms to the current scene.

[0157] In some embodiments, the sensory data includes visual data and audio data, and the visual data and the audio data are of different modalities; see Fig. 9 , Fig. 9 yes Figure 7 Another flowchart diagram of step S601 further included in the embodiment. In some embodiments, step S601 may further include steps S801 to S802:

[0158] Step S801, encoding the visual data to obtain a first subtext feature, and encoding the audio data to obtain a second subtext feature;

[0159] Step S802: The first text feature, the first sub-text feature and the second sub-text feature are fused to obtain a fused feature.

[0160] In the above steps, the perception data includes visual data and audio data. The visual data comes from the environmental image input, while the audio data comes from the environmental sound input, so the modalities of the visual data and the audio data are different. For example, the visual data can be acquired by a camera deployed on the robot or a camera deployed in the environment where the robot is located, and the audio data can be acquired by a microphone sensor deployed on the robot or a microphone sensor deployed in the environment where the robot is located.

[0161] Furthermore, in the embodiments of the present disclosure, the perceptual data of different modalities can be encoded and processed separately, including encoding the visual data to obtain the first sub-text feature, encoding the audio data to obtain the second sub-text feature, and finally fusing the first text feature, the first sub-text feature and the second sub-text feature to obtain the fused feature.

[0162] It should be noted that, in the process of forming the first subtext feature and the second subtext feature, a mapping process may be performed by a multi-layer perceptron mapper. The complete data processing process in the embodiment of the present disclosure is described in detail below:

[0163] Please refer to Fig.10 , Fig.10 It is a complete schematic diagram of the data processing process provided in the embodiment of the present disclosure. In the embodiment of the present disclosure, by setting a visual encoder, an audio encoder and a multi-layer perceptron mapper, the behavior tree generation model forms a multimodal behavior tree model. When the multimodal behavior tree model receives visual data from the environment picture input, it can be encoded by the visual encoder under the corresponding visual modality to obtain visual features. Similarly, when the multimodal behavior tree model receives audio data from the environment sound input, it can be encoded by the audio encoder under the corresponding audio modality to obtain audio features. Subsequently, after the visual features and audio features are input into the multi-layer perceptron mapper, these features are mapped to the language embedding space, and aligned and fused with the features mapped by the word segmenter of the control instruction, so as to achieve the combination of the perceptual data when inputting the control instruction, and understand the current environment, so as to enhance the description of the task that conforms to the current scene.

[0164] See also Fig.11 , Fig.11 1 is a flowchart of a large model training process for generating a behavior tree according to an embodiment of the present disclosure. In some embodiments, the large model generated by the behavior tree is trained by the following steps, which may further include steps S901 to S903:

[0165] Step S901, obtaining a preset behavior tree data set;

[0166] The behavior tree data set includes multiple sample behavior tree data, each sample behavior tree data includes a sample description text and a corresponding label behavior tree;

[0167] Step S902: Input each sample description text into the initial behavior tree generation model, generate a behavior tree for each sample description text, and obtain a corresponding sample behavior tree;

[0168] Step S903: Perform supervised fine-tuning on the initial behavior tree generation model based on the sample behavior tree and the corresponding label behavior tree to obtain a supervised fine-tuned behavior tree generation model.

[0169] In the above steps, in the process of training to generate a large model of behavior tree, you first need to obtain a preset behavior tree dataset, which contains multiple sample behavior tree data. Each sample behavior tree data consists of two parts, namely, sample description text and corresponding label behavior tree. The sample description text is a text description of a specific task, which is used to guide the generation of behavior tree, and the label behavior tree is a constructed behavior tree structure corresponding to the sample description text, which serves as a standard answer or reference in the training process.

[0170] Subsequently, the embodiment of the present disclosure inputs each sample description text into the initial behavior tree generation model. The initial behavior tree generation model may be a large model that has not been fully trained, or a pre-trained model with basic functions. Therefore, the initial behavior tree generation model needs to be fine-tuned before it can be used. During the training process, the corresponding sample behavior tree can be generated according to the input sample description text.

[0171] Next, the disclosed embodiment can supervise and fine-tune the initial behavior tree generation model based on the sample behavior tree and the corresponding label behavior tree. Specifically, for each sample description text, the generated behavior tree is compared with the label behavior tree, and then the difference or loss value between the two is calculated. This loss value reflects the accuracy of the model generation behavior tree. Finally, according to the loss value, the parameters of the model are adjusted using optimization algorithms such as back propagation, and the above process is repeated until the performance of the model reaches the preset standard or is no longer significantly improved. Finally, a supervised fine-tuned behavior tree generation model can be obtained. This model can generate a behavior tree structure that matches the task requirements based on the input control instructions and scene description text, thereby achieving precise control of the robot.

[0172] Furthermore, in the disclosed embodiment, a lightweight base model is used as the initial behavior tree generation model. For example, a lightweight model such as llama-3-7B, Qwen2-7B or MiniCPM can be selected as the initial behavior tree generation model, and during the training process, the Quantized Low-Rank Adaptation efficient fine-tuning method in the parameter-efficient fine-tuning method is used, and the additional weight matrix of the multi-layer perceptron layer is unfrozen at the same time, so as to achieve more robust and efficient fine-tuning through a limited data set. Through the fine-tuning operation, a lightweight behavior tree generation model can be obtained as the behavior tree generator of the entire system.

[0173] See also Fig.12 , Fig.12 1001 to 1002:

[0174] Step S1001, obtaining a plurality of preset tag behavior trees;

[0175] Step S1002: construct a description text generation prompt text of the tag behavior tree, and input the description text generation prompt text and each tag behavior tree into a preset description text generation model to generate sample description text corresponding to each tag behavior tree.

[0176] In the above steps, the embodiment of the present disclosure provides a method for generating sample description text in sample data. The behavior tree dataset in the embodiment of the present disclosure is saved in the json file format, wherein each sample behavior tree data contains three elements, namely "Instruction", "Input" and "Output", wherein "Instruction" is a prompt text for generating the description text, which may be "You will get a summary of the task performed by the behavior tree, and your goal is to express this behavior tree in XML format.", "Input" is the sample description text, which is a natural language description of the tagged behavior tree, and "Output" is the tagged behavior tree, and the content in the tagged behavior tree is the behavior tree code in XML format. It can be understood that it is relatively easy to obtain the tagged behavior tree, and the behavior tree code in XML format in "Output" can be obtained from open source datasets, but these datasets themselves do not have natural language descriptions, so it is necessary to provide a method for generating sample description text.

[0177] Specifically, the disclosed embodiment can input the description text generation prompt text and each label behavior tree into a preset description text generation model. The description text generation model is a model used when generating sample description texts. It can also be a large-scale parameter model or a behavior tree generation model. For this, the disclosed embodiment only takes the description text generation model as a large-scale parameter model as an example. The description text generation model will try to generate descriptive text that matches the label behavior tree based on the input information, so as to obtain a set of sample description texts corresponding to the label behavior tree. These texts will serve as important data resources for training the behavior tree generation model, and are used to improve the model's ability to understand complex control instructions and scene descriptions, thereby generating a more accurate and reliable behavior tree structure.

[0178] See also Fig.13 , Fig.13 yes Fig.12 In some embodiments, step S1002 may further include steps S1101 to S1103:

[0179] Step S1101, selecting multiple target behavior trees containing different behavior tree elements from multiple tag behavior trees;

[0180] Step S1102, constructing a corresponding standard description text for each target behavior tree, and performing supervised fine-tuning on a preset description text generation model based on each target behavior tree and the corresponding standard description text, to obtain a supervised fine-tuned description text generation model;

[0181] Step S1103, constructing a description text generation prompt text of the label behavior tree, and inputting the description text generation prompt text and each label behavior tree into the description text generation large model after supervised fine-tuning to generate sample description text corresponding to each label behavior tree.

[0182] In the above steps, the large model for generating description texts is fine-tuned after supervision. Specifically, the embodiment of the present disclosure can select multiple target behavior trees containing different behavior tree elements from multiple label behavior trees. For example, five simple sequential execution behavior trees, simple parallel execution behavior trees, and complex behavior trees containing subtrees are selected from the behavior tree data set as target behavior trees. These target behavior trees try to contain all elements in the behavior tree, such as "Sequence", "Selector", "Parallel", "Decorator", "Action" and "Condition", etc. Then, a corresponding standard description text is constructed for each target behavior tree. Based on each target behavior tree and the corresponding standard description text, the preset large model for generating description texts is fine-tuned through supervision, including prompting the large language model in a one-shot manner, so as to obtain the large model for generating description texts after supervision and fine-tuning.

[0183] Finally, the embodiment of the present disclosure constructs the description text generation prompt text of the label behavior tree, and inputs the description text generation prompt text and each label behavior tree into the description text generation large model after supervised fine-tuning to generate sample description text corresponding to each label behavior tree.

[0184] It should be noted that, in the process of generating sample description texts corresponding to each label behavior tree, the embodiment of the present disclosure may also set a system-level description text generation prompt text, which begins with "You will get a behavior tree in XML format. Your task is to summarize the task type of this behavior tree and describe the actions performed by the robot." Then, several additional requirements may be set as constraints, including "The summary and description of the behavior tree should not exceed 200 words", "Compatible with the format requirements of the behavior tree in ROS2", "Comply with the syntax and format requirements of the BehaviorTree.CPP behavior tree library", "The task description must clearly express the overall summary of the task in natural language", "You need to do a secondary check on the generated description to ensure that there are no problems", etc., to generate high-quality sample description text.

[0185] Finally, the disclosed embodiment will traverse the behavior tree dataset, take each XML-formatted tagged behavior tree as a prompt for the user role, input it into the description text generation model through the description text generation model API, and select the assistant role of the description text generation model to output the description of the input behavior tree. Therefore, through the above method, the complete data of the sample behavior data can be obtained, so that these data can be stored in the behavior tree dataset file in json format in sequence, thus completing the construction of the behavior tree fine-tuning dataset.

[0186] See also Fig.14The present disclosure also provides a robot control device, which can implement the above robot control method. The robot control device includes:

[0187] The data acquisition module 1401 is used to acquire the control instructions of the robot and the perception data of any mode in the environment where the robot is located;

[0188] The scene description module 1402 is used to input the perception data into the behavior tree generation model after the supervised fine-tuning, perform scene description on the robot's environment, and obtain a scene description text;

[0189] The instruction decomposition module 1403 is used to input the control instruction and the scenario description text into the behavior tree generation model, decompose and pre-plan the control instruction in combination with the scenario description text, generate multiple sub-description texts, and obtain the target description text based on the multiple sub-description texts;

[0190] The behavior tree generation module 1404 is used to input the target description text into the behavior tree generation model to generate a behavior tree, and obtain a target behavior tree containing multiple action nodes;

[0191] Wherein, each action node corresponds to at least one sub-description text;

[0192] The control module 1405 is used to control the robot to execute the corresponding action node based on the target behavior tree.

[0193] In summary, the control device of the robot executes the control method of the robot, and after obtaining the control instructions of the robot and the perception data in any mode of the robot's environment, the perception data can be input into the behavior tree generation model after supervised fine-tuning, and the scene description of the robot's environment is performed to obtain the scene description text, thereby enhancing the understanding of the robot's surrounding environment. Subsequently, the control instructions and the scene description text are input into the behavior tree generation model, and the control instructions are decomposed and pre-planned in combination with the scene description text to generate multiple sub-description texts, and the target description text is obtained based on the multiple sub-description texts, so that the complex instructions can be decomposed into multiple executable tasks, thereby enhancing the understanding of the instructions. Subsequently, the target description text is input into the behavior tree generation model for behavior tree generation to obtain a target behavior tree containing multiple action nodes, wherein each action node corresponds to at least one sub-description text. Finally, based on the target behavior tree, the robot is controlled to execute the corresponding action node, which can improve the robot's work efficiency and the quality of task completion.

[0194] The specific implementation of the robot control device is basically the same as the specific implementation of the robot control method described above, and will not be repeated here. On the premise of meeting the requirements of the disclosed embodiment, the robot control device can also be provided with other functional modules to implement the robot control method described above.

[0195] The disclosed embodiment also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the control method of the robot when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0196] See also Fig.15 , Fig.15 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0197] The processor 1501 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present disclosure;

[0198] The memory 1502 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1502 can store operating devices and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1502, and the processor 1501 calls and executes the control method of the robot in the embodiment of the present disclosure;

[0199] Input / output interface 1503, used to implement information input and output;

[0200] Communication interface 1504, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0201] A bus 1505 that transmits information between the various components of the device (e.g., the processor 1501, the memory 1502, the input / output interface 1503, and the communication interface 1504);

[0202] The processor 1501 , the memory 1502 , the input / output interface 1503 and the communication interface 1504 are connected to each other in communication within the device via the bus 1505 .

[0203] The embodiment of the present disclosure also provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the control method of the robot is implemented.

[0204] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0205] The embodiments described in the embodiments of the present disclosure are intended to more clearly illustrate the technical solutions of the embodiments of the present disclosure and do not constitute a limitation on the technical solutions provided by the embodiments of the present disclosure. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present disclosure are also applicable to similar technical problems.

[0206] Those skilled in the art will appreciate that the technical solutions shown in the figures do not limit the embodiments of the present disclosure and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0207] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0208] Those skilled in the art will appreciate that all or some of the steps, devices, and functional modules / units in the above disclosed methods may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0209] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0210] It should be understood that in the present disclosure, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0211] In the several embodiments provided in the present disclosure, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0212] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0213] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0214] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method of each embodiment of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0215] The preferred embodiments of the present disclosure are described above with reference to the accompanying drawings, but the scope of the rights of the present disclosure is not limited thereto. Any modification, equivalent substitution and improvement made by those skilled in the art without departing from the scope and essence of the present disclosure should be within the scope of the rights of the present disclosure.

Claims

1. A robot control method, characterized in that: include: Obtain control instructions for the robot and perception data of the robot in any mode of the environment; Inputting the perception data into the behavior tree generation model after supervised fine-tuning, performing a scene description on the environment where the robot is located, and obtaining a scene description text; Inputting the control instruction and the scenario description text into the behavior tree generation model, decomposing and pre-planning the control instruction in combination with the scenario description text, generating a plurality of sub-description texts, and obtaining a target description text based on the plurality of sub-description texts; Inputting the target description text into the behavior tree generation model to generate a behavior tree, thereby obtaining a target behavior tree including a plurality of action nodes, wherein each action node corresponds to at least one of the sub-description texts; The robot is controlled to execute the corresponding action node based on the target behavior tree.

2. The robot control method according to claim 1, characterized in that: The control instruction and the scenario description text are input into the behavior tree generation model, and the control instruction is decomposed and pre-planned in combination with the scenario description text to generate multiple sub-description texts, including: Construct multiple levels of thinking prompt texts based on the preset thinking prompt framework; The control instructions, the scenario description text, and the multiple levels of thought prompt texts are input into the behavior tree generation model in sequence respectively, and the behavior tree generation model is guided by the multiple levels of thought prompt texts to decompose and pre-plan the control instructions in combination with the scenario description text to generate multiple sub-description texts.

3. The robot control method according to claim 2, characterized in that: The multiple levels of thought prompt texts include thought decomposition prompt texts, thought generation prompt texts and state evaluation prompt texts; The control instruction, the scenario description text and the thought prompt texts of multiple levels are input into the behavior tree generation model in sequence, and the behavior tree generation model is guided by the thought prompt texts of multiple levels to decompose and pre-plan the control instruction in combination with the scenario description text to generate multiple sub-description texts, including: Inputting the control instruction, the scenario description text and the thought decomposition prompt text into the behavior tree generation model, guiding the behavior tree generation model to perform thought decomposition on the control instruction in combination with the scenario description text through the thought decomposition prompt text, and generating a plurality of thought step information; Inputting the plurality of thought step information and the thought generation prompt text into the behavior tree generation model, guiding the behavior tree generation model to generate thoughts for the plurality of thought step information through the thought generation prompt text, and generating a plurality of corresponding candidate execution actions; Inputting the plurality of candidate execution actions and the state evaluation prompt text into the behavior tree generation macromodel, guiding the behavior tree generation macromodel to generate thoughts for the plurality of candidate execution actions through the state evaluation prompt text, and generating a plurality of corresponding action scores; Based on the plurality of action scores, the corresponding plurality of candidate execution actions are screened to determine a plurality of target execution actions that meet preset requirements, and corresponding multiple sub-description texts are generated based on the plurality of target execution actions.

4. The robot control method according to claim 3, characterized in that: The step of screening the corresponding candidate execution actions based on the plurality of action scores to determine a plurality of target execution actions that meet preset requirements includes: Determine a start node and a target node from a plurality of candidate execution actions; By using a preset heuristic search algorithm, the shortest path between the start node and the target node is determined according to the plurality of action scores, and each of the candidate execution actions in the shortest path is determined as a target execution action.

5. The robot control method according to claim 3, characterized in that: The step of inputting the plurality of thought step information and the thought generation prompt text into the behavior tree generation macro model, guiding the behavior tree generation macro model to generate thoughts for the plurality of thought step information through the thought generation prompt text, and generating a plurality of corresponding candidate execution actions, includes: Inputting a plurality of the thought step information and the thought generation prompt text into the behavior tree generation model, and calling an action library matching the robot, wherein the action library includes a plurality of executable actions of the robot and an action description corresponding to each of the executable actions; The behavior tree generation large model is guided by the thought generation prompt text to generate thoughts for the plurality of thought step information, and based on the plurality of executable actions and the corresponding action descriptions, a plurality of candidate execution actions matching at least one executable action are generated.

6. The robot control method according to claim 1, characterized in that: The control instruction and the scenario description text are input into the behavior tree generation model, and the control instruction is decomposed and pre-planned in combination with the scenario description text to generate multiple sub-description texts, including: Inputting the control instruction and the scene description text into the behavior tree generation model, performing word segmentation mapping on the control instruction to obtain a first text feature, encoding the perception data to obtain a second text feature, and fusing the first text feature and the second text feature to obtain a fusion feature; The fusion features are decomposed and pre-planned based on the behavior tree generation model to generate multiple sub-description texts.

7. The robot control method according to claim 6, characterized in that: The encoding of the perception data to obtain a second text feature, and fusing the first text feature and the second text feature to obtain a fused feature, include: Encoding the perception data in a corresponding mode to obtain corresponding perception features; Mapping the perceptual feature to a preset language embedding space to obtain a corresponding second text feature; The first text feature is aligned with the second text feature, and after the alignment, the first text feature and the second text feature are fused to obtain a fused feature.

8. The robot control method according to claim 6 or 7, characterized in that: The sensory data includes visual data and audio data, and the visual data and the audio data have different modalities; The encoding of the perception data to obtain a second text feature, and fusing the first text feature and the second text feature to obtain a fused feature, include: Encoding the visual data to obtain a first subtext feature, and encoding the audio data to obtain a second subtext feature; The first text feature, the first sub-text feature and the second sub-text feature are fused to obtain a fused feature.

9. The robot control method according to claim 1, characterized in that: The behavior tree generation large model is trained by the following steps, including: Obtain a preset behavior tree data set, wherein the behavior tree data set includes a plurality of sample behavior tree data, each of the sample behavior tree data includes a sample description text and a corresponding label behavior tree; Input each of the sample description texts into the initial behavior tree generation model, generate a behavior tree for each of the sample description texts, and obtain a corresponding sample behavior tree; The initial behavior tree generation large model is supervised and fine-tuned based on the sample behavior tree and the corresponding label behavior tree to obtain the behavior tree generation large model after supervised and fine-tuned.

10. The robot control method according to claim 9, characterized in that: The sample description text is obtained by the following steps, including: Get multiple preset label behavior trees; Construct a description text generation prompt text of the label behavior tree, and input the description text generation prompt text and each of the label behavior trees into a preset description text generation model to generate sample description texts corresponding to each of the label behavior trees.

11. The robot control method according to claim 10, characterized in that: The description text generation prompt text of the label behavior tree is constructed, and the description text generation prompt text and each label behavior tree are input into a preset description text generation model to generate sample description text corresponding to each label behavior tree, including: Selecting multiple target behavior trees containing different behavior tree elements from the multiple tagged behavior trees; Constructing a corresponding standard description text for each target behavior tree, and performing supervised fine-tuning on a preset description text generation model based on each target behavior tree and the corresponding standard description text, to obtain the description text generation model after supervised fine-tuning; Construct a description text generation prompt text of the label behavior tree, and input the description text generation prompt text and each of the label behavior trees into the description text generation large model after supervised fine-tuning to generate sample description texts corresponding to each of the label behavior trees.

12. A robot control device, characterized in that: include: A data acquisition module, used to acquire control instructions of the robot and perception data of any mode in the environment where the robot is located; A scene description module, used to input the perception data into the behavior tree generation model after supervised fine-tuning, perform scene description on the environment where the robot is located, and obtain a scene description text; An instruction decomposition module, used for inputting the control instruction and the scenario description text into the behavior tree generation model, decomposing and pre-planning the control instruction in combination with the scenario description text, generating a plurality of sub-description texts, and obtaining a target description text based on the plurality of sub-description texts; a behavior tree generation module, configured to input the target description text into the behavior tree generation macromodel to generate a behavior tree, and obtain a target behavior tree including a plurality of action nodes, wherein each of the action nodes corresponds to at least one of the sub-description texts; A control module is used to control the robot to execute the corresponding action node based on the target behavior tree.

13. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the robot control method according to any one of claims 1 to 11 when executing the computer program.

14. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the robot control method according to any one of claims 1 to 11 is implemented.

Images