Robot control method, device, equipment and medium

Abstract

The present disclosure provides a robot control method, device, equipment and medium, the method comprises: obtaining a control instruction of a robot and perception data of the robot in any one mode of an environment in which the robot is located; inputting the perception data into a supervised fine-tuned behavior tree generation large model to describe a scene of the environment in which the robot is located, and obtaining a scene description text; inputting the control instruction and the scene description text into the behavior tree generation large model, decomposing and pre-planning the control instruction in combination with the scene description text, generating multiple sub-description texts, and obtaining a target description text based on the multiple sub-description texts; inputting the target description text into the behavior tree generation large model to generate a behavior tree, and obtaining a target behavior tree comprising multiple action nodes, each action node corresponding to at least one sub-description text; and controlling the robot to execute the corresponding action node based on the target behavior tree, thereby improving the work efficiency and task completion quality of the robot.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of robot control, and particularly relates to a robot control method and device, equipment and medium. BACKGROUND

[0002] With the rapid development of science and technology, robots have become an indispensable part of modern industry and daily life, and the application range of robots is also increasingly wide, from simple automated production lines to complex smart home management, unmanned driving, unknown environment exploration, disaster rescue and medical surgery, etc.

[0003] However, with the continuous expansion of application fields, higher requirements are put forward for the ability of robots to autonomously execute and complete complex tasks. In the related art, robots can execute simple control instructions, but due to the one-sided understanding of robots to the control instructions, there is obvious limitation in processing complex control instructions, thereby reducing the work efficiency and task completion quality of the robot. SUMMARY

[0004] The main purpose of the embodiments of the present disclosure is to provide a robot control method, device, equipment and medium, which can improve the work efficiency and task completion quality of the robot.

[0005] To achieve the above purpose, a first aspect of the embodiments of the present disclosure provides a robot control method, comprising:

[0006] obtaining a control instruction of a robot and perception data in any one mode of an environment where the robot is located;

[0007] inputting the perception data into a supervised fine-tuned behavior tree generation large model to describe a scene of the environment where the robot is located, to obtain a scene description text;

[0008] inputting the control instruction and the scene description text into the behavior tree generation large model, decomposing and pre-planning the control instruction in combination with the scene description text, generating a plurality of sub-description texts, and obtaining a target description text based on the plurality of sub-description texts;

[0009] inputting the target description text into the behavior tree generation large model to generate a behavior tree, to obtain a target behavior tree comprising a plurality of action nodes, wherein each action node corresponds to at least one sub-description text;

[0010] controlling the robot to execute the corresponding action node based on the target behavior tree.

[0011] In some embodiments, the control instruction and the scene description text are input into the behavior tree generation large model, the control instruction is decomposed and pre-planned in combination with the scene description text, and a plurality of sub-description texts are generated, including:

[0012] A plurality of levels of thinking prompt texts are constructed based on a preset thinking prompt framework;

[0013] The control instruction, the scene description text, and the plurality of levels of thinking prompt texts are sequentially input into the behavior tree generation large model, and the behavior tree generation large model is guided by the plurality of levels of thinking prompt texts to decompose and pre-plan the control instruction in combination with the scene description text, and a plurality of sub-description texts are generated.

[0014] In some embodiments, the plurality of levels of thinking prompt texts include thinking decomposition prompt texts, thinking generation prompt texts, and state evaluation prompt texts;

[0015] The control instruction, the scene description text, and the plurality of levels of thinking prompt texts are sequentially input into the behavior tree generation large model, and the behavior tree generation large model is guided by the plurality of levels of thinking prompt texts to decompose and pre-plan the control instruction in combination with the scene description text, and a plurality of sub-description texts are generated.

[0016] The control instruction, the scene description text, and the thinking decomposition prompt text are input into the behavior tree generation large model, and the behavior tree generation large model is guided by the thinking decomposition prompt text to decompose the control instruction in combination with the scene description text, and a plurality of thinking step information are generated;

[0017] The plurality of thinking step information and the thinking generation prompt text are input into the behavior tree generation large model, and the behavior tree generation large model is guided by the thinking generation prompt text to generate a plurality of candidate execution actions corresponding to the plurality of thinking step information;

[0018] The plurality of candidate execution actions and the state evaluation prompt text are input into the behavior tree generation large model, and the behavior tree generation large model is guided by the state evaluation prompt text to generate a plurality of action scores corresponding to the plurality of candidate execution actions;

[0019] Based on the plurality of action scores, a plurality of target execution actions satisfying a preset requirement are selected from the plurality of candidate execution actions, and a plurality of sub-description texts corresponding to the plurality of target execution actions are generated.

[0020] In some embodiments, the filtering, from the plurality of candidate execution actions, a plurality of target execution actions that meet preset requirements based on the plurality of action scores comprises:

[0021] determining a starting node and a target node from the plurality of candidate execution actions;

[0022] determining, by a preset heuristic search algorithm, a shortest path between the starting node and the target node according to the plurality of action scores, and determining each candidate execution action in the shortest path as a target execution action.

[0023] In some embodiments, the inputting, into the behavior tree generation large model, the plurality of thought step information and the thought generation prompt text, and guiding the behavior tree generation large model to generate thoughts based on the plurality of thought step information and the thought generation prompt text to generate a plurality of candidate execution actions comprises:

[0024] inputting, into the behavior tree generation large model, the plurality of thought step information and the thought generation prompt text, and calling an action library matched with the robot, wherein the action library comprises a plurality of executable actions of the robot and an action description corresponding to each executable action;

[0025] guiding the behavior tree generation large model to generate thoughts based on the plurality of thought step information and the thought generation prompt text, and generating a plurality of candidate execution actions matched with at least one executable action based on the plurality of executable actions and the corresponding action descriptions.

[0026] In some embodiments, the inputting, into the behavior tree generation large model, the control instruction and the scene description text, and decomposing and pre-planning the control instruction based on the scene description text to generate a plurality of sub-description texts comprises:

[0027] inputting, into the behavior tree generation large model, the control instruction and the scene description text, performing word segmentation mapping on the control instruction to obtain first text features, and encoding the perception data to obtain second text features, and fusing the first text features and the second text features to obtain fused features;

[0028] decomposing and pre-planning, based on the behavior tree generation large model, the fused features to generate a plurality of sub-description texts.

[0029] In some embodiments, the encoding the perception data to obtain second text features, and fusing the first text features and the second text features to obtain fused features comprises:

[0030] The perceived data is encoded in the corresponding modality to obtain the corresponding perceived features;

[0031] The perceptual features are mapped to a preset language embedding space to obtain the corresponding second text features;

[0032] Align the first text feature with the second text feature, and then merge the first text feature and the second text feature after alignment to obtain the merged feature.

[0033] In some embodiments, the perceived data includes visual data and audio data, wherein the visual data and the audio data are modally different;

[0034] The process of encoding the perceived data to obtain a second text feature, and fusing the first text feature and the second text feature to obtain a fused feature, includes:

[0035] The visual data is encoded to obtain a first sub-text feature, and the audio data is encoded to obtain a second sub-text feature;

[0036] The first text feature, the first sub-text feature, and the second sub-text feature are fused to obtain the fused feature.

[0037] In some embodiments, the behavior tree generation model is trained through the following steps:

[0038] Obtain a preset behavior tree dataset, wherein the behavior tree dataset contains multiple sample behavior tree data, and each sample behavior tree data includes sample description text and corresponding label behavior tree;

[0039] Each of the sample description texts is input into the initial behavior tree generation model, and a behavior tree is generated for each of the sample description texts to obtain the corresponding sample behavior tree.

[0040] Based on the sample behavior tree and the corresponding label behavior tree, the initial behavior tree generation model is subjected to supervised fine-tuning to obtain the supervised fine-tuned behavior tree generation model.

[0041] In some embodiments, the sample description text is obtained through the following steps:

[0042] Retrieve multiple preset tag behavior trees;

[0043] The descriptive text of the label behavior tree is constructed to generate prompt text, and the descriptive text and each label behavior tree are input into a preset descriptive text generation model to generate sample descriptive text corresponding to each label behavior tree.

[0044] In some embodiments, the description text generation prompt text of the label behavior tree is generated, and the description text generation prompt text and each label behavior tree are input into a preset description text generation large model to generate sample description texts corresponding to each label behavior tree, including:

[0045] A plurality of target behavior trees containing different behavior tree elements are selected from the plurality of label behavior trees;

[0046] A corresponding standard description text is constructed for each target behavior tree, and a preset description text generation large model is supervised and fine-tuned based on each target behavior tree and the corresponding standard description text to obtain a supervised and fine-tuned description text generation large model;

[0047] The description text generation prompt text of the label behavior tree is constructed, and the description text generation prompt text and each label behavior tree are input into the supervised and fine-tuned description text generation large model to generate sample description texts corresponding to each label behavior tree.

[0048] To achieve the above-mentioned purpose, a second aspect of the embodiments of the present disclosure proposes a control device of a robot, including:

[0049] A data acquisition module is configured to acquire a control instruction of a robot and perception data of the robot in any one mode of an environment in which the robot is located;

[0050] A scene description module is configured to input the perception data into a supervised and fine-tuned behavior tree generation large model to describe a scene of the environment in which the robot is located, and obtain a scene description text;

[0051] An instruction decomposition module is configured to input the control instruction and the scene description text into the behavior tree generation large model, decompose and pre-plan the control instruction in combination with the scene description text, generate a plurality of sub-description texts, and obtain a target description text based on the plurality of sub-description texts;

[0052] A behavior tree generation module is configured to input the target description text into the behavior tree generation large model to generate a behavior tree, and obtain a target behavior tree containing a plurality of action nodes, wherein each action node corresponds to at least one sub-description text.

[0053] A control module is configured to control the robot to perform the corresponding action nodes based on the target behavior tree.

[0054] To achieve the above object, a third aspect of the embodiments of the present disclosure provides an electronic device, comprising a memory and a processor, the memory stores a computer program, and the processor implements the robot control method of the first aspect of the embodiments when executing the computer program.

[0055] To achieve the above object, a fourth aspect of the embodiments of the present disclosure provides a storage medium, which is a computer readable storage medium, and the storage medium stores a computer program, and the computer program is executed by a processor to implement the robot control method of the first aspect of the embodiments.

[0056] The robot control method, device, equipment and medium provided by the embodiments of the present disclosure can be applied to the robot control device. By executing the robot control method, after obtaining the control instruction of the robot and the perception data in any one mode of the environment where the robot is located, the perception data can be input into the supervised fine-tuned behavior tree generation large model to describe the scene of the environment where the robot is located, and obtain the scene description text, thereby enhancing the understanding ability of the environment around the robot. Then, the control instruction and the scene description text are input into the behavior tree generation large model, the control instruction is decomposed and pre-planned in combination with the scene description text, a plurality of sub-description texts are generated, and a target description text is obtained based on the plurality of sub-description texts. The complex instruction can be decomposed into a plurality of executable tasks, thereby enhancing the understanding of the instruction. Then, the target description text is input into the behavior tree generation large model to generate a behavior tree, and a target behavior tree containing a plurality of action nodes is obtained, wherein each action node corresponds to at least one sub-description text. Finally, the robot is controlled based on the target behavior tree to execute the corresponding action node, thereby improving the working efficiency of the robot and the quality of task completion. BRIEF DESCRIPTION OF DRAWINGS

[0057] Figure 1 is a flowchart of the robot control method provided by the embodiments of the present disclosure;

[0058] Figure 2 is a schematic diagram of perception data in a visual mode provided by the embodiments of the present disclosure;

[0059] Figure 3 is Figure 2 is a flowchart of step S103 further included in the robot control method provided by the embodiments of the present disclosure;

[0060] Figure 4 is Figure 3 is a flowchart of step S202 further included in the robot control method provided by the embodiments of the present disclosure;

[0061] Figure 5 is Figure 4 is a flowchart of step S304 further included in the robot control method provided by the embodiments of the present disclosure;

[0062] Figure 6 is Figure 4 a flowchart further comprising step S302 in the method 1000;

[0063] Figure 7 is Figure 2 another flowchart further comprising step S103 in the method 1000;

[0064] Figure 8 is Figure 7 a flowchart further comprising step S601 in the method 1000;

[0065] Figure 9 is Figure 7 another flowchart further comprising step S601 in the method 1000;

[0066] Figure 10 is a complete schematic diagram of the data processing process provided in the embodiments of the present disclosure;

[0067] Figure 11 is a flowchart of the behavior tree generation large model training process provided in the embodiments of the present disclosure;

[0068] Figure 12 is a flowchart of the sample description text generation process provided in the embodiments of the present disclosure;

[0069] Figure 13 is Figure 12 a flowchart further comprising step S1002 in the method 1000;

[0070] Figure 14 is a functional module schematic diagram of the control device of the robot provided in the embodiments of the present disclosure;

[0071] Figure 15 is a hardware structure schematic diagram of the electronic device provided in the embodiments of the present disclosure. DETAILED DESCRIPTION

[0072] In order to make the objectives, technical solutions and advantages of the present disclosure clearer, further detailed descriptions will be given to the present disclosure in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present disclosure, and are not intended to limit the present disclosure.

[0073] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present disclosure belongs. The terms used herein are only for the purpose of describing the embodiments of the present disclosure, and are not intended to limit the present disclosure.

[0074] First, several terms involved in the present disclosure are analyzed:

[0075] Artificial intelligence (AI): is a new technical science that studies, develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence; artificial intelligence is a branch of computer science, artificial intelligence attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence, including robots, language recognition, image recognition, natural language processing and expert systems. Artificial intelligence can simulate the information process of human consciousness and thinking. Artificial intelligence is also the theory, method, technology and application system of using digital computers or digital computer controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0076] The basic technology of artificial intelligence generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. The software technology of artificial intelligence mainly includes computer vision technology, robot technology, biometric technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc. several major directions.

[0077] Behavior trees (BTs) organize behaviors through tree structures, providing a modular, flexible and extensible method to define complex behaviors. A behavior tree is composed of multiple nodes, each representing a behavior or decision, which are connected together in a tree structure in a specific way. In a behavior tree, the root node is the starting point, and the system's behavior is decided by traversing the child nodes.

[0078] Tree of Thoughts (ToT) is a framework that uses large language models to solve problems by breaking down problems, generating thought steps, evaluating states, and exploring solutions using search algorithms. It mimics the human decision-making process and is general and modular, but is limited by the capabilities of language models and search strategies.

[0079] In the related art, robots can execute simple control instructions, but due to the one-sided understanding of robots to control instructions, there are obvious limitations in processing complex control instructions, thereby reducing the work efficiency and task completion quality of the robot.

[0080] Based on this, the embodiments of the present disclosure provide a robot control method, device, equipment and medium, which can improve the work efficiency and task completion quality of the robot.

[0081] The robot control method in the embodiments of the present disclosure can be described as follows.

[0082] The embodiments of the present disclosure can acquire and process related data based on artificial intelligence technology.

[0083] The robot control method provided by the embodiments of the present disclosure can be applied in a terminal, a server, or software running in the terminal or the server. In some embodiments, the terminal can be a smartphone, a tablet computer, a notebook computer, a desktop computer, etc.; the server can be configured as a standalone physical server, a server cluster composed of multiple physical servers, or a distributed system, and can also be configured as a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN, and big data and artificial intelligence platform; and the software can be an application implementing the robot control method, but is not limited to the above forms.

[0084] It should be noted that when the embodiments of the present disclosure are applied in a terminal, a server, or software, the robot control method can be executed to generate a target behavior tree, and then the target behavior tree is issued to the robot to enable the robot to execute the corresponding action nodes on the target behavior tree; or the terminal can be the robot itself, and the robot can execute the robot control method to generate a target behavior tree and then execute the corresponding action nodes on the target behavior tree.

[0085] The present disclosure can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present disclosure can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present disclosure can also be practiced in a distributed computing environment in which tasks are performed by remote processing devices connected by a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0086] It should be noted that in various specific embodiments of the present disclosure, when the control instruction and the perception data are obtained for correlation processing, the user's permission or consent is obtained first. Moreover, the collection, use and processing of these data comply with relevant laws, regulations and standards. In addition, when the embodiments of the present disclosure need to obtain the control instruction and the perception data, the user's separate permission or separate consent can be obtained through a pop-up window or jumping to a confirmation page, and after obtaining the separate permission or separate consent of the user, the necessary control instruction and perception data for enabling the embodiments of the present disclosure to normally run are obtained.

[0087] Please refer to Figure 1 , Figure 1 is a flowchart of a control method of a robot provided by the embodiments of the present disclosure, Figure 1 The method in the above embodiment can include but is not limited to steps S101-S105.

[0088] Step S101, obtaining a control instruction of a robot, and perception data in any one mode of an environment where the robot is located;

[0089] Step S102, inputting the perception data into the supervised fine-tuned behavior tree generation large model to describe the scene of the environment where the robot is located, and obtaining a scene description text;

[0090] Step S103, inputting the control instruction and the scene description text into the behavior tree generation large model, decomposing and pre-planning the control instruction combined with the scene description text, generating a plurality of sub-description texts, and obtaining a target description text based on the plurality of sub-description texts;

[0091] Step S104, inputting the target description text into the behavior tree generation large model to generate a behavior tree, and obtaining a target behavior tree containing a plurality of action nodes;

[0092] Each action node corresponds to at least one sub-description text.

[0093] Step S105, controlling the robot to execute the corresponding action node based on the target behavior tree.

[0094] For the above step S101, the robot refers to a mechanical device or system capable of autonomously performing operations or responding to instructions. Illustratively, the robot can be equipped with various sensors and actuators for perceiving the environment, receiving instructions, and performing corresponding actions. The control instruction refers to an instruction for operating and controlling the robot. The control instruction can be simple, such as instructing the robot to move forward or backward, or complex, such as instructing the robot to perform a series of complex actions in a specific environment to complete a task. The control instruction can be generated by a human operator or a system and sent to the robot to guide its behavior. For example, the robot can be a household robot, and the control instruction can be "get a bottle of coke".

[0095] The perception data is information about the environment around the robot collected by various sensors, which can include but are not limited to cameras, lidar, infrared sensors, sonar, temperature sensors, humidity sensors, pressure sensors, microphones, etc. They can detect and measure different physical quantities or phenomena in the environment, so the perception data has multiple modalities, i.e. different data types or information representations. For example, a camera can capture visual images (visual modality), a lidar can generate point cloud data of the environment (three-dimensional space modality), and a microphone sensor can detect audio data in the environment around the robot (audio modality).

[0096] Illustratively, there are various ways to obtain perception data. For example, the perception data is collected by sensors deployed in the environment where the robot is located, and the collected perception data is transmitted to the system in real time after the control instruction of the robot is obtained; or the perception data can also be collected by sensors deployed on the robot. Further, in the embodiments of the present disclosure, the perception data of any one modality in the environment is obtained by the sensors set on the robot after the control instruction of the robot is obtained, which is an example and does not represent a limitation of the embodiments of the present disclosure.

[0097] For the above step S102, the behavior tree is a data structure used to represent the behavior decision-making process of the robot, which usually contains a series of nodes representing actions or decision points that the robot can perform. In the embodiments of the present disclosure, a behavior tree generation large model is used to automatically generate these behavior trees. It should be noted that the behavior tree generation large model has been supervised fine-tuned, which means it has learned on a large amount of training data to optimize its ability to generate behavior trees.

[0098] In the embodiments of the present disclosure, the perception data is input into the behavior tree generation large model, which can generate a scene description text of the environment where the robot is located using the perception data. This text is a concise and structured representation that captures the most important features of the environment, such as the position, size, shape, color, etc. of objects, as well as their positional relationships.

[0099] Further, a corresponding scene description prompt text can be constructed for the perception data and input into the behavior tree generation large model together with the perception data. For example, please refer to Figure 2 , Figure 2 A schematic diagram of perception data in a visual modality is provided for the embodiments of the present disclosure. If the perception data is visual perception data in a visual modality, the scene description prompt text can be constructed as "You will be given a perception picture representing the current environmental information acquired by the robot's sensors in real time. Your goal is to generate an environmental description of the current environment where the robot is located based on this picture. The description includes the attributes of the current environment contained in the picture, the semantic information of the objects in the environment, the state of the objects, the attributes that the objects may have, and the spatial relationships between the objects." Finally, the behavior tree generation large model can generate a scene description text based on the prompt of the scene description prompt text, which includes "This picture shows a modern kitchen. The design of the kitchen is simple and practical, with wooden texture cabinets and black countertops, giving a warm and fashionable feel. The walls are painted pure white, reflecting sufficient light, making the entire space appear more spacious and bright. The floor is paved with white ceramic tiles, easy to clean and maintain. On one side of the kitchen, there is a storage system consisting of high and low cabinets. The high cabinet has an open shelf on top for storing commonly used cooking spices or decorations, while the lower part has closed drawers and cabinets for storing tableware and other kitchen supplies. On the other side of the wall, there is an integrated stove and sink, making daily cooking operations more convenient. Next to it is a large refrigerator, providing sufficient cold storage space for food and beverages. In addition, a small work area is set up in the corner, equipped with a sink and a cutting board. It can be used not only for washing vegetables, but also for temporarily placing freshly cut vegetables or food being prepared."

[0100] For the above step S103, in the embodiments of the present disclosure, the control instruction and the scene description text are input into the behavior tree generation large model, and the control instruction is decomposed and pre-planned in combination with the scene description text. This process not only enhances the understanding ability of the robot for the instruction, but also ensures that the robot can execute the task in the correct order and under the correct conditions, thereby improving the work efficiency and the quality of task completion.

[0101] The decomposition refers to the behavior tree generating large model decomposing the complex control instruction into multiple smaller executable sub-tasks, which can be simple actions such as "moving to a certain position", "grabbing an object", etc. The pre-planning refers to the behavior tree generating large model sequencing and planning the sub-tasks according to the information in the scene description text to ensure that they can be executed in the correct order and conditions, including considering obstacles, object accessibility, time constraints, etc., which are not limited in the embodiments of the present disclosure.

[0102] During the decomposition and pre-planning, the behavior tree generating large model generates multiple sub-description texts, each corresponding to a sub-task. Then, the behavior tree generating large model generates a target description text based on these sub-description texts, which is a complete description of the entire control instruction and contains all sub-tasks and their order and conditions.

[0103] For the above step S104, in the embodiments of the present disclosure, the target description text is input into the behavior tree generating large model for behavior tree generation. The behavior tree generating large model takes the target description text as input and generates a target behavior tree, which is a structured representation containing multiple action nodes, each corresponding to an action or decision point that the robot needs to execute. The action node is the basic unit of the behavior tree, representing specific actions that the robot can perform, such as "moving to a certain position", "grabbing an object", etc. Further, conditions can be included in the action node, which will determine which sub-tree or action to execute based on specific conditions.

[0104] It should be noted that the multiple action nodes can be in a sequential or concurrent relationship, i.e., the action nodes in the target behavior tree can be executed sequentially or concurrently, depending on the task requirements and the capabilities of the robot. Each action node corresponds to at least one sub-description text, i.e., it can correspond to one sub-description text and execute one action, or it can correspond to multiple sub-description texts and execute multiple actions, ensuring that the robot can accurately understand and execute each action.

[0105] For the above step S105, after generating the target behavior tree, the embodiments of the present disclosure can control the robot to execute. The control system of the robot can interpret the target behavior tree and control the behavior of the robot according to the action nodes therein, which includes the motion controller, sensor interface, actuator interface, etc. of the robot, which cooperate together to ensure that the robot can accurately execute the action nodes in the target behavior tree. Through this process, the robot can accurately understand and execute complex task instructions, thereby improving work efficiency and task completion quality.

[0106] It should be noted that in the target behavior tree, if the action nodes are arranged in a certain order, the robot will execute each action node in this order. If the target behavior tree contains conditional nodes, the robot will make conditional judgments when executing these nodes, and according to the judgment result, the robot can choose to execute the action nodes in a certain sub-tree or skip some action nodes. If the target behavior tree includes action nodes with concurrent relationships, the target behavior tree also allows concurrent execution of action nodes, so that the robot can execute multiple actions at the same time to improve efficiency.

[0107] For example, take the example of Figure 2 When the control instruction is "get a bottle of coke", if the robot only executes according to the control instruction, in the environment shown in Figure 2 , there is no coke, so the prior art can only control the robot to search for coke, and if there is no coke on the surface of the entire room, the final task will fail. In contrast, in the embodiments of the present disclosure, by introducing multi-modal perception data and by decomposing and pre-planning the control instruction in combination with the perception data, the target description text obtained is a better task description, so that the embodiments of the present disclosure can combine the scene analysis result and the optimal task description to prompt the multi-modal large model, and generate a more reasonable target behavior tree. This behavior tree will control the robot to check the refrigerator and search for coke through multiple action nodes, including searching the environment for coke, if there is no coke, then moving to the vicinity of the refrigerator and searching for coke in the refrigerator after opening the refrigerator, if there is coke in the refrigerator, then taking the coke in the refrigerator to the user, etc. The task will be completed quickly, thereby improving the work efficiency and task completion quality of the robot. It should be noted that the action nodes in this embodiment are only examples and do not represent a limitation on the embodiments of the present disclosure.

[0108] In addition, it should be noted that in the embodiments of the present disclosure, feedback from the environment can also be received by the sensors of the robot during the execution of the action nodes, and used to adjust the behavior of the robot to ensure that the task can be completed smoothly. For example, if the robot encounters an obstacle when executing a certain action, it can adjust its motion trajectory according to the feedback of the sensor.

[0109] When the robot successfully executes all the action nodes in the target behavior tree, the task indicated by the control instruction is considered to be completed. At this time, the robot can update its internal state to reflect the completion of the task, and at the same time, the robot can also send a task completion signal to the external system (such as the task planner, the overall control system, etc.).

[0110] To sum up, by steps S101 to S105, by executing the robot control method, after obtaining the control instruction of the robot and the perception data in any one mode of the environment where the robot is located, the perception data can be input into the supervised fine-tuned behavior tree generated large model to describe the scene of the environment where the robot is located, and the scene description text is obtained, thereby enhancing the understanding ability of the environment around the robot. Then, the control instruction and the scene description text are input into the behavior tree generated large model, the control instruction is decomposed and pre-planned in combination with the scene description text, a plurality of sub-description texts are generated, and the target description text is obtained based on the plurality of sub-description texts, so as to decompose the complex instruction into a plurality of executable tasks, thereby enhancing the understanding of the instruction. Then, the target description text is input into the behavior tree generated large model to generate a behavior tree, and the target behavior tree containing a plurality of action nodes is obtained, wherein each action node corresponds to at least one sub-description text. Finally, the robot is controlled based on the target behavior tree to execute the corresponding action node, so as to improve the working efficiency and the quality of task completion of the robot.

[0111] Please refer to Figure 3 , Figure 3 is Figure 2 The step S103 further includes a flowchart. In some embodiments, the step S103 can further include steps S201 to S202:

[0112] In step S201, a plurality of hierarchical thinking prompt texts are constructed based on a preset thinking prompt framework.

[0113] In step S202, the control instruction, the scene description text and the plurality of hierarchical thinking prompt texts are input into the behavior tree generated large model in sequence, and the behavior tree generated large model is guided by the plurality of hierarchical thinking prompt texts to decompose and pre-plan the control instruction in combination with the scene description text, to generate a plurality of sub-description texts.

[0114] In the above steps, the thinking prompt framework is a framework designed in advance by the embodiments of the present disclosure to guide the behavior tree generated large model to decompose and plan complex tasks, which includes a plurality of levels, each level representing a different thinking depth or task decomposition granularity. These levels can be from high to low, from abstract to concrete, etc., so as to gradually guide the behavior tree generated large model to decompose complex control instructions into executable sub-tasks. In the embodiments of the present disclosure, a plurality of hierarchical thinking prompt texts are constructed under the thinking prompt framework, which are presented in the form of natural language, and each level has a corresponding text description for guiding the behavior tree generated large model to think at this level, thereby providing a structured thinking path for the behavior tree generated large model to better understand and process complex control instructions.

[0115] Specifically, the control instruction, the scene description text, and the previously constructed multiple levels of thinking prompt texts are input into the behavior tree generation large model. Through the multiple levels of thinking prompt texts, the behavior tree generation large model is guided to think and plan according to the preset thinking framework. These texts not only provide clues for task decomposition, but also help the behavior tree generation large model to consider the environmental information in the scene description text during the decomposition process, so as to generate more accurate and executable sub-description texts.

[0116] Finally, under the guidance of the behavior tree generation large model, the control instruction is gradually decomposed into multiple executable sub-tasks or actions, and the corresponding multiple sub-description texts are obtained. Therefore, by introducing the multiple levels of thinking prompt texts, the decomposition and planning ability of the behavior tree generation large model for complex control instructions can be improved, so that more accurate and executable sub-description texts can be generated, and a more efficient and reliable target behavior tree can be constructed.

[0117] In some embodiments, the thinking tree prompt framework in the embodiments of the present disclosure mainly consists of four modules of thinking decomposition, thinking generation, state evaluation, and search algorithm. Therefore, the multiple levels of thinking prompt texts include thinking decomposition prompt texts, thinking generation prompt texts, and state evaluation prompt texts. Among them, the thinking prompt algorithm is used to prompt the large model to perform thinking decomposition, the thinking generation prompt text is used to guide the large model to perform thinking generation, and the state evaluation prompt text is used to guide the large model to perform state evaluation on the candidate actions. Further, the multiple levels of thinking prompt texts can also include action description prompt texts, which are used to prompt the large model to output description texts in clear and logical natural language according to the process in the screened actions after scoring.

[0118] Based on this, please refer to Figure 4 , Figure 4 is Figure 3 the flowchart further included in step S202. In some embodiments, step S202 can further include steps S301 to S304:

[0119] In step S301, the control instruction, the scene description text, and the thinking decomposition prompt text are input into the behavior tree generation large model. The behavior tree generation large model is guided to perform thinking decomposition on the control instruction in combination with the scene description text through the thinking decomposition prompt text, and multiple thinking step information is generated.

[0120] In step S302, the multiple thinking step information and the thinking generation prompt text are input into the behavior tree generation large model. The behavior tree generation large model is guided to perform thinking generation on the multiple thinking step information through the thinking generation prompt text, and multiple candidate execution actions are generated.

[0121] Step S303, inputting the plurality of candidate execution actions and state evaluation prompt text into the behavior tree generation large model, guiding the behavior tree generation large model to generate thoughts for the plurality of candidate execution actions through the state evaluation prompt text, and generating a plurality of action scores corresponding thereto;

[0122] Step S304, screening the plurality of candidate execution actions based on the plurality of action scores, determining a plurality of target execution actions meeting the preset requirements, and generating a plurality of sub-description texts corresponding to the plurality of target execution actions.

[0123] In the above steps, the control instruction, the scene description text and the thought decomposition prompt text are first input into the behavior tree generation large model. The behavior tree generation large model decomposes the control instruction according to the guidance of the thought decomposition prompt text, combines the scene description text, uses natural language processing technology to understand the instruction and the scene description, and generates a series of thought step information according to the logical structure in the prompt text. Exemplarily, the thought decomposition prompt text can be "I need you to help me {task_instruction}, let's think step by step, and save each step in a json file, where step is the number of steps, and name is the step name corresponding to each step", so as to prompt the behavior tree generation large model to perform thought decomposition, wherein the parameter task_instruction represents the input control instruction. Further, task_instruction can be the information fused from the control instruction and the scene description text. Through the thought decomposition prompt text, the control instruction can be decomposed into a plurality of thought step information in combination with the scene description text.

[0124] Subsequently, on the basis of thought decomposition, the plurality of thought step information and thought generation prompt text are input into the behavior tree generation large model. The behavior tree generation large model further processes each thought step information according to the guidance of the thought generation prompt text, and generates a candidate execution action corresponding thereto. The candidate execution action is a specific command that can be directly executed by the robot. Exemplarily, the thought generation prompt text can be "{step_name} step corresponds to {robot_action} action. Output in array form to ensure that the output can be parsed by json.loads", so as to guide the behavior tree generation large model to generate thoughts, wherein step_name in the prompt text corresponds to the step name obtained by thought decomposition, and robot_action is the candidate execution action to be output, such as "move from point A to point B", "pick up an object", etc.

[0125] Next, after generating a plurality of candidate execution actions, the embodiments of the present disclosure input the candidate execution actions and the state evaluation prompt text into the behavior tree generation large model, and the behavior tree generation large model performs detailed evaluation on each execution action according to the guidance of the state evaluation prompt text. Exemplarily, the state evaluation prompt text can be "analyze the requirements of the {step_name} step for the {robot_action} action, score these actions, total score 10 points, and directly output the score", and finally, the scores of the respective candidate execution actions are output as action scores.

[0126] Finally, after obtaining a plurality of action scores, the embodiments of the present disclosure screen the execution actions according to the scores, determine which actions are feasible, efficient or safe, determine target execution actions that meet the preset requirements, and generate a plurality of sub-description texts corresponding to the target execution actions based on the target execution actions. Finally, the behavior tree generation large model can guide the decomposition, action generation and evaluation process of the control instruction by using the thought decomposition prompt text, the thought generation prompt text and the state evaluation prompt text, so as to generate a series of target actions that meet the preset requirements and corresponding sub-description texts, and the sub-description texts will be used as input information for the generation of the behavior tree, to further guide the behavior planning and execution of the robot.

[0127] Further, after determining the target execution actions, the embodiments of the present disclosure can input the target execution actions and the action description prompt text into the behavior tree generation large model, and the behavior tree generation large model performs detailed natural language text generation on each target execution action according to the guidance of the action description prompt text, to obtain a plurality of sub-description texts corresponding to the target execution actions. Exemplarily, the action description prompt text can be "read the {tot_json} file, use clear and logical natural language, and output the task description of the control instruction {task_instruction} according to the flow in the file", wherein the {tot_json} file is the text stored after determining the target execution node, and finally the embodiments of the present disclosure obtain the optimal description of the control instruction.

[0128] Please refer to Figure 5 , Figure 5 is Figure 4 a flowchart further included in step S304. In some embodiments, step S304 can further include steps S401 to S402:

[0129] Step S401, determining a starting node and a target node from a plurality of candidate execution actions;

[0130] In step S402, a preset heuristic search algorithm is used to determine the shortest path between the start node and the target node according to the scores of the plurality of actions, and determine each candidate execution action in the shortest path as a target execution action.

[0131] In the above steps, it is necessary to first determine the start node and the target node from a plurality of candidate execution actions. The start node is the current state of the robot or the first action to be executed, representing the starting point of the robot to execute the action sequence, and the target node is the final state to be reached by the robot or the final action to be completed, representing the end point of the robot to execute the action sequence.

[0132] For example, the A* algorithm can be used as the preset heuristic search algorithm to determine the shortest path between the start node and the target node.

[0133] In the process of determining the shortest path using the A* algorithm, an open list is first created to record all nodes to be processed, and a closed list is created to record the nodes that have been processed, and then the start node is added to the open list with a cost of 0. Then, a loop processing process is entered, and the node with the lowest cost is selected from the open list as the current node. If the current node is the target node, the algorithm ends, the shortest path can be found by backtracking, and the current node is removed from the open list and added to the closed list. Then, the node is expanded, and all adjacent nodes of the current node, i.e. candidate execution actions, are traversed. If the adjacent node is already in the closed list, the node is skipped. If the adjacent node is not in the open list, its cost is calculated and added to the open list, and its parent node is recorded. If the adjacent node is already in the open list but the new path cost is lower, the cost and parent node information of the adjacent node are updated.

[0134] Subsequently, the cost is calculated. In the present embodiment, the cost function F of the A* algorithm is G+H, where G represents the actual cost from the starting point to the current node, and H represents the estimated cost from the current node to the target node. In the present embodiment, G can represent the total time or total energy consumption of the robot executing the action sequence to reach the current node, and H can be calculated using heuristic functions such as Manhattan distance, diagonal distance or Euclidean distance, depending on the movement mode and environmental characteristics of the robot. When the target node is found, the shortest path from the start node to the target node can be constructed by backtracking the parent node information.

[0135] Through the above steps, the embodiment of the present disclosure can effectively determine the shortest path between the starting node and the target node by using the A* algorithm, and determine the candidate execution actions in the shortest path as the target execution actions, which will serve as the basis for the robot execution action sequence in the subsequent steps. Finally, the flow of the shortest path is written into a json file, step_name is the step name, and robot_action is the robot action selected for the step, i.e., the target execution action.

[0136] It should be noted that the above embodiment shows a scenario in which the A* algorithm is used as a preset heuristic search algorithm to determine the shortest path between the starting node and the target node. In addition to this, the embodiment of the present disclosure can also use other heuristic search algorithms, such as genetic algorithms, simulated annealing algorithms, ant colony algorithms, and greedy algorithms, and the embodiment of the present disclosure does not make specific limitations.

[0137] Based on this, please refer to Figure 6 , Figure 6 is Figure 4 the flowchart further included in step S302. In some embodiments, step S302 can further include steps S501 to S502:

[0138] Step S501, input the plurality of thinking step information and thinking generation prompt text into the behavior tree generation large model, and call the action library matched with the robot;

[0139] Among them, the action library contains a plurality of executable actions of the robot and the action description corresponding to each executable action;

[0140] Step S502, guide the behavior tree generation large model to generate thinking for the plurality of thinking step information through the thinking generation prompt text, and generate a plurality of candidate execution actions matched with at least one executable action based on the plurality of executable actions and the corresponding action description.

[0141] In some embodiments, the embodiment of the present disclosure can also pre-construct a robot action library to prompt the behavior tree generation large model in one-shot manner. Specifically, in order to ensure that the action nodes contained in the behavior tree generated by the behavior tree generation large model are actions that can be executed by the current robot, and to ensure that each action is executed in place, the embodiment of the present disclosure saves the action information of the robot in the format of a json file, constructs an action library of the robot, which contains a plurality of executable actions of the robot, such as "action name", and the "action description" corresponding to each executable action. Further, the action library can also contain "robot type" and "action parameter" and other types of information, and the embodiment of the present disclosure does not make specific limitations.

[0142] Then, the embodiment of the present disclosure takes the json file of the above-mentioned action library as a parameter of a system-level prompt word, and prompts the behavior tree generation large model in a one-shot manner. The prompt word starts with "You will get a json format robot action library, which contains robot executable actions and action descriptions. If the user asks you to generate a behavior tree according to the task description, please refer to the robot action library to ensure that the action nodes on the behavior tree can match the action library.", and then appends the label and parameter information "<robot_action_library> ${robot_action_lib} < / robot_action_library>", where the parameter robot_acition_lib will be assigned the global path of the action library through the code. This step will guide the behavior tree generation large model to perform feasibility checking on the action nodes in the behavior tree when generating the behavior tree, so as to ensure that the behavior tree is executable.

[0143] In the above steps, after introducing the action library, the thought generation prompt text in the embodiment of the present disclosure can be "Refer to the robot action library {robot_action_lib}, what are the {robot_action} actions corresponding to the {step_name} step. Output in array form to ensure that the output can be parsed by json.loads", so as to guide the behavior tree generation large model to generate thoughts based on the multiple thought step information, and generate multiple candidate execution actions matched with at least one executable action based on the multiple executable actions and corresponding action descriptions, so as to ensure that the generated candidate execution actions can be executed by the robot. Among them, robot_action_lib in the prompt text corresponds to the action library constructed in the above-mentioned embodiment, step_name corresponds to the step name obtained by thought decomposition, and robot_action is the candidate execution action to be output, such as "move from point A to point B", "pick up an object", etc.

[0144] Please refer to Figure 7 , Figure 7 is Figure 2 Another flowchart further included in step S103. In some embodiments, step S103 can further include steps S601 to S602:

[0145] Step S601, input the control instruction and the scene description text into the behavior tree generation large model, perform word segmentation mapping on the control instruction to obtain first text features, and encode the perception data to obtain second text features, and fuse the first text features and the second text features to obtain fused features;

[0146] Step S602, based on the behavior tree generation large model, decompose and pre-plan the fused features to generate multiple sub-description texts.

[0147] In the above step, when the behavior tree generation large model receives the control instruction, the control instruction text is subjected to word segmentation processing, that is, long sentences or paragraphs are decomposed into independent words or phrases, which are then mapped into a vector space that the model can understand to form the first text feature. The behavior tree generation large model in the embodiment of the disclosure configures corresponding encoders for the perception data of different modalities. When the behavior tree generation large model receives the perception data, the perception data can be encoded by the encoder of the corresponding modality to convert into a vector form that the model can understand to form the second text feature.

[0148] Then, the behavior tree generation large model fuses the first text feature and the second text feature to form a fusion feature, so as to realize the conversion of the control instruction and the perception data into a format that the model can process, and enable the model to consider the intention of the instruction and the context of the environment at the same time through feature fusion, thereby providing a basis for subsequent task decomposition and pre-planning.

[0149] Finally, the embodiment of the disclosure utilizes the behavior tree generation large model to analyze and pre-plan the fusion feature, and decomposes it into a plurality of executable sub-tasks or actions, so as to convert the complex control instruction and environmental information into a series of clear and executable sub-tasks, and make a detailed execution plan for each sub-task, which helps to improve the work efficiency and the quality of task completion of the robot, so as to achieve the understanding of the current environment when the input control instruction is combined with the perception data, so as to enhance the description of the task that conforms to the current scene.

[0150] Please refer to Figure 8 , Figure 8 is Figure 7 The step S601 further includes a flowchart. In some embodiments, the step S601 can further include steps S701 to S703:

[0151] Step S701, encoding the perception data in the corresponding modality to obtain the corresponding perception feature;

[0152] Step S702, mapping the perception feature into a preset language embedding space to obtain a corresponding second text feature;

[0153] Step S703, aligning the first text feature and the second text feature, and fusing the first text feature and the second text feature after alignment to obtain a fusion feature.

[0154] In the above steps, the behavior tree generation large model configures corresponding encoders for the perception data of different modalities. When the behavior tree generation large model receives the perception data, the perception data can be encoded in the corresponding modality first to obtain the corresponding perception features. The preset language embedding space is the language embedding space in the word segmentation mapping process of the first text feature. Since the behavior tree generation large model maps the control instruction to obtain the first text feature in the process of word segmentation mapping, each vector represents the meaning of a word or phrase. Therefore, in order to map the perception features to the space so that similar words or phrases are close to each other in the vector space, the perception features need to be mapped to the preset language embedding space to obtain the corresponding second text feature.

[0155] Exemplarily, in the embodiments of the present disclosure, a multi-layer perception machine mapper (MLP Projector) can be introduced. When the behavior tree generation large model encodes the perception data in the corresponding modality to obtain the corresponding perception features, the features can be mapped to the language embedding space by inputting the perception features into the multi-layer perception machine mapper to obtain the corresponding second text feature.

[0156] It should be noted that after obtaining the second text feature, the length of the second text feature and the first text feature is not necessarily the same, but since the first text feature and the second text feature are now represented in the same vector space, the correspondence between them can be found by calculating their similarity or distance, which is called alignment. The aligned features can be merged by concatenation, weighted summation or other more complex fusion strategies to form a unified feature representation, thereby obtaining the fusion feature, so that the fusion feature contains both the semantic information of the control instruction and the perception information of the environment, so as to understand the current environment by combining the perception data when inputting the control instruction, thereby enhancing the description of the task in accordance with the current scene.

[0157] In some embodiments, the perception data includes visual data and audio data, and the modalities of the visual data and the audio data are different; please refer to Figure 9 , Figure 9 is Figure 7 Another flowchart further included in step S601. In some embodiments, step S601 can further include steps S801 to S802:

[0158] Step S801, encoding the visual data to obtain a first sub-text feature and encoding the audio data to obtain a second sub-text feature;

[0159] Step S802, fusing the first text feature, the first sub-text feature and the second sub-text feature to obtain the fusion feature.

[0160] In the above steps, the perception data includes visual data and audio data, the visual data is derived from the environment picture input, and the audio data is derived from the environment sound input, so the modalities of the visual data and the audio data are different. For example, the visual data can be obtained by a camera deployed on the robot or in the environment where the robot is located, and the audio data can be obtained by a microphone sensor deployed on the robot or in the environment where the robot is located.

[0161] Further, in the embodiments of the present disclosure, the perception data of different modalities can be encoded respectively, including encoding the visual data to obtain the first sub-text feature, and encoding the audio data to obtain the second sub-text feature, and finally fusing the first text feature, the first sub-text feature and the second sub-text feature to obtain the fused feature.

[0162] It should be noted that, in the process of forming the first sub-text feature and the second sub-text feature, mapping processing by the multi-layer perception machine mapper can be included, and the complete data processing process in the embodiments of the present disclosure will be described in detail as follows:

[0163] Please refer to Figure 10 , Figure 10 is a complete schematic diagram of the data processing process provided in the embodiments of the present disclosure. In the embodiments of the present disclosure, by setting the visual encoder, the audio encoder and the multi-layer perception machine mapper, the behavior tree generation large model forms a multi-modal behavior tree large model. When the multi-modal behavior tree large model receives the visual data derived from the environment picture input, the visual encoder corresponding to the visual modality can be used for encoding processing to obtain the visual feature. Similarly, when the multi-modal behavior tree large model receives the audio data derived from the environment sound input, the audio encoder corresponding to the audio modality can be used for encoding processing to obtain the audio feature. Subsequently, by inputting the visual feature and the audio feature into the multi-layer perception machine mapper, these features are mapped to the language embedding space, and are aligned and fused with the features mapped by the tokenizer of the large model, so as to achieve that when the control instruction is input, the current environment is understood in combination with the perception data, so as to enhance the description of the task conforming to the current scene.

[0164] Please refer to Figure 11 , Figure 11 is a flowchart of the behavior tree generation large model training process provided in the embodiments of the present disclosure. In some embodiments, the behavior tree generation large model is trained by the following steps, which can further include steps S901 to S903:

[0165] Step S901, obtaining a preset behavior tree data set;

[0166] The behavior tree dataset includes a plurality of sample behavior tree data, and each sample behavior tree data includes sample description text and a corresponding label behavior tree.

[0167] In step S902, each sample description text is input into an initial behavior tree generation large model, and a behavior tree is generated for each sample description text to obtain a corresponding sample behavior tree.

[0168] In step S903, the initial behavior tree generation large model is supervised and fine-tuned based on the sample behavior tree and the corresponding label behavior tree to obtain a supervised and fine-tuned behavior tree generation large model.

[0169] In the above steps, during the training of the behavior tree generation large model, a preset behavior tree dataset is first obtained. The behavior tree dataset includes a plurality of sample behavior tree data, and each sample behavior tree data is composed of two parts, i.e., sample description text and a corresponding label behavior tree. The sample description text is a textual description of a specific task, which is used to guide the generation of a behavior tree. The label behavior tree is a behavior tree structure corresponding to the sample description text, which is already constructed and serves as a standard answer or reference in the training process.

[0170] Subsequently, each sample description text is input into an initial behavior tree generation large model. The initial behavior tree generation large model can be a large model that has not been fully trained or a pre-trained model with basic functions. Therefore, the initial behavior tree generation large model needs to be supervised and fine-tuned before it can be used. During the training process, a sample behavior tree corresponding to the input sample description text can be generated.

[0171] Then, the initial behavior tree generation large model can be supervised and fine-tuned based on the sample behavior tree and the corresponding label behavior tree. Specifically, for each sample description text, the behavior tree generated by the sample description text is compared with the label behavior tree, and then the difference or loss value between the two is calculated. The loss value reflects the accuracy of the model in generating the behavior tree. Finally, according to the loss value, the parameters of the model are adjusted using an optimization algorithm such as backpropagation, and the above process is repeated until the performance of the model reaches a preset standard or no longer improves significantly. Finally, a supervised and fine-tuned behavior tree generation large model can be obtained. This model can generate a behavior tree structure that matches the task requirements according to the input control instructions and scene description text, thereby achieving precise control of the robot.

[0172] Further, the lightweight base large model is used as an initial behavior tree generation large model in the embodiments of the present disclosure. For example, a lightweight large model such as llama-3-7B, Qwen2-7B or MiniCPM can be selected as the initial behavior tree generation large model, and in the training process, the Quantized Low-Rank Adaptation high-efficiency fine-tuning method in the Parameter-Efficient-Fine-Tuning method is used, and the additional weight matrix of the multi-layer perception layer is unfrozen, so as to obtain more robust and efficient fine-tuning through limited data sets. Through the fine-tuning operation, a lightweight behavior tree generation large model can be obtained as a behavior tree generator of the entire system.

[0173] Referring to Figure 12 , Figure 12 is a flowchart of a sample description text generation process provided by the embodiments of the present disclosure. In some embodiments, the sample description text is obtained through the following steps, which can further include steps S1001 to S1002:

[0174] In step S1001, a plurality of preset label behavior trees are obtained.

[0175] In step S1002, a description text generation prompt text of the label behavior tree is constructed, and the description text generation prompt text and each label behavior tree are input into a preset description text generation large model to generate a sample description text corresponding to each label behavior tree.

[0176] In the above steps, the embodiments of the present disclosure provide a method for generating sample description text in sample data. The behavior tree data set in the embodiments of the present disclosure is saved in a json file format, wherein each sample behavior tree data contains three elements, namely “Instruction”, “Input” and “Output”. “Instruction” is a description text generation prompt text, which can be “You will get a summary of a task executed by a behavior tree, and your goal is to express this behavior tree in XML format.” “Input” is a sample description text, which is a natural language description of a label behavior tree. “Output” is a label behavior tree, and the content in the label behavior tree is an XML format behavior tree code. It can be understood that the label behavior tree is relatively easy to obtain, and the XML format behavior tree code in “Output” can be obtained from an open source data set. However, these data sets do not have natural language descriptions, so it is necessary to provide a way to generate sample description text.

[0177] Specifically, the embodiments of the present disclosure can input the description text generation prompt text and the respective label behavior trees into a preset description text generation large model. The description text generation large model is a large model used when generating sample description texts, and can also be a large-scale parameter large model or a behavior tree generation large model. The embodiments of the present disclosure only take the description text generation large model as an example of a large-scale parameter large model. The description text generation large model will try to generate descriptive texts matching the label behavior trees according to the input information, so as to obtain a set of sample description texts corresponding to the label behavior trees. These texts will be important data resources for training the behavior tree generation large model, and are used to improve the understanding ability of the model to complex control instructions and scene descriptions, so as to generate more accurate and reliable behavior tree structures.

[0178] Please refer to Figure 13 , Figure 13 is Figure 12 The step S1002 further includes a flowchart. In some embodiments, the step S1002 can further include steps S1101 to S1103:

[0179] Step S1101, selecting a plurality of target behavior trees containing different behavior tree elements from the plurality of label behavior trees;

[0180] Step S1102, constructing a corresponding standard description text for each target behavior tree, and performing supervised fine-tuning on the preset description text generation large model based on each target behavior tree and the corresponding standard description text to obtain a supervised fine-tuned description text generation large model;

[0181] Step S1103, constructing a description text generation prompt text of the label behavior tree, and inputting the description text generation prompt text and the respective label behavior trees into the supervised fine-tuned description text generation large model to generate sample description texts corresponding to the respective label behavior trees.

[0182] In the above steps, the description text generation large model is supervised fine-tuned. Specifically, the embodiments of the present disclosure can select multiple target behavior trees containing different behavior tree elements from multiple label behavior trees, for example, from the behavior tree dataset, respectively select 5 simple sequential execution behavior trees, simple parallel execution behavior trees, and complex behavior trees containing sub-trees as target behavior trees, and these target behavior trees contain all elements in the behavior tree as much as possible, such as “Sequence”, “Selector”, “Parallel”, “Decorator”, “Action” and “Condition”, etc. Then, a standard description text corresponding to each target behavior tree is constructed, and the preset description text generation large model is supervised fine-tuned based on each target behavior tree and the corresponding standard description text, including prompting the large language model in one-shot manner, thereby obtaining the supervised fine-tuned description text generation large model.

[0183] Finally, the embodiments of the present disclosure construct the description text generation prompt text of the label behavior tree, and input the description text generation prompt text and each label behavior tree into the supervised fine-tuned description text generation large model to generate the sample description text corresponding to each label behavior tree.

[0184] It should be noted that in the process of generating the sample description text corresponding to each label behavior tree, the embodiments of the present disclosure can also set a system-level description text generation prompt text, which starts with “You will get an XML format behavior tree, your task is to summarize the task type of this behavior tree and describe the actions performed by the robot.” Then, several requirements can be additionally set as constraints, including “the summary and description of the behavior tree should not exceed 200 words”, “compatible with the format requirements of the ROS2 behavior tree”, “consistent with the syntax and format requirements of the BehaviorTree.CPP behavior tree library”, “the task description must clearly represent the overall summary of the task in natural language”, “you need to perform a second check on the generated description to ensure that there are no problems” and the like, so as to generate high-quality sample description text.

[0185] Finally, the embodiments of the present disclosure traverse the behavior tree dataset, take each XML format label behavior tree as a prompt for the user role, input it into the description text generation large model through the API of the description text generation large model, and select the assistant role of the description text generation large model to output the description of the input behavior tree. Therefore, through the above method, the complete data of the sample behavior data can be obtained, so as to store these data in the behavior tree dataset file in json format in sequence, thereby completing the construction of the behavior tree fine-tuning dataset.

[0186] Please refer to Figure 14The robot control method and the robot control device can be used to implement the robot control method.

[0187] The data acquisition module 1401 is configured to acquire a control instruction of the robot and perception data in any one mode of an environment in which the robot is located.

[0188] The scene description module 1402 is configured to input the perception data into the supervised fine-tuned behavior tree generation large model to describe a scene of the environment in which the robot is located, to obtain a scene description text.

[0189] The instruction decomposition module 1403 is configured to input the control instruction and the scene description text into the behavior tree generation large model, decompose and pre-plan the control instruction in combination with the scene description text, generate a plurality of sub-description texts, and obtain a target description text based on the plurality of sub-description texts.

[0190] The behavior tree generation module 1404 is configured to input the target description text into the behavior tree generation large model to generate a behavior tree, to obtain a target behavior tree comprising a plurality of action nodes.

[0191] Each action node corresponds to at least one sub-description text.

[0192] The control module 1405 is configured to control the robot to execute a corresponding action node based on the target behavior tree.

[0193] In summary, the robot control device can enhance the understanding of the environment around the robot by inputting the perception data into the supervised fine-tuned behavior tree generation large model to describe the scene of the environment in which the robot is located, to obtain the scene description text, after acquiring the control instruction of the robot and the perception data in any one mode of the environment in which the robot is located. Then, the robot control device can enhance the understanding of the instruction by inputting the control instruction and the scene description text into the behavior tree generation large model, decomposing and pre-planning the control instruction in combination with the scene description text, generating a plurality of sub-description texts, and obtaining a target description text based on the plurality of sub-description texts. Then, the robot control device can improve the working efficiency and the quality of task completion of the robot by inputting the target description text into the behavior tree generation large model to generate a behavior tree, obtaining a target behavior tree comprising a plurality of action nodes, wherein each action node corresponds to at least one sub-description text, and controlling the robot to execute a corresponding action node based on the target behavior tree.

[0194] The specific implementation of the robot control device is basically the same as the specific embodiment of the robot control method described above, and will not be repeated here. The robot control device can also be provided with other functional modules to realize the robot control method in the above embodiments, as long as the requirements of the embodiments of the present disclosure are met.

[0195] The embodiments of the present disclosure also provide an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor implements the robot control method described above when executing the computer program. The electronic device can be any intelligent terminal, such as a tablet computer or a vehicle-mounted computer.

[0196] Please refer to Figure 15 , Figure 15 The hardware structure of the electronic device of another embodiment is illustrated, which includes:

[0197] The processor 1501 can be implemented in the form of a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, etc., for executing related programs to implement the technical solutions provided by the embodiments of the present disclosure.

[0198] The memory 1502 can be implemented in the form of a ROM (ReadOnly Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory), etc. The memory 1502 can store operating devices and other application programs. When the technical solutions provided by the embodiments of the present disclosure are implemented by software or firmware, the related program codes are stored in the memory 1502 and called and executed by the processor 1501 to implement the robot control method of the embodiments of the present disclosure.

[0199] The input / output interface 1503 is used to realize information input and output.

[0200] The communication interface 1504 is used to realize the communication interaction between the device and other devices. The communication can be realized by wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0201] The bus 1505 transmits information between various components (such as the processor 1501, the memory 1502, the input / output interface 1503, and the communication interface 1504) of the device.

[0202] The processor 1501, the memory 1502, the input / output interface 1503, and the communication interface 1504 are communicatively connected with each other within the device through a bus 1505.

[0203] The disclosure embodiments further provide a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the control method of the robot.

[0204] The memory, as a non-transitory computer readable storage medium, can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0205] The embodiments described in the disclosure embodiments are used to more clearly illustrate the technical solutions of the disclosure embodiments, and do not constitute a limitation on the technical solutions provided by the disclosure embodiments. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the disclosure embodiments are also applicable to similar technical problems.

[0206] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the disclosure embodiments, and can include more or fewer steps than the figures, or combine certain steps, or different steps.

[0207] The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separated, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments.

[0208] Those skilled in the art can understand that all or some of the steps in the above disclosed method, the functional modules / units in the device, and the equipment can be implemented as software, firmware, hardware, and appropriate combinations thereof.

[0209] The terms "first", "second", "third", "fourth", and the like in the description of the disclosure and the above drawings, if any, are used to distinguish similar objects, and do not necessarily have to be used to describe a particular order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the disclosure described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, device, product or apparatus including a series of steps or units does not have to be limited to only those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or apparatuses.

[0210] It should be understood that in the present disclosure, "at least one" means one or more, and "multiple" means two or more. "And / or" is used to describe the relationship between the associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can mean a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0211] In several embodiments provided by the present disclosure, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the above units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another device, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed objects can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0212] The units described above as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e. they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0213] In addition, each function unit in various embodiments of the present disclosure can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit.

[0214] When the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present disclosure, essentially or in part, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions used to cause a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods in the various embodiments of the present disclosure. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various other media that can store programs.

[0215] The preferred embodiments of the embodiments of the present disclosure are described above with reference to the accompanying drawings, and are not intended to limit the scope of the embodiments of the present disclosure. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present disclosure shall be within the scope of the embodiments of the present disclosure.

Claims

1. A control method of a robot characterized by, The method comprises the following steps: obtaining a control instruction of a robot and perception data of an environment in which the robot is located in any one mode; inputting the perception data into a supervised fine-tuned behavior tree generation large model to describe a scene of the environment in which the robot is located, and obtaining a scene description text; inputting the control instruction and the scene description text into the behavior tree generation large model, decomposing and pre-planning the control instruction in combination with the scene description text, generating a plurality of sub-description texts, and obtaining a target description text based on the plurality of sub-description texts; inputting the target description text into the behavior tree generation large model to generate a target behavior tree comprising a plurality of action nodes, wherein each action node corresponds to at least one sub-description text; controlling the robot to perform the corresponding action node based on the target behavior tree.

2. The control method of the robot according to claim 1, characterized by, The step of inputting the control instruction and the scene description text into the behavior tree generation large model, decomposing and pre-planning the control instruction in combination with the scene description text, and generating a plurality of sub-description texts comprises the following steps: constructing a plurality of levels of thinking prompt texts based on a preset thinking prompt framework; inputting the control instruction, the scene description text and the plurality of levels of thinking prompt texts into the behavior tree generation large model in sequence, guiding the behavior tree generation large model to decompose and pre-plan the control instruction in combination with the scene description text through the plurality of levels of thinking prompt texts, and generating a plurality of sub-description texts.

3. The control method of the robot according to claim 2, characterized by, The plurality of levels of thinking prompt texts comprise thinking decomposition prompt texts, thinking generation prompt texts and state evaluation prompt texts. The step of inputting the control instruction, the scene description text and the plurality of levels of thinking prompt texts into the behavior tree generation large model in sequence, guiding the behavior tree generation large model to decompose and pre-plan the control instruction in combination with the scene description text through the plurality of levels of thinking prompt texts, and generating a plurality of sub-description texts comprises the following steps: inputting the control instruction, the scene description text and the thinking decomposition prompt texts into the behavior tree generation large model, guiding the behavior tree generation large model to decompose the control instruction in combination with the scene description text through the thinking decomposition prompt texts, and generating a plurality of thinking step information; inputting the plurality of thinking step information and the thinking generation prompt texts into the behavior tree generation large model, guiding the behavior tree generation large model to generate a plurality of candidate execution actions corresponding to the plurality of thinking step information through the thinking generation prompt texts; inputting the plurality of candidate execution actions and the state evaluation prompt texts into the behavior tree generation large model, guiding the behavior tree generation large model to generate a plurality of action scores corresponding to the plurality of candidate execution actions through the state evaluation prompt texts; Based on the plurality of action scores, the corresponding plurality of candidate execution actions are screened to determine a plurality of target execution actions meeting preset requirements, and a plurality of sub-description texts corresponding to the plurality of target execution actions are generated.

4. The control method of the robot according to claim 3, characterized by, The screening of the plurality of candidate execution actions based on the plurality of action scores to determine a plurality of target execution actions meeting preset requirements comprises: determining a starting node and a target node from the plurality of candidate execution actions; determining the shortest path between the starting node and the target node according to the plurality of action scores by a preset heuristic search algorithm, and determining each candidate execution action in the shortest path as a target execution action.

5. The control method of the robot according to claim 3, characterized by, The inputting of the plurality of thought step information and the thought generation prompt text into the behavior tree generation large model, the guiding of the behavior tree generation large model to generate thoughts based on the plurality of thought step information through the thought generation prompt text, and the generation of a plurality of candidate execution actions corresponding to the plurality of thought step information comprise: inputting the plurality of thought step information and the thought generation prompt text into the behavior tree generation large model, and calling an action library matched with the robot, wherein the action library comprises a plurality of executable actions of the robot and an action description corresponding to each executable action; guiding the behavior tree generation large model to generate thoughts based on the plurality of thought step information through the thought generation prompt text, and generating a plurality of candidate execution actions matched with at least one executable action based on the plurality of executable actions and the corresponding action description.

6. The control method of the robot according to claim 1, characterized by, The inputting of the control instruction and the scene description text into the behavior tree generation large model, the decomposition and pre-planning of the control instruction in combination with the scene description text, and the generation of a plurality of sub-description texts comprise: inputting the control instruction and the scene description text into the behavior tree generation large model, performing word segmentation mapping on the control instruction to obtain first text features, and encoding the perception data to obtain second text features, and fusing the first text features and the second text features to obtain fused features; decomposing and pre-planning the fused features based on the behavior tree generation large model to generate a plurality of sub-description texts.

7. The control method of the robot according to claim 6, characterized by, The encoding of the perception data to obtain second text features, and the fusing of the first text features and the second text features to obtain fused features comprise: encoding the perception data in a corresponding mode to obtain corresponding perception features; mapping the perception features to a preset language embedding space to obtain corresponding second text features; aligning the first text features and the second text features, and fusing the first text features and the second text features after alignment to obtain fused features.

8. The control method of the robot according to claim 6 or 7, characterized by, The perception data comprises visual data and audio data, and the modalities of the visual data and the audio data are different. The encoding of the perception data to obtain second text features, and the fusing of the first text features and the second text features to obtain fused features comprise: The visual data is encoded to obtain first sub-text features, and the audio data is encoded to obtain second sub-text features; The first text features, the first sub-text features, and the second sub-text features are fused to obtain fusion features.

9. The control method of the robot according to claim 1, characterized by, The behavior tree generation large model is trained by the following steps, comprising: obtaining a preset behavior tree data set, wherein the behavior tree data set comprises a plurality of sample behavior tree data, and each sample behavior tree data comprises a sample description text and a corresponding label behavior tree; inputting each sample description text into an initial behavior tree generation large model to generate a behavior tree for each sample description text to obtain a corresponding sample behavior tree; based on the sample behavior tree and the corresponding label behavior tree, the initial behavior tree generation large model is supervised and fine-tuned to obtain the behavior tree generation large model after supervised fine-tuning.

10. The control method of the robot according to claim 9, characterized by, The sample description text is obtained by the following steps, comprising: obtaining a plurality of preset label behavior trees; constructing a description text generation prompt text for the label behavior tree, and inputting the description text generation prompt text and each label behavior tree into a preset description text generation large model to generate a sample description text corresponding to each label behavior tree.

11. The control method of the robot according to claim 10, characterized by, The construction of the description text generation prompt text for the label behavior tree, and the input of the description text generation prompt text and each label behavior tree into the preset description text generation large model to generate the sample description text corresponding to each label behavior tree, comprises: selecting a plurality of target behavior trees containing different behavior tree elements from a plurality of label behavior trees; constructing a corresponding standard description text for each target behavior tree, and based on each target behavior tree and the corresponding standard description text, the preset description text generation large model is supervised and fine-tuned to obtain the description text generation large model after supervised fine-tuning; constructing a description text generation prompt text for the label behavior tree, and inputting the description text generation prompt text and each label behavior tree into the description text generation large model after supervised fine-tuning to generate a sample description text corresponding to each label behavior tree.

12. A control device of a robot characterized by comprising: comprising: a data acquisition module for acquiring a control instruction of a robot and perception data of the robot in any one mode of an environment; a scene description module for inputting the perception data into the behavior tree generation large model after supervised fine-tuning to describe the environment of the robot to obtain a scene description text; an instruction decomposition module for inputting the control instruction and the scene description text into the behavior tree generation large model, decomposing and pre-planning the control instruction based on the scene description text, generating a plurality of sub-description texts, and obtaining a target description text based on the plurality of sub-description texts; a behavior tree generation module for inputting the target description text into the behavior tree generation large model to generate a behavior tree, and obtaining a target behavior tree comprising a plurality of action nodes, wherein each action node corresponds to at least one sub-description text. A control module is configured to control the robot to perform the corresponding action node based on the target behavior tree.

13. An electronic device, comprising: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the control method of the robot in any one of claims 1 to 11 when executing the computer program.

14. A computer readable storage medium, the storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the control method of the robot in any one of claims 1 to 11. The computer program is executed by the processor to implement the control method of the robot in any one of claims 1 to 11.

Images