Robot control method, device and equipment, robot, medium and product

Abstract

The invention provides a robot control method, device and equipment, a robot, a medium and a product, and belongs to the technical field of robots, and the method comprises the steps: generating a behavior tree based on task intention information corresponding to a user instruction; the task intention information represents the intention of the user for indicating the robot to execute the operation task, tree nodes of the behavior tree represent subtasks of the operation task executed by the robot, and node information of the tree nodes represents execution conditions of the subtasks corresponding to the tree nodes; controlling the robot to execute the operation task based on the behavior tree; task execution state information fed back by the robot in the operation task execution process is received; and when the task execution state information represents that the execution of the operation task is abnormal, dynamically adjusting the behavior tree to obtain an updated behavior tree, and controlling the robot to continuously execute the operation task based on the updated behavior tree. According to the method, the robot can efficiently work in a dynamically changing complex scene.

Description

Technical Field

[0001] The present application relates to the field of robotics technology, and in particular to a robot control method, device, equipment, robot, medium and product. Background Art

[0002] Robot task planning and execution have always played an important role in the contemporary field of artificial intelligence and automation technology. In related technologies, robot control systems usually use a single instruction set to control the robot to respond to user input for task planning, and use a fixed task execution sequence to control the robot to execute tasks. As a result, as the application scenarios of robots become more and more complex, the robot control system will easily lack flexible task adjustment and execution strategies, making it difficult to control the robot to smoothly plan and execute tasks. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose a robot control method, device, medium and program product, aiming to improve the flexibility and accuracy of controlling the robot to plan and execute tasks, so that the robot can adapt to dynamically changing complex scenarios and work efficiently.

[0004] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application provides a robot control method, the method comprising:

[0005] Generate a behavior tree based on task intention information corresponding to the user instruction; the task intention information is used to represent the user's intention to instruct the robot to perform the operation task, the tree nodes of the behavior tree are used to represent the subtasks of the operation task performed by the robot, and the node information of each tree node is used to represent the execution condition of the subtask corresponding to the tree node;

[0006] Controlling the robot to perform the operation task based on the behavior tree;

[0007] Receiving task execution status information fed back by the robot during the process of executing the operation task;

[0008] When the task execution status information indicates that the operation task has an execution exception, a dynamic adjustment operation is performed on the behavior tree to obtain an updated behavior tree, and the robot is controlled to continue to execute the operation task based on the updated behavior tree, wherein the dynamic adjustment operation includes: adjusting a subtask represented by a tree node of the behavior tree, and / or adjusting an execution condition represented by node information of the tree node.

[0009] In some embodiments, the method further comprises:

[0010] Acquire first environmental information of the environment where the robot is located;

[0011] The dynamically adjusting the behavior tree to obtain an updated behavior tree includes:

[0012] Based on the first environment information, a dynamic adjustment operation is performed on at least one tree node of the behavior tree.

[0013] In some embodiments, the dynamically adjusting at least one tree node of the behavior tree based on the first environment information includes:

[0014] Performing a state evaluation on a subtask represented by a tree node in the behavior tree based on the first environment information to obtain an execution state of the subtask represented by the tree node;

[0015] When the execution status indicates that the target subtask represented by the target tree node in each of the tree nodes has failed to execute, determining the target tree node as a problem node;

[0016] Dynamically adjust the problem node.

[0017] In some embodiments, after determining the target tree node as a problem node, the method further includes:

[0018] Generate an adjustment strategy based on the failure cause of the target subtask represented by the problem node;

[0019] The dynamically adjusting operation on the problem node includes:

[0020] Dynamically adjust the problem node based on the adjustment strategy.

[0021] In some embodiments, the dynamically adjusting the problem node based on the adjustment strategy includes at least one of the following:

[0022] When the adjustment strategy is to re-plan the task, the sub-task represented by the problem node is switched to the re-planned sub-task;

[0023] When the adjustment strategy is to change the task execution order, the task execution order in the execution condition represented by the node information of the problem node is adjusted;

[0024] In the case where the adjustment strategy is to add a new task, a new subtask is added to the subtask represented by the problem node, and an execution condition of the new subtask is added to the execution condition represented by the condition information of the problem node.

[0025] In some embodiments, the method further comprises:

[0026] Acquire second environmental information of the environment where the robot is located;

[0027] The generating of the behavior tree based on the task intention information corresponding to the user instruction includes:

[0028] Performing robot task planning based on the task intention information corresponding to the user instruction and the second environment information to obtain at least one subtask of the robot and an execution condition of the at least one subtask;

[0029] The at least one subtask is bound to a tree node of the behavior tree, and the execution condition is merged with the node condition information of the tree node to construct a behavior tree for the robot.

[0030] In some embodiments, the method further comprises:

[0031] Inputting the acquired user instruction into a preset user intention macro model, so as to parse the user instruction through the user intention macro model and output task intention information corresponding to the user instruction;

[0032] The user intention large model is obtained by model training based on constructed user samples, and the user samples include: user instruction samples and user intention samples corresponding to the user instruction samples.

[0033] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a control device for a robot, the device comprising:

[0034] A task planning module, used to generate a behavior tree based on task intention information corresponding to a user instruction; the task intention information is used to represent the user's intention to instruct the robot to perform an operation task, the tree nodes of the behavior tree are used to represent subtasks of the operation task performed by the robot, and the node information of each tree node is used to represent the execution condition of the subtask corresponding to the tree node;

[0035] A task execution module, used for controlling the robot to execute the operation task based on the behavior tree;

[0036] The task dynamic adjustment module is used to receive task execution status information fed back by the robot during the execution of the operation task; and, when the task execution status information indicates that the operation task has an execution abnormality, dynamically adjust the behavior tree to obtain an updated behavior tree; wherein the dynamic adjustment operation includes: adjusting the subtask represented by the tree node of the behavior tree, and / or adjusting the execution condition represented by the node information of the tree node;

[0037] The task execution module is further used to control the robot to continue to execute the operation task based on the updated behavior tree.

[0038] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented. When the electronic device is applied to a cloud server, when the processor executes the computer program, the method described in the second aspect is implemented.

[0039] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a robot, wherein the robot is configured with an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the method described in the first aspect when executing the computer program.

[0040] To achieve the above objectives, the fifth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0041] To achieve the above-mentioned purpose, the sixth aspect of the embodiments of the present application proposes a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, the method provided in the first aspect above is implemented.

[0042] The robot control method, device, electronic device, robot, computer-readable storage medium and computer program product proposed in the present application generate a behavior tree based on task intention information corresponding to a user instruction; the task intention information is used to represent the user's intention to instruct the robot to perform an operation task, the tree nodes of the behavior tree are used to represent the subtasks of the operation task performed by the robot, and the node information of each tree node is used to represent the execution condition of the subtask corresponding to the tree node; the robot is controlled to perform the operation task based on the behavior tree; task execution status information fed back by the robot during the execution of the operation task is received; when the task execution status information represents that the operation task has an execution abnormality, the behavior tree is dynamically adjusted to obtain an updated behavior tree, and the robot is controlled to continue to perform the operation task based on the updated behavior tree, wherein the dynamic adjustment operation includes: adjusting the subtask represented by the tree nodes of the behavior tree, and / or adjusting the execution condition represented by the node information of the tree nodes.

[0043] Thus, compared with the method in which the robot uses a single instruction set and a fixed execution sequence to control the robot to respond to user input for task planning and execution, the embodiment of the present application converts the task intention information corresponding to the user instruction into a task in the behavior tree, thereby controlling the robot to perform the operation task indicated by the user intention through the behavior tree. Afterwards, by receiving the task execution status information fed back by the robot during the execution of the operation task, when the task execution status information indicates that the robot's operation task has an execution abnormality, the behavior tree is dynamically adjusted to obtain an updated behavior tree, and the robot is controlled to continue to perform the operation task through the updated behavior tree. Among them, the dynamic adjustment of the behavior tree is mainly to adjust the subtask represented by the tree node of the behavior tree, and / or to adjust the execution condition represented by the node information of the tree node, that is, by dynamically adjusting the execution strategy and / or execution order of the operation task. In this way, it is possible to achieve more flexible and accurate response to user instructions for robot task planning and execution, and even in increasingly complex scenarios, the operation task strategy and / or order can be dynamically adjusted to plan and execute the operation task with high adaptability. That is to say, the embodiments of the present application improve the flexibility and accuracy of controlling the robot to plan and execute tasks, thereby enabling the robot to adapt to dynamically changing complex scenarios and work efficiently. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A schematic diagram of the steps of the robot control method provided in some embodiments of the present application;

[0045] Figure 2 A schematic flow chart of the steps of dynamically adjusting a behavior tree involved in some embodiments of the robot control method provided in the embodiments of the present application;

[0046] Figure 3 A schematic flow chart of steps for dynamically adjusting a behavior tree based on an adjustment strategy in some embodiments of the robot control method provided in the embodiments of the present application;

[0047] Figure 4 A flow chart of a dynamic adjustment strategy in the execution of a behavior tree involved in some embodiments of the robot control method provided in the embodiments of the present application;

[0048] Figure 5 for Figure 1 Schematic diagram of detailed process flow of step S101;

[0049] Figure 6 A schematic diagram of the overall structure involved in some embodiments of the robot control method provided in the embodiments of the present application;

[0050] Figure 7A schematic diagram of the application flow of the robot control method provided in the embodiment of the present application;

[0051] Figure 8 A schematic diagram of the structure of a control device for a robot provided in an embodiment of the present application;

[0052] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0054] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0056] First, the definitions of key technical terms involved in the embodiments of the present application are explained.

[0057] Behavior Trees.

[0058] Behavior tree is a decision tree used to represent the hierarchical structure of tasks and subtasks. It manages the task execution process and logical judgment through behavior nodes and node information (also called conditional nodes). As a task decomposition and decision management tool, behavior tree provides clear and intuitive task execution logic and dynamic adjustment capabilities through hierarchical behavior nodes and conditional nodes.

[0059] Intent big model.

[0060] The intention big model is a deep learning model that uses natural language processing technology to extract and understand user intentions from user instructions. The intention big model can extract complex intention information from user input, enhancing the naturalness and effectiveness of human-computer interaction.

[0061] Modular control architecture.

[0062] Modular control architecture is a design method that divides the robot control system into several independent functional modules. Modular control architecture simplifies system design and maintenance by decomposing the system into independent functional modules, such as navigation, operation, perception, etc. Each module in the modular control architecture can be independently developed, tested and optimized, and communicate and collaborate through standard interfaces.

[0063] Next, the overall concept of the robot control method provided in the embodiment of the present application is explained.

[0064] Robot task planning and execution have always played an important role in the contemporary field of artificial intelligence and automation technology. In related technologies, robot control systems usually use a single instruction set to control the robot to respond to user input for task planning, and use a fixed task execution sequence to control the robot to execute tasks, but lack flexibility and adaptability in task adjustment and execution. As the application scenarios of robots become more and more complex, robot control systems will easily lack flexible task adjustment and execution strategies, making it difficult to control robots to smoothly plan and execute tasks.

[0065] Based on this, the embodiments of the present application provide a robot control method, device, medium and program product, which aim to improve the flexibility and accuracy of controlling the robot to plan and execute tasks, so that the robot can adapt to dynamically changing complex scenarios and work efficiently.

[0066] The embodiment of the present application generates a behavior tree based on task intention information corresponding to a user instruction; the task intention information is used to represent the user's intention to instruct the robot to perform an operation task, the tree nodes of the behavior tree are used to represent the subtasks of the operation task performed by the robot, and the node information of each of the tree nodes is used to represent the execution conditions of the subtasks corresponding to the tree nodes; the robot is controlled to perform the operation task based on the behavior tree; task execution status information fed back by the robot during the execution of the operation task is received; when the task execution status information represents that the operation task has an execution abnormality, the behavior tree is dynamically adjusted to obtain an updated behavior tree, and the robot is controlled to continue to perform the operation task based on the updated behavior tree, wherein the dynamic adjustment operation includes: adjusting the subtasks represented by the tree nodes of the behavior tree, and / or adjusting the execution conditions represented by the node information of the tree nodes.

[0067] Thus, compared with the method in which the robot uses a single instruction set and a fixed execution sequence to control the robot to respond to user input for task planning and execution, the embodiment of the present application converts the task intention information corresponding to the user instruction into a task in the behavior tree, thereby controlling the robot to perform the operation task indicated by the user intention through the behavior tree. Afterwards, by receiving the task execution status information fed back by the robot during the execution of the operation task, when the task execution status information indicates that the robot's operation task has an execution abnormality, the behavior tree is dynamically adjusted to obtain an updated behavior tree, and the robot is controlled to continue to perform the operation task through the updated behavior tree. Among them, the dynamic adjustment of the behavior tree is mainly to adjust the subtask represented by the tree node of the behavior tree, and / or to adjust the execution condition represented by the node information of the tree node, that is, by dynamically adjusting the execution strategy and / or execution order of the operation task. In this way, it is possible to achieve more flexible and accurate response to user instructions for robot task planning and execution, and even in increasingly complex scenarios, the operation task strategy and / or order can be dynamically adjusted to plan and execute the operation task with high adaptability. That is to say, the embodiments of the present application improve the flexibility and accuracy of controlling the robot to plan and execute tasks, thereby enabling the robot to adapt to dynamically changing complex scenarios and work efficiently.

[0068] Next, the control method, device, medium and program product of the robot provided in the embodiments of the present application are specifically described through the following embodiments, and the control method of the robot provided in the embodiments of the present application is first described in detail.

[0069] It should be noted that the control method of the robot provided in the embodiment of the present application relates to the field of robot technology. The control method of the robot provided in the embodiment of the present application can be applied to the terminal, can also be applied to the server side, and can also be software running in the terminal or the server side. In some embodiments, the terminal can be a terminal configured by the robot itself, or it can be an electronic device such as a smart phone, a tablet computer, a laptop computer, a desktop computer, etc. associated with the robot. The terminal and the robot are associated with each other, which means that the terminal can communicate and interact with the robot based on the network. The server side can be the background server terminal device of the robot, which can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN) and cloud servers that provide basic cloud computing services such as big data and artificial intelligence platforms. The software can be an application that implements the control method of the robot, a computer program, and a storage medium that carries the computer program. It should be understood that, based on different design requirements of actual applications, in different feasible embodiments, the terminal, server, and software that apply the robot control method provided in the embodiments of the present application may of course also be in other forms not listed here, and the robot control method provided in the embodiments of the present application does not specifically limit this.

[0070] In addition, the present application can also be used in many general or special computer system environments or configurations. For example: robots, personal computers, server computers, handheld devices or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, personal computers (PC), minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0071] For ease of understanding and explanation, the following text will take the application of the robot control method provided by the embodiment of the present application to the electronic device configured by the robot itself as an example to explain in detail each specific embodiment of the present application. The implementation of the robot control method provided by the embodiment of the present application in any of the above-mentioned forms of the subject matter can refer to the process of applying the robot control method to the electronic device described later.

[0072] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0073] Please refer to Figure 1 , Figure 1 The following is a flow chart of the steps of the robot control method provided in some embodiments of the present application. Figure 1 The following other steps flow chart shows the execution order of some method steps, but based on different design requirements of actual applications, the robot control method provided in the embodiment of the present application can certainly adopt an execution order different from the method steps shown in the figure. That is, Figure 1 The order of the steps in the method shown does not constitute a limitation on the execution logic order of the robot control method provided in the embodiment of the present application. Figure 1 Reasonable changes in the sequence of steps in the method shown should be included in the protection scope of the robot control method provided in the embodiments of the present application.

[0074] like Figure 1 As shown, in some embodiments, the electronic device applies the robot control method provided by the embodiments of the present application, which may include steps S101 to S104.

[0075] Step S101: Generate a behavior tree based on task intention information corresponding to the user instruction; the task intention information is used to represent the user's intention to instruct the robot to perform an operation task, the tree nodes of the behavior tree are used to represent the subtasks of the operation task performed by the robot, and the node information of each tree node is used to represent the execution condition of the subtask corresponding to the tree node.

[0076] After receiving the instruction (user instruction) issued by the user to the robot, the electronic device understands the intention of the user to instruct the robot to perform the operation task (task intention information). Then, the electronic device generates a behavior tree based on the task intention information, that is, the electronic device represents the subtasks of the operation task performed by the robot through the tree nodes of the behavior tree, and represents the execution conditions of the subtasks corresponding to the tree nodes through the node information of each tree node of the behavior tree.

[0077] In some embodiments, the user instruction may be a natural voice instruction issued by the user to the robot. The electronic device may receive the voice instruction input by the user to the robot through a microphone of the robot.

[0078] In other embodiments, the user instruction may also be a text instruction issued by the user to the robot. The electronic device may receive the text instruction input by the user to instruct the robot to perform the operation task through the visual human-computer interaction interface provided by the robot. Alternatively, the electronic device may also receive the text instruction input by the user on the visual human-computer interaction interface provided by the terminal device that is connected to the robot for communication.

[0079] The electronic device can recognize the intention of the user's instructions through the intention model, thereby obtaining the task intention information of the user instructing the robot to perform the operation task.

[0080] In some embodiments, the technical architecture of the robot control method provided in the embodiments of the present application may include an intention parsing module. The perception module may take user instructions as input, and then parse the user instructions using natural language processing technology, thereby extracting user intentions and converting them into structured task descriptions (task intention information).

[0081] It should be noted that the intention parsing module communicates and collaborates with the task planning module, dynamic adjustment module, navigation module, operation module and perception module in the technical architecture of the robot control method provided by the embodiment of the present application mentioned later through a standard interface to form an overall solution. Among them, the intention parsing module parses the user instructions to generate task intention information, and then the task planning module converts the task intention into a behavior tree. The dynamic adjustment module optimizes the behavior tree in real time according to the environment and task execution feedback, the navigation module and the operation module specifically execute the task, and the perception module provides environmental data support for the intention parsing module and the dynamic adjustment module.

[0082] In some embodiments, the robot control method provided in the embodiments of the present application may further include the following steps:

[0083] The acquired user instructions are input into a preset user intention macro model, so as to parse the user instructions through the user intention macro model and output task intention information corresponding to the user instructions.

[0084] After obtaining the instructions input by the user to instruct the robot to perform an operation task, the electronic device inputs the user instructions into a pre-trained user intention model, thereby analyzing and judging the intention of the user instructions through the user intention model, and then outputs the task intention information corresponding to the user instructions.

[0085] It should be noted that the user intention large model can be obtained by the electronic device through model training based on the constructed user samples. Among them, the user samples include: user command samples and user intention samples corresponding to the user command samples. For example, the electronic device pre-trains the intention model through user samples, that is, the intention of the user instruction sample is judged through the intention model, and the model parameters of the intention model are continuously adjusted and optimized based on the user intention sample corresponding to the user instruction sample, so that the user intention output by the intention model is basically consistent with the user intention sample.

[0086] In some embodiments, considering the generalization ability of the large model, in the process of training the large model of user intention, the electronic device can introduce more fine-grained intentions according to the scenario in which the robot performs the task, in addition to the two intentions of the user instructing the robot to perform navigation tasks and performing specific action tasks, so as to help the robot complete the task more accurately. It should be understood that based on the different design needs of actual applications, the electronic device can of course introduce specific different intentions when designing the intention model to understand the intention information of the user instructing the robot to perform the operation task. The robot control method provided in the embodiment of the present application does not limit the specific types of fine-grained intentions introduced by the electronic device in the intention model.

[0087] Step S102: controlling the robot to perform the operation task based on the behavior tree.

[0088] After the electronic device generates a behavior tree based on the task intention information corresponding to the user instruction, the electronic device immediately controls the robot to perform the operation task instructed by the user based on the behavior tree. For example, the electronic device sequentially controls the electronic device to perform the subtask represented by each tree node in the entire behavior tree according to each tree node of the behavior tree and the node information of the tree node, until the goal of the operation task represented by the entire behavior tree is completed, or the maximum number of execution steps of the operation task is reached.

[0089] Step S103: receiving task execution status information fed back by the robot during the execution of the operation task.

[0090] After the electronic device controls the robot to perform the operation task based on the behavior tree, the robot will continuously and uninterruptedly feed back the task execution status information of the currently executed task to the electronic device during the process of performing the operation task. The electronic device receives the task execution status information fed back by the robot.

[0091] It should be noted that the task execution status information is information used to reflect the task execution status of the subtask currently executed by the robot (such as successful execution, failed execution, or intermediate execution status). When the robot executes each subtask represented by a tree node in the behavior tree, it can feedback the task pointing status information to the electronic device.

[0092] Step S104: When the task execution status information indicates that the operation task has an execution exception, dynamically adjust the behavior tree to obtain an updated behavior tree, and control the robot to continue to execute the operation task based on the updated behavior tree, wherein the dynamic adjustment operation includes: adjusting the subtask represented by the tree node of the behavior tree, and / or adjusting the execution condition represented by the node information of the tree node.

[0093] It should be noted that when the electronic device is controlling the robot to perform an operation task, if the robot fails to execute the current subtask or there is a risk of failure due to its own reasons or changes in the external environment, then the task execution status information fed back by the robot to the electronic device indicates that there is an execution abnormality in the operation task.

[0094] After receiving the task execution status information fed back by the robot, if the task execution status information fed back by the robot in real time currently received by the electronic device indicates that the operation task has an execution abnormality, the electronic device immediately performs a dynamic adjustment operation on the currently used behavior tree to obtain an updated behavior tree, and then the electronic device controls the robot to continue to perform the operation task instructed by the user based on the updated behavior tree until the goal of the operation task is completed or the maximum number of execution steps of the operation task is reached. The dynamic adjustment operation performed by the electronic device on the used behavior tree may be: only adjusting the subtask represented by the tree node of the behavior tree, only adjusting the execution condition represented by the node information of the tree node, or adjusting the subtask represented by the tree node and the execution condition represented by the node information of the tree node at the same time.

[0095] Exemplarily, the electronic device can specifically control the robot to plan and execute robot tasks in complex scenes according to the detailed technical steps shown below, so that the robot can adapt to the dynamically changing environment and smoothly execute the operation tasks instructed by the user. That is, the electronic device first performs the operation of initial behavior tree generation to generate an initial behavior tree BT = GenerateInitialBehaviorTree (I, E) to represent the operation tasks instructed by the user to perform the robot according to the task intention information obtained by parsing the user's instructions and the environmental information of the robot's environment. In the behavior tree BT, each tree node represents a task or subtask, and the relationship between the nodes represents the order and conditions of task execution. In addition, before performing the initial behavior tree generation, the electronic device can perform the operation of environmental perception to continuously collect the environmental data of the robot's environment through the perception module Et = PerceptionModule.getEnvironmentData(). Afterwards, in the process of controlling the robot to perform the operation task based on the behavior tree BT, the electronic device performs the operation of task execution monitoring to monitor the robot's execution of each task node in the behavior tree BT, thereby obtaining task execution feedback Ft = MonitorTaskExecution (Ti). The electronic device performs feedback analysis and adjustment operations for the acquired task execution feedback Ft, that is, according to the environmental data Et and the task execution feedback Ft, analyzes the execution status of the current behavior tree and identifies the task nodes and sequence that need to be adjusted. AnalysisResult=AnalyzeFeedback(BT,Et,Ft). If the analysis result shows that adjustment is required, the behavior tree BT is updated: BT′=AdjustBehaviorTree(BT,AnalysisResult). When the electronic device performs the operation of updating the behavior tree BT, it can dynamically adjust the task nodes and execution sequence in the behavior tree BT according to the analysis results and the preset adjustment rules, thereby obtaining an updated new behavior tree BT′=UpdateTree(BT,AnalysisResult). Then, the electronic device continues to control the robot to execute the adjusted behavior tree BT′, so that the robot continues to execute the operation task ExecuteBehaviorTree(BT′) instructed by the user. While the robot continues to execute the operation task, the electronic device continues to monitor the execution of the task, thereby cyclically performing the operation of dynamically adjusting the behavior tree until the robot achieves the purpose of executing the operation task or reaches the maximum number of execution steps.

[0096] In the embodiment of the present application, the task intention information corresponding to the user instruction is converted into the task in the behavior tree, so as to control the robot to perform the operation task indicated by the user intention through the behavior tree. Afterwards, by receiving the task execution status information fed back by the robot during the execution of the operation task, when the task execution status information indicates that the operation task of the robot has an execution abnormality, the behavior tree is dynamically adjusted to obtain an updated behavior tree, and the robot is controlled to continue to perform the operation task through the updated behavior tree. Among them, the dynamic adjustment of the behavior tree is mainly to adjust the subtask represented by the tree node of the behavior tree, and / or to adjust the execution condition represented by the node information of the tree node, that is, by dynamically adjusting the execution strategy and / or execution order of the operation task. In this way, it is possible to achieve more flexible and accurate response to user instructions for robot task planning and execution. Even in increasingly complex scenarios, the strategy and / or sequence of the operation task can be dynamically adjusted, so as to plan and execute the operation task with high adaptability, so as to respond to environmental changes in real time and adapt to complex scenarios. In other words, the embodiment of the present application improves the flexibility and accuracy of controlling the robot to plan and execute tasks, so that the robot can adapt to dynamically changing complex scenarios and work efficiently.

[0097] In addition, the embodiment of the present application uses the intention big model to recognize the intention of the user's instruction through the electronic device, thereby obtaining the task intention information of the user instructing the robot to perform the operation task, and then generates and constructs the behavior tree according to the different intentions in the task intention information, and controls the robot to perform the operation task instructed by the user by executing the behavior tree. In this way, the embodiment of the present application significantly improves the flexibility and accuracy of the robot's task planning and execution by combining the intention big model with the behavior tree.

[0098] When the controller robot is executing an operation task instructed by the user, the electronic device can dynamically adjust the operation of the behavior tree based on the task execution status information characterizing the task execution status fed back by the robot and the environmental state changes of the robot's environment.

[0099] In some embodiments, the robot control method provided in the embodiments of the present application may further include the following steps:

[0100] Acquire first environmental information of the environment in which the robot is located.

[0101] It should be noted that the first environmental information is used to characterize environmental factors that may have an impact on the robot's execution of an operation task if changed, for example, obstacles that appear on the robot's navigation path, etc.

[0102] When the electronic device controls the robot to perform an operation task based on a behavior tree, the robot continuously collects first environmental information of its environment through its own environmental perception device (such as an image acquisition device, a microphone, a temperature sensor, a pressure sensor, etc.), and then uploads the first environmental information to the electronic device, and the electronic device obtains the first environmental information.

[0103] In some embodiments, the electronic device may use the perception module in the above technical architecture to obtain the first environmental information. That is, the electronic device uses the environmental data of the environment in which the robot is located obtained by collecting environmental information of the robot or a terminal device associated with the robot through the perception module as input, and then processes the environmental data to output the first environmental information of the environment in which the robot is located.

[0104] It should be noted that the function of the perception module used by the electronic device can collect environmental data through sensors, thereby providing real-time environmental information to support task execution for the robot's task planning, execution and dynamic adjustment.

[0105] In some embodiments, in the above step S104, "dynamically adjusting the behavior tree to obtain an updated behavior tree" may include the following steps:

[0106] Based on the first environment information, a dynamic adjustment operation is performed on at least one tree node of the behavior tree.

[0107] When the electronic device receives the task execution status information fed back by the robot in real time, and the task execution status indicates that the operation task has an execution abnormality, the electronic device dynamically adjusts the behavior tree currently in use based on the first environment information collected and uploaded by the robot through the environment perception device at the current moment to obtain an updated behavior tree. For example, the electronic device only adjusts the subtask represented by the tree node of the currently used behavior tree based on the first environment information, or only adjusts the execution condition of the subtask represented by the node information of the tree node, or simultaneously adjusts the subtask represented by the tree node and the execution condition represented by the node information of the tree node.

[0108] In some embodiments, the electronic device can dynamically adjust at least one tree node of the behavior tree through the dynamic adjustment module in the above technical architecture. For example, the electronic device uses the behavior tree for controlling the robot to perform the operation task and the first environment information as inputs to the dynamic adjustment module, so that the dynamic adjustment module dynamically adjusts the task execution order and strategy in the behavior tree according to the real-time environment feedback reflected by the first environment information, and outputs the updated behavior tree.

[0109] Please refer to Figure 2 , Figure 2A schematic flowchart of the steps of dynamically adjusting a behavior tree involved in some embodiments of the robot control method provided in the embodiments of the present application.

[0110] like Figure 2 As shown, in some embodiments, the above-mentioned step of "dynamically adjusting at least one tree node of the behavior tree based on the first environment information" may include steps S201 to S203 as shown below.

[0111] Step S201: performing a status evaluation on a subtask represented by a tree node in the behavior tree based on the first environment information to obtain an execution status of the subtask represented by the tree node.

[0112] When the electronic device dynamically adjusts the robot's behavior tree based on the first environmental information, it first performs a status evaluation on the subtask represented by the tree node in the behavior tree based on whether the first environmental information affects the robot's smooth execution of the subtask currently being executed, thereby obtaining the execution status of the subtask represented by the tree node.

[0113] It should be noted that the execution state of the subtask obtained by the electronic device through the state evaluation of the subtask represented by the tree node can indicate that the subtask is successfully executed, that the subtask fails to execute, that the subtask is blocked, and that the subtask is being executed. For example, if the electronic device confirms based on the first environmental information that the environment in which the robot is located is an environment in which the operation task is completed after the current subtask is completed, then the electronic device can evaluate that the execution state of the subtask currently executed by the robot is successful. For another example, if the electronic device confirms based on the first environmental information that the environment in which the robot is located is an environment in which the current subtask is ready to be executed, then the electronic device can evaluate that the execution state of the subtask currently executed by the robot is being executed. For another example, if the electronic device confirms based on the first environmental information that the environment in which the robot is located has obstacles that affect the smooth execution of the current subtask by the robot, and the existence of the obstacle may cause the robot to be unable to complete the current subtask, then the electronic device can evaluate that the execution state of the subtask currently executed by the robot is failed. For another example, if the electronic device confirms, based on the first environmental information, that the environment in which the robot is located contains obstacles that affect the robot's smooth execution of the current subtask, and the existence of the obstacle causes the robot to be unable to start executing the current subtask, then the electronic device can evaluate that the execution status of the subtask currently being executed by the robot is blocked.

[0114] It should be noted that the execution status of the subtask obtained by the electronic device may be a result obtained by the electronic device through unified status evaluation of the subtasks represented by all tree nodes of the behavior tree, thereby indicating the execution status of the subtask represented by each tree node. Alternatively, the execution status of the subtask obtained by the electronic device may also be a result obtained by the electronic device through separate status evaluation of the subtask represented by a tree node in the behavior tree, thereby indicating only the execution status of the subtask represented by the tree node.

[0115] Step S202: When the execution status indicates that the target subtask represented by the target tree node in each of the tree nodes fails to execute, the target tree node is determined as a problem node.

[0116] After the electronic device performs a status evaluation on the subtask represented by the tree node in the behavior tree, if the execution status of the subtask represented by one or more target tree nodes in the behavior tree is task execution failure, in this case, the electronic device determines the one or more target tree nodes as problem nodes that need to be dynamically adjusted.

[0117] In some embodiments, after the electronic device performs a status evaluation on the subtask represented by the tree node in the behavior tree, if the execution status of the subtask represented by one or more target tree nodes in the behavior tree is that the task execution is blocked, then in this case, the electronic device may also determine the one or more target tree nodes as problem nodes that need to be dynamically adjusted.

[0118] Step S203: Dynamically adjust the problem node.

[0119] After confirming one or more problem nodes in the behavior tree that need to be dynamically adjusted, the electronic device immediately performs a dynamic adjustment operation on the one or more problem nodes to obtain an updated behavior tree. The electronic device may adjust only the subtask represented by the problem node, or only adjust the execution condition of the subtask represented by the node information of the problem node, or adjust both the subtask represented by the problem node and the execution condition represented by the node information of the problem node.

[0120] It should be noted that the electronic device may adjust the subtask represented by the problem node by deleting the subtask and binding a re-planned new subtask to the problem node. In addition, the electronic device may adjust the execution condition of the subtask represented by the node information of the problem node by adjusting the task execution order or other logic in the execution condition.

[0121] In some embodiments, the dynamic adjustment operation performed by the electronic device on the problem node can also be to further split the subtask represented by the current problem node into multiple subtasks, and then generate a new tree node and the node information of the tree node based on the subtasks obtained by the splitting, and at the same time establish a hierarchical relationship between the upper-level node and the lower-level node of the current problem node and the newly generated tree node.

[0122] In other embodiments, the dynamic adjustment operation performed by the electronic device on the problem node may also be to plan one or more new subtasks after the subtask represented by the current problem node, and then generate a new tree node and the node information of the tree node based on the newly planned subtasks, and at the same time establish a hierarchical relationship between the current problem node and the newly generated tree nodes.

[0123] When the electronic device dynamically adjusts the problem node in the behavior tree, it can adjust the subtask and / or the execution condition of the subtask in a targeted manner based on the specific reason why the robot fails to execute the subtask represented by the problem node.

[0124] Based on this, in the above step S202, when the execution status indicates that the target subtask represented by the target tree node in each of the tree nodes fails to execute, after the target tree node is determined as a problem node, the control method of the robot provided in the embodiment of the present application may further include the following steps:

[0125] An adjustment strategy is generated based on the failure cause of the target subtask represented by the problem node.

[0126] Before dynamically adjusting the problem node in the behavior tree, the electronic device further analyzes the failure reason of the target subtask represented by the problem node. For example, the robot may fail to execute the target subtask represented by the problem node due to factors such as environmental changes (such as the sudden appearance of obstacles) and insufficient resources (such as insufficient power). In this case, the electronic device will consider environmental changes, insufficient resources or other factors as the failure reason.

[0127] Then, the electronic device generates a corresponding adjustment strategy based on the analyzed failure reason. For example, when the failure reason is an environmental change in which an obstacle suddenly appears in front of the robot, the electronic device can generate an adjustment strategy for re-planning the task (specifically, re-planning the navigation path of the robot). In addition, the electronic device can also generate adjustment strategies such as changing the order of tasks and adding new tasks based on other failure reasons.

[0128] Based on this, the above step S203: dynamically adjusting the problem node may include the following steps:

[0129] Dynamically adjust the problem node based on the adjustment strategy.

[0130] After analyzing the failure cause of the target subtask represented by the problem node and generating an adjustment strategy corresponding to the failure cause, the electronic device can dynamically adjust the problem node based on the adjustment strategy when dynamically adjusting the problem node.

[0131] Please refer to Figure 3 , Figure 3 A schematic flowchart of the steps of dynamically adjusting a behavior tree based on an adjustment strategy involved in some embodiments of the robot control method provided in the embodiments of the present application.

[0132] like Figure 3 As shown, in some embodiments, the above-mentioned step of "dynamically adjusting the problem node based on the adjustment strategy" may include at least one of the following steps S301 to S303.

[0133] Step S301: When the adjustment strategy is to re-plan tasks, the subtask represented by the problem node is switched to the re-planned subtask.

[0134] When the electronic device dynamically adjusts the problem node in the behavior tree based on the adjustment strategy, if the electronic device generates an adjustment strategy corresponding to the failure reason of the target subtask represented by the problem node based on the analysis of the failure reason of the target subtask represented by the problem node, which is to re-plan the task, then in this case, the robot switches the target subtask represented by the problem node to a new subtask obtained by re-planning. For example, the electronic device deletes the target subtask and binds the re-planned new subtask to the problem node. For another example, the electronic device further splits the target subtask into multiple subtasks, and then generates a new tree node and the node information of the tree node based on the subtasks obtained by the split, and at the same time establishes a hierarchical relationship between the upper-level node and the lower-level node of the current problem node and the newly generated tree node.

[0135] Step S302: When the adjustment strategy is to change the task execution order, the task execution order in the execution condition represented by the node information of the problem node is adjusted.

[0136] When the electronic device dynamically adjusts the problem node in the behavior tree based on the adjustment strategy, if the adjustment strategy generated by the electronic device is to change the task execution order, then in this case, the robot adjusts the execution conditions of the target subtask represented by the node information of the problem node, that is, the electronic device adjusts the task execution order of the target subtask in the execution condition.

[0137] Step S303: when the adjustment strategy is to add a new task, a new subtask is added to the subtask represented by the problem node, and an execution condition of the new subtask is added to the execution condition represented by the condition information of the problem node.

[0138] When the electronic device dynamically adjusts the problem node in the behavior tree based on the adjustment strategy, if the adjustment strategy generated by the electronic device is to add a new task, then in this case, the electronic device adds the new subtask to the target subtask represented by the problem node, and at the same time adds the execution condition of the new subtask to the execution condition represented by the condition information of the problem node.

[0139] In some embodiments, if the adjustment strategy generated by the electronic device is to add a new task, the electronic device can also plan one or more new subtasks after the subtask represented by the current problem node, and then generate a new tree node and the node information of the tree node based on the newly planned subtasks, and at the same time establish a hierarchical relationship between the current problem node and the newly generated tree nodes.

[0140] Please refer to Figure 4 , Figure 4 A flow chart of a strategy for dynamically adjusting behavior tree execution involved in some embodiments of the robot control method provided in the embodiments of the present application.

[0141] like Figure 4 As shown, in some embodiments, the process of constructing and dynamically adjusting the behavior tree that the electronic device refers to when the robot performs an operation task may be:

[0142] After the electronic device generates the initial behavior tree based on the intention judgment of the user's instructions, in the process of controlling the robot to perform the operation task based on the behavior tree, the electronic device evaluates the state of each tree node in the behavior tree, thereby determining the execution state (success, failure, in progress, blocked, etc.) of the subtask represented by the tree node, and then identifies the problem node. That is, according to the result of the state evaluation of the tree node, the problem node where the robot has problems (failed to execute or blocked) when executing the task is identified. Then, the electronic device analyzes the failure reason of the subtask represented by the problem node, which may be due to environmental changes, insufficient resources, etc., resulting in the failure of the subtask execution, and generates a corresponding adjustment strategy (re-planning tasks, changing the order of tasks, adding new tasks, etc.) according to the failure reason. Finally, the electronic device adjusts the problem node based on the generated adjustment strategy to achieve the update of the behavior tree, and the electronic device controls the robot to execute the adjusted new behavior tree, so as to continue to execute the operation task instructed by the user.

[0143] Exemplarily, when the electronic device generates an adjustment strategy based on the failure reason of the target subtask represented by the problem node, it is assumed that the target subtask is "carrying item A to location B" performed by the robot, and the failure reason of the target subtask is: the robot encounters an obstacle when performing the task of "carrying item A to location B", resulting in path planning failure. Then the electronic device can generate a corresponding adjustment strategy based on the following processes 1 to 5: 1. Obtain environmental data Et = ObstacleDetected and feedback Ft = PathPlanningFailure of the robot performing the task, 2. Perform a status evaluation on the current tree node based on Et and Ft: NodeStatus (NavigateToA) = Failure, 3. Identify the problem node based on the result of the status evaluation: ProblemNodes = {NavigateToA}, 4. Analyze the reason for the failure of the task execution of the problem node: FailureReasons = ObstacleInPath, 5. Generate a corresponding adjustment strategy (replan the path around the obstacle) based on the failure reason: AdjustmentStrategy = ReplanPathAroundObstacle.

[0144] In some embodiments, the electronic device can use the navigation module and the operation module in the above technical architecture to control the robot to perform specific tasks. For example, the electronic device inputs the target position that the robot needs to reach when performing a certain task into the navigation module, and the navigation module calculates and outputs the optimal navigation path based on the target position and the environmental map of the robot's environment, and controls the robot to move along the navigation path. In addition, the electronic device inputs the operation instruction that instructs the robot to perform a specific operation into the operation module, so that the operation module responds to the operation instruction to perform a specific operation task (such as grabbing, carrying, etc.) and outputs the execution result.

[0145] In this embodiment, the electronic device combines the real-time environmental feedback reflected by the first environmental information and the task execution status (task execution status information) fed back by the robot to dynamically adjust the behavior tree in real time, such as adjusting the task execution order and strategy of the subtasks represented by one or more tree nodes in the behavior tree in real time, thereby improving the flexibility and adaptability of planning robot tasks and controlling the robot to perform tasks.

[0146] In addition to adjusting the behavior tree based on the task execution status information fed back by the robot and the changing environmental information (first environmental information), the electronic device can also combine the task intention information corresponding to the user's instructions and the environmental information of the robot's environment (second environmental information) to generate an initial behavior tree to represent the operation task that the user instructs the robot to perform.

[0147] In some embodiments, the robot control method provided in the embodiments of the present application may further include the following steps:

[0148] Acquire second environmental information of the environment where the robot is located.

[0149] It should be noted that the second environmental information is environmental information that can be collected by the robot or a terminal device connected to the robot through the environmental sensing device for information collection of the environment in which the robot is located. For example, the second environmental information can be one or more of environmental image information, environmental text description information, or even environmental point cloud information of the environment in which the robot is located.

[0150] The electronic device can control the robot's environment perception device to collect environment information while obtaining user instructions, thereby obtaining the second environment information.

[0151] Please refer to Figure 5 , Figure 5 for Figure 1 Schematic diagram of the detailed process flow of step S101.

[0152] like Figure 5 As shown, in some embodiments, step S101: generating a behavior tree based on task intention information corresponding to a user instruction may include steps S501 and S502 as shown below.

[0153] Step S501: Perform robot task planning based on the task intention information corresponding to the user instruction and the second environment information to obtain at least one subtask of the robot and the execution condition of the at least one subtask.

[0154] When receiving a user instruction input to a robot and simultaneously obtaining the second environment information of the robot's environment, the electronic device inputs the task intention information and the second environment information obtained by parsing the user instruction into a large language model (LLM), thereby using the LLM model to perform robot task planning based on the task intention information and the second environment information, and outputs at least one subtask and the execution condition of the at least one subtask. In this way, the electronic device can use the at least one subtask and the execution condition of the at least one subtask output by the LLM model as the at least one subtask and the execution condition of the at least one subtask of the operation task that the user instructs the robot to perform in the current environment where the robot is located through the input instruction.

[0155] Step S502: Binding the at least one subtask to a tree node of the behavior tree, and fusing the execution condition with the node condition information of the tree node to construct a behavior tree for the robot.

[0156] After obtaining at least one subtask of the operation task and the execution condition of the at least one subtask, the electronic device binds the at least one subtask to the tree nodes of the behavior tree one by one, and merges the execution condition of the at least one subtask with the node information of the tree node bound to the subtask, thereby constructing an initial behavior tree of the robot for characterizing the operation task of the robot.

[0157] In some embodiments, after obtaining at least one subtask of the operation task and the execution condition of the at least one subtask, the electronic device can also directly generate a behavior tree based on the hierarchical structure between at least one subtask, that is, each subtask is a tree node of the behavior tree, and the execution condition of each subtask is the node information of the tree node.

[0158] In some embodiments, the electronic device can use the task planning module in the above architecture to generate the initial behavior tree of the robot. That is, the electronic device uses the task planning module to take the task intention information and the second environment information obtained by parsing the user's instructions as input, so that the task planning module generates an initial behavior tree according to the task intention information and the second environment information, so as to represent the hierarchical structure of the tasks and subtasks that the user instructs the robot to perform through the tree nodes of the behavior tree and the node information of the tree nodes. In this way, the electronic device can obtain the behavior tree output by the task planning module.

[0159] In this embodiment, the electronic device inputs the task intention information and the second environment information into the LLM large model by combining the LLM large model and the behavior tree, so as to use the LLM large model to perform robot task planning and output the behavior tree to characterize the operation task performed by the robot. In this way, the user's natural language instructions can be accurately converted into a specific task execution plan for the robot, thereby improving the flexibility and adaptability of the robot's task execution in a complex environment. That is, this embodiment can not only enhance the accuracy of user intention analysis through the LLM large model, but also optimize the coordination and execution efficiency of the robot's multiple tasks by using the behavior tree.

[0160] Next, a complete embodiment of the robot control method proposed in the embodiment of the present application is proposed based on the above-mentioned technical architecture.

[0161] Please refer to Figure 6 , Figure 6 A schematic diagram of the overall structure involved in some embodiments of the robot control method provided in the embodiments of the present application.

[0162] like Figure 6As shown, in some embodiments, the electronic device uses the intention parsing module to judge the intention of the user instruction by the LLM large model (using natural language processing technology to parse the user instruction, extract the user intention from the user instruction and convert it into a structured task description), so as to obtain the task intention information (navigation intention and operation intention) corresponding to the user instruction to instruct the robot to perform the operation task. Then, the electronic device generates and constructs a behavior tree according to the different intentions in the task intention information through the task planning module, and then controls the robot to perform the operation task represented by the behavior tree through the navigation module and the operation model. During the robot's operation, the electronic device uses the dynamic adjustment module to dynamically adjust the execution smoothness and strategy of the operation task represented by the behavior tree according to the environmental state and the execution result, until the final goal of the operation task is completed or the number of steps taken by the robot to perform the operation task reaches the maximum number of execution steps.

[0163] In some embodiments, the navigation module in the above technical architecture can achieve optimal path planning by calling mature technical solutions and third-party APIs. For example: when an electronic device uses the navigation module, the navigation module first performs the target position input operation to receive the target position G = GetGoalPosition(), and then the navigation module further performs the operation of building an environment map of the robot's environment, that is, using sensor data to build an environment map M = BuildMapFromSensors(). After that, the navigation module performs path planning by calling a third-party API, that is, using a third-party API (such as Google Maps API, ROS Navigation Stack, etc.) for path planning to generate an optimal path P = ThirdPartyAPI.planPath(M,G)). Finally, the navigation module performs a path execution operation based on the optimal path P to control the robot to move along the planned optimal path P: ExecutePath(P).

[0164] It should be noted that in the process of controlling the robot to plan and execute tasks based on the above technical architecture, the electronic device always uses sensors to collect environmental data through the perception module, thereby providing real-time environmental data support for the robot to plan and execute tasks.

[0165] In this embodiment, the electronic device can understand the user instructions more accurately and convert them into specific tasks and subtasks in the behavior tree by introducing the intention model to parse the user instructions. In addition, the electronic device dynamically adjusts the behavior tree through the dynamic adjustment module, that is, dynamically adjusts the task execution order and strategy in the behavior tree according to environmental feedback and task execution status, thereby improving the flexibility and adaptability of the control robot to plan and execute tasks. In addition, the electronic device uses a modular control architecture to enable independent functional modules to collaborate through standard interfaces, which can simplify the overall system maintenance and expansion, and improve the ability of the control robot to perform multi-task parallel processing.

[0166] Next, another complete embodiment of the robot control method provided by the embodiment of the present application is proposed.

[0167] Please refer to Figure 7 , Figure 7 A schematic diagram of the application flow of the robot control method provided in an embodiment of the present application.

[0168] like Figure 7 As shown, in some embodiments, when the electronic device controls the robot to plan and execute tasks, it first receives the instruction U=UserInput() input by the user to the robot through natural language. After that, the electronic device performs intent parsing processing on the instruction U, thereby generating the task intent I=IntentModel(U) by parsing the user instruction U. Then, the electronic device controls the robot to perform task planning operations to generate an initial behavior tree BT=GenerateInitialBehaviorTree(I,E) according to the task intent I and environmental information. After that, the electronic device controls the robot to execute the operation task ExecuteBehaviorTree(BT) represented by the behavior tree BT, and continuously monitors the task execution status and environmental changes in the process, so as to dynamically adjust the behavior tree to obtain the updated behavior tree BT′=AdjustBehaviorTree(BT,E,F). Finally, after the electronic device controls the robot to complete the task, it feedbacks the execution result to the user.

[0169] For example, suppose that in a complex warehouse management scenario, the user instructs the robot to complete multiple tasks, including object handling and path navigation, etc. The electronic device can control the robot to specifically implement the following steps.

[0170] First, the electronic device controls the robot to perform the task of carrying items. That is, the electronic device receives the user instruction U = "walk from the location of item A to the location of item B". The electronic device generates the task intent I = navigation through intent analysis. Then, the electronic device performs the operation generated by the behavior tree based on the task intent I to generate the initial behavior tree BT = {locate the location of item A → navigate to location A → locate the location of item B → navigate to location B} according to the intent I. After that, the electronic device controls the robot to perform the task in the behavior tree BT. In the process of the robot navigating to item A, the electronic device detects an obstacle Et = {obstacle detected} on the path through the perception module, and the electronic device immediately dynamically adjusts the behavior tree to obtain a new behavior tree BT' = {plan a new path → navigate to the location of item A → locate the location of B → navigate to location B → complete the task}, so that the electronic device controls the robot to continue to perform the tasks in the behavior tree BT' to plan a new path to bypass the obstacle.

[0171] It should be noted that, in practical applications, the control method of the robot provided in the embodiment of the present application can be applied to a variety of scenarios such as service robots, industrial robots and mobile robots. Among them, the control method of the robot provided in the embodiment of the present application can make the robot system or the system controlling the robot flexibly expand and maintain through modular design. In addition, each functional module (intention parsing module, task planning module, dynamic adjustment module, navigation module, operation module and perception module can be independently developed and optimized. Among them, the intention big model enables the robot to interact with the user more naturally, the task planning module generates a behavior tree and the dynamic adjustment module dynamically adjusts the behavior tree, ensuring the flexibility and adaptability of the robot to task execution. In this way, in a complex environment, the electronic device can dynamically optimize the task execution strategy according to real-time feedback, and improve the overall performance and user experience of the robot.

[0172] See also Figure 8 The present application also provides a robot control device that can implement the above robot control method. The device includes: a task planning module 801, a task execution module 802 and a task dynamic adjustment module 803.

[0173] The task planning module 801 is used to generate a behavior tree based on the task intention information corresponding to the user instruction; the task intention information is used to represent the user's intention to instruct the robot to perform the operation task, the tree nodes of the behavior tree are used to represent the subtasks of the operation task performed by the robot, and the node information of each tree node is used to represent the execution condition of the subtask corresponding to the tree node;

[0174] A task execution module 801, used for controlling the robot to execute the operation task based on the behavior tree;

[0175] The task dynamic adjustment module 803 is used to receive task execution status information fed back by the robot during the execution of the operation task; and, when the task execution status information indicates that the operation task has an execution abnormality, dynamically adjust the behavior tree to obtain an updated behavior tree; wherein the dynamic adjustment operation includes: adjusting the subtask represented by the tree node of the behavior tree, and / or adjusting the execution condition represented by the node information of the tree node;

[0176] The task execution module 802 is further configured to control the robot to continue to execute the operation task based on the updated behavior tree.

[0177] In some embodiments, the control device of the robot provided in the embodiments of the present application further includes:

[0178] An acquisition module, used to acquire first environmental information of the environment where the robot is located;

[0179] The task dynamic adjustment module 803 is further configured to dynamically adjust at least one tree node of the behavior tree based on the first environment information.

[0180] In some embodiments, the task dynamic adjustment module 803 is also used to perform a status evaluation on the subtask represented by the tree node in the behavior tree based on the first environmental information to obtain the execution status of the subtask represented by the tree node; when the execution status indicates that the target subtask represented by the target tree node in each of the tree nodes has failed to execute, the target tree node is determined as a problem node; and, a dynamic adjustment operation is performed on the problem node.

[0181] In some embodiments, the task dynamic adjustment module 803 is further used to generate an adjustment strategy based on the failure cause of the target subtask represented by the problem node; and to perform a dynamic adjustment operation on the problem node based on the adjustment strategy.

[0182] In some embodiments, the task dynamic adjustment module 803 is also used to switch the subtask represented by the problem node to the re-planned subtask when the adjustment strategy is to re-plan the task; to adjust the task execution order in the execution conditions represented by the node information of the problem node when the adjustment strategy is to change the task execution order; and to add a new subtask to the subtask represented by the problem node and add the execution condition of the new subtask to the execution condition represented by the condition information of the problem node when the adjustment strategy is to add a new task.

[0183] In some embodiments, the acquisition module is further used to acquire second environmental information of the environment in which the robot is located;

[0184] The task planning module 801 is further used to perform robot task planning based on the task intention information corresponding to the user instruction and the second environment information, to obtain at least one subtask of the robot and an execution condition of the at least one subtask; and to bind the at least one subtask to a tree node of the behavior tree, and to merge the execution condition with the node condition information of the tree node, so as to construct a behavior tree for the robot.

[0185] In some embodiments, the task planning module 801 is also used to input the acquired user instructions into a preset user intention big model, so as to parse the user instructions through the user intention big model and output the task intention information corresponding to the user instructions; wherein, the user intention big model is obtained by model training based on constructed user samples, and the user samples include: user instruction samples and user intention samples corresponding to the user instruction samples.

[0186] The specific implementation of the robot control device provided in the embodiment of the present application is basically the same as the specific implementation of the robot control method described above, and will not be repeated here.

[0187] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the control method of the robot when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0188] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0189] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0190] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the control method of the robot in the embodiment of this application;

[0191] Input / output interface 903, used to implement information input and output;

[0192] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0193] A bus 905 that transmits information between the various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0194] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0195] An embodiment of the present application also provides a robot, which is equipped with an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the control method of the above-mentioned robot when executing the computer program.

[0196] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the control method of the robot is implemented.

[0197] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0198] The embodiment of the present application also provides a computer program product, including a computer program. The steps implemented when the computer program is executed by a processor are basically the same as the specific embodiments of the above-mentioned robot control method, and will not be repeated here.

[0199] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0200] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0201] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0202] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0203] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0204] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0205] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0206] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0207] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0208] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0209] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A robot control method, characterized in that: The method comprises: Generate a behavior tree based on task intention information corresponding to the user instruction; the task intention information is used to represent the user's intention to instruct the robot to perform the operation task, the tree nodes of the behavior tree are used to represent the subtasks of the operation task performed by the robot, and the node information of each tree node is used to represent the execution condition of the subtask corresponding to the tree node; Controlling the robot to perform the operation task based on the behavior tree; Receiving task execution status information fed back by the robot during the process of executing the operation task; When the task execution status information indicates that the operation task has an execution exception, a dynamic adjustment operation is performed on the behavior tree to obtain an updated behavior tree, and the robot is controlled to continue to execute the operation task based on the updated behavior tree, wherein the dynamic adjustment operation includes: adjusting a subtask represented by a tree node of the behavior tree, and / or adjusting an execution condition represented by node information of the tree node.

2. The method according to claim 1, characterized in that The method further comprises: Acquire first environmental information of the environment where the robot is located; The dynamically adjusting the behavior tree to obtain an updated behavior tree includes: Based on the first environment information, a dynamic adjustment operation is performed on at least one tree node of the behavior tree.

3. The method according to claim 2, characterized in that The dynamically adjusting the at least one tree node of the behavior tree based on the first environment information includes: Performing a state evaluation on a subtask represented by a tree node in the behavior tree based on the first environment information to obtain an execution state of the subtask represented by the tree node; When the execution status indicates that the target subtask represented by the target tree node in each of the tree nodes has failed to execute, determining the target tree node as a problem node; Dynamically adjust the problem node.

4. The method according to claim 3, characterized in that After determining the target tree node as a problem node, the method further includes: Generate an adjustment strategy based on the failure cause of the target subtask represented by the problem node; The dynamically adjusting operation on the problem node includes: Dynamically adjust the problem node based on the adjustment strategy.

5. The method according to claim 4, characterized in that The dynamically adjusting the problem node based on the adjustment strategy includes at least one of the following: When the adjustment strategy is to re-plan the task, the sub-task represented by the problem node is switched to the re-planned sub-task; When the adjustment strategy is to change the task execution order, the task execution order in the execution condition represented by the node information of the problem node is adjusted; In the case where the adjustment strategy is to add a new task, a new subtask is added to the subtask represented by the problem node, and an execution condition of the new subtask is added to the execution condition represented by the condition information of the problem node.

6. The method according to claim 1, characterized in that The method further comprises: Acquire second environmental information of the environment where the robot is located; The generating of the behavior tree based on the task intention information corresponding to the user instruction includes: Performing robot task planning based on the task intention information corresponding to the user instruction and the second environment information to obtain at least one subtask of the robot and an execution condition of the at least one subtask; The at least one subtask is bound to a tree node of the behavior tree, and the execution condition is merged with the node condition information of the tree node to construct a behavior tree for the robot.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: Inputting the acquired user instruction into a preset user intention macro model, so as to parse the user instruction through the user intention macro model and output task intention information corresponding to the user instruction; The user intention large model is obtained by model training based on constructed user samples, and the user samples include: user instruction samples and user intention samples corresponding to the user instruction samples.

8. A robot control device, characterized in that: The device comprises: A task planning module, used to generate a behavior tree based on task intention information corresponding to a user instruction; the task intention information is used to represent the user's intention to instruct the robot to perform an operation task, the tree nodes of the behavior tree are used to represent subtasks of the operation task performed by the robot, and the node information of each tree node is used to represent the execution condition of the subtask corresponding to the tree node; A task execution module, used for controlling the robot to execute the operation task based on the behavior tree; The task dynamic adjustment module is used to receive task execution status information fed back by the robot during the execution of the operation task; and, when the task execution status information indicates that the operation task has an execution abnormality, dynamically adjust the behavior tree to obtain an updated behavior tree; wherein the dynamic adjustment operation includes: adjusting the subtask represented by the tree node of the behavior tree, and / or adjusting the execution condition represented by the node information of the tree node; The task execution module is further used to control the robot to continue to execute the operation task based on the updated behavior tree.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the robot control method according to any one of claims 1 to 7 when executing the computer program.

10. A robot, characterized in that: The robot includes an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the robot control method according to any one of claims 1 to 7 when executing the computer program.

11. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the robot control method according to any one of claims 1 to 7 is implemented.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the robot control method according to any one of claims 1 to 7 are implemented.

Images