Robot control method, device, equipment, robot, medium and product

Abstract

This application provides a robot control method, apparatus, device, robot, medium, and product, belonging to the field of robotics technology. The method includes: generating a behavior tree based on task intent information corresponding to a user instruction; the task intent information represents the user's intention to instruct the robot to perform an operation task, the tree nodes of the behavior tree represent the subtasks of the operation task to be performed by the robot, and the node information of the tree nodes represents the execution conditions of the subtasks corresponding to the tree nodes; controlling the robot to perform the operation task based on the behavior tree; receiving task execution status information fed back by the robot during the execution of the operation task; and dynamically adjusting the behavior tree to obtain an updated behavior tree when the task execution status information represents an execution anomaly in the operation task, and controlling the robot to continue performing the operation task based on the updated behavior tree. This application enables the robot to operate efficiently in dynamically changing and complex scenarios.

Description

Technical Field

[0001] The present application relates to the field of robotics technology, and in particular to a robot control method, device, equipment, robot, medium, and product. Background Art

[0002] Robotic task planning and execution have always played a vital role in contemporary artificial intelligence and automation technologies. In related technologies, robot control systems typically employ a single instruction set to control the robot's response to user input for task planning, and a fixed task execution sequence to control the robot's execution. As robotic application scenarios become increasingly complex, robot control systems can easily struggle to smoothly plan and execute tasks due to a lack of flexible task adjustment and execution strategies. Summary of the Invention

[0003] The main purpose of the embodiments of the present application is to propose a robot control method, device, medium and program product, aiming to improve the flexibility and accuracy of controlling the robot to plan and execute tasks, so that the robot can adapt to dynamically changing complex scenarios and work efficiently.

[0004] To achieve the above objectives, a first aspect of an embodiment of the present application provides a robot control method, the method comprising:

[0005] Generate a behavior tree based on task intent information corresponding to a user instruction; the task intent information is used to represent the user's intention to instruct the robot to perform an operation task; the tree nodes of the behavior tree are used to represent subtasks of the operation task performed by the robot; and the node information of each tree node is used to represent the execution condition of the subtask corresponding to the tree node;

[0006] controlling the robot to perform the operation task based on the behavior tree;

[0007] receiving task execution status information fed back by the robot during the execution of the operation task;

[0008] If the task execution status information indicates that an execution exception has occurred in the operation task, dynamically adjusting the behavior tree to obtain an updated behavior tree, and controlling the robot to continue executing the operation task based on the updated behavior tree. The dynamic adjustment operation includes adjusting subtasks represented by tree nodes of the behavior tree and / or adjusting execution conditions represented by node information of the tree nodes.

[0009] In some embodiments, the method further comprises:

[0010] Acquire first environmental information of the environment in which the robot is located;

[0011] The dynamically adjusting the behavior tree to obtain an updated behavior tree includes:

[0012] Based on the first environment information, a dynamic adjustment operation is performed on at least one tree node of the behavior tree.

[0013] In some embodiments, dynamically adjusting at least one tree node of the behavior tree based on the first environment information includes:

[0014] performing a status evaluation on a subtask represented by a tree node in the behavior tree based on the first environment information to obtain an execution status of the subtask represented by the tree node;

[0015] When the execution status indicates that the target subtask represented by the target tree node in each of the tree nodes has failed to execute, determining the target tree node as a problem node;

[0016] Dynamically adjust the problem node.

[0017] In some embodiments, after determining the target tree node as a problem node, the method further includes:

[0018] generating an adjustment strategy based on the failure cause of the target subtask represented by the problem node;

[0019] The dynamically adjusting operation on the problem node includes:

[0020] Dynamically adjust the problem node based on the adjustment strategy.

[0021] In some embodiments, the dynamically adjusting the problem node based on the adjustment strategy includes at least one of the following:

[0022] In the case where the adjustment strategy is to re-plan the task, switching the subtask represented by the problem node to the re-planned subtask;

[0023] In the case where the adjustment strategy is to change the task execution order, adjusting the task execution order in the execution condition represented by the node information of the problem node;

[0024] In the case where the adjustment strategy is to add a new task, a new subtask is added to the subtask represented by the problem node, and an execution condition of the new subtask is added to the execution condition represented by the condition information of the problem node.

[0025] In some embodiments, the method further comprises:

[0026] Acquiring second environmental information of the environment in which the robot is located;

[0027] Generating a behavior tree based on task intention information corresponding to a user instruction includes:

[0028] Performing robot task planning based on task intention information corresponding to the user instruction and the second environment information to obtain at least one subtask of the robot and an execution condition of the at least one subtask;

[0029] The at least one subtask is bound to a tree node of the behavior tree, and the execution condition is integrated with the node condition information of the tree node to construct a behavior tree for the robot.

[0030] In some embodiments, the method further comprises:

[0031] Inputting the acquired user instruction into a preset user intention macro model, parsing the user instruction through the user intention macro model and outputting task intention information corresponding to the user instruction;

[0032] The user intention large model is obtained by model training based on constructed user samples, and the user samples include: user instruction samples and user intention samples corresponding to the user instruction samples.

[0033] To achieve the above-mentioned objectives, a second aspect of an embodiment of the present application provides a robot control device, the device comprising:

[0034] A task planning module is configured to generate a behavior tree based on task intent information corresponding to a user instruction; the task intent information is configured to represent the user's intention to instruct the robot to perform an operation task, the tree nodes of the behavior tree are configured to represent subtasks of the operation task to be performed by the robot, and the node information of each tree node is configured to represent the execution conditions of the subtask corresponding to the tree node;

[0035] A task execution module, configured to control the robot to execute the operation task based on the behavior tree;

[0036] A task dynamic adjustment module is configured to receive task execution status information fed back by the robot during execution of the operation task; and, if the task execution status information indicates an execution anomaly in the operation task, dynamically adjust the behavior tree to obtain an updated behavior tree; wherein the dynamic adjustment operation includes adjusting subtasks represented by tree nodes of the behavior tree and / or adjusting execution conditions represented by node information of the tree nodes;

[0037] The task execution module is further configured to control the robot to continue executing the operation task based on the updated behavior tree.

[0038] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, the memory storing a computer program, and the processor implementing the method described in the first aspect when executing the computer program. When the electronic device is applied to a cloud server, the processor implements the method described in the second aspect when executing the computer program.

[0039] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a robot, which is configured with an electronic device, the electronic device including a memory and a processor, the memory storing a computer program, and the processor implementing the method described in the first aspect when executing the computer program.

[0040] To achieve the above-mentioned purpose, the fifth aspect of the embodiments of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the method described in the first aspect above.

[0041] To achieve the above-mentioned purpose, the sixth aspect of the embodiments of the present application proposes a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the method provided in the first aspect above.

[0042] The robot control method, apparatus, electronic device, robot, computer-readable storage medium, and computer program product proposed in the present application generate a behavior tree based on task intention information corresponding to a user instruction; the task intention information is used to represent the user's intention to instruct the robot to perform an operation task, the tree nodes of the behavior tree are used to represent subtasks of the operation task performed by the robot, and the node information of each tree node is used to represent the execution condition of the subtask corresponding to the tree node; the robot is controlled to perform the operation task based on the behavior tree; task execution status information fed back by the robot during the execution of the operation task is received; and when the task execution status information represents an execution abnormality of the operation task, the behavior tree is dynamically adjusted to obtain an updated behavior tree, and the robot is controlled to continue to perform the operation task based on the updated behavior tree, wherein the dynamic adjustment operation includes: adjusting the subtasks represented by the tree nodes of the behavior tree, and / or adjusting the execution conditions represented by the node information of the tree nodes.

[0043] Thus, compared to methods where a robot uses a single instruction set and a fixed execution sequence to control task planning and execution in response to user input, embodiments of the present application convert the task intent information corresponding to user instructions into tasks in a behavior tree, thereby controlling the robot to execute the operation task indicated by the user's intention through a behavior tree. Subsequently, by receiving task execution status information fed back by the robot during the operation task, and when the task execution status information indicates an execution anomaly in the robot's operation task, the behavior tree is dynamically adjusted to obtain an updated behavior tree, and the robot is then controlled to continue executing the operation task using the updated behavior tree. Dynamic adjustments to the behavior tree primarily involve adjusting the subtasks represented by the behavior tree nodes and / or the execution conditions represented by the node information within the tree nodes, that is, by dynamically adjusting the execution strategy and / or execution sequence of the operation tasks. This allows for more flexible and accurate robot task planning and execution in response to user instructions. Even in increasingly complex scenarios, dynamic adjustments to the strategy and / or sequence of the operation tasks allow for highly adaptable task planning and execution. In other words, the embodiments of the present application improve the flexibility and accuracy of controlling the robot to plan and execute tasks, thereby enabling the robot to adapt to dynamically changing complex scenarios and work efficiently. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A schematic flow chart of the steps of the robot control method provided in some embodiments of the present application;

[0045] Figure 2 A schematic flow chart of the steps involved in dynamically adjusting a behavior tree in some embodiments of the robot control method provided in the present application;

[0046] Figure 3 A flowchart illustrating the steps of dynamically adjusting a behavior tree based on an adjustment strategy in some embodiments of the robot control method provided in the embodiments of the present application;

[0047] Figure 4 A flowchart of a dynamic adjustment strategy during behavior tree execution involved in some embodiments of the robot control method provided in the embodiments of the present application;

[0048] Figure 5 for Figure 1 Schematic diagram of the detailed steps of step S101;

[0049] Figure 6 A schematic diagram of the overall structure involved in some embodiments of the robot control method provided in the embodiments of the present application;

[0050] Figure 7A schematic diagram of the application flow of the robot control method provided in an embodiment of the present application;

[0051] Figure 8 A schematic diagram of the structure of a robot control device provided in an embodiment of the present application;

[0052] Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0054] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and so on, in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0056] First, the definitions of key technical terms involved in the embodiments of this application are explained.

[0057] Behavior Tree.

[0058] A behavior tree is a decision tree used to represent a hierarchical structure of tasks and subtasks. It manages task execution flow and logical judgment through behavior nodes and node information (also called conditional nodes). As a tool for task decomposition and decision management, behavior trees provide clear and intuitive task execution logic and dynamic adjustment capabilities through hierarchical behavior nodes and conditional nodes.

[0059] Intentional big model.

[0060] The Intent Big Model is a deep learning model that uses natural language processing to extract and understand user intent from user commands. The Intent Big Model can extract complex intent information from user input, enhancing the naturalness and effectiveness of human-computer interaction.

[0061] Modular control architecture.

[0062] Modular control architecture is a design approach that divides a robotic control system into several independent functional modules. By breaking down the system into independent functional modules, such as navigation, manipulation, and perception, modular control architecture simplifies system design and maintenance. Each module in a modular control architecture can be independently developed, tested, and optimized, while communicating and collaborating through standard interfaces.

[0063] Next, the overall concept of the robot control method provided in the embodiment of the present application is explained.

[0064] Robotic task planning and execution have always played a crucial role in contemporary artificial intelligence and automation technologies. In related technologies, robot control systems typically employ a single instruction set to control the robot's response to user input for task planning, and a fixed task execution sequence to control the robot's execution. However, these systems lack flexibility and adaptability in task adjustment and execution. As robotic application scenarios become increasingly complex, robot control systems can easily struggle to smoothly plan and execute tasks due to a lack of flexible task adjustment and execution strategies.

[0065] Based on this, the embodiments of the present application provide a robot control method, device, medium and program product, which aim to improve the flexibility and accuracy of controlling the robot to plan and execute tasks, so that the robot can adapt to dynamically changing complex scenarios and work efficiently.

[0066] An embodiment of the present application generates a behavior tree based on task intention information corresponding to a user instruction; the task intention information is used to represent the user's intention to instruct the robot to perform an operation task, the tree nodes of the behavior tree are used to represent subtasks of the operation task performed by the robot, and the node information of each tree node is used to represent the execution condition of the subtask corresponding to the tree node; the robot is controlled to perform the operation task based on the behavior tree; task execution status information fed back by the robot during the execution of the operation task is received; and when the task execution status information represents an execution abnormality of the operation task, the behavior tree is dynamically adjusted to obtain an updated behavior tree, and the robot is controlled to continue to perform the operation task based on the updated behavior tree, wherein the dynamic adjustment operation includes: adjusting the subtasks represented by the tree nodes of the behavior tree, and / or adjusting the execution conditions represented by the node information of the tree nodes.

[0067] Thus, compared to methods where a robot uses a single instruction set and a fixed execution sequence to control task planning and execution in response to user input, embodiments of the present application convert the task intent information corresponding to user instructions into tasks in a behavior tree, thereby controlling the robot to execute the operation task indicated by the user's intention through a behavior tree. Subsequently, by receiving task execution status information fed back by the robot during the operation task, and when the task execution status information indicates an execution anomaly in the robot's operation task, the behavior tree is dynamically adjusted to obtain an updated behavior tree, and the robot is then controlled to continue executing the operation task using the updated behavior tree. Dynamic adjustments to the behavior tree primarily involve adjusting the subtasks represented by the behavior tree nodes and / or the execution conditions represented by the node information within the tree nodes, that is, by dynamically adjusting the execution strategy and / or execution sequence of the operation tasks. This allows for more flexible and accurate robot task planning and execution in response to user instructions. Even in increasingly complex scenarios, dynamic adjustments to the strategy and / or sequence of the operation tasks allow for highly adaptable task planning and execution. In other words, the embodiments of the present application improve the flexibility and accuracy of controlling the robot to plan and execute tasks, thereby enabling the robot to adapt to dynamically changing complex scenarios and work efficiently.

[0068] Next, the control method, device, medium and program product of the robot provided in the embodiments of the present application are specifically described through the following embodiments, and the control method of the robot provided in the embodiments of the present application is first described in detail.

[0069] It should be noted that the control method of the robot provided in the embodiment of the present application relates to the field of robot technology. The control method of the robot provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server side, and can also be software running in a terminal or a server side. In some embodiments, the terminal can be a terminal configured by the robot itself, or can be an electronic device such as a smart phone, tablet computer, laptop computer, desktop computer, etc. associated with the robot. The association of the terminal with the robot means that the terminal can communicate and interact with the robot based on the network. The server side can be the background server terminal device of the robot, which can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers. It can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The software can be an application that implements the control method of the robot, a computer program, and a storage medium that carries the computer program. It should be understood that, based on different design requirements of actual applications, in different feasible embodiments, the terminal, server side, and software that apply the robot control method provided in the embodiments of the present application may of course also be in other forms not listed here, and the robot control method provided in the embodiments of the present application does not specifically limit this.

[0070] In addition, the present application can also be used in many general or special computer system environments or configurations. For example: robots, personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, personal computers (PCs), minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0071] For ease of understanding and explanation, the following text will use the electronic device configured by the robot itself to apply the robot control method provided by the embodiment of the present application as an example to describe in detail various specific embodiments of the present application. The implementation of the robot control method provided by the embodiment of the present application in any of the above-mentioned forms of subject matter can refer to the process of applying the robot control method provided by the electronic device described below.

[0072] It should be noted that in each specific embodiment of the present application, when it comes to the need to perform relevant processing based on data related to the user's identity or characteristics, such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of such data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0073] Please refer to Figure 1 , Figure 1 The following is a flow chart of the steps in some embodiments of the robot control method provided in the present application. It should be understood that although Figure 1 The following flowcharts of other steps illustrate the execution order of some method steps, but based on the different design requirements of actual applications, the robot control method provided in the embodiment of the present application can certainly adopt an execution order different from the method steps shown in the figure. Figure 1 The order of the steps in the method shown does not constitute a limitation on the execution logic order of the robot control method provided in the embodiment of the present application. Figure 1 Reasonable changes in the sequence of steps in the method shown should be included in the scope of protection of the robot control method provided in the embodiments of the present application.

[0074] like Figure 1 As shown, in some embodiments, the electronic device applies the robot control method provided by the embodiments of the present application, which may include steps S101 to S104.

[0075] Step S101: Generate a behavior tree based on task intention information corresponding to the user instruction; the task intention information is used to represent the user's intention to instruct the robot to perform an operation task, the tree nodes of the behavior tree are used to represent the subtasks of the operation task performed by the robot, and the node information of each tree node is used to represent the execution condition of the subtask corresponding to the tree node.

[0076] After receiving a user instruction (user instruction) directed at the robot, the electronic device interprets the user's intent to instruct the robot to perform an operation task (task intention information) by interpreting the user's intent. Based on this task intention information, the electronic device then generates a behavior tree. Specifically, the electronic device uses the behavior tree's tree nodes to represent the subtasks of the operation task performed by the robot, and uses the node information of each behavior tree node to represent the execution conditions of the subtask corresponding to the tree node.

[0077] In some embodiments, the user instruction may be a natural voice instruction issued by the user to the robot. The electronic device may receive the voice instruction input by the user to the robot in natural language through the microphone of the robot.

[0078] In other embodiments, the user instructions may also be text instructions issued by the user to the robot. The electronic device may receive text instructions input by the user to instruct the robot to perform an operation task through a visual human-computer interaction interface provided by the robot. Alternatively, the electronic device may receive text instructions input by the user through a visual human-computer interaction interface provided by a terminal device in communication with the robot.

[0079] The electronic device can use the intention model to identify the intention of the user's instructions, thereby obtaining the task intention information of the user instructing the robot to perform the operation task.

[0080] In some embodiments, the technical architecture of the robot control method provided by the embodiments of the present application may include an intent parsing module. The perception module may take user instructions as input and then parse the user instructions using natural language processing technology to extract user intent and convert it into a structured task description (task intent information).

[0081] It should be noted that the intent parsing module communicates and collaborates with the task planning module, dynamic adjustment module, navigation module, operation module, and perception module in the technical architecture of the robot control method provided by the embodiment of the present application mentioned later through standard interfaces to form an overall solution. Among them, the intent parsing module parses user instructions to generate task intent information, and then the task planning module converts the task intent into a behavior tree. The dynamic adjustment module optimizes the behavior tree in real time based on the environment and task execution feedback, the navigation module and the operation module specifically execute the task, and the perception module provides environmental data support for the intent parsing module and the dynamic adjustment module.

[0082] In some embodiments, the robot control method provided in the embodiments of the present application may further include the following steps:

[0083] The acquired user instruction is input into a preset user intention macro model, so as to parse the user instruction through the user intention macro model and output the task intention information corresponding to the user instruction.

[0084] After obtaining the instructions input by the user to instruct the robot to perform an operation task, the electronic device inputs the user instructions into a pre-trained user intention model, thereby analyzing and judging the intention of the user instructions through the user intention model, and then outputs the task intention information corresponding to the user instructions.

[0085] It should be noted that the user intent large model can be obtained by electronic devices through model training based on constructed user samples. Among them, user samples include: user command samples and user intent samples corresponding to the user command samples. For example, the electronic device pre-trains the intent model with user samples, that is, the intent model judges the intent of the user command samples and continuously adjusts and optimizes the model parameters of the intent model based on the user intent samples corresponding to the user command samples, so that the user intent output by the intent model is basically consistent with the user intent sample.

[0086] In some embodiments, considering the generalization ability of the large model, in the process of training the large model of user intention, the electronic device can not only introduce the two intentions of the user instructing the robot to perform navigation tasks and perform specific action tasks, but also introduce more fine-grained intentions according to the scenario in which the robot performs the task, thereby helping the robot to complete the task more accurately. It should be understood that based on the different design needs of actual applications, the electronic device can of course introduce specific different intentions when designing the intention model to understand the intention information of the user instructing the robot to perform the operation task. The robot control method provided in the embodiment of the present application does not limit the specific types of fine-grained intentions introduced by the electronic device in the intention model.

[0087] Step S102: controlling the robot to perform the operation task based on the behavior tree.

[0088] After the electronic device generates a behavior tree based on the task intent information corresponding to the user's instruction, it immediately controls the robot to perform the operation task instructed by the user based on the behavior tree. For example, the electronic device sequentially controls the electronic device to perform the subtask represented by each tree node in the behavior tree according to the node information of each tree node until the goal of the operation task represented by the entire behavior tree is achieved or the maximum number of execution steps for the operation task is reached.

[0089] Step S103: receiving task execution status information fed back by the robot during the execution of the operation task.

[0090] After the electronic device controls the robot to perform an operation task based on the behavior tree, the robot will continuously and uninterruptedly feedback the task execution status information of the current task to the electronic device during the execution of the operation task. The electronic device receives the task execution status information fed back by the robot.

[0091] It should be noted that task execution status information reflects the execution status of the robot's currently executing subtask (e.g., success, failure, or in progress). The robot can provide feedback to the electronic device regarding task execution status information for each subtask represented by a node in the behavior tree.

[0092] Step S104: When the task execution status information indicates that the operation task has an execution exception, dynamically adjusting the behavior tree to obtain an updated behavior tree, and controlling the robot to continue executing the operation task based on the updated behavior tree, wherein the dynamic adjustment operation includes: adjusting the subtasks represented by the tree nodes of the behavior tree, and / or adjusting the execution conditions represented by the node information of the tree nodes.

[0093] It should be noted that when the electronic device is controlling the robot to perform an operation task, if the robot fails to execute the subtask currently being executed or there is a risk of failure due to its own reasons or changes in the external environment, then the task execution status information fed back by the robot to the electronic device indicates that there is an execution abnormality in the operation task.

[0094] After receiving task execution status information from the robot, if the real-time task execution status information currently received by the electronic device indicates an execution anomaly in the operation task, the electronic device immediately dynamically adjusts the currently used behavior tree to obtain an updated behavior tree. The electronic device then controls the robot based on the updated behavior tree to continue executing the operation task instructed by the user until the task goal is completed or the maximum number of execution steps for the operation task is reached. The dynamic adjustment of the used behavior tree by the electronic device may include adjusting only the subtasks represented by the tree nodes of the behavior tree, adjusting only the execution conditions represented by the node information of the tree nodes, or adjusting both the subtasks represented by the tree nodes and the execution conditions represented by the node information of the tree nodes.

[0095] For example, an electronic device can specifically control a robot to plan and execute tasks in complex scenarios according to the detailed technical steps described below, enabling the robot to adapt to dynamically changing environments and smoothly execute user-directed tasks. Specifically, the electronic device first performs an initial behavior tree generation operation. Based on the task intent information obtained from parsing the user's instructions and the environmental information of the robot's environment, it generates an initial behavior tree (BT = GenerateInitialBehaviorTree(I, E)) to represent the user-directed task. In the behavior tree (BT), each tree node represents a task or subtask, and the relationships between nodes indicate the order and conditions for task execution. Furthermore, before performing the initial behavior tree generation operation, the electronic device can perform an environmental perception operation to continuously collect environmental data about the robot's environment through the perception module (Et = PerceptionModule.getEnvironmentData()). Subsequently, while controlling the robot to perform the task based on the behavior tree (BT), the electronic device performs a task execution monitoring operation to monitor the robot's execution of each task node in the behavior tree (BT), thereby obtaining task execution feedback (Ft = MonitorTaskExecution(Ti)). The electronic device performs feedback analysis and adjustments based on the acquired task execution feedback Ft. Specifically, it analyzes the execution status of the current behavior tree based on the environmental data Et and the task execution feedback Ft, identifying the task nodes and order that require adjustment. AnalysisResult = AnalyzeFeedback(BT, Et, Ft). If the analysis results indicate that adjustment is necessary, the behavior tree BT is updated: BT′ = AdjustBehaviorTree(BT, AnalysisResult). During the behavior tree BT update, the electronic device dynamically adjusts the task nodes and execution order within the behavior tree BT based on the analysis results and pre-set adjustment rules, resulting in an updated new behavior tree BT′ = UpdateTree(BT, AnalysisResult). The electronic device then controls the robot to execute the adjusted behavior tree BT′, causing it to continue executing the user-instructed task ExecuteBehaviorTree(BT′). While the robot continues executing the task, the electronic device continues to monitor task execution, repeating the dynamic behavior tree adjustment process until the robot achieves its task goal or reaches the maximum number of execution steps.

[0096] In an embodiment of the present application, the task intent information corresponding to a user instruction is converted into a task in a behavior tree, thereby controlling the robot to execute the operation task indicated by the user's intent through the behavior tree. Subsequently, by receiving task execution status information fed back by the robot during the operation task, if the task execution status information indicates that the robot's operation task has an execution anomaly, the behavior tree is dynamically adjusted to obtain an updated behavior tree, and the robot is controlled to continue executing the operation task using the updated behavior tree. The dynamic adjustment of the behavior tree primarily involves adjusting the subtasks represented by the behavior tree nodes and / or adjusting the execution conditions represented by the node information of the tree nodes, that is, by dynamically adjusting the execution strategy and / or execution order of the operation task. This allows for more flexible and accurate response to user instructions for robot task planning and execution. Even in increasingly complex scenarios, by dynamically adjusting the strategy and / or order of the operation task, highly adaptable operation task planning and execution can be achieved, responding to environmental changes in real time and adapting to complex scenarios. In other words, the embodiments of the present application improve the flexibility and accuracy of robot task planning and execution, enabling the robot to adapt to dynamically changing complex scenarios and operate efficiently.

[0097] Furthermore, in an embodiment of the present application, an electronic device uses a large model of intent to identify the intent of a user's command, thereby obtaining information about the user's task intent for instructing the robot to perform an operation task. Based on the different intents in the task intent information, a behavior tree is then generated and constructed, and the robot is controlled to perform the user-instructed operation task by executing the behavior tree. Thus, by combining the large model of intent with the behavior tree, the embodiment of the present application significantly improves the flexibility and accuracy of the robot's task planning and execution.

[0098] When the controller robot is performing an operation task instructed by the user, the electronic device can dynamically adjust the operation of the behavior tree based on the task execution status information characterizing the task execution status fed back by the robot and the changes in the environmental state of the robot's environment.

[0099] In some embodiments, the robot control method provided in the embodiments of the present application may further include the following steps:

[0100] Acquire first environmental information of the environment in which the robot is located.

[0101] It should be noted that the first environmental information is used to represent environmental factors that may affect the robot's execution of an operation task if changed, such as obstacles appearing on the robot's navigation path.

[0102] When an electronic device controls a robot to perform an operation task based on a behavior tree, the robot continuously collects first environmental information of its environment through its own environmental perception device (such as an image acquisition device, a microphone, a temperature sensor, a pressure sensor, etc.), and then uploads the first environmental information to the electronic device, which then obtains the first environmental information.

[0103] In some embodiments, the electronic device may use the perception module in the above technical architecture to obtain the first environmental information. That is, the electronic device uses the perception module to receive environmental data of the robot's environment, obtained by collecting environmental information from the robot or a terminal device associated with the robot, as input, and then processes the environmental data to output the first environmental information of the robot's environment.

[0104] It should be noted that the function of the perception module used by electronic devices can collect environmental data through sensors, thereby providing real-time environmental information to support task execution for the robot's task planning, execution and dynamic adjustment.

[0105] In some embodiments, in step S104 above, "dynamically adjusting the behavior tree to obtain an updated behavior tree" may include the following steps:

[0106] Based on the first environment information, a dynamic adjustment operation is performed on at least one tree node of the behavior tree.

[0107] When the electronic device receives real-time task execution status information from the robot, and the task execution status indicates an execution anomaly in the task, the electronic device dynamically adjusts the currently used behavior tree based on first environmental information collected and uploaded by the robot via an environmental sensing device at that moment to obtain an updated behavior tree. For example, the electronic device may adjust only the subtask represented by a tree node of the currently used behavior tree, or only the execution condition of the subtask represented by the node information of the tree node, or may adjust both the subtask represented by the tree node and the execution condition represented by the node information of the tree node.

[0108] In some embodiments, the electronic device can dynamically adjust at least one node in a behavior tree using the dynamic adjustment module in the aforementioned technical architecture. For example, the electronic device can input a behavior tree for controlling a robot to perform an operational task and first environmental information into the dynamic adjustment module. The dynamic adjustment module then dynamically adjusts the task execution order and strategy in the behavior tree based on real-time environmental feedback reflected in the first environmental information, and outputs an updated behavior tree.

[0109] Please refer to Figure 2 , Figure 2A flowchart illustrating the steps for dynamically adjusting a behavior tree in some embodiments of the robot control method provided in the present application.

[0110] like Figure 2 As shown, in some embodiments, the above-mentioned step of "dynamically adjusting at least one tree node of the behavior tree based on the first environment information" may include steps S201 to S203 as shown below.

[0111] Step S201: performing a status evaluation on the subtask represented by the tree node in the behavior tree based on the first environment information to obtain the execution status of the subtask represented by the tree node.

[0112] When the electronic device dynamically adjusts the robot's behavior tree based on the first environmental information, it first performs a status evaluation on the subtask represented by the tree node in the behavior tree based on whether the first environmental information affects the robot's smooth execution of the subtask currently being executed, thereby obtaining the execution status of the subtask represented by the tree node.

[0113] It should be noted that the execution status of the subtask obtained by the electronic device through the status evaluation of the subtask represented by the tree node can indicate that the subtask execution was successful, that the subtask execution failed, that the subtask execution was blocked, or that the subtask is being executed. For example, if the electronic device confirms based on the first environmental information that the robot's environment is an environment in which the operation task is completed after the current subtask is completed, then the electronic device can evaluate that the execution status of the subtask currently being executed by the robot is successful. For another example, if the electronic device confirms based on the first environmental information that the robot's environment is an environment in which the robot is ready to execute the current subtask, then the electronic device can evaluate that the execution status of the subtask currently being executed by the robot is being executed. For another example, if the electronic device confirms based on the first environmental information that the robot's environment has obstacles that affect the robot's smooth execution of the current subtask, and the presence of the obstacles may cause the robot to be unable to complete the current subtask, then the electronic device can evaluate that the execution status of the subtask currently being executed by the robot is failed. For another example, if the electronic device confirms, based on the first environmental information, that the environment in which the robot is located contains obstacles that affect the robot's smooth execution of the current subtask, and the existence of the obstacle prevents the robot from starting to execute the current subtask, then the electronic device can evaluate that the execution status of the subtask currently being executed by the robot is blocked.

[0114] It should be noted that the subtask execution status obtained by the electronic device can be the result of a unified status evaluation of the subtasks represented by all tree nodes of the behavior tree, resulting in a result that indicates the execution status of the subtask represented by each tree node. Alternatively, the subtask execution status obtained by the electronic device can be the result of a separate status evaluation of the subtask represented by a tree node in the behavior tree, resulting in a result that indicates only the execution status of the subtask represented by that tree node.

[0115] Step S202: When the execution status indicates that the target subtask represented by the target tree node in each of the tree nodes has failed to execute, the target tree node is determined as a problem node.

[0116] After the electronic device performs a status evaluation on the subtasks represented by the tree nodes in the behavior tree, if the execution status of the subtasks represented by one or more target tree nodes in the behavior tree is task execution failure, in this case, the electronic device determines that the one or more target tree nodes are problem nodes that need to be dynamically adjusted.

[0117] In some embodiments, after the electronic device performs a status evaluation on the subtasks represented by the tree nodes in the behavior tree, if the execution status of the subtasks represented by one or more target tree nodes in the behavior tree is blocked, then in this case, the electronic device may also determine the one or more target tree nodes as problem nodes that need to be dynamically adjusted.

[0118] Step S203: Dynamically adjust the problem node.

[0119] After identifying one or more problematic nodes in the behavior tree that require dynamic adjustment, the electronic device immediately performs dynamic adjustment operations on the one or more problematic nodes to obtain an updated behavior tree. The electronic device may adjust only the subtask represented by the problematic node, or only the execution conditions of the subtask represented by the node information of the problematic node, or simultaneously adjust the execution conditions of both the subtask represented by the problematic node and the node information of the problematic node.

[0120] It should be noted that the electronic device's adjustment to the subtask represented by the problem node may involve deleting the subtask and binding a newly planned subtask to the problem node. Furthermore, the electronic device's adjustment to the execution conditions of the subtask represented by the node information of the problem node may involve adjusting the order of task execution or other logic within the execution conditions.

[0121] In some embodiments, the dynamic adjustment operation performed by the electronic device on the problem node can also be to further split the subtask represented by the current problem node into multiple subtasks, and then generate a new tree node and the node information of the tree node based on the subtasks obtained by the split, and at the same time establish a hierarchical relationship between the upper-level node and the lower-level node of the current problem node and the newly generated tree node.

[0122] In other embodiments, the dynamic adjustment operation performed by the electronic device on the problem node can also be to plan one or more new subtasks after the subtask represented by the current problem node, and then generate a new tree node and the node information of the tree node based on the newly planned subtasks, and at the same time establish a hierarchical relationship between the current problem node and the newly generated tree nodes.

[0123] When dynamically adjusting a problem node in a behavior tree, the electronic device can adjust the subtask and / or the execution conditions of the subtask in a targeted manner based on the specific reason why the robot failed to execute the subtask represented by the problem node.

[0124] Based on this, in the above-mentioned step S202, when the execution status indicates that the target subtask represented by the target tree node in each of the tree nodes has failed to execute, after determining the target tree node as a problem node, the robot control method provided by the embodiment of the present application may further include the following steps:

[0125] An adjustment strategy is generated based on the failure cause of the target subtask represented by the problem node.

[0126] Before dynamically adjusting the problem node in the behavior tree, the electronics analyze the failure reason for the target subtask represented by the problem node. For example, if the robot fails to execute the target subtask represented by the problem node due to factors such as environmental changes (such as the sudden appearance of an obstacle) or insufficient resources (such as low battery), the electronics will identify environmental changes, insufficient resources, or other factors as the failure reason.

[0127] The electronic device then generates a corresponding adjustment strategy based on the analyzed failure cause. For example, if the failure is caused by an environmental change such as a sudden obstacle appearing in front of the robot, the electronic device can generate an adjustment strategy to replan the task (specifically, replan the robot's navigation path). Furthermore, the electronic device can also generate adjustment strategies based on other failure causes, such as changing the order of tasks or adding new tasks.

[0128] Based on this, the above-mentioned step S203: dynamically adjusting the problem node may include the following steps:

[0129] Dynamically adjust the problem node based on the adjustment strategy.

[0130] After analyzing the failure cause of the target subtask represented by the problem node and generating an adjustment strategy corresponding to the failure cause, the electronic device can dynamically adjust the problem node based on the adjustment strategy when performing a dynamic adjustment operation on the problem node.

[0131] Please refer to Figure 3 , Figure 3 A flowchart illustrating the steps for dynamically adjusting a behavior tree based on an adjustment strategy involved in the robot control method provided in some embodiments of the present application.

[0132] like Figure 3 As shown, in some embodiments, the above-mentioned step of "dynamically adjusting the problem node based on the adjustment strategy" may include at least one of the following steps S301 to S303.

[0133] Step S301: When the adjustment strategy is to re-plan tasks, the subtask represented by the problem node is switched to the re-planned subtask.

[0134] When the electronic device dynamically adjusts the problem node in the behavior tree based on the adjustment strategy, if the electronic device generates an adjustment strategy corresponding to the failure reason of the target subtask represented by the problem node based on the analysis of the failure reason of the target subtask represented by the problem node, which is to re-plan the task, then in this case, the robot switches the target subtask represented by the problem node to the new subtask obtained by re-planning. For example, the electronic device deletes the target subtask and binds the re-planned new subtask to the problem node. For another example, the electronic device further splits the target subtask into multiple subtasks, and then generates a new tree node and the node information of the tree node based on the subtasks obtained by the split, and at the same time establishes a hierarchical relationship between the upper-level node and the lower-level node of the current problem node and the newly generated tree node.

[0135] Step S302: When the adjustment strategy is to change the task execution order, the task execution order in the execution condition represented by the node information of the problem node is adjusted.

[0136] When the electronic device dynamically adjusts the problem node in the behavior tree based on the adjustment strategy, if the adjustment strategy generated by the electronic device is to change the task execution order, then in this case, the robot adjusts the execution conditions of the target subtask represented by the node information of the problem node, that is, the electronic device adjusts the task execution order of the target subtask in the execution condition.

[0137] Step S303: When the adjustment strategy is to add a new task, a new subtask is added to the subtask represented by the problem node, and an execution condition of the new subtask is added to the execution condition represented by the condition information of the problem node.

[0138] When an electronic device dynamically adjusts a problem node in a behavior tree based on an adjustment strategy, if the adjustment strategy generated by the electronic device is to add a new task, then in this case, the electronic device adds a new subtask to the target subtask represented by the problem node, and at the same time adds the execution condition of the new subtask to the execution condition represented by the condition information of the problem node.

[0139] In some embodiments, if the adjustment strategy generated by the electronic device is to add a new task, the electronic device can also plan one or more new subtasks after the subtask represented by the current problem node, and then generate a new tree node and the node information of the tree node based on the newly planned subtasks, and at the same time establish a hierarchical relationship between the current problem node and the newly generated tree nodes.

[0140] Please refer to Figure 4 , Figure 4 A flowchart of a dynamic adjustment strategy during behavior tree execution involved in some embodiments of the robot control method provided in the embodiments of the present application.

[0141] like Figure 4 As shown, in some embodiments, the process of constructing and dynamically adjusting the behavior tree referenced by the electronic device when the robot performs an operation task may be:

[0142] After generating an initial behavior tree based on intent judgment of user commands, the electronic device then performs a status evaluation on each node in the behavior tree while controlling the robot to perform an operational task based on the behavior tree. This evaluation determines the execution status (success, failure, in progress, blocked, etc.) of the subtask represented by the node, and then identifies problematic nodes. That is, based on the results of the tree node status evaluation, the electronic device identifies problematic nodes where the robot encounters problems (failed or blocked) while performing the task. The electronic device then analyzes the failure cause of the subtask represented by the problematic node, which may be due to factors such as environmental changes or insufficient resources, and generates a corresponding adjustment strategy based on the failure cause (replanning tasks, changing the order of tasks, adding new tasks, etc.). Finally, the electronic device adjusts the problematic node based on the generated adjustment strategy to update the behavior tree, and then controls the robot to execute the adjusted new behavior tree, thereby continuing to perform the operational task instructed by the user.

[0143] For example, when an electronic device generates an adjustment strategy based on analyzing the failure cause of a target subtask represented by a problem node, assuming that the target subtask is "transporting item A to location B" performed by a robot, and the failure cause of the target subtask is that the robot encountered an obstacle while performing the "transporting item A to location B" task, resulting in a path planning failure, the electronic device can generate a corresponding adjustment strategy based on the following processes 1 to 5: 1. Obtaining environmental data Et = ObstacleDetected and feedback Ft = PathPlanningFailure from the robot's task execution; 2. Based on Et and Ft, performing a status evaluation on the current tree node: NodeStatus(NavigateToA) = Failure; 3. Identifying the problem node based on the status evaluation result: ProblemNodes = {NavigateToA}; 4. Analyzing the reasons for the failure of the task execution of the problem node: FailureReasons = ObstacleInPath; 5. Generating a corresponding adjustment strategy (replanning the path around the obstacle) based on the failure cause: AdjustmentStrategy = ReplanPathAroundObstacle.

[0144] In some embodiments, the electronic device can use the navigation module and operation module in the above-mentioned technical architecture to control the robot to perform specific tasks. For example, the electronic device inputs the target location that the robot needs to reach when performing a certain task into the navigation module, and the navigation module calculates and outputs the optimal navigation path based on the target location and the environmental map of the robot's environment, and controls the robot to move along the navigation path. In addition, the electronic device inputs an operation instruction that instructs the robot to perform a specific operation into the operation module, so that the operation module responds to the operation instruction to perform a specific operation task (such as grasping, carrying, etc.) and outputs the execution result.

[0145] In this embodiment, the electronic device combines the real-time environmental feedback reflected by the first environmental information and the task execution status (task execution status information) fed back by the robot to dynamically adjust the behavior tree in real time. For example, the task execution order and strategy of the subtasks represented by one or more tree nodes in the behavior tree are adjusted in real time, thereby improving the flexibility and adaptability of planning robot tasks and controlling the robot to perform tasks.

[0146] In addition to adjusting the behavior tree based on the task execution status information and changing environmental information (first environmental information) fed back by the robot, the electronic device can also generate an initial behavior tree based on the task intention information corresponding to the user's instructions and the environmental information of the robot's environment (second environmental information) to represent the operation task that the user instructs the robot to perform.

[0147] In some embodiments, the robot control method provided in the embodiments of the present application may further include the following steps:

[0148] Acquire second environmental information of the environment in which the robot is located.

[0149] It should be noted that the second environmental information is environmental information that can be collected by the robot or a terminal device connected to the robot through the environmental perception device. For example, the second environmental information can be one or more of environmental image information, environmental text description information, or environmental point cloud information of the robot's environment.

[0150] The electronic device can control the robot's environmental perception device to collect environmental information while obtaining user instructions, thereby obtaining second environmental information.

[0151] Please refer to Figure 5 , Figure 5 for Figure 1 Schematic diagram of the detailed process flow of step S101.

[0152] like Figure 5 As shown, in some embodiments, step S101: generating a behavior tree based on task intention information corresponding to a user instruction may include steps S501 and S502 as shown below.

[0153] Step S501: performing robot task planning based on the task intention information corresponding to the user instruction and the second environment information, and obtaining at least one subtask of the robot and an execution condition of the at least one subtask.

[0154] When the electronic device receives a user instruction input to the robot and simultaneously obtains second environmental information of the robot's environment, it inputs the task intent information and the second environmental information obtained by parsing the user instruction into a large language model (LLM), thereby using the LLM model to perform robot task planning based on the task intent information and the second environmental information, and outputs at least one subtask and the execution condition of the at least one subtask. In this way, the electronic device can use the at least one subtask and the execution condition of the at least one subtask output by the LLM model as the at least one subtask and the execution condition of the at least one subtask of the operation task that the user instructs the robot to perform in the current environment of the robot through the input instruction.

[0155] Step S502: Bind the at least one subtask to a tree node of the behavior tree, and fuse the execution condition with the node condition information of the tree node to construct a behavior tree for the robot.

[0156] After obtaining at least one subtask of the operation task and the execution condition of the at least one subtask, the electronic device binds the at least one subtask to the tree nodes of the behavior tree one by one, and fuses the execution condition of the at least one subtask with the node information of the tree node bound to the subtask, thereby constructing an initial behavior tree for the robot to represent the operation task of the robot.

[0157] In some embodiments, after obtaining at least one subtask of the operation task and the execution condition of the at least one subtask, the electronic device can also directly generate a behavior tree based on the hierarchical structure between the at least one subtask, that is, each subtask is a tree node of the behavior tree, and the execution condition of each subtask is the node information of the tree node.

[0158] In some embodiments, the electronic device can use the task planning module in the aforementioned architecture to generate an initial behavior tree for the robot. Specifically, the electronic device uses the task planning module to take the task intent information and second environmental information obtained by parsing user instructions as input. Based on this information, the task planning module generates an initial behavior tree. The behavior tree's nodes and their node information represent the hierarchical structure of the tasks and subtasks that the user instructs the robot to perform. In this way, the electronic device can obtain the behavior tree output by the task planning module.

[0159] In this embodiment, the electronic device combines the LLM large model with a behavior tree, inputting task intent information and secondary environmental information into the LLM large model. The LLM large model then performs robot task planning and outputs a behavior tree to represent the robot's operational tasks. This accurately translates the user's natural language instructions into specific task execution plans, thereby improving the robot's flexibility and adaptability in complex environments. In other words, this embodiment not only enhances the accuracy of user intent parsing through the LLM large model, but also optimizes the coordination and execution efficiency of the robot's multiple tasks through the use of behavior trees.

[0160] Next, a complete embodiment of the robot control method proposed in the embodiment of this application is proposed based on the above technical architecture.

[0161] Please refer to Figure 6 , Figure 6 Schematic diagram of the overall structure involved in some embodiments of the robot control method provided in the embodiments of the present application.

[0162] like Figure 6As shown, in some embodiments, the electronic device uses the LLM large model to judge the intention of the user instruction through the intention parsing module (using natural language processing technology to parse the user instruction, extract the user intention from the user instruction and convert it into a structured task description), thereby obtaining the task intention information (navigation intention and operation intention) corresponding to the user instruction that instructs the robot to perform the operation task. Then, the electronic device generates and constructs a behavior tree based on the different intentions in the task intention information through the task planning module, and then controls the robot to perform the operation task represented by the behavior tree through the navigation module and the operation model. During the robot's operation, the electronic device uses the dynamic adjustment module to dynamically adjust the execution smoothness and strategy of the operation task represented by the behavior tree according to the environmental state and execution results until the final goal of the operation task is completed or the number of steps the robot takes to perform the operation task reaches the maximum number of execution steps.

[0163] In some embodiments, the navigation module in the above technical architecture can achieve optimal path planning by calling mature technical solutions and third-party APIs. For example: when an electronic device uses the navigation module, the navigation module first performs the target position input operation to receive the target position G = GetGoalPosition(), and then the navigation module further performs the operation of building an environment map of the robot's environment, that is, using sensor data to build an environment map M = BuildMapFromSensors(). Afterwards, the navigation module performs path planning by calling a third-party API, that is, using a third-party API (such as Google Maps API, ROS Navigation Stack, etc.) to perform path planning and generate an optimal path P = ThirdPartyAPI.planPath(M,G)). Finally, the navigation module performs a path execution operation based on the optimal path P to control the robot to move along the planned optimal path P: ExecutePath(P).

[0164] It should be noted that in the process of controlling the robot to plan and execute tasks based on the above technical architecture, the electronic device always uses sensors through the perception module to collect environmental data, thereby providing real-time environmental data support for the robot to plan and execute tasks.

[0165] In this embodiment, the electronic device incorporates a large intent model to parse user commands, enabling a more accurate understanding of user commands and translating them into specific tasks and subtasks within a behavior tree. Furthermore, the electronic device dynamically adjusts the behavior tree through a dynamic adjustment module. Specifically, the electronic device dynamically adjusts the task execution order and strategy within the behavior tree based on environmental feedback and task execution, improving the flexibility and adaptability of the robot's task planning and execution. Furthermore, the electronic device's modular control architecture enables independent functional modules to collaborate through standard interfaces, simplifying overall system maintenance and expansion and enhancing the robot's ability to perform multi-task parallel processing.

[0166] Next, another complete embodiment of the robot control method provided by the embodiment of the present application is proposed.

[0167] Please refer to Figure 7 , Figure 7 A schematic diagram of the application flow of the robot control method provided in an embodiment of the present application.

[0168] like Figure 7 As shown, in some embodiments, when controlling a robot to plan and execute a task, the electronic device first receives a command U = UserInput() input to the robot via natural language from the user. The electronic device then performs intent parsing on the command U, thereby generating a task intent I = IntentModel(U) by parsing the user command U. The electronic device then controls the robot to perform task planning operations, generating an initial behavior tree BT = GenerateInitialBehaviorTree(I, E) based on the task intent I and environmental information. The electronic device then controls the robot to execute the operational task represented by the behavior tree BT, ExecuteBehaviorTree(BT), and continuously monitors task execution and environmental changes during this process, thereby dynamically adjusting the behavior tree to obtain an updated behavior tree BT′ = AdjustBehaviorTree(BT, E, F). Finally, after controlling the robot to complete the task, the electronic device provides feedback to the user on the execution results.

[0169] For example, assume that in a complex warehouse management scenario, a user instructs a robot to complete multiple tasks, including item handling and path navigation. The electronic device can control the robot to specifically implement the following steps.

[0170] First, the electronic device controls the robot to perform an item transport task. Specifically, the electronic device receives a user instruction U = "walk from item A to item B." Through intent analysis, the electronic device generates a task intent I = navigation. Then, based on task intent I, the electronic device executes the operations generated by the behavior tree, generating an initial behavior tree BT = {locate item A → navigate to location A → locate item B → navigate to location B}. The electronic device then controls the robot to execute the tasks in the behavior tree BT = {locate item A → navigate to location A → locate item B → navigate to location B}. The electronic device then controls the robot to execute the tasks in the behavior tree BT = {locate item A → navigate to location A → locate item B → navigate to location B}. While the robot is navigating to item A, the electronic device's perception module detects an obstacle in the path (Et = {obstacle detected}). The electronic device then dynamically adjusts the behavior tree to produce a new behavior tree BT′ = {plan new path → navigate to item A → locate item B → navigate to location B → complete task}. This allows the electronic device to continue executing the tasks in the behavior tree BT′, planning a new path to circumvent the obstacle.

[0171] It should be noted that, in actual applications, the control method of the robot provided in the embodiment of the present application can be applied to a variety of scenarios such as service robots, industrial robots and mobile robots. Among them, the control method of the robot provided in the embodiment of the present application can make the robot system or the system that controls the robot flexibly expand and maintain through modular design. In addition, each functional module (intention parsing module, task planning module, dynamic adjustment module, navigation module, operation module and perception module) can be independently developed and optimized. Among them, the intention big model enables the robot to interact with the user more naturally, the task planning module generates a behavior tree and the dynamic adjustment module dynamically adjusts the behavior tree, ensuring the flexibility and adaptability of the robot for task execution. In this way, in a complex environment, the electronic device can dynamically optimize the task execution strategy based on real-time feedback, thereby improving the overall performance of the robot and the user experience.

[0172] See also Figure 8 The embodiment of the present application also provides a robot control device that can implement the above robot control method. The device includes: a task planning module 801, a task execution module 801 and a task dynamic adjustment module 803.

[0173] The task planning module 801 is configured to generate a behavior tree based on task intent information corresponding to a user instruction; the task intent information is used to represent the user's intention to instruct the robot to perform an operation task, the tree nodes of the behavior tree are used to represent subtasks of the operation task to be performed by the robot, and the node information of each tree node is used to represent the execution conditions of the subtask corresponding to the tree node;

[0174] A task execution module 801 is configured to control the robot to execute the operation task based on the behavior tree;

[0175] The task dynamic adjustment module 803 is configured to receive task execution status information fed back by the robot during the execution of the operation task; and, if the task execution status information indicates an execution anomaly in the operation task, dynamically adjust the behavior tree to obtain an updated behavior tree. The dynamic adjustment operation may include adjusting subtasks represented by tree nodes of the behavior tree and / or adjusting execution conditions represented by node information of the tree nodes.

[0176] The task execution module 802 is further configured to control the robot to continue executing the operation task based on the updated behavior tree.

[0177] In some embodiments, the robot control device provided by the embodiments of the present application further includes:

[0178] An acquisition module, configured to acquire first environmental information of an environment in which the robot is located;

[0179] The task dynamic adjustment module 803 is further configured to dynamically adjust at least one tree node of the behavior tree based on the first environment information.

[0180] In some embodiments, the task dynamic adjustment module 803 is also used to perform a status evaluation on the subtask represented by the tree node in the behavior tree based on the first environmental information to obtain the execution status of the subtask represented by the tree node; when the execution status indicates that the target subtask represented by the target tree node in each of the tree nodes has failed to execute, the target tree node is determined as a problem node; and a dynamic adjustment operation is performed on the problem node.

[0181] In some embodiments, the task dynamic adjustment module 803 is further configured to generate an adjustment strategy based on the failure cause of the target subtask represented by the problem node; and perform a dynamic adjustment operation on the problem node based on the adjustment strategy.

[0182] In some embodiments, the task dynamic adjustment module 803 is also used to switch the subtask represented by the problem node to the re-planned subtask when the adjustment strategy is to re-plan the task; to adjust the task execution order in the execution conditions represented by the node information of the problem node when the adjustment strategy is to change the task execution order; and to add a new subtask to the subtask represented by the problem node and add the execution conditions of the new subtask to the execution conditions represented by the condition information of the problem node when the adjustment strategy is to add a new task.

[0183] In some embodiments, the acquisition module is further configured to acquire second environmental information of the environment in which the robot is located;

[0184] The task planning module 801 is further configured to perform robot task planning based on the task intent information corresponding to the user instruction and the second environment information, obtain at least one subtask of the robot and an execution condition for the at least one subtask; and bind the at least one subtask to a tree node of a behavior tree, and fuse the execution condition with the node condition information of the tree node to construct a behavior tree for the robot.

[0185] In some embodiments, the task planning module 801 is also used to input the acquired user instructions into a preset user intention big model, so as to parse the user instructions through the user intention big model and output the task intention information corresponding to the user instructions; wherein, the user intention big model is obtained by model training based on the constructed user samples, and the user samples include: user instruction samples and user intention samples corresponding to the user instruction samples.

[0186] The specific implementation of the robot control device provided in the embodiment of the present application is basically the same as the specific implementation of the robot control method described above, and will not be repeated here.

[0187] The present application also provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the aforementioned robot control method. The electronic device can be any intelligent terminal, such as a tablet computer or an in-vehicle computer.

[0188] See also Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes:

[0189] The processor 901 can be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0190] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program codes are stored in the memory 902 and are called by the processor 901 to execute the control method of the robot in the embodiments of this application.

[0191] Input / output interface 903, used to implement information input and output;

[0192] Communication interface 904, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0193] Bus 905 , which transmits information between various components of the device (e.g., processor 901 , memory 902 , input / output interface 903 , and communication interface 904 );

[0194] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0195] An embodiment of the present application also provides a robot, which is equipped with an electronic device, the electronic device including a memory and a processor, the memory storing a computer program, and the control method of the above-mentioned robot is implemented when the processor executes the computer program.

[0196] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the control method of the robot is implemented.

[0197] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0198] An embodiment of the present application also provides a computer program product, including a computer program. The steps implemented when the computer program is executed by a processor are basically the same as the specific embodiments of the above-mentioned robot control method and will not be repeated here.

[0199] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0200] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0201] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0202] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0203] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0204] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0205] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0206] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0207] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0208] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0209] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A robot control method, characterized in that: The method comprises: Generate a behavior tree based on task intent information corresponding to a user instruction; the task intent information is used to represent the user's intention to instruct the robot to perform an operation task; the tree nodes of the behavior tree are used to represent subtasks of the operation task performed by the robot; and the node information of each tree node is used to represent the execution condition of the subtask corresponding to the tree node; controlling the robot to perform the operation task based on the behavior tree; receiving task execution status information fed back by the robot during the execution of the operation task; If the task execution status information indicates that an execution exception has occurred in the operation task, dynamically adjusting the behavior tree to obtain an updated behavior tree, and controlling the robot to continue executing the operation task based on the updated behavior tree. The dynamic adjustment operation includes adjusting subtasks represented by tree nodes of the behavior tree and / or adjusting execution conditions represented by node information of the tree nodes.

2. The method according to claim 1, characterized in that The method further comprises: Acquire first environmental information of the environment in which the robot is located; The dynamically adjusting the behavior tree to obtain an updated behavior tree includes: Based on the first environment information, a dynamic adjustment operation is performed on at least one tree node of the behavior tree.

3. The method according to claim 2, characterized in that The dynamically adjusting at least one tree node of the behavior tree based on the first environment information includes: performing a status evaluation on a subtask represented by a tree node in the behavior tree based on the first environment information to obtain an execution status of the subtask represented by the tree node; When the execution status indicates that the target subtask represented by the target tree node in each of the tree nodes has failed to execute, determining the target tree node as a problem node; Dynamically adjust the problem node.

4. The method according to claim 3, characterized in that After determining the target tree node as a problem node, the method further includes: generating an adjustment strategy based on the failure cause of the target subtask represented by the problem node; The dynamically adjusting operation on the problem node includes: Dynamically adjust the problem node based on the adjustment strategy.

5. The method according to claim 4, characterized in that The dynamically adjusting the problem node based on the adjustment strategy includes at least one of the following: In the case where the adjustment strategy is to re-plan the task, switching the subtask represented by the problem node to the re-planned subtask; In the case where the adjustment strategy is to change the task execution order, adjusting the task execution order in the execution condition represented by the node information of the problem node; In the case where the adjustment strategy is to add a new task, a new subtask is added to the subtask represented by the problem node, and an execution condition of the new subtask is added to the execution condition represented by the condition information of the problem node.

6. The method according to claim 1, characterized in that The method further comprises: Acquiring second environmental information of the environment in which the robot is located; Generating a behavior tree based on task intention information corresponding to a user instruction includes: Performing robot task planning based on task intention information corresponding to the user instruction and the second environment information to obtain at least one subtask of the robot and an execution condition of the at least one subtask; The at least one subtask is bound to a tree node of the behavior tree, and the execution condition is integrated with the node condition information of the tree node to construct a behavior tree for the robot.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: Inputting the acquired user instruction into a preset user intention macro model, parsing the user instruction through the user intention macro model and outputting task intention information corresponding to the user instruction; The user intention large model is obtained by model training based on constructed user samples, and the user samples include: user instruction samples and user intention samples corresponding to the user instruction samples.

8. A robot control device, characterized in that: The device comprises: A task planning module is configured to generate a behavior tree based on task intent information corresponding to a user instruction; the task intent information is configured to represent the user's intention to instruct the robot to perform an operation task, the tree nodes of the behavior tree are configured to represent subtasks of the operation task to be performed by the robot, and the node information of each tree node is configured to represent the execution conditions of the subtask corresponding to the tree node; A task execution module, configured to control the robot to execute the operation task based on the behavior tree; A task dynamic adjustment module is configured to receive task execution status information fed back by the robot during execution of the operation task; and, if the task execution status information indicates an execution anomaly in the operation task, dynamically adjust the behavior tree to obtain an updated behavior tree; wherein the dynamic adjustment operation includes adjusting subtasks represented by tree nodes of the behavior tree and / or adjusting execution conditions represented by node information of the tree nodes; The task execution module is further configured to control the robot to continue executing the operation task based on the updated behavior tree.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the robot control method according to any one of claims 1 to 7 when executing the computer program.

10. A robot, characterized in that: The robot includes an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the robot control method according to any one of claims 1 to 7 when executing the computer program.

11. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the robot control method according to any one of claims 1 to 7 is implemented.

12. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the robot control method according to any one of claims 1 to 7 are implemented.

Images