Robot control method, device and equipment

Abstract

The embodiment of the invention provides a robot control method, device and equipment, the architecture design of a robot at least comprises a data layer and a decision-making layer, the data layer is configured with various task background data, task types supported by the robot and various meta-actions supported by the robot, and the decision-making layer deploys a large language model. The robot control method comprises the following steps: acquiring real-time sensing information of a task scene; in the decision-making layer, a meta-action sequence is generated through a large language model on the basis of cue words including data, real-time perception information, task types and various meta-actions, and the data is obtained by searching from various task background data according to initial perception information of a task instruction and a task scene; and controlling the robot to execute the meta-action sequence. Therefore, the expandability and compatibility of a robot control mode to various task scenes and various robot hardware systems are effectively improved by utilizing personalized information configuration of the data layer and the model capability of a large language model in the decision-making layer.

Description

Technical Field

[0001] This application relates to the field of robotics, and more particularly to a robot control method, apparatus, and device. Background Technology

[0002] With the development of artificial intelligence, robots can achieve automatic movement, autonomous operation, and human interaction.

[0003] In one approach, multiple functional modules, such as perception, localization, and planning modules, are hard-coded for a specific task scenario, task, and robot hardware system. Through the coordinated operation of these modules, the robot is controlled to perform a series of actions, including autonomous movement, autonomous operation, and human interaction. If the task scenario, task, or robot hardware system changes, secondary development of these functional modules is required.

[0004] It is evident that the above methods have poor scalability and compatibility. Summary of the Invention

[0005] The main purpose of this application is to propose a robot control method, device, and equipment, which at least solves the problem of poor scalability and compatibility of robot control methods.

[0006] In a first aspect, this application provides a robot control method, wherein the robot's architecture design includes at least a data layer and a decision layer, the data layer is configured with various task background information, the types of tasks that the robot supports to perform, and various meta-actions that the robot supports to perform, and the decision layer deploys a large language model; The robot control method includes: Acquire real-time perception information of the task scenario; In the decision-making layer, based on the prompts containing data, real-time perception information, task type, and multiple meta-actions, a meta-action sequence is generated through the large language model. The data is obtained from multiple task background data according to the task instructions and the initial perception information of the task scenario. The initial perception information is the real-time perception information obtained at the start of the task. Control the robot to execute the meta-action sequence.

[0007] Secondly, this application provides a robot control device, wherein the robot's architecture design includes at least a data layer and a decision layer. The data layer is configured with various task background information, the types of tasks that the robot supports to perform, and various meta-actions that the robot supports to perform. The decision layer deploys a large language model. The robot control device includes: The sensing unit is used to acquire real-time sensing information about the task scenario; A decision-making unit is used in the decision-making layer to generate a sequence of meta-actions through the large language model based on the data, the real-time perception information, the task type, and various meta-action prompts. The data is obtained from various task background data according to the task instructions and the initial perception information of the task scenario. The initial perception information is the real-time perception information at the start of the task. A control unit is used to control the robot to execute the meta-action sequence.

[0008] Thirdly, this application provides an electronic device, comprising: One or more processors; A memory that stores one or more programs, which, when executed by one or more processors, cause the one or more processors to perform the following: The robot control method described in the first aspect.

[0009] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the following: The robot control method described in the first aspect.

[0010] Fifthly, this application provides a computer program product, including a computer program, which, when executed by a processor, implements, as follows: The robot control method described in the first aspect.

[0011] The robot control method, apparatus, and device proposed in this application have a robot architecture design that includes at least a data layer and a decision layer. The data layer is configured with various task background information, task types supported by the robot, and various meta-actions supported by the robot. The task types and meta-actions supported by the robot are personalized data related to the robot's hardware system, while the various task background information are personalized data related to the task scenario and the task itself. These personalized data configured in the data layer can be flexibly modified when the robot's hardware system, task scenario, and / or task change. The decision layer deploys a large language model. During robot control, initial perception information based on task instructions and task scenarios can be used to retrieve data from various task background information. Based on prompts containing data, real-time perception information of the task scenario, task types supported by the robot, and various meta-actions supported by the robot, a sequence of meta-actions is generated through the large language model, controlling the robot to execute the sequence of meta-actions. Thus, by utilizing the personalized information configuration of the data layer and the model capabilities of the large language model in the decision layer, the scalability and compatibility of the robot control method are effectively improved. Attached Figure Description

[0012] Figure 1 This is a flowchart illustrating the robot control method provided in the embodiments of this application.

[0013] Figure 2 This is a schematic diagram of the robot control device provided in the embodiments of this application.

[0014] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0015] To enable those skilled in the art to better understand the technical solutions of this application, the technical solutions provided in this application will be described in detail below with reference to the accompanying drawings.

[0016] Exemplary embodiments will be described more fully below with reference to the accompanying drawings; however, the described exemplary embodiments may be embodied in different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete, and will enable those skilled in the art to fully understand the scope of this application.

[0017] As used herein, the term "and / or" includes any and all combinations of one or more related enumerated purposes.

[0018] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of a feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded.

[0019] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0020] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in common dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this application, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined in the embodiments of this application.

[0021] The robot can achieve autonomous movement, operation, and interaction under remote control. To enhance the robot's intelligence, autonomous movement, operation, and interaction can be achieved based on multiple functional modules such as perception, localization, and planning. These modules are implemented through hard coding, requiring code debugging for specific scenarios, robot hardware systems, and even specific tasks and manipulated objects to achieve optimal performance. If the task scenario or robot hardware system changes, secondary development of these modules is necessary. Therefore, this solution has significant limitations: it is difficult to directly migrate to other task scenarios and robot hardware systems, resulting in high migration costs and poor scalability and compatibility.

[0022] In addition, an end-to-end vision-language-action (VLA) model can be used for robot control: features are extracted from the robot's perceived information, and the extracted features are input into the end-to-end VLA model to predict the robot's next action. However, a robot is a complex system, and the end-to-end VLA model cannot fit the overall mapping relationship from the robot's complex inputs to its complex outputs. Furthermore, it cannot adapt to various task scenarios and robot hardware systems; its generalization and accuracy still need improvement.

[0023] This application provides a robot control method, apparatus, and device, employing a novel architectural design for the robot. This architecture includes at least a data layer and a decision layer. The data layer deploys various task background information, task types supported by the robot, and meta-actions supported by the robot. The decision layer deploys a large language model. Based on prompts containing information, real-time perception information of the robot, task types supported by the robot, and meta-actions supported by the robot, the large language model generates a sequence of meta-actions, controlling the robot to execute the sequence. The information is obtained from various background data. Thus, by using personalized information related to the task scenario, task, and robot hardware system as configuration information in the data layer, and flexibly applying it to the prompts in the large language model, combined with the large language model in the decision layer, robot control is achieved. This not only ensures the accuracy and precision of robot control but also allows for low-cost migration when changes occur in the robot hardware system or task scenario. Only the personalized data configured in the data layer and / or the appropriate information retrieved from the data layer need to be modified. It exhibits high scalability, compatibility, and generalization.

[0024] The embodiments of this application are applicable to robots, and in particular to embodied robots, such as wheeled / tracked robots, bipedal robots, quadrupedal robots, and biomimetic robots (such as humanoid robots and underwater biomimetic robots).

[0025] Please see Figure 1 , Figure 1 This is a flowchart illustrating the robot control method provided in an embodiment of this application. Figure 1 As shown, the robot control method includes at least the following steps S101 to S103.

[0026] S101, acquire real-time perception information of the task scenario.

[0027] Real-time perception information refers to the perception information that the robot acquires in real time within the task scenario.

[0028] In this step, the task scene may change dynamically. For example, surrounding obstacles may be dynamic and move around within the task scene; the scene may change due to changes in the robot's own position; or the scene may change due to interactive operations performed by the robot (picking up or putting down an object). By performing this step, the task scene can be perceived in real time, and real-time perception information of the task scene can be obtained, which may include 3D spatial information of the task scene, information about surrounding obstacles, etc.

[0029] S102, in the decision-making layer, based on prompt words containing data, real-time perception information, task type and multiple meta-actions, a meta-action sequence is generated through a large language model. The data is obtained from multiple task background data based on the initial perception information of the task instructions and task scenario.

[0030] The prompts follow a predetermined structured format, typically including: system role settings (e.g., "You are a robot task planner"), task background (including retrieved data), current environmental state (i.e., real-time perception information), a list of available tools or capabilities (i.e., the task types and meta-actions supported by the robot), and prior model performance (e.g., the accuracy of optional modular models). This structured prompting engineering transforms unstructured environmental information and task instructions into inputs that a large language model can efficiently understand and accurately plan.

[0031] The robot's architecture design includes at least a data layer and a decision layer. The data layer is configured with various task background information, the types of tasks the robot can perform, and various meta-actions the robot can execute. The decision layer deploys a large language model. The various task background information are personalized information related to the task scenario; different task scenarios may require different background information. The types of tasks the robot can perform and the various meta-actions the robot can execute are personalized information related to the robot's hardware system; robots using different hardware systems may support different task types and different meta-actions. By configuring personalized data in the data layer and deploying the large language model in the decision layer, personalized data and the model are decoupled. Flexible configuration of personalized data enables compatibility with various task scenarios and robot hardware systems.

[0032] Initial perception information refers to real-time perception information of the task scenario acquired at the task start moment, which is the starting point when the robot enters the task execution phase after receiving the task instructions. The robot acquires the task instructions and initial perception information of the task scenario; based on these, it can search for data in various task background materials. For example, it can use the task instructions and initial perception information as search keywords to find data from various task background materials, or it can use feature information extracted from the task instructions and initial perception information as search keywords to find data from various task background materials. Because the initial perception information is related to the task scenario in which the robot operates, the retrieved data is personalized data adapted to the task instructions and task scenario.

[0033] For example, various task background information includes, but is not limited to: scene information (such as scene type, scene identifier, etc., scene type such as home scene, restaurant scene, shopping mall scene, factory scene, logistics scene, office scene, scene identifier such as scene number, scene name), mobile mapping information (such as a map describing the robot's activity range, the location information of various objects in the map), and task interaction information (such as chat information, task instruction set, performance action set, and navigation background information. Task interaction information is related to the specific task and can be configured according to the specific task).

[0034] For example, the types of tasks that the robot supports performing include, but are not limited to: manipulation tasks (such as object grasping, object picking and placing, and various manipulation objects such as screwdrivers, wrenches, barcode scanners, brooms, irons, etc.), cleaning tasks (such as floor cleaning, table cleaning, sofa cleaning, and bed cleaning), delivery tasks (such as moving an object from point A to point B), and interactive tasks (such as chatting, guiding, and performing).

[0035] For example, the meta-actions that the robot supports include, but are not limited to: lower limb movement meta-actions (e.g., using a chassis, bipedal or quadrupedal locomotion structure to drive the robot to move from a starting point A to an ending point B), body posture meta-actions (e.g., bending and extending the lower limbs, bending and rotating the waist, bending and rotating the head), robotic arm movement meta-actions (e.g., the robotic arm end effector moves from position A to position B, and the joint posture changes of each joint of the robotic arm), and dexterous manipulation meta-actions (e.g., using a dexterous hand or gripper to perform grasping actions, releasing actions, and various hand gestures).

[0036] In this step, prompt words are generated based on at least the data, real-time perception information, the types of tasks the robot can perform, and the various meta-actions the robot can perform. These prompt words are then input into a large language model, which generates a sequence of meta-actions. The large language model, having mastered the data related to the task instructions and scenario, the real-time perception information of the task scenario, the types of tasks the robot can perform, and the various meta-actions the robot can perform, has a thorough understanding of the task instructions, the task scenario, and the robot itself. Utilizing its own model capabilities, it can accurately generate a sequence of meta-actions that conforms to the current task and the task scenario and is executable by the robot, ensuring the precision of robot control.

[0037] S103 controls the robot to execute a sequence of basic actions.

[0038] The meta-action sequence includes multiple meta-actions ordered by execution time.

[0039] In this step, corresponding control instructions can be generated based on the meta-action sequence, and the robot can complete the execution of the meta-action sequence by executing the control instructions.

[0040] In this embodiment, a novel architecture design is adopted for the robot. This architecture design includes at least a data layer and a decision layer. The data layer deploys various task background information, the types of tasks the robot can perform, and the meta-actions the robot can perform. The decision layer deploys a large language model. In a task scenario, data related to the task instructions and the task scenario can be obtained from various task background information. Based on prompts containing data, real-time perception information of the task scenario, the types of tasks the robot can perform, and the meta-actions the robot can perform, a sequence of meta-actions is generated through the large language model, and the robot is controlled to execute the sequence of meta-actions. Thus, by using personalized information related to the task scenario, task, and robot hardware system as configuration information in the data layer, and flexibly applying it to the prompts of the large language model, combined with the large language model in the decision layer, robot control is achieved. This not only ensures the accuracy and precision of robot control, but also allows for low-cost migration when the robot hardware system or task scenario changes. Only the personalized data configured in the data layer needs to be modified and / or the appropriate information retrieved from the data layer needs to be obtained. This results in high scalability and compatibility.

[0041] Below, we provide some implementation examples for task scenario perception.

[0042] In some embodiments, the robot's architecture design further includes a perception layer, which deploys multiple perception models; acquiring real-time perception information of the task scenario, including: acquiring multiple detection information collected in real time by multiple sensors, the multiple detection information including at least two of the following: voice information, image information, point cloud information, tactile information, position information, posture information, and motion information; processing the multiple detection information through multiple perception models to obtain real-time perception information in a multimodal form, the multiple perception models including at least two of the following: voice information processing model, visual information processing model, tactile information processing model, pose information processing model, and motion information processing model. Thus, on the one hand, multiple perception models provide multiple basic perception capabilities, which can be applied to multiple task scenarios, multiple tasks, and multiple robot hardware systems, allowing for low-cost migration, improving the portability and scalability of the overall architecture design, and possessing strong generalization ability; on the other hand, multiple perception models enable the processing of multiple detection information to obtain real-time perception information in a multimodal form, improving the richness of real-time perception information, thereby contributing to improving the accuracy and precision of robot control.

[0043] The perception layer is located above the decision layer. The real-time perception information output by the perception layer serves as one of the inputs to the decision layer, assisting the decision layer in determining the sequence of meta-actions to be executed by the robot more accurately based on the actual situation of the task scenario.

[0044] Among them, various sensors can be deployed in robotic and / or mission environments.

[0045] The location information may include obstacle location information and / or robot location information. Obstacle location information includes the position of the obstacle in the task scene and the relative position of the obstacle and the robot. Robot location information includes the position of the robot in the task scene and the relative position of the robot's end effector and the object to be operated. The posture information may include obstacle posture information and / or robot posture information. Obstacle posture information includes the placement angle of the obstacle and the walking posture of a dynamic obstacle. Robot posture information includes the joint posture of each joint of the robot and the posture of the robot's end effector (such as the opening and closing angle of the gripper and the posture of the finger joints). The motion information may include obstacle motion information and / or robot motion information. Obstacle motion information includes the moving speed of the obstacle and the displacement of the obstacle. Robot motion information includes the overall moving speed of the robot and the moving speed of the robot's end effector.

[0046] In this embodiment, multiple sensors are used to collect various detection information in real time. This real-time collection of various detection information using multiple sensors may include one or more of the following steps: real-time acquisition of voice information in the task scene using a voice sensor (e.g., a microphone); real-time acquisition of image information, point cloud information, obstacle position information, obstacle posture information, and / or obstacle motion information using a vision sensor (e.g., a camera, radar sensor, ultrasonic sensor); real-time acquisition of robot position information, robot posture information, and / or robot motion information using a pose sensor (e.g., an inertial measurement unit, angle sensor, encoder, odometry, positioning sensor). For example, robot posture information includes the joint angles of each joint of the robot, which are collected in real-time using angle sensors (e.g., potentiometer-type angle sensors, Hall effect angle sensors) and encoders (e.g., incremental encoders, absolute encoders); and real-time acquisition of the robot's tactile information using tactile sensors (e.g., force sensors, array-type tactile sensors, slip sensors) deployed on the robot's end effector or joints.

[0047] In this embodiment, multiple detection information is processed through various perception models to obtain real-time perception information in a multimodal form. This may include at least two steps: processing speech information using a speech information processing model to obtain a speech information processing result; processing visual information (including image information, point cloud information, obstacle position information, obstacle posture information, and / or obstacle motion information) using a visual information processing model to obtain a visual information processing result; processing tactile information using a tactile information processing model to obtain a tactile information processing result; processing robot position information and / or robot posture information using a pose information processing model to obtain a processing result corresponding to the robot position information and / or the robot posture information; and processing robot motion information using a motion information processing model to obtain a robot motion information processing result. Finally, real-time perception information containing the processing results corresponding to each of the multiple detection information is obtained, i.e., real-time perception information in a multimodal form.

[0048] For example, speech information processing models can include speech preprocessing models, automatic speech recognition (ASR), etc. Speech preprocessing models include filtering models, noise reduction models, and / or echo cancellation models; visual information processing models include object detection models, object recognition models, object tracking models, image segmentation models, image classification models, depth estimation models, point cloud segmentation models, point cloud classification models, point cloud reconstruction models, etc.; tactile information processing models include pressure signal analysis models, slip detection models, tactile texture recognition models, etc.; pose information processing models include various pose estimation models, localization models, etc.; motion information processing models include various kinematic and dynamic models.

[0049] In some embodiments, the multiple perception models include multiple local models deployed on the robot and multiple cloud models deployed in the cloud. The model accuracy of the cloud models is better than that of the local models, and the computation time of the cloud models is longer than that of the local models. Multiple detection information is processed through the multiple perception models to obtain real-time perception information in a multimodal form, including: processing the multiple detection information through multiple local models to obtain output data and confidence levels corresponding to each local model; determining whether the confidence levels corresponding to the multiple local models, the task timeliness corresponding to the task instructions, and the network signal of the task scenario meet preset requirements. The requirements include that the confidence level of various local models is less than a preset confidence threshold, the task timeliness is greater than a preset timeliness threshold, and the network signal quality is greater than a preset quality threshold. If the confidence level, task timeliness, and network signal quality of various local models all meet the preset requirements, then various cloud models are used to process various detection information to obtain the output data corresponding to each cloud model. Real-time perception information is obtained based on the output data of each cloud model. If any of the confidence level, task timeliness, or network signal quality of various local models does not meet the preset requirements, then real-time perception information is obtained based on the output data of each local model. Therefore, for various perception models, not only the robot's local model but also the cloud model is deployed. When the confidence level of the local model is low, the task timeliness corresponding to the task instruction is long, and the network signal quality of the task scenario is good, the cloud model is selected to process various detection information to improve the accuracy and precision of real-time perception information; otherwise, the local model is selected to process various detection information, prioritizing task completion time and robot response speed, effectively improving the flexibility of the perception process.

[0050] Among them, the task timeliness corresponding to the task instruction refers to the task completion time corresponding to the task instruction. This task completion time can be the task deadline or the task completion duration.

[0051] The output data corresponding to the various local models are the processing results obtained after processing various detection information through various local models. For example, the positions of multiple detection boxes output by the object detection model. The confidence scores corresponding to the various local models are also output by the various local models. For example, the confidence score corresponding to the distribution of multiple detection box positions output by the object detection model, or the confidence score corresponding to the obstacle category output by the image classification model.

[0052] The output data corresponding to the various cloud models are the processing results obtained after the various detection models have been processed by the various cloud models. They will not be listed one by one here.

[0053] In this embodiment, compared to the robot's local storage, the cloud has more hardware resources, with stronger computing and storage capabilities. For multiple perception models, larger and more accurate cloud models can be deployed, resulting in higher accuracy than local models. Consequently, cloud models consume more computational power and take longer to compute than local models. After acquiring various detection information, the robot first processes this information using multiple local models to obtain output data and confidence levels for each model. The confidence levels of each local model are then compared to preset confidence thresholds, the task timeliness of the task instructions is compared to a preset timeliness threshold, and the network signal quality of the task scenario is compared to a preset quality threshold. If the confidence levels of the local models are lower than the preset confidence threshold (indicating low confidence), the task timeliness is greater than the preset timeliness threshold (indicating long task time), or the network signal quality is greater than the preset quality threshold, the robot will be considered to have lower confidence. If a quality threshold is set (indicating good network quality), the robot can send various detection data to the cloud. In the cloud, multiple cloud models process the detection data to obtain output data corresponding to each cloud model. The robot can receive the output data returned from the cloud and determine it as real-time perception information. If any of the following three conditions are not met: the confidence level of the multiple local models is less than the preset confidence threshold, the task timeliness is greater than the preset timeliness threshold, or the network signal quality is greater than the preset quality threshold, then the robot can determine the output data of the multiple local models as real-time perception information.

[0054] Below are some examples for decision-making and prompt words.

[0055] In some embodiments, the decision layer also deploys an end-to-end model and multiple modular models; the prompt words also include model performance information corresponding to each of the multiple modular models; in the decision layer, based on the prompt words containing data, real-time perception information, task type, and multiple meta-actions, a meta-action sequence is generated through a large language model, including: in the large language model, selecting a target model from the end-to-end model and multiple modular models according to the prompt words, where the target model is either an end-to-end model or multiple modular models; and generating the meta-action sequence by calling the target model through the large language model. Thus, the large language model can select and call a suitable target model from the end-to-end model and multiple modular models, improving the flexibility of the decision layer, ensuring the accuracy of the meta-action sequence, and improving the accuracy of robot control.

[0056] The end-to-end model and multiple modular models are used to complete the meta-action generation task under the invocation of the large language model. The end-to-end model directly maps the raw input of the meta-action generation task to the final output of the meta-action generation task. The raw input is provided by the large language model to the end-to-end model, and the final output is the meta-action sequence. The multiple modular models are responsible for the multiple task stages obtained from the decomposition of the meta-action generation task, with one modular model responsible for one task stage.

[0057] The end-to-end model is a large-scale model. The accuracy of the end-to-end model is superior to that of multiple modular models, but its computation time is longer than that of multiple modular models. For example, the end-to-end model is a VLA model or a vision-tactile-language-action (VTLA) model.

[0058] Furthermore, the end-to-end model can be targeted at a local range of the task scenario. For example, if the task scenario is a home scenario, the end-to-end model can be used for the living room area of ​​that home scenario. That is, it is only used when the task execution range is within the living room area, so as to avoid the end-to-end model dealing with too wide a spatial range, which would cause too long a time consumption and affect the robot's response speed.

[0059] The model performance information corresponding to the multiple modular models may include: the accuracy of the multiple modular models (for example, the accuracy of the multiple modular models can be determined by multiple preliminary tests) and / or the confidence of the multiple modular models (the confidence of the multiple modular models can be output by themselves).

[0060] In this embodiment, prompt words can be generated based on the model performance information, data, real-time perception information, robot-supported task types, and various meta-actions corresponding to multiple modular models. For example, a preset template for prompt words can be obtained, and the model performance information, data, real-time perception information, robot-supported task types, and various meta-actions corresponding to multiple modular models can be filled into the corresponding positions of the preset template to form prompt words. Alternatively, the model performance information, data, real-time perception information, robot-supported task types, and various meta-actions corresponding to multiple modular models can be directly combined to obtain prompt words. The prompt words are input into a large language model. In the large language model, the prompt words are analyzed to select a target model from the end-to-end model and multiple modular models. For example, an end-to-end model can be selected, or multiple modular models can be selected. If the target model is an end-to-end model, the end-to-end model is called through the large language model, and the meta-action sequence is generated by the end-to-end model. If the target model is multiple modular models, the multiple modular models are called through the large language model, and the meta-action sequence is generated by the multiple modular models.

[0061] In some embodiments, the prompt words also include task intents related to the task instructions. In the decision layer, before generating a meta-action sequence using a large language model based on prompt words containing data, real-time perception information, task type, and multiple meta-actions, the robot control method further includes: performing intent recognition based on task instructions and real-time perception information to obtain the task intent. This further enhances the richness of the prompt words and improves the accuracy of the meta-action sequence generated by the large language model.

[0062] In this embodiment, task instructions and real-time perception information can be input into the intent recognition model. Based on the real-time perception information, the intent of the task instructions is recognized to obtain the task intent. Real-time perception information can provide multimodal information about the task scenario, which helps improve the accuracy of the intent recognition model in recognizing the intent of task instructions.

[0063] In cases where the task instruction is a voice instruction, the recognized text of the task instruction (such as the voice recognition of the task instruction obtained by the perception model of the perception layer) and real-time perception information can be input into the intent recognition model.

[0064] In some embodiments, the intent recognition model may employ a large language model to leverage its capabilities, thereby improving the accuracy of intent recognition and consequently the accuracy of the meta-action sequence. This large language model can be the same as or a different model from the large language model used to generate the meta-action sequence.

[0065] In some embodiments, intent recognition based on task instructions and real-time sensing information to obtain the task intent includes: when the task instruction indicates the execution of a preset long-term task, decomposing the task instruction according to the multiple sub-tasks included in the preset long-term task to obtain multiple sub-task instructions, wherein the preset long-term task is a task whose number of included sub-tasks is greater than a preset threshold; and performing intent recognition based on the multiple sub-task instructions and real-time sensing information to obtain the task intent. Thus, for long-term tasks, the accuracy of task intent recognition is improved by decomposing the task instruction. For example, the preset long-term task is a grocery shopping task. The grocery shopping task includes multiple sub-tasks such as going to the corresponding location of the vegetables according to the categories on the shopping list, selecting vegetables according to the quantity on the shopping list at the location, and checking out at the cashier. The user's task instruction is to buy vegetables according to the shopping list. Then, according to the sub-tasks included in the grocery shopping task, the task instruction can be broken down into multiple sub-task instructions such as movement instructions, grab instructions, recognition instructions, and payment instructions. The intent recognition is performed by combining multiple sub-task instructions and real-time perception information.

[0066] In some embodiments, when a task instruction directs the execution of a preset timed task, the prompt word also includes a task description of the timed task. Compared to preset long-term tasks or temporarily assigned tasks, preset timed tasks are usually simpler, and the robot may already have execution experience. Therefore, task intent recognition can be omitted, and the task description of the timed task can be directly used as part of the prompt word to improve the accuracy of the large language model in generating meta-action sequences for the timed task.

[0067] The task description of the scheduled task includes the task name (such as scheduled watering task, scheduled cleaning task, scheduled light-off task), the task triggering conditions (such as executing the scheduled task at 8 o'clock every day, or executing the scheduled task every 4 hours), and / or the execution content of the scheduled task (such as going to the balcony to get water and watering the plants).

[0068] Below, we provide some examples for information retrieval and the data layer.

[0069] In some embodiments, the data retrieval process includes: extracting keywords from task instructions to obtain instruction keywords; extracting features from the task instructions and initial perceived information to obtain text feature vectors of the task instructions and perceptual feature vectors of the initial perceived information; and retrieving data from various task background materials based on the instruction keywords, text feature vectors, and perceptual feature vectors. Thus, by utilizing keywords from task instructions, feature vectors from task instructions, and feature vectors from initial perceived information, data related to the task and task scenario can be retrieved from various task background materials, improving the accuracy of data retrieval.

[0070] Keyword extraction algorithms can be used to extract keywords from task instructions, resulting in instruction keywords. These algorithms include those based on predefined grammatical rules and pre-trained deep learning models. Instruction keywords can include action keywords related to the action requested by the task instruction, item keywords related to the item requested by the task instruction, location keywords related to the task execution location, and so on.

[0071] Specifically, features can be extracted from task instructions and initial perceptual information using pre-trained models to obtain textual feature vectors for task instructions and perceptual feature vectors for initial perceptual information. The feature extraction of task instructions and initial perceptual information can use the same or different pre-trained models; no specific model information is limited here.

[0072] In some embodiments, the data layer includes a first database, a second database, and a third database. The first database stores multiple task keywords extracted from various task background data, as well as other attributes and descriptive information corresponding to each keyword. The second database stores multiple feature vectors extracted from various task background data, as well as text information, image information, and voice information corresponding to each feature vector. The third database stores multiple relationship graphs extracted from various task background data. Data is retrieved from various task background data based on instruction keywords, text feature vectors, and perceptual feature vectors, including: selecting target keywords from multiple task keywords based on the matching results of instruction keywords and multiple task keywords; selecting target feature vectors from multiple feature vectors based on the matching results of text feature vectors and multiple feature vectors; determining entity names based on instruction keywords and / or perceptual feature vectors; and retrieving a target relationship graph from multiple relationship graphs based on the entity names and the node names contained in the multiple relationship graphs. The data includes target keywords, target feature vectors, and target relationship graphs. Therefore, by extracting various forms of data (keywords, feature vectors, relationship graphs) from multiple task background materials and storing them in multiple databases, and by retrieving the various forms of data extracted from task instructions and initial perception information in the corresponding databases, the accuracy of information retrieval is improved. The final retrieved information also includes various forms of data, thus improving the richness of the information.

[0073] In this embodiment, the instruction keywords of the task instruction can be matched with multiple task keywords in the first database to obtain the matching results of the instruction keywords and multiple task keywords. The matching results of the instruction keywords and multiple task keywords indicate the similarity between the instruction keywords and multiple task keywords. Based on the matching results of the instruction keywords and multiple task keywords, the task keyword with the highest similarity to the instruction keyword can be selected as the target keyword from the multiple task keywords. Alternatively, based on the matching results of the instruction keywords and multiple task keywords, the task keyword with a similarity greater than a first similarity threshold to the instruction keyword can be selected as the target keyword from the multiple task keywords.

[0074] In this embodiment, the text feature vector of the task instruction can be matched with multiple feature vectors in the second database to obtain the matching result between the text feature vector and the multiple feature vectors. The matching result indicates the similarity between the text feature vector and the multiple feature vectors. Based on the matching result, the feature vector with the highest similarity to the text feature vector can be selected as the target feature vector. Alternatively, based on the matching result, the feature vector with a similarity greater than a second similarity threshold to the text feature vector can be selected as the target feature vector.

[0075] Among them, the first similarity threshold and the second similarity threshold are preset thresholds.

[0076] In this embodiment, entity names can be obtained from instruction keywords. For example, if the task instruction is "go...get a coffee cup," then instruction keywords such as "get" and "coffee cup" can be extracted, and the entity name "coffee cup" can be obtained from these instruction keywords. Alternatively, entity names can be extracted based on perceptual feature vectors. For example, if the perceptual feature vector includes an image feature vector of a coffee cup, the entity name "coffee cup" can be extracted based on the image feature vector of the coffee cup. Multiple relationship graphs include multiple nodes and their corresponding node names. After obtaining the entity name, it can be matched with the node names contained in the multiple relationship graphs to find the relationship graph to which the entity name belongs, i.e., the target relationship graph.

[0077] Each relational graph also includes multiple edges, and the connection weights of the edges can be determined based on the similarity between the two nodes connected by the edge. The similarity between two nodes can refer to the similarity between the feature vectors corresponding to the two nodes (such as the image feature vector corresponding to the coffee cup).

[0078] Below, some implementation examples are provided for robot control.

[0079] In some embodiments, the architecture design further includes a control layer, which deploys a control model and controls the robot to execute a sequence of basic actions, including: converting the sequence of basic actions into motor control signals for the robot using the control model; and controlling the robot to execute the motor control signals. Thus, the control model enables accurate conversion between the sequence of basic actions and the motor control signals, providing the robot with executable motor control signals and improving the accuracy of robot control.

[0080] The meta-action sequences are described in text form. Since the robot cannot execute them directly, they need to be converted into motor control signals that the robot can execute.

[0081] The control model can be a large language model or other models, such as a pre-trained deep learning model.

[0082] Based on the description of the robot's architecture design through the above embodiments, the robot's architecture design includes at least a data layer and a decision layer, and may also include a perception layer that deploys multiple perception models and / or a model layer that deploys control models.

[0083] In some embodiments, the robot's architecture further includes a learning layer for training robot-related models using at least one training algorithm. If the robot's architecture includes a perception layer with multiple perception models deployed thereon, the learning layer trains these multiple perception models. If the robot's architecture includes a control layer with a control model deployed thereon, the learning layer trains the control model. If the decision layer deploys an end-to-end model and multiple modular models, the learning layer trains both the end-to-end model and the multiple modular models. Thus, by optimizing the models deployed in the perception, control, and / or decision layers within the learning layer, the model capabilities of these models are improved, thereby enhancing the accuracy and intelligence of robot control.

[0084] In some embodiments, at least one training algorithm in the learning layer may include: a supervised learning algorithm, a keypoint imitation learning algorithm, a telekinetic imitation learning algorithm, and / or a reinforcement learning algorithm.

[0085] For example, in a monitoring learning algorithm, models deployed in the perception, control, and / or decision layers are trained based on training samples and sample labels labeled for the training samples, so that the models can fit the mapping relationship between the training samples and the sample labels. In a keypoint imitation learning algorithm, models deployed in the perception, control, and / or decision layers are trained by detecting the body keypoints and gesture keypoints of the task executor on the video stream used as training data, thereby improving the similarity between the robot's body keypoints and the task executor's body keypoints during the execution of task instructions, and the similarity between the robot's dexterous hand keypoints and the task execution. The similarity of the key points of the operator's gestures; in the teleoperation imitation learning algorithm, based on the task execution information recorded in the teleoperation (such as task instructions, perception information, posture information, position information, action information, etc. in the teleoperation), the model deployed in the perception layer, control layer and / or decision layer is trained to fit the relationship between these task execution information; in the reinforcement learning algorithm, training data (such as perception information and action information) collected in the simulation environment and tested in the real environment can be obtained, and the model deployed in the perception layer, control layer and / or decision layer is trained based on the simulation data.

[0086] In some embodiments, after controlling the robot to execute a meta-action sequence, the robot control method further includes: acquiring the execution result of the robot executing the meta-action sequence; storing the task instructions, real-time perception information, and execution result as training data; wherein the training data is used to train a discriminator model, the discriminator model is used to predict the result of the robot's action execution, and in a real environment and / or a simulation environment, the discriminator model is used to assist reinforcement learning training in the learning layer. Thus, the execution result of the robot executing the meta-action sequence, together with the task instructions and real-time perception information, is used to train the discriminator model, and then the discriminator model is used to train the models deployed in the perception layer, control layer, and / or decision layer, realizing closed-loop feedback of robot control, achieving continuous optimization of robot control, and improving the accuracy of robot control.

[0087] The execution result of the meta-action sequence can be either execution failure or execution success.

[0088] In one implementation, after controlling the robot to execute the meta-action sequence, visual and tactile information of the robot can be acquired. Based on this visual and tactile information, it can be determined whether the robot has successfully executed the meta-action sequence, thus obtaining the execution result. For example, based on the visual and tactile information, it can be determined that the robot successfully grasped the task item and transported it from location A to location B; therefore, the execution result is determined to be successful.

[0089] The task instructions and real-time perception information are training samples in the training data, which are used to input the discriminator model. The execution result is the sample label in the training data, which is used to compare with the output data of the discriminator model to determine the model loss value of the discriminator model. Then, the parameters of the discriminator model are adjusted according to the model loss value.

[0090] In some embodiments, the real-time sensing information includes multiple modal information in various modal forms. Before storing the task instructions, real-time sensing information, and execution results as training data, the robot control method further includes: extracting features from the multiple modal information in each modal form for each modal form to obtain feature vectors corresponding to each modal information in each modal form; and deduplicating the multiple modal information in each modal form based on the feature similarity between the feature vectors corresponding to each modal information in each modal form for each modal form. Thus, by deduplicating the modal information in various modal forms, the redundancy of the real-time sensing information is reduced, thereby reducing the amount of data storage.

[0091] Among them, multiple modal forms include image modality, text modality, video modality, etc., and multiple modal information under multiple modal forms, such as multiple images in image modality, multiple texts in text modality, and multiple videos in video modality.

[0092] In this embodiment, for various modalities, feature extraction is performed on multiple modal information sets under each modality to obtain feature vectors corresponding to each modality. The feature similarity between these feature vectors is calculated. Based on the feature similarity, duplicate modal information sets with a feature similarity greater than a third similarity threshold are deduplicated. For example, only one image is retained from multiple images with high similarity. This effectively reduces duplicate data in real-time sensing information and significantly lowers data redundancy.

[0093] Please see Figure 2 , Figure 2 This is a schematic diagram of the robot control device provided in an embodiment of this application. The robot's architecture design includes at least a data layer and a decision layer. The data layer is configured with various task background information, the types of tasks the robot supports performing, and various meta-actions the robot supports performing. The decision layer deploys a large language model. For example... Figure 2 As shown, the robot control device 200 includes: The perception unit 201 is used to acquire real-time perception information of the task scene; the decision unit 202 is used in the decision layer to generate a sequence of meta-actions through a large language model based on prompts containing data, real-time perception information, task type and multiple meta-actions. The data is obtained from multiple task background data based on the task instructions and the initial perception information of the task scene at the initial moment. The initial perception information is the real-time perception information acquired at the start of the task; the control unit 203 is used to control the robot to execute the sequence of meta-actions.

[0094] In some embodiments, the architecture design further includes a perception layer, which deploys multiple perception models; the perception unit 201 is specifically used to: acquire multiple detection information collected in real time by multiple sensors, the multiple detection information including at least two of the following: voice information, image information, point cloud information, tactile information, position information, posture information and motion information; process the multiple detection information through multiple perception models to obtain real-time perception information in a multimodal form, the multiple perception models including at least two of the following: voice information processing model, visual information processing model, pose information processing model and motion information processing model.

[0095] In some embodiments, the multiple perception models include multiple local models deployed on one side of the robot and multiple cloud models deployed on the cloud side. The model accuracy of the cloud models is better than that of the local models, and the computation time of the cloud models is longer than that of the local models. The perception unit 201 is specifically used to: process multiple detection information through multiple local models to obtain output data and confidence levels corresponding to each local model; determine whether the confidence levels, task timeliness, and network signal of the task scene meet preset requirements, including that the confidence levels of the multiple local models are less than a preset confidence threshold, the task timeliness is greater than a preset timeliness threshold, and the signal quality of the network signal is greater than a preset quality threshold; if the confidence levels, task timeliness, and network signal of the multiple local models all meet the preset requirements, then the multiple detection information is processed through multiple cloud models to obtain output data corresponding to each cloud model, and real-time perception information is obtained based on the output data of the multiple cloud models; if any one of the confidence levels, task timeliness, and network signal of the multiple local models does not meet the preset requirements, then real-time perception information is obtained based on the output data of the multiple local models.

[0096] In some embodiments, the decision layer also deploys an end-to-end model and multiple modular models; the prompt words also include model performance information corresponding to the multiple modular models respectively; the decision unit 202 is specifically used to: in the large language model, select a target model from the end-to-end model and multiple modular models according to the prompt words, the target model being an end-to-end model or multiple modular models; and call the target model through the large language model to generate a meta-action sequence.

[0097] In some embodiments, the prompt words may also include task intent related to the task instruction; the robot control device may also include: an intent recognition unit (not shown in the figure), used to perform intent recognition based on the task instruction and real-time perception information to obtain the task intent.

[0098] In some embodiments, the intent recognition unit is specifically used to: when a task instruction indicates the execution of a preset long-term task, decompose the task instruction according to the multiple sub-tasks included in the preset long-term task to obtain multiple sub-task instructions, wherein the preset long-term task is a task whose number of included sub-tasks is greater than a preset number threshold; and perform intent recognition based on the multiple sub-task instructions and real-time perception information to obtain the task intent.

[0099] In some embodiments, when a task instruction instructs the execution of a preset timed task, the prompt word may also include a task description of the timed task.

[0100] In some embodiments, the data retrieval process includes: extracting keywords from task instructions to obtain instruction keywords; extracting features from task instructions and initial perception information to obtain text feature vectors of task instructions and perception feature vectors of initial perception information; and retrieving data from various task background information based on instruction keywords, text feature vectors, and perception feature vectors.

[0101] In some embodiments, the data layer includes a first database, a second database, and a third database. The first database stores multiple task keywords extracted from various task background materials, the second database stores multiple feature vectors extracted from various task background materials, and the third database stores multiple relationship graphs extracted from various task background materials. The data retrieval process, based on instruction keywords, text feature vectors, and perceptual feature vectors, involves: selecting target keywords from multiple task keywords based on the matching results between instruction keywords and multiple task keywords; selecting target feature vectors from multiple feature vectors based on the matching results between text feature vectors and multiple feature vectors; determining entity names based on instruction keywords and / or perceptual feature vectors; and retrieving a target relationship graph from multiple relationship graphs based on the entity names and the node names contained in each relationship graph. The data retrieval process includes target keywords, target feature vectors, and target relationship graphs.

[0102] In some embodiments, the architecture design also includes a control layer, which deploys a control model; the control unit 203 is specifically used to: convert the meta-action sequence into motor control signals for the robot through the control model; and control the robot to execute the motor control signals.

[0103] In some embodiments, the architecture design further includes a learning layer for training robot-related models using at least one training algorithm; in the case where the architecture design includes a perception layer and the perception layer deploys multiple perception models, the learning layer is used to train multiple perception models; in the case where the architecture design includes a control layer and the control layer deploys a control model, the learning layer is used to train a control model; in the case where the decision layer deploys an end-to-end model and multiple modular models, the learning layer is used to train the end-to-end model and multiple modular models.

[0104] In some embodiments, after controlling the robot to execute the meta-action sequence, the robot control device further includes a training data generation unit (not shown in the figure). The training data generation unit is used to: obtain the execution result of the robot executing the meta-action sequence; and store the task instructions, real-time perception information and execution results as training data. The training data is used to train a discriminator model, which is used to predict the result of the robot's action execution. In real and / or simulation environments, the discriminator model is used to assist reinforcement learning training in the learning layer.

[0105] In some embodiments, the real-time sensing information includes multiple modal information in multiple modal forms; the training data generation unit is further configured to: extract features from the multiple modal information in each modal form for each modal form to obtain feature vectors corresponding to the multiple modal information in each modal form; and deduplicate the multiple modal information in each modal form based on the feature similarity between the feature vectors corresponding to the multiple modal information in each modal form for each modal form.

[0106] The robot control device 200 provided in this application embodiment can execute the technical solution shown in the above-described robot control method embodiment. Its implementation principle and beneficial effects are similar, and will not be repeated here.

[0107] Please see Figure 3 , Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 3 As shown, the electronic device 300 includes: One or more processors 310; The memory 320 stores one or more programs that, when executed by one or more processors 310, cause the one or more processors 310 to implement the robot control method described in any of the above embodiments.

[0108] Memory 320, as a non-transitory network system, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory 320 may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory 320 may optionally include remotely located memories 320 relative to processor 310, which can be connected to processor 310 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0109] The memory 320 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 320 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 320 and is called and executed by the processor 310.

[0110] The processor 310 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0111] In some embodiments, the electronic device further includes: Input / output interfaces are used to implement information input and output; The communication interface is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). A bus transmits information between various components of a device, such as processor 310, memory 320, input / output interfaces, and communication interfaces. The processor 310, memory 320, input / output interface, and communication interface can communicate with each other within the device via a bus.

[0112] One embodiment of this application also provides a computer-readable storage medium storing computer-executable instructions for performing the robot control method described in any of the above embodiments.

[0113] An embodiment of this application also provides a computer program product, including a computer program or computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer program or computer instructions from the computer-readable storage medium and executes the computer program or computer instructions, causing the computer device to perform the robot control method described in any of the above embodiments.

[0114] The system architecture and application scenarios described in this application are intended to more clearly illustrate the technical solutions of this application and do not constitute a limitation on the technical solutions provided in this application. Those skilled in the art will understand that as system architectures evolve and new application scenarios emerge, the technical solutions provided in this application are also applicable to similar technical problems.

[0115] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0116] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0117] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0118] The above description, with reference to the accompanying drawings, illustrates some embodiments of this application, but does not limit the scope of the invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and spirit of this invention should be considered within the scope of this application.

Claims

1. A robot control method, characterized in that, The robot's architecture design includes at least a data layer and a decision layer. The data layer is configured with various task background information, the types of tasks that the robot can perform, and various meta-actions that the robot can perform. The decision layer deploys a large language model. The robot control method includes: Acquire real-time perception information of the task scenario; In the decision-making layer, based on the prompts containing data, real-time perception information, task type, and multiple meta-actions, a meta-action sequence is generated through the large language model. The data is obtained from multiple task background data according to the task instructions and the initial perception information of the task scenario. The initial perception information is the real-time perception information obtained at the start of the task. Control the robot to execute the meta-action sequence.

2. The robot control method according to claim 1, characterized in that, The architecture design also includes a perception layer, which deploys multiple perception models; The acquisition of real-time perception information of the task scenario includes: The system acquires various detection information collected in real time from multiple sensors, including at least two of the following: voice information, image information, point cloud information, tactile information, position information, posture information, and motion information. The detection information is processed by multiple perception models to obtain real-time perception information in a multimodal form. The multiple perception models include at least two of the following: speech information processing model, visual information processing model, tactile information processing model, pose information processing model, and motion information processing model.

3. The robot control method according to claim 2, characterized in that, The various perception models include multiple local models deployed on one side of the robot and multiple cloud models deployed on the cloud side. The model accuracy of the cloud models is better than that of the local models, and the computation time of the cloud models is longer than that of the local models. The process of processing multiple detection information through multiple perception models to obtain real-time perception information in a multimodal form includes: By processing the detection information using multiple local models, output data and confidence levels corresponding to the various local models are obtained respectively. Determine whether the confidence levels corresponding to the various local models, the task timeliness corresponding to the task instruction, and the network signal of the task scenario meet preset requirements. The preset requirements include that the confidence levels corresponding to the various local models are less than a preset confidence threshold, the task timeliness is greater than a preset timeliness threshold, and the signal quality of the network signal is greater than a preset quality threshold. If the confidence level, task timeliness, and network signal corresponding to the various local models all meet the preset requirements, then the various detection information are processed by the various cloud models to obtain the output data corresponding to the various cloud models. Based on the output data corresponding to the various cloud models, the real-time perception information is obtained. If any one of the confidence level, task timeliness, and network signal corresponding to the various local models fails to meet the preset requirements, then the real-time perception information is obtained based on the output data corresponding to the various local models.

4. The robot control method according to any one of claims 1 to 3, characterized in that, The decision-making layer also deploys an end-to-end model and multiple modular models; The prompt words also include model performance information corresponding to each of the modular models; In the decision-making layer, based on cue words containing data, real-time perceived information, task type, and various meta-actions, a sequence of meta-actions is generated through the large language model, including: In the large language model, based on the prompt word, a target model is selected from the end-to-end model and the multiple modular models, wherein the target model is the end-to-end model or the multiple modular models; The target model is invoked through the large language model to generate the meta-action sequence.

5. The robot control method according to any one of claims 1 to 3, characterized in that, The prompt words also include the task intent related to the task instruction; In the decision-making layer, before generating a sequence of meta-actions using the large language model based on data, real-time perception information, task type, and various meta-action prompts, the robot control method further includes: The task intent is obtained by performing intent recognition based on the task instructions and the real-time perception information.

6. The robot control method according to claim 5, characterized in that, The step of performing intent recognition based on the task instructions and the real-time perceived information to obtain the task intent includes: When the task instruction indicates the execution of a preset long-term task, the task instruction is decomposed according to the multiple sub-tasks contained in the preset long-term task to obtain multiple sub-task instructions. The preset long-term task is a task whose number of sub-tasks is greater than a preset number threshold. The task intent is obtained by performing intent recognition based on multiple sub-task instructions and the real-time perception information.

7. The robot control method according to any one of claims 1 to 3, characterized in that, When the task instruction indicates the execution of a preset timed task, the prompt word also includes a task description of the timed task.

8. The robot control method according to any one of claims 1 to 3, characterized in that, The data retrieval process includes: Keyword extraction is performed on the task instructions to obtain the instruction keywords of the task instructions; Feature extraction is performed on the task instruction and the initial perception information respectively to obtain the text feature vector of the task instruction and the perception feature vector of the initial perception information; Based on the instruction keywords, the text feature vector, and the perception feature vector, the data is retrieved from various task background materials.

9. The robot control method according to claim 8, characterized in that, The data layer includes a first database, a second database, and a third database. The first database stores multiple task keywords extracted from various task background materials. The second database stores multiple feature vectors extracted from various task background materials. The third database stores multiple relationship graphs extracted from various task background materials. The step of searching for data from various task background materials based on the instruction keywords, the text feature vector, and the perceptual feature vector includes: Based on the matching results between the instruction keyword and multiple task keywords, a target keyword is selected from the multiple task keywords; Based on the matching results between the text feature vector and multiple feature vectors, a target feature vector is selected from the multiple feature vectors; The entity name is determined based on the instruction keywords and / or the perception feature vector; Based on the entity name and the node names contained in the multiple relationship graphs, the target relationship graph is obtained from the multiple relationship graphs; The data includes the target keywords, the target feature vector, and the target relationship graph.

10. The robot control method according to any one of claims 1 to 3, characterized in that, The architecture design also includes a control layer, which deploys a control model. The control of the robot to execute the meta-action sequence includes: The control model converts the meta-action sequence into motor control signals for the robot. Control the robot to execute the motor control signals.

11. The robot control method according to any one of claims 1 to 3, characterized in that, The architecture design also includes a learning layer, which is used to train a model related to the robot using at least one training algorithm; In the case where the architecture design includes a perception layer and the perception layer is deployed with multiple perception models, the learning layer is used to train multiple perception models; In the case where the architecture design includes a control layer and the control layer is equipped with a control model, the learning layer is used to train the control model; In the case where the decision layer deploys an end-to-end model and multiple modular models, the learning layer is used to train the end-to-end model and the multiple modular models.

12. The robot control method according to claim 11, characterized in that, After controlling the robot to execute the meta-action sequence, the robot control method further includes: Obtain the execution result of the robot executing the meta-action sequence; The task instructions, the real-time sensing information, and the execution results are stored as training data. The training data is used to train a discriminator model, which is used to predict the results of the robot's action execution. In real and / or simulated environments, the discriminator model is used to assist reinforcement learning training in the learning layer.

13. The robot control method according to claim 12, characterized in that, The real-time sensing information includes multiple modal information in multiple modal forms; Before storing the task instructions, the real-time perception information, and the execution results as training data, the robot control method further includes: For each of the aforementioned modal forms, feature extraction is performed on multiple modal information sets under each modal form to obtain feature vectors corresponding to the multiple modal information sets under each modal form; For each of the aforementioned modal forms, the multiple modal information under the aforementioned modal forms is deduplicated based on the feature similarity between the feature vectors corresponding to the multiple modal information under the aforementioned modal forms.

14. A robot control device, characterized in that, The robot's architecture design includes at least a data layer and a decision layer. The data layer is configured with various task background information, the types of tasks that the robot can perform, and various meta-actions that the robot can perform. The decision layer deploys a large language model. The robot control device includes: The sensing unit is used to acquire real-time sensing information about the task scenario; A decision-making unit is used in the decision-making layer to generate a sequence of meta-actions through the large language model based on the data, the real-time perception information, the task type, and various meta-action prompts. The data is obtained from various task background data according to the task instructions and the initial perception information of the task scenario. The initial perception information is the real-time perception information obtained at the start of the task. A control unit is used to control the robot to execute the meta-action sequence.

15. An electronic device comprising: One or more processors; A memory having stored one or more programs, which, when executed by one or more processors, cause the one or more processors to perform the following: The robot control method according to any one of claims 1 to 13.

Images