Robot control method and device, storage medium and robot

Abstract

The invention discloses a robot control method and device, a storage medium and a robot, and relates to the technical field of robots. A first task instruction, a first scene image and a first state of a robot body at the current moment are acquired; processing the first task instruction and the first scene image through a visual language model, and determining a first sub-task sequence corresponding to the first task instruction; and processing based on the first sub-task sequence and a second sub-task sequence corresponding to the previous moment, and determining a target sub-task to be executed by the robot. Wherein the robot does not complete at least one second subtask in the second subtask sequence. And processing the first scene image, the first state and the target subtask through a motion control model, determining an action instruction to be executed by the robot, and controlling the robot to execute the action instruction. In this way, the real-time interaction capacity of the robot and the outside can be improved, and the understanding capacity of the robot for complex instructions and the task execution success rate can be improved based on the double models.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of robots, and in particular to a robot control method and device, a storage medium and a robot. BACKGROUND

[0002] With the rapid development of robot technology, robots have made significant progress in autonomous movement and environmental perception capabilities. Robot operation tasks include complex long-range tasks. After the robot understands the complex instructions, it needs to perform long-time actions to complete the complex long-range tasks.

[0003] Currently, the existing robot control method makes the robot have weak ability to interact with the outside, and when facing the complex long-range task of interaction, the robot lacks accurate understanding of the complex instructions, resulting in poor robot control accuracy and low task execution success rate. SUMMARY

[0004] To solve the above technical problems, the present disclosure provides a robot control method, device, storage medium and robot, which can improve the robot's understanding of complex instructions, thereby improving the robot's success rate for complex long-range tasks. At the same time, the robot can interact with the outside in real time during task execution, which can improve the robot's interactive safety, controllability, task execution efficiency and interactive friendliness.

[0005] The first aspect of the present disclosure provides a robot control method, comprising: obtaining a first task instruction, a first scene image and a first state of a robot body at a current time; processing the first task instruction and the first scene image through a visual language model to determine a first sub-task sequence corresponding to the first task instruction; performing task planning processing based on the first sub-task sequence and a second sub-task sequence corresponding to a previous time to determine a target sub-task to be executed by the robot; wherein the robot has not completed at least one second sub-task in the second sub-task sequence corresponding to the previous time; processing the first scene image, the first state and the target sub-task through a motion control model to determine an action instruction to be executed by the robot; and controlling the robot to execute the action instruction.

[0006] The second aspect of the present disclosure provides a training method of a motion control model, comprising: obtaining a sample set, the sample set comprising: a single task instruction, a scene image, a robot body state, a label of an action instruction and a label of a task completion flag; processing the scene image, the robot body state and the single task instruction using a motion control model to be trained to determine a predicted action instruction and a predicted task completion flag; training the motion control model to be trained based on the predicted action instruction, the predicted task completion flag, the label of the action instruction and the label of the task completion flag to obtain a target motion control model.

[0007] In a third aspect, an embodiment of the present disclosure provides a robot control device, comprising: a first acquisition module configured to acquire a first task instruction, a first scene image, and a first state of a robot body at a current time; a first processing module configured to process the first task instruction and the first scene image by a visual language model to determine a first subtask sequence corresponding to the first task instruction; a second processing module configured to perform task planning processing based on the first subtask sequence and a second subtask sequence corresponding to a previous time to determine a target subtask to be executed by the robot, wherein the robot has not completed at least one second subtask in the second subtask sequence corresponding to the previous time; a third processing module configured to process the first scene image, the first state, and the target subtask by a motion control model to determine an action instruction to be executed by the robot; and a first control module configured to control the robot to execute the action instruction.

[0008] In a fourth aspect, an embodiment of the present disclosure provides a training device of a motion control model, comprising: a second acquisition module configured to acquire a sample set, the sample set comprising: a single task instruction, a scene image, a state of a robot body, a label of an action instruction, and a label of a task completion flag; a fourth processing module configured to process the scene image, the state of the robot body, and the single task instruction by a motion control model to be trained to determine a predicted action instruction and a predicted task completion flag; and a training module configured to train the motion control model to be trained based on the predicted action instruction, the predicted task completion flag, the label of the action instruction, and the label of the task completion flag to obtain a target motion control model.

[0009] In a fifth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, the storage medium storing a computer program, the computer program being executed by a processor to implement the robot control method of the first aspect or the training method of the motion control model of the second aspect.

[0010] In a sixth aspect, an embodiment of the present disclosure provides a robot, comprising: a processor; a memory for storing processor-executable instructions; and the processor configured to read the executable instructions from the memory and execute the instructions to implement the robot control method of the first aspect or the training method of the motion control model of the second aspect.

[0011] This disclosure provides a robot control method. For any given moment, it can acquire a first task instruction, a first scene image, and a first state of the robot body in real time. Then, using a visual language model, it decomposes the first task instruction into subtasks based on the first scene image to determine a first subtask sequence corresponding to the first task instruction. If the robot has not completed at least one second subtask in the second subtask sequence corresponding to the previous moment, it performs subtask planning by combining the first subtask sequence and the second subtask sequence corresponding to the previous moment to determine the target subtask to be executed by the robot. Finally, using a motion control model, it makes action decisions based on the first scene image, the first state, and the target subtask to determine the action instruction to be executed by the robot, and controls the robot body to execute the action instruction to complete the task corresponding to the first task instruction or the second subtask. Therefore, the robot control method provided in this disclosure, by performing subtask planning on the first subtask sequence and the second subtask sequence corresponding to the previous moment to determine the target subtask to be executed by the robot, enables the robot to respond to user task instructions in real time and interact with the outside world in real time during task execution, thereby improving the robot's interactivity and task execution efficiency. In addition, the upper-level visual language model responds to the real-time first task instruction and breaks down the sub-tasks. After the sub-task planning is completed, the lower-level motion control model determines the action instructions. This allows the upper and lower model structures to focus on their respective specific tasks. Based on the dual upper and lower models, more accurate action instructions can be generated for processing complex long-term tasks, improving the robot's accuracy in understanding complex instructions and thus increasing the success rate of the robot in performing complex long-term tasks. Attached Figure Description

[0012] FIG. 1A This is a schematic diagram of the system architecture provided in an exemplary embodiment of this disclosure.

[0013] FIG. 1B This is a schematic diagram of the structure of a robot provided in an exemplary embodiment of this disclosure.

[0014] FIG. 1C This is a schematic diagram of the structure of a robot provided in another exemplary embodiment of this disclosure.

[0015] FIG. 2 This is a flowchart illustrating a robot control method provided in an exemplary embodiment of this disclosure.

[0016] FIG. 3 This is a flowchart illustrating a robot control method provided in another exemplary embodiment of this disclosure.

[0017] FIG. 4 This is a flowchart illustrating a robot control method provided in another exemplary embodiment of this disclosure.

[0018] FIG. 5 is a flowchart of a robot control method provided by another example embodiment of the present disclosure.

[0019] FIG. 6 is a flowchart of a robot control method provided by another example embodiment of the present disclosure.

[0020] FIG. 7A is a flowchart of a robot control method provided by another example embodiment of the present disclosure.

[0021] FIG. 7B is a flowchart of a task planning process provided by an example embodiment of the present disclosure.

[0022] FIG. 7C is a flowchart of a robot control method provided by another example embodiment of the present disclosure.

[0023] FIG. 8 is a flowchart of a training method of a motion control model provided by an example embodiment of the present disclosure.

[0024] FIG. 9 is a structural diagram of a robot control device provided by an example embodiment of the present disclosure.

[0025] FIG. 10 is a structural diagram of a robot control device provided by another example embodiment of the present disclosure.

[0026] FIG. 11 is a structural diagram of a training device of a motion control model provided by an example embodiment of the present disclosure.

[0027] FIG. 12 is a structural diagram of a robot provided by another example embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] In order to explain the present disclosure, the example embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings, obviously, the described embodiments are only part of the embodiments of the present disclosure, but not all the embodiments, it should be understood that the present disclosure is not limited by the example embodiments.

[0029] It should be noted that: unless otherwise specifically stated, the relative arrangement, numerical expressions and values of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0030] SUMMARY With the rapid development of robot technology, robots have made significant progress in autonomous movement and environmental perception capabilities. In the field of robot operation, robot operation tasks include complex long-range tasks. Complex long-range tasks mainly reflect that after the robot understands complex abstract instructions, it needs to perform long-time actions to complete complex long-range tasks. For example, after the household service robot receives the task instruction of "cleaning the kitchen", it analyzes the actions to be performed and then completes these actions one by one to complete the task instruction.

[0031] In the face of complex long-range tasks, robots can use end-to-end models to obtain robot action instructions. However, the inventors found that using end-to-end models to process complex long-range tasks makes the robot have poor understanding of complex instructions and low success rate of executing complex long-range tasks. At the same time, the robot receives the next instruction only after completing a task instruction, which makes the robot have poor interaction ability with the outside.

[0032] It can be understood that the defects of the above solutions are the results obtained by the inventors after practice and careful study. Therefore, the discovery process of the above problems and the solutions proposed by the embodiments of the present disclosure to the above problems in the following should be the contributions made by the applicant to the embodiments of the present disclosure in the process of the present application.

[0033] Exemplary System FIG. 1A is a system architecture diagram provided by an exemplary embodiment of the present disclosure. As shown in FIG. 1A , the system architecture can include an electronic device 101, a network 102 and a robot 103. The network 102 is a medium for providing a communication link between the electronic device 101 and the robot 103. The network 102 can include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0034] The electronic device 101 is a device other than the robot 103, which can be a terminal device, a server, or another robot, which is not limited herein. The terminal device includes, but is not limited to, a mobile terminal such as a vehicle terminal (e.g., a car machine), a mobile phone, a notebook computer, a digital broadcast receiver, a personal digital assistant (PDA), a portable Android device (PAD), a portable media player (PMP), and the like, and a fixed terminal such as a digital television (TV), a desktop computer, and the like. Various communication client applications, such as a robot control application, can be installed on the terminal device. The server can be a single server or a server cluster composed of multiple servers. The other robot can be a robot of the same type as the robot 103, such as a humanoid robot, or a robot of a different type.

[0035] The robot 103 is a robot supporting dual-arm operation. For example, a humanoid robot, an industrial robot, a service robot, a medical robot, and the like supporting dual-arm operation, which is not limited herein. The robot 103 is internally deployed with a computing platform supporting a deep learning algorithm, such as a graphics processing unit, a neural network processing unit, and the like, which is not limited herein.

[0036] The user can use the electronic device 101 to interact with the robot 103 through the network 102, and the electronic device 101 can also interact with the robot 103 through the network 102. For example, training a model on the electronic device 101 and then deploying the model on the robot 103 through the network 102. Or, updating the model on the electronic device 101 and then transmitting the updated model parameters to the robot 103 through the network 102. Or, the electronic device 101 issues a task (a task input by the user or a task generated by the electronic device 101) and sends the task to the robot 103 through the network 102. Or, the user remotely controls the robot 103 through the electronic device 103. Or, the electronic device 101 and the robot 103 perform collaborative work, and the like.

[0037] In some examples, the robot control method provided by the embodiments of the present disclosure can be executed by the robot 103.

[0038] Referring to FIG. 1B , FIG. 1Bis a structural schematic diagram of a robot provided by an example embodiment of the present disclosure. The robot 103 includes a processor 10 and an interactive terminal 13. The processor 10 can be a central processing unit (CPU) or other forms of processing units with data processing capability and / or instruction execution capability, and can control other components in the robot 103 to perform desired functions. The interactive terminal 13 can include a microphone, a touch display screen, a keyboard / mouse, an indicator light, a speaker, etc. The microphone, the touch display screen, and the keyboard / mouse can be used to receive task instructions, such as voice instructions or text instructions of a user. The speaker can be used for voice feedback, such as playing task execution status, etc. The indicator light is used to display task execution status or perform alarm.

[0039] The processor 10 is deployed with a visual language model and a motion control model to control the robot based on the visual language model and the motion control model to complete a task. For example, the visual language model and the motion control model can be trained by the electronic device 101, and then the trained visual language model and the motion control model are deployed to the robot 103, or the robot 103 trains the visual language model and the motion control model by itself. The processor 10 is used to realize task understanding and task decomposition based on the visual language model to obtain a subtask sequence, and then realize subtask scheduling based on the subtask sequence to process the subtasks based on the motion control model to obtain motion instructions. The subtask scheduling includes functions such as planning of a target subtask, distribution of the target subtask, and detection of execution status of the target subtask.

[0040] It should be noted that the robot can also include an image acquisition device and a joint encoder, etc. The image acquisition device is used to acquire scene images in the environment around the robot. The joint encoder is used to acquire joint angles of the robot. The image acquisition device can be a camera, a laser radar, etc. The joint encoder can be an absolute value encoder or an incremental encoder. The present disclosure does not make specific limitation on the types of the image acquisition device and the joint encoder. The robot can also include a controller, which is used to control the robot to move based on the motion instructions, so as to realize interaction with the surrounding environment.

[0041] In the present disclosure, based on the robot structure shown in FIG. 1B The robot control method implemented by the robot structure is exemplarily described.

[0042] The interaction terminal 13 in the robot can collect a first task instruction issued by a user at a current time, and send the first task instruction to the processor 10 for processing. For example, the microphone included in the interaction terminal 13 is used to collect a task instruction indicated by a user voice. The image acquisition device in the robot can collect a first scene image at the current time, and send the first scene image to the processor 10 for processing. The joint encoder in the robot can collect a joint angle of the robot at the current time, determine a first state of the robot body according to the joint angle, and send the first state to the processor 10 for processing. After the processor 10 receives the first task instruction, the first scene image, and the first state, the first task instruction and the first scene image are processed by using a visual language model, so that a task corresponding to the first task instruction is disassembled, and a first subtask sequence corresponding to the first task instruction is obtained. The processor 10 performs task planning processing based on the first subtask sequence and a second subtask sequence corresponding to a previous time, to determine a target subtask to be executed by the robot. The robot has not completed at least one second subtask in the second subtask sequence corresponding to the previous time. The processor 10 processes the first scene image, the first state, and the target subtask by using a motion control model, so as to realize action decision and determine an action instruction to be executed by the robot. Further, the processor 10 sends the action instruction to a controller in the robot, and the controller controls an execution end of the robot to execute a related action based on the action instruction, so as to realize interaction with a surrounding environment. The execution end can be one or more devices in a mechanical arm and a mechanical hand, a mechanical leg, or a mechanical foot of the robot.

[0043] participate FIG. 1C , FIG. 1C is a structural schematic diagram of a robot provided by another exemplary embodiment of the present disclosure. The robot 103 includes a first processor 11, a second processor 12, and an interaction terminal 13. The first processor 11 can be a central processing unit (CPU), which can control other components in the robot 103 to perform a desired function. The second processor 12 can be another form of processing unit having a data processing capability and / or an instruction execution capability, such as a graphics processing unit (GPU), a neural network processing unit (NPU), and the like, except for the CPU, which is not limited here. The visual language model and the motion control model are deployed in the first processor 11, so as to control the robot based on the visual language model and the motion control model to complete a task. The first processor 11 is configured to implement task understanding and task disassembly based on the visual language model, to obtain a subtask sequence, and to process the subtask based on the motion control model to obtain an action instruction. The second processor 12 is configured to implement subtask scheduling based on the subtask sequence, to facilitate the first processor 11 to process the subtask based on the motion control model to obtain the action instruction.

[0044] In the present disclosure, based on FIG. 1CThe robot structure shown is illustrated with an example of the robot control method implemented therein.

[0045] The interactive terminal 13 in the robot can collect the first task instruction issued externally at the current moment and send the first task instruction to the first processor 11 for processing. For example, the microphone included in the interactive terminal 13 can be used to collect the task instruction indicated by the user's voice. The image acquisition device in the robot can collect the first scene image at the current moment and send the first scene image to the first processor 11 for processing. The joint encoder in the robot can collect the joint angles of the robot at the current moment, determine the first state of the robot body based on the joint angles, and send the first state to the first processor 11 for processing. After receiving the first task instruction, the first scene image, and the first state, the first processor 11 uses the internally deployed visual language model to process the first task instruction and the first scene image, thereby decomposing the task corresponding to the first task instruction and obtaining the first sub-task sequence corresponding to the first task instruction. The first processor 11 sends the first sub-task sequence to the second processor 12.

[0046] The second processor 12 performs task planning based on the first subtask sequence and the second subtask sequence corresponding to the previous time step, determining the target subtask to be executed by the robot. Specifically, the robot has not completed at least one second subtask in the second subtask sequence corresponding to the previous time step. The second processor 12 returns the target subtask to the first processor 11.

[0047] The first processor 11 uses an internally deployed motion control model to process the first scene image, the first state, and the target sub-task to make action decisions and determine the action instructions to be executed by the robot. The first processor 11 sends the action instructions to the controller in the robot, and the controller controls the robot's execution end to perform relevant actions based on the action instructions, thereby enabling interaction with the surrounding environment.

[0048] The robot structure provided above is for illustrative purposes only, and other structures may also be used. For example, the second processor 12 may be replaced with a server with stronger data processing capabilities, or the functions of the second processor 12 may be integrated into the first processor 11. The interactive terminal 13 may also include other devices (such as end effectors) for the robot to interact with the outside world, which is not limited here.

[0049] Exemplary Method FIG. 2 This is a flowchart illustrating a robot control method provided in an exemplary embodiment of this disclosure. This embodiment can be applied to robots (including but not limited to robot 103). FIG. 2 As shown, it includes the following steps 201-205: Step 201: Obtain the first task instruction, the first scene image, and the first state of the robot body at the current moment.

[0050] The robot provided in this disclosure can interact with the user in real time, thus acquiring the first task instruction, the first scene image, and the first state of the robot itself in real time. For ease of description, the information acquired by the robot in real time can be interpreted and explained based on the current time dimension. Of course, the current time can be any time. The current time refers to the moment that is happening or in progress, and the current time can be time t. It is understood that the robot control method provided in this disclosure is not limited to being executed only at time t; the robot can execute the control method at any time.

[0051] The following is based on FIG. 1B and FIG. 1C The robot shown is used as an example to explain the robot control method disclosed herein.

[0052] The robot 103 obtains the first task instruction at the current moment through its interactive terminal 13, the first scene image at the current moment through its image acquisition device, and the first state of the robot body at the current moment through its joint encoder. Subsequently, the interactive terminal 13, the image acquisition device, and the joint encoder will send their respective first task instructions, first scene image, and first state of the robot body to... FIG. 1B The processor 10 in the robot structure shown FIG. 1C The robot structure shown includes a first processor 11. It is understood that when the robot receives the first task instruction, there may be other tasks that the robot is executing but have not yet completed.

[0053] The first task instruction at the current moment is the instruction the robot is currently receiving in real time. As an example, the first task instruction can be an operation task instruction, belonging to the execution type, used to enable the robot to perform physical actions. The first task instruction can also be other task instructions, which are not limited here. When the first task instruction is an operation task instruction, it may have several possibilities. First, the first task instruction can be a complex operation task instruction, which needs to be broken down into multiple sub-tasks. For example, a complex task instruction like "clean the table" can be broken down into sub-tasks such as putting away headphones or putting away a cup. Second, the first task instruction can be a single operation task instruction, such as putting away headphones or putting away a cup. Third, the first task instruction can also be a combination of single operation task instructions and complex operation task instructions. These are all exemplary descriptions of the first task instruction, and are not limited here.

[0054] In some examples, the first task instruction can be an instruction issued by a user to the robot to be executed. For example, the first task instruction can be issued by the user to the robot through a microphone in the interactive terminal 13, or can be issued by the user to the robot through a touch display screen or a keyboard / mouse input device in the interactive terminal 13, and the like, which will not be further limited herein.

[0055] In other examples, the first task instruction can also be an instruction sent by another robot. The other robot issues the first task instruction to the robot of the present disclosure to complete a collaborative task. The robot of the present disclosure can collect the first task instruction sent by the other robot through the interactive terminal 13, or can receive the first task instruction sent by the other robot through a network. In addition, the first task instruction can also be issued by a robot scheduling platform, and the robot can receive the first task instruction issued by the robot scheduling platform through a network, which will not be further limited herein. The first scene image is a two-dimensional image collected at the current time, which can be obtained by collecting a preset scene in the surrounding environment by an image collection device installed on the robot. The scene image is exemplified as an RGB image. The image collection device is exemplified as a visual sensor. The visual sensor is, for example, various devices for collecting image data, such as a monocular camera, a binocular stereo camera, a three-dimensional structured light imaging device, an all-around camera system, and the like. The preset scene is an operating scene of the robot, which is related to the task instruction. For example, the first task instruction is “clean the table”, and the preset scene can be an environment including a table. In some examples, the device collection parameters (such as a scene area to be shot, a shooting angle, a resolution, and the like) can be determined by the first task instruction, and the robot controls the image collection device to collect images according to the collection parameters. The obtained scene image is related to the first task instruction, and can provide environmental information of the surrounding environment for the robot, which is helpful for the robot to perceive the surrounding environment, understand the task, and make action decisions.

[0056] The first state of the robot body is a robot body state collected at the current time, which can include angles of joints of the robot, and can also include positions of the joints of the robot, speeds of the joints of the robot, and the like, which will not be limited herein. The first state of the robot body is helpful for the robot to accurately perceive the motion state of itself in space after obtaining the first task instruction, and to make action decisions, so as to realize accurate control of the motion of the robot. The joint angle of the robot can be exemplarily collected by a joint encoder installed on the robot.

[0057] During the execution of the task by the robot, the robot can collect the scene image of the preset scene and the state of the robot body in real time, so as to help the robot understand the surrounding environment, and determine an action instruction based on the surrounding environment, so as to complete the task corresponding to the first task instruction. In addition, the robot can also collect a new task instruction issued by a user or another robot in real time through the interactive terminal, so as to enhance the interaction ability of the robot with the outside.

[0058] At step 202, the first task instruction and the first scene image are processed by a vision language model (VLM) to determine a first subtask sequence corresponding to the first task instruction.

[0059] In the embodiments of the present disclosure, the vision language model is an upper model deployed in the robot, and the vision language model can be exemplarily deployed in a processor 10 of the robot as shown in the figure, or deployed in a first processor 11 of the robot as shown in the figure. FIG. 1B FIG. 1C In actual applications, step 202 can be exemplarily executed by any processor in the robot. In addition, the vision language model in step 202 is a pre-trained model.

[0060] In step 202, after the robot obtains the first task instruction at the current time, the first scene image at the current time, and the first state of the robot body at the current time, the robot can process the first task instruction at the current time and the first scene image at the current time by the vision language model, implement the disassembly process of the task corresponding to the first task instruction, and determine a plurality of first subtasks, which form a first subtask sequence.

[0061] In some examples, the vision language model can adopt a large language model. The large language model is a large-scale deep learning model based on the Transformer architecture and pre-trained through massive text data, which is used to understand and generate natural language and can be adapted to various natural language processing and cross-modal tasks through fine-tuning or prompt engineering. When the vision language model adopts the large language model, the robot can realize different functions of the vision language model by constructing different prompt words. For example, the vision language model is a GPT (Generative Pre-Trained Transformer) model. For another example, the vision language model can also be a SOTA (State of the Art) model, such as a GPT-4o or other SOTA multi-modal large model. The SOTA model is the model or method with the optimal performance and effect under a specific task (such as action generation of robot VLA, visual language alignment of VLM) and specific evaluation benchmark. In this way, the latest model achievements can be fully utilized to improve the reasoning ability and interaction ability of the upper model.

[0062] ​In some implementations, if the visual language model is a large language model. The robot generates a complete prompt word based on the preset prompt word template, the first task instruction and the first scene image through the large language model. The prompt word is used to guide the inference direction, output content and output format of the large language model, so that the large language model can complete the task decomposition under the guidance of the prompt word, combined with the first task instruction and the first scene image. Wherein, different types of prompt word templates can be preset to enrich the language recognition of different scenes and improve the understanding of the large language model to diversified instructions or scenes. The prompt word template is used to indicate the relevant rules and constraint information of the language or text understanding process of the large language model. The prompt word template can be stored in the memory of the robot or the memory of the server for calling when used.

[0063] Specifically, the prompt word template includes fixed content and fill bits. The fixed content can include description sentences, structured framework fields and field contents, etc., which are not limited here. The fill bits can include filling task instructions, etc., which are not limited here. Wherein, the description sentence is used to indicate the content to be understood by the large language model and / or constrain the content of the large language model, guide the inference direction of the large language model. The structured framework field can include various fields such as core operation rules, planning logic steps, constraint conditions and output format requirements, as well as the field content corresponding to each field. The structured framework is used to constrain the target to be achieved by the large language model, constrain the output range and output format, and guide the inference direction of the large language model. In actual application, different description sentences can be set in the prompt word template according to different types of task instructions, as well as different field contents of the fields such as core operation rules, planning logic steps, constraint conditions and output format requirements, which are not limited here.

[0064] In an optional example, based on the prompt word template, the first task instruction and the first scene image are processed by the visual language model to determine the first subtask sequence corresponding to the first task instruction, including: based on the prompt word template, the first task instruction and the first scene image, constructing a first prompt word, and processing the first prompt word, the first task instruction and the first scene image through the visual language model to determine the first subtask sequence corresponding to the first task instruction. Wherein, the first prompt word is used to instruct the visual language model to decompose the task corresponding to the first task instruction into a plurality of first subtasks to determine the first subtask sequence.

[0065] It can be understood that the robot can interact with the user or other robots in real time because the robot is provided with the interaction terminal. When the robot receives the first task instruction through the interaction terminal, the robot may be executing a task received at the last time, and the instruction corresponding to the task received at the last time is referred to as a second task instruction in the disclosure. Among them, the last time in the disclosure is the last second, the last few seconds or the last few minutes of the current time. Among them, the current time can be t time, and the last time can be t-n time. n is the time difference between the last time and the current time, which is not limited here.

[0066] The first task instruction received at the current time (such as t time) is “Earphone does not need to be stored, please put it back to the original position”, and the second task instruction received at the last time (such as t-n time) is “Please tidy up the desktop”. The prompt word template is the template corresponding to the task instruction of the storage type. The process of constructing the first prompt word is explained by examples as follows: The task instruction examples are the first task instruction and the second task instruction. For example, the content about the task instruction in the first prompt word is {user instruction: [please tidy up the desktop / earphone does not need to be stored, please put it back to the original position]}. When the task instruction is the second task instruction at t-n time, the filling position [] behind “user instruction” is filled with “please tidy up the desktop”; when the task instruction is the first task instruction at t time, the filling position [] behind “user instruction” is filled with “earphone does not need to be stored, please put it back to the original position”.

[0067] The description sentence in the first prompt word exemplarily includes description sentence 1 and description sentence 2. The description sentence 1 is placed after the “user instruction” in the first prompt word, and is used to guide, constrain and prompt the visual language model in advance. The description sentence 2 is placed at the last position of the first prompt word, and is used to guide, constrain and prompt the visual language model afterwards. The description sentence 1 is an example: {You are a robot that tidies up the desktop, and your abilities include two types of core actions: storage action (for example, storing grapes, storing cups, storing cubes, etc.); Put back / inverse action (for example, put the grapes back on the desktop, put the cup back on the desktop, put the cube back on the desktop). You need to output [the list of actions to be executed] according to the list of items provided below and the user's needs; when the user inputs a “put back” or “interrupt” type instruction, preferentially execute the put back / inverse action}. The description sentence 2 is an example: {You need to fully understand all the rules and examples, and the output result needs to strictly meet all the above requirements: the action list only contains 6 types of legal values (for example, store grapes, store cups, store cubes, put grapes back on the desktop, put cups back on the desktop, or put cubes back on the desktop); preferentially respond to “put back / interrupt” type instructions, and then respond to storage instructions}.

[0068] The core operation rules in the first prompt word can include a storage action rule, a put back / reverse action rule, or a general rule, etc. Among them, the storage action rule includes an executable scene, a storage container matching, and an exclusion rule. For example, the first prompt word includes {executable scene: put the "not in the storage container" object on the desktop into the corresponding storage container; storage container matching: grapes match to the bowl, cups match to the dish, and blocks match to the shelf; exclusion rule: the object is already in the storage container (such as grapes in the bowl), and the storage action is not executed}. Among them, the put back / reverse action rule includes a trigger condition, an executable scene, and an exclusion rule. For example, the first prompt word includes {trigger condition: user input contains keywords such as "put back", "interrupt", "take out", "put back", etc.; executable scene: put the "in the storage container" object back on the desktop; exclusion rule: the object is already on the desktop (such as grapes on the desktop), and the put back action is not executed}. Among them, the general rule includes a storage container definition, an object range, and an occlusion rule. For example, the first prompt word includes {storage container definition: bowls, dishes, shelves, drawers, etc. belong to storage containers and do not need to be stored / put back; object range: only handle grapes, cups, and blocks, and other objects are not responded to; occlusion rule: object occlusion does not affect the judgment of "existence" (the list shows that the object exists, and it is recognized as having the object)}.

[0069] The planning logic steps in the first prompt word are, for example: {identify the state of the object in the object list (on the desktop / in the storage container / does not exist); analyze the user instruction keywords (preferentially judge whether it contains "put back", "interrupt", and then judge the storage requirement); filter the target object that meets the action execution condition (storage requires "desktop + not stored", and put back requires "storage container + non-desktop"); select the final action to be executed from the capability range; generate a corresponding real-time Chinese reply string for each action}.

[0070] The strict constraint conditions in the first prompt word are, for example: {objects that have been stored (such as grapes in the bowl): only the "put back" action can be executed, and the storage action cannot be executed; objects that are already on the desktop (such as cups on the desktop): only the "storage" action can be executed, and the put back action cannot be executed; when the user specifies a specific object, only that object is handled, and other objects are not considered; if the user's requirement exceeds the capability range (such as "give me the grapes" or "throw away the cup") or the object does not exist, an empty list is output; the action list and the real-time reply list must be consistent in length and one-to-one correspondence, and an empty action list corresponds to an empty reply list; the "interrupt" instruction is equivalent to "put back": for example, the user says "interrupt the storage and put the grapes back", and the "put the grapes back on the desktop" action is executed}.

[0071] The output format requirement in the first prompt word is as follows: an example is as follows: the output result needs to clearly distinguish the "action list", and examples are as follows: example 1: the user inputs "clean the table", the item list displays "grapes and a cup on the table (not stored), and a square on the shelf", and the action list is [ "store the grapes", "store the cup" ]; example 2: the user inputs "put the grapes back on the table", the item list displays "grapes in a bowl", and the action list is [ "put the grapes back on the table" ]; example 3: the user inputs "interrupt and put the cup back", the item list displays "the cup is in a dish", and the action list is [ "put the cup back on the table" ]; and example 4: the user inputs "store the grapes", the item list displays "grapes in a bowl", and the action list is [].

[0072] It should be noted that the content in the above-described square brackets "[]" is the content in the first prompt word, which is used to jointly build the first prompt word.

[0073] In addition, the information displayed in the item list in the first prompt word is obtained by analyzing the first scene image by the visual language model. The content displayed in the "action list" is an exemplary sub-task sequence obtained by splitting, for example, including two sub-tasks of "store the grapes" and "store the cup". The first sub-task sequence output by the visual language model needs to be output in the format of the action list.

[0074] In some other implementations, the visual language model can also be a reinforcement learning model, which is obtained based on a reward signal to optimize the decision strategy of the model. The model type of the visual language model is not limited in the embodiments of the present disclosure, and can be set according to actual conditions. If the visual language model is a trained reinforcement learning model, the robot inputs the first task instruction and the first scene image into the visual language model to obtain the first sub-task sequence corresponding to the first task instruction output by the visual language model.

[0075] In step 203, the first sub-task sequence and the second sub-task sequence corresponding to the last time are used for task planning processing to determine a target sub-task to be executed by the robot; wherein the robot does not complete at least one second sub-task in the second sub-task sequence corresponding to the last time.

[0076] In some examples, the second sub-task sequence corresponding to the last time is determined by the following method, including: determining the second task instruction and the second scene image of the last time; processing the second task instruction and the second scene image by the visual language model to determine the second sub-task sequence corresponding to the second task instruction.

[0077] It is understandable that the current moment is the moment that is happening or in progress, and the first task instruction is the task instruction that the robot receives in real time at the current moment. The previous moment is the time node before the current moment, which can be, for example, the previous second, a few seconds, or a few minutes ago, without being limited here.

[0078] The second task instruction is the most recently stored task instruction in the robot's internal memory at the previous moment, which can be exemplified by the task instruction received at the previous moment. The second scene image is the scene image captured by the robot at the previous moment. The acquisition process of the second task instruction and the second scene image can refer to the various implementations of the first task instruction and the first scene image in step 201, and will not be elaborated here. The process of generating the second sub-task sequence can refer to the implementation of generating the first sub-task sequence in step 202, and will not be detailed here. In practical applications, the robot can also obtain a second state of the robot body.

[0079] In some implementations, step 203 can be... FIG. 1B The processor 10 of the robot shown is implemented or is made by FIG. 1C The robot shown is implemented by a second processor 12. Taking processor 10 as an example, after obtaining the first subtask sequence based on the visual language model, processor 10 sorts the first subtasks in the first subtask sequence to generate a sorted first subtask sequence, or generates an acyclic directed subtask graph. In the subtask graph, a node represents a first subtask that the robot is capable of executing, and the arrow indicates the execution direction of the first subtask. Taking the second processor 12 as an example, after the first processor 11 obtains the first subtask sequence based on the visual language model, it sends the first subtask sequence to the second processor 12. The second processor 12 can sort the first subtasks in the first subtask sequence to generate a sorted first subtask sequence, or generate an acyclic directed subtask graph.

[0080] For example, the first subtasks can be sorted according to their safety priority. For instance, if the first subtasks include obstacle avoidance and grabbing, the obstacle avoidance subtask will be executed first. Alternatively, the first subtasks can be sorted according to their urgency. For instance, if the first subtasks include cleaning the ground and cleaning the desktop, the cleaning ground subtask will be executed first due to its higher urgency. Or, the first subtasks can be sorted according to their importance. For instance, if the first subtasks include heating water and cleaning the ground, the heating water subtask will be executed first due to its higher importance. It is understood that safety priority, urgency priority, and importance priority can be preset in processor 10 or the second processor 12, or determined by processor 10 or the second processor 12 through its own evaluation.

[0081] For example, the first sub-tasks include a sub-task of grabbing a left cup and a sub-task of grabbing a right spoon, the position and posture of the left cup and the robot body are more matched, and the robot is convenient to grab the left cup first, so the sub-task of grabbing the left cup is executed preferentially. For example, the first sub-tasks can also be sorted according to the resource occupation situation (such as resource occupation time) of the first sub-tasks, and the first sub-task with less resource occupation time is executed preferentially.

[0082] It can be understood that the above examples are exemplary descriptions of the embodiments of the present disclosure and are not limiting.

[0083] The first sub-task in the sorted first sub-task sequence or the first sub-task in the acyclic directed sub-task graph is usually the next sub-task to be executed by the robot. If the step 203 is implemented by the second processor 12, the second processor 12 will return the sub-task to the first processor 11 of the robot for processing. If the first sub-tasks are sorted from left to right according to the priority, the first sub-task in the sorted first sub-task sequence or the first sub-task in the acyclic directed sub-task graph is the leftmost sub-task.

[0084] In actual application, after the first sub-task sequence is determined, if the robot does not complete at least one second sub-task in the second sub-task sequence corresponding to the last time, the first sub-task sequence and the second sub-task sequence corresponding to the last time are also comprehensively planned when the sub-tasks are sorted. Specifically, the target sub-task to be executed by the robot is determined based on the task planning processing of the first sub-task sequence and the second sub-task sequence corresponding to the last time. The target sub-task is the next sub-task to be executed by the robot.

[0085] For example, the second sub-task sub-sequence that is not completed in the second sub-task sequence corresponding to the last time is determined. The target sub-task to be executed by the robot is determined as the first sub-task in the sorted combined sub-task sequence based on the task planning processing of the first sub-task sequence and the second sub-task sub-sequence that is not completed at the last time. For example, the first sub-task sequence includes sub-task 1, sub-task 2 and sub-task 3. The second sub-task sequence includes sub-task 4, sub-task 5 and sub-task 6. If the second sub-task sub-sequence that is not completed in the second sub-task sequence includes sub-task 5 and sub-task 6. The robot performs task planning processing on sub-task 1, sub-task 2, sub-task 3, sub-task 5 and sub-task 6, determines the combined sub-task sequence as {sub-task 2, sub-task 3, sub-task 1, sub-task 5, sub-task 6}, and determines the first sub-task in the combined sub-task sequence as sub-task 2, so the target sub-task is sub-task 2.

[0086] It should be noted that the method for determining the sequence of combined subtasks can refer to the method for sorting the first subtasks described above, and will not be repeated here. In some implementations, the priority of the first subtask sequence is higher than that of the incomplete second subtask sequence, and each first subtask in the first subtask sequence can be executed first.

[0087] Understandably, once a robot completes a subtask, the completed subtask can be removed from the sequence or subtask graph to simplify the sequence or subtask graph and free up storage space.

[0088] The robot interacts with the user in real time through an interactive terminal to obtain initial task instructions. After obtaining the initial task instructions, it then... FIG. 1B For example, after the robot's processor 10 processes the first task instruction and the first scene image based on a visual language model and determines the first sub-task sequence, it updates the sequence of sub-tasks to be completed based on the real-time obtained first sub-task sequence and the incomplete second sub-task sequence, and determines the target sub-task from the sequence of sub-tasks to be completed. FIG. 1C For example, the robot's first processor 11 processes the first task instruction and the first scene image based on the visual language model and determines the first sub-task sequence, which is then sent to the second processor 12. The second processor 12 updates the sub-task sequence to be completed based on the first sub-task sequence obtained in real time and the uncompleted second sub-task sequence, and determines the target sub-task from the sub-task sequence to be completed, and returns the target sub-task to the first processor 11 for processing.

[0089] Step 204: Process the first scene image, the first state, and the target sub-task using the motion control model to determine the action instructions to be executed by the robot.

[0090] In this embodiment of the disclosure, the motion control model is a lower-level model deployed on the robot. Specifically, the motion control model can be deployed on... FIG. 1B The robot's processor 10 shown, or deployed in FIG. 1C In the first processor 11 of the robot shown, the visual language model is an upper-level model deployed in processor 10 or the first processor 11, and the motion control model is a lower-level model deployed in processor 10 or the first processor 11. The upper and lower level model structures focus on their respective specific tasks, and by processing complex long-range tasks based on the dual upper and lower level models, more precise motion commands can be generated subsequently. In practical applications, step 204 can be exemplarily performed by… FIG. 1B The processor 10 shown executes or FIG. 1C The first processor 11 shown executes.

[0091] The robot can achieve various functions through the motion control model. In some examples, the robot can process the scene image, the state of the robot body, and the subtask through the motion control model, achieve action decision, and determine the action instruction to be executed by the robot. It can be understood that the robot can also achieve other functions through the motion control model, which is not limited further here. In addition, the motion control model in the present disclosure can be a pre-trained motion control model.

[0092] In some examples, the motion control model can be a small Vision Language Action (VLA) model or a language-guided Vision Action (VA) model to ensure the convergence speed and execution efficiency of the model. For example, the motion control model can include a diffusion model, which is a step-by-step denoising generative model. For example, the diffusion model can include the structure of Diffusion Transformer (DiT), and can also include the structure of DiT combined with Feature-wise Linear Modulation (FiLM), which is not limited here. If the motion control model includes a diffusion model, a noise signal needs to be added. Specifically, the robot processes the first scene image, the first state, the target subtask, and the noise signal through the motion control model to determine the action instruction to be executed by the robot. The noise signal can be random noise, and can also be Gaussian noise, which is independent of the input of the scene image, the task instruction, and the state of the robot body, and can be generated independently, and the content of the noise signal is not limited here.

[0093] The action instruction is the control parameter instruction required for a single action to be executed by the robot. For example, the target subtask is "grabbing a cup on the table", which can be achieved by the robot executing multiple actions, such as raising the arm, opening the fingers, closing the fingers to grab, raising the arm, etc. If the single action to be executed by the robot is "raising the arm", the robot needs to determine the control parameters corresponding to this action before executing the action. For example, the control parameters include the angles of the joints of the robot, and can also include the positions of the joints of the robot, the speeds of the joints of the robot, the position of the end effector of the robot, the attitude of the end effector of the robot, or the opening degree of the end effector of the robot, etc. which is not limited here.

[0094] It can be understood that in order to complete the sub-task instruction, the robot needs to execute one or more action instructions. In some examples, the robot can acquire a scene image in real time and acquire a state of the robot body in real time, and can also acquire other task instructions in real time. The robot can repeatedly execute steps 202-204 based on the task instructions, the scene image and the state of the robot body acquired in real time to generate a next action instruction to be executed.

[0095] In some scenarios, if the target sub-task determined based on step 203 is "grab the cup on the table", the action instruction determined based on step 204 is "lift the arm", and within a period of time after time t, the robot does not receive a new task instruction but can receive a scene image and a state of the robot body in real time, then when the target sub-task "grab the cup on the table" executed by the robot is not completed and the action instruction "lift the arm" required by the target sub-task is not completed, the next sub-task determined by processing the first task instruction and the scene image received in real time based on the visual language model is still the target sub-task "grab the cup on the table", and the action instruction determined by processing the scene image received in real time, the state of the robot body received in real time and the target sub-task based on the motion control model is still the action instruction "lift the arm". Only after the current action instruction "lift the arm" executed by the robot is completed, the action instruction determined by the robot based on the motion control model will be the next action instruction to complete the target sub-task, such as the next action instruction "open the fingers"; only after the target sub-task "grab the cup on the table" executed by the robot is completed, the next sub-task determined by the robot based on the visual language model will be another sub-task, such as the next sub-task "grab the toy on the table". After each sub-task is completed, the robot completes the overall task.

[0096] Step 205, controlling the robot to execute the action instruction.

[0097] After acquiring the action instruction, the execution end of the robot can be controlled by the action instruction to complete the action instruction. The execution end can be one or more devices of a mechanical arm, a mechanical hand, a mechanical leg or a mechanical foot of the robot. Controlling the robot to execute the action instruction includes: FIG. 3 The processor 10 shown in the figure or FIG. 2 The first processor 11 shown in the figure sends the action instruction to the controller, and the controller controls the execution end of the robot to execute the action instruction, thereby realizing interaction with the surrounding environment.

[0098] In the embodiments of the present disclosure, the robot can respond to the task instructions issued by the user in real time during the execution of the task, and the robot can interact with the user in real time, thereby improving the interaction controllability, task execution efficiency and user experience friendliness of the robot. Through the upper visual language model, the first task instruction is responded and the sub-task is split, and after the sub-task planning is completed, the action instruction is determined through the lower motion control model, so that the upper and lower model structures focus on specific tasks respectively, and the complex long-range task processing based on the upper and lower double models can generate more accurate action instructions, improve the accuracy of the robot in understanding complex instructions, and further improve the success rate of the robot in executing complex long-range tasks.

[0099] As shown in FIG. 1B the above FIG. 1C based on the embodiments shown in the above, the robot control method provided by the embodiments of the present disclosure further includes the following steps 301-302: Step 301: determining a voice instruction based on the first task instruction and the first scene image through the visual language model.

[0100] In some examples, the robot can also determine a voice instruction for responding to the first task instruction based on the visual language model, and the robot can perform voice feedback to the outside based on the voice instruction, output the voice corresponding to the voice instruction, so as to enhance the interaction ability of the robot with the user.

[0101] Specifically, FIG. 1B the processor 10 shown in FIG. 1C the first processor 11 shown in, processes the first task instruction and the first scene image through the visual language model to determine a reply statement for the first task instruction. Further, the text type reply statement is converted into a voice instruction through the related technology of text to voice. The voice instruction is a voice reply instruction for the first task instruction, which can be used for the robot to interact with the outside to improve the interaction ability of the robot.

[0102] If the visual language model is a large language model, in some examples, the robot can construct a second prompt word based on the first task instruction and the first scene image, and the second prompt word is used to guide the visual language model to obtain a reply statement for responding to the first task instruction. The robot processes the second prompt word, the first task instruction and the first scene image through the visual language model to determine the reply statement, and converts the reply statement into a voice instruction, so that the voice instruction is played through the loudspeaker in the interaction terminal. The construction method of the second prompt word can refer to the construction method of the first prompt word described above, which will not be described here. In addition, the visual language model can simultaneously determine the first sub-task sequence based on the first task instruction and the first scene image while generating the reply statement for the first task instruction.

[0103] If the visual language model is a large language model, in some examples, the first prompt word can include the core content of the second prompt word, and the first prompt word is used to guide the visual language model to disassemble the first task instruction to determine the first sub-task sequence, and also used to guide the visual language model to determine the reply sentence for the first task instruction. Then the robot processes the first prompt word, the first task instruction and the first scene image through the visual language model to determine the reply sentence and the first sub-task sequence, and converts the reply sentence into a voice instruction. It should be noted that the description sentence in the prompt word template can also increase the content related to the reply sentence, and the prompt word template can also include the field of the real-time reply character string template and the field content thereof, and the output format in the prompt word template can also increase the example of the reply sentence.

[0104] For the case where the first prompt word is used to guide the visual language model to disassemble the first task instruction to determine the first sub-task sequence, and also used to guide the visual language model to determine the reply sentence for the first task instruction, taking the prompt word template shown in step 202 as an example, the increased content is illustrated as follows: Description sentence 1, example: {You are a robot that arranges the desktop, your ability includes two types of core actions: storage action: store grapes, store cups, store blocks; Put back / inverse action: put grapes back on the desktop, put cups back on the desktop, put blocks back on the desktop. You need to output [list of actions to be performed] and [list of real-time Chinese reply character strings corresponding to the actions] according to the list of items provided below and the user's needs; when the user inputs instructions containing "put back" or "interrupt", prefer to perform the put back / inverse action}. Description sentence 2, example: {You need to fully understand all the rules and examples, and the output result must strictly meet all the above requirements: the action list only contains 6 legal values: store grapes, store cups, store blocks, put grapes back on the desktop, put cups back on the desktop, and put blocks back on the desktop; The real-time reply completely matches the corresponding template and does not modify the style of the speech; prefer to respond to "put back / interrupt" instructions, and then respond to storage instructions}.

[0105] The real-time reply character string template includes storage action replies, for example, {storage action replies: the reply corresponding to storing grapes is "I will now store the grapes on the desktop in the bowl"; the reply corresponding to storing cups is "I will now store the cups on the desktop in the dish"; the reply corresponding to storing blocks is "I will now store the blocks on the desktop on the shelf"}.

[0106] The output format requirement example is: the output result needs to clearly distinguish the "action list", for example: example 1: the user inputs "clean the table", the item list shows "grapes, cup on the table (not stored), square on the shelf", the action list: ["store grapes", "store cup"], the real-time reply string list: ["I will now store the grapes on the table in the bowl", "I will now store the cup on the table in the dish"] ; example 2: the user inputs "put the grapes back on the table", the item list shows "grapes in the bowl", the action list: ["put the grapes back on the table"], the real-time reply string list: ["I will now put the grapes in the bowl back on the table"] ; example 3: the user inputs "interrupt, put the cup back", the item list shows "cup in the dish", the action list: ["put the cup back on the table"], the real-time reply string list: ["I will now put the cup in the dish back on the table"] ; example 4: the user inputs "store the grapes", the item list shows "grapes in the bowl", the action list: [], the real-time reply string list: [].

[0107] Step 302: based on the voice instruction, controlling the robot to output the voice corresponding to the voice instruction.

[0108] For example, the interactive terminal of the robot can include a loudspeaker, after determining the voice instruction, FIG. 4 The processor 10 or FIG. 2 The first processor 11 can send the voice instruction to the interactive terminal 13, and the loudspeaker in the interactive terminal 13 can be used to output the voice corresponding to the voice instruction. For example, the first task instruction is "help me tidy up the table", and the voice instruction output by the loudspeaker of the interactive terminal 13 is "I will now tidy up the table".

[0109] It can be understood that when the robot receives the second task instruction, it will also determine the corresponding voice instruction based on the second task instruction and the second scene image. For example, the second task instruction is "the earphone does not need to be stored, put it back to the original position", and the voice instruction output by the loudspeaker of the interactive terminal is "I will leave the earphone on the table".

[0110] It can be understood that before the robot executes the action instruction corresponding to the target sub-task, the robot can also determine the corresponding voice instruction based on the target sub-task and perform voice output to inform the user of the content of the target sub-task to be executed by the robot, further improving the interaction ability of the robot and the user. For example, the target sub-task is "grab the cup", and the voice instruction is "I will grab the cup".

[0111] In the embodiment, the robot can obtain a reply statement for the task instruction based on the visual language model, and give real-time feedback on the task instruction of the user through voice based on the reply statement, so as to improve the interaction ability of the robot with the user, and facilitate the user to determine that the robot has received the task instruction, thereby improving the task execution efficiency.

[0112] As FIG. 1B shown in the above FIG. 1C based on the embodiment shown in the above, step 202 can include steps 401-403 as follows: Step 401, vectorizing the first task instruction to determine the target text feature vector.

[0113] The first task instruction can be natural language or text input by the user, FIG. 1B The processor 10 shown in the above or FIG. 1C The first processor 11 shown in the above can first vectorize the first task instruction to convert voice or text into a feature vector representation. Wherein, the vectorization processing refers to performing word segmentation processing on the first task instruction, splitting it into a sequence of minimum semantic units recognizable by the model, and then standardizing the sequence, and mapping the standardized sequence into a dense sequence vector to determine the target text feature vector.

[0114] Exemplarily, the target text feature vector can include basic vocabulary features, vocabulary attribute features, etc., which are not limited here.

[0115] Step 402, based on an image encoder, encoding the first scene image to determine an image feature vector.

[0116] In some examples, the image encoder can adopt a structure combining Sigmoid Loss for Language-Image Pre-training (SigLIP) and Distillation with No Labels v2 (DINO v2), which is not limited here.

[0117] Exemplarily, FIG. 1B The processor 10 shown in the above or FIG. 1C The first processor 11 shown in the above encodes the first scene image through the image encoder to determine the image feature vector, so as to convert the pixel information of the first scene image into a feature vector rich in global and local semantics. Wherein, the image feature vector can include visual features such as color features, texture features, edge features, and semantic features obtained on the basis of the visual features, such as semantic category features, etc., which are not limited here.

[0118] In step 403, the target text feature vector and the image feature vector are processed by the visual language model to determine the first subtask sequence corresponding to the first task instruction.

[0119] In some examples, FIG. 5 The processor 10 shown or FIG. 2 The first processor 11 shown constructs a first prompt word based on the first task instruction and the first scene image, and vectorizes the first prompt word to obtain a prompt word feature vector. Further, the prompt word feature vector, the target text feature vector, and the image feature vector are processed by the visual language model to determine the first subtask sequence corresponding to the first task instruction.

[0120] For example, the prompt word feature vector, the target text feature vector, and the image feature vector can be fused to obtain a fused feature vector, and then the fused feature vector is input into the visual language model for inference. The visual language model performs correlation calculation on the features in the fused feature vector to capture the correlation between different features, and finally determines the first subtask sequence corresponding to the first task instruction.

[0121] In this embodiment, the robot vectorizes the first task instruction and the first scene image to unify the representation space of multi-modal information, obtain a feature representation that can be understood by the visual language model and can be cross-modal inference, thereby improving the processing efficiency of the subsequent visual language model. In addition, the target text feature vector and the image feature vector both remove irrelevant interference information, so that the first subtask sequence obtained by processing the filtered feature vectors by the visual language model is more accurate.

[0122] As FIG. 1B Based on the above FIG. 1C The step 204 can include the following steps 501-503 based on the above Step 501, based on the state encoder, the first state is encoded to determine the state feature vector.

[0123] In some examples, FIG. 1B The processor 10 shown or FIG. 1C After obtaining the first state, the first processor 11 shown can encode the first state by the state encoder to obtain the state feature vector. The state feature vector can include joint angle features, joint position features, joint speed features, etc. For example, the state encoder can be implemented based on a multi-layer perception algorithm, which is not limited here.

[0124] Step 502, vectorizing the target subtask to determine a subtext feature vector.

[0125] FIG. 1BThe processor 10 or FIG. 1C The first processor 11 shown determines the target subtask and outputs the corresponding target subtask instruction (i.e., instruction form). The target subtask instruction can be in the form of text. Further, the target subtask instruction can be processed by vectorization to convert the target subtask instruction in the form of text into a feature vector representation. The vectorization processing refers to processing the target subtask instruction by word segmentation to split the target subtask instruction into a sequence of minimum semantic units recognizable by the model, and then standardizing the sequence and mapping the standardized sequence into a dense sequence vector to determine a subtext feature vector. For example, the subtext feature vector can include basic vocabulary features, vocabulary attribute features, etc., which are not limited herein.

[0126] At step 503, the image feature vector corresponding to the first scene image, the state feature vector, and the subtext feature vector are processed by the motion control model to determine the action instruction to be executed by the robot.

[0127] FIG. 1B The processor 10 or FIG. 1C After obtaining the state feature vector and the subtext feature vector, the first processor 11 shown processes the image feature vector corresponding to the first scene image, the state feature vector, and the subtext feature vector by the motion control model to determine the action instruction to be executed by the robot. The image feature vector corresponding to the first scene image can be the image feature vector obtained at step 402. The image feature vector obtained by processing the first scene image once can be reused in the information processing process of the visual language model and the motion control model, thereby improving the processing efficiency of the model.

[0128] For example, FIG. 6 The processor 10 or FIG. 2 The first processor 11 shown can further process the subtext feature vector by encoding based on an encoder to convert the sequence vector into a semantic vector, and then process the image feature vector, the state feature vector, and the semantic vector by the motion control model to determine the action instruction to be executed by the robot. For example, the encoding processing can be implemented based on a T5 encoder, which is not limited herein.

[0129] The encoding processing of the subtext feature vector can also be implemented by the motion control model, which is not limited herein.

[0130] In some examples, if the motion control model is a diffusion model, the first processor 11 shown can further process the subtext feature vector by encoding based on the diffusion model to convert the sequence vector into a semantic vector, and then process the image feature vector, the state feature vector, and the semantic vector by the motion control model to determine the action instruction to be executed by the robot. FIG. 1B The processor 10 or FIG. 1CThe first processor 11 shown can also encode the noise signal to obtain a noise feature vector. For example, the noise signal can be encoded based on a multi-layer perception. After obtaining the noise feature vector, the image feature vector, the state feature vector, the noise feature vector, and the subtext feature vector (or semantic vector) are processed by the motion control model to determine the action instruction to be executed by the robot. The implementation process of the diffusion model is not repeated here.

[0131] In this embodiment, the robot vectorizes the first state and the target subtask to unify the representation space of the multi-modal information, obtain a feature representation that can be understood by the motion control model and cross-modal inference, thereby improving the processing efficiency of the subsequent motion control model. In addition, the image feature vector, the state feature vector, and the subtext feature vector remove irrelevant interference information, so that the action instruction obtained by processing the filtered feature vector by the motion control model is more accurate, thereby helping to improve the success rate of task execution of the robot.

[0132] As shown in FIG. 1B Based on the above FIG. 1C The step 203 can include the following steps 601-603 based on the embodiment shown in the above Step 601: Determine the subtask switching signal corresponding to the subtask executed at the previous time.

[0133] Each subtask executed by the robot can correspond to a subtask switching signal. The subtask switching signal is used to indicate whether the executed subtask is completed. For example, if the subtask switching signal is 1, it indicates that the subtask executed by the robot has been completed; if the subtask switching signal is 0, it indicates that the subtask executed by the robot has not been completed.

[0134] The subtask switching signal can be determined by one or more of the subtask detection result of the executed subtask, the subtask completion flag of the executed subtask, and the robot body state at the time of executing the subtask. The subtask detection result can be obtained by the robot based on the visual language model, and the subtask completion flag can be obtained by the robot based on the motion control model.

[0135] For example, the subtask detection result can be obtained by FIG. 7AThe processor 10 in the robot shown performs loop closure detection based on a visual language model to obtain subtask detection results, detects the executed subtask based on a motion control model to obtain a subtask completion flag, and then determines a subtask switching signal based on one or more of the subtask detection results, the subtask completion flag, and the robot's body state during the execution of the subtask. The robot's body state during the execution of the subtask can be collected by sensors such as joint encoders on the robot and sent to the processor 10. For example, it can be obtained by... FIG. 6 The first processor 11 in the robot shown performs loop closure detection based on a visual language model to obtain subtask detection results, and detects the executed subtask based on a motion control model to obtain a subtask completion flag. It then sends the subtask detection results and the subtask completion flag to the second processor 12. The second processor 12 then determines a subtask switching signal based on one or more of the subtask detection results, the subtask completion flag, and the robot's body state during the execution of the subtask. The robot's body state during the execution of the subtask can be collected by sensors such as joint encoders on the robot and sent to the second processor 12.

[0136] During the execution of a subtask, the robot determines the subtask switching signal corresponding to the subtask being executed in real time. In step 601, the robot can determine the subtask switching signal corresponding to the subtask executed at the previous moment (the previous moment can be exemplified as moment tn). Here, the subtask executed at the previous moment is a second subtask executed at the previous moment.

[0137] In some examples, the subtask switching signal corresponding to the subtask executed at the previous moment can be determined according to steps 701-703, which will not be described in detail here.

[0138] Step 602: Based on the subtask switching signal of the previous time step and the second subtask sequence corresponding to the previous time step, determine the unfinished second subtask subsequence in the second subtask sequence corresponding to the previous time step.

[0139] If the subtask switching signal corresponding to the subtask executed at the previous time step (tn) is 1, and the robot receives the first task instruction at the current time step (t), then the completed subtask (i.e., the subtask executed at tn) can be cleared from the second subtask sequence. Second subtasks completed before the previous time step will also be cleared from the second subtask sequence, thus identifying the incomplete second subtask subsequence in the second subtask sequence corresponding to the previous time step. The incomplete second subtask subsequence does not include the subtask executed at the previous time step (i.e., the subtask executed at tn). Then, step 603 is executed.

[0140] If the subtask switching signal corresponding to the subtask executed in the previous moment is 0, and the robot receives the first task instruction at the current moment (time t), then the subtask executed in the previous moment is retained, and the completed second subtasks before the previous moment are removed from the second subtask sequence. The incomplete second subtask subsequence corresponding to the second subtask sequence at the previous moment is determined. The incomplete second subtask subsequence includes the subtask executed in the previous moment. Then, step 603 is executed.

[0141] Step 603: Perform task planning based on the first subtask sequence and the second subtask sequence that was not completed in the previous time step to determine the target subtask to be executed by the robot.

[0142] After determining the first subtask sequence and the second subtask sequence that was not completed in the previous time step, the robot (specifically, it can be...) FIG. 7B The processor 10 shown or FIG. 1C The second processor 12 shown performs comprehensive planning on the subtasks in the two sequences to determine the target subtask to be executed by the robot.

[0143] In some examples, task planning is performed based on the first subtask sequence and the second subtask sequence that was not completed in the previous time step to determine the sorted combined subtask sequence. From the combined subtask sequence, the first subtask is determined as the target subtask to be executed by the robot. The method for determining the combined subtask sequence can refer to the method for sorting the first subtasks in step 203, which will not be repeated here.

[0144] In this embodiment, the completion status of the subtask executed in the previous moment is detected based on the subtask switching signal, which enables effective supervision of the subtask completion status. Furthermore, based on the subtask completion status, a second subtask subsequence that was not completed in the previous moment is determined, and task planning processing for all subtasks is performed based on the first subtask sequence and the second subtask subsequence that was not completed in the previous moment, so that no subtask is lost during task planning.

[0145] It can be understood that if the subtask switching signal corresponding to the subtask executed at the last moment is 1, and the robot does not receive the first task instruction, the completed subtask (i.e., the subtask executed at the t-n moment) can be removed from the second subtask sequence, and the second subtask subsequence that has not been completed in the second subtask sequence corresponding to the last moment is determined. The first subtask in the second subtask subsequence that has not been completed is determined as the subtask to be executed at present. Based on the current subtask to be executed, the real-time obtained scene image, the real-time obtained state of the robot body and the motion control model, a new action instruction to be executed by the robot is determined, which can be referred to step 204. In addition, if the subtask switching signal corresponding to the subtask executed at the last moment is 0, and the robot does not receive the new first task instruction, the subtask executed at the last moment is continued to be executed.

[0146] As shown in the above FIG. 7B based on the embodiment shown in FIG. 1B The step 601 can include the following steps 701-703 based on the above embodiment: Step 701, based on the visual language model, determining the subtask detection result of the subtask executed at the last moment.

[0147] In some examples, the robot can also implement loop detection based on the visual language model to determine the subtask detection result of the subtask executed at the last moment. The subtask detection result is used to indicate whether the subtask execution is successful. For example, the subtask detection result is 1, indicating that the subtask execution is successful, and also indicating that the subtask execution is completed; if the task detection result is 0, it indicates that the subtask execution fails.

[0148] In an alternative implementation, the embodiment of the present disclosure provides a specific implementation of determining the subtask detection result of the subtask executed at the last moment based on the visual language model, comprising: Through the visual language model, the subtask detection is performed based on the subtask executed at the last moment, the first scene image and the scene image at the last moment, and the subtask detection result is determined.

[0149] If the visual language model is a large language model, in some examples, the robot can construct a third prompt word based on the subtask executed at the last moment, the first scene image and the scene image at the last moment and the corresponding prompt word template. The third prompt word is used to guide the visual language model to detect whether the execution of the subtask executed at the last moment is successful. The robot processes the third prompt word, the subtask executed at the last moment, the first scene image and the scene image at the last moment through the visual language model to determine the subtask detection result. The construction of the third prompt word can refer to the construction of the first prompt word described above, which will not be described here.

[0150] The third prompt word is exemplarily described as follows: The description sentence in the third prompt word is exemplified as: {You need to judge whether the subtask is completed based on the provided picture}, {Please check whether the subtask is completed based on the provided image}, {Output only a single numerical value: output 1 when the task is completed, and output 0 when the task is not completed}. The content of the subtask performed at the last time in the third prompt word can be exemplified as: {The current subtask is: {put on the earphone}}.

[0151] Among them, each description sentence can be placed in different positions in the third prompt word according to the actual situation. For example, the first description sentence can be placed before {The current subtask is: {put on the earphone}}, the second description sentence can be placed after {The current subtask is: {put on the earphone}}, and the third description sentence can be placed at the end of the third prompt word to prompt the output format.

[0152] The detection examples provided in the third prompt word can be: {Example 1: If the subtask is “put the grapes on the table into the bowl”, and the grapes have been placed in the bowl and are no longer on the table, then the task is determined to be completed, and the output result is 1; Example 2: If the subtask is “put the cup on the table into the cup and saucer”, and the cup is not placed in the cup and saucer, but is still on the table or placed in other positions, then the task is determined to be not completed, and the output result is 0; Example 3: If the subtask is “put the cup on the table into the cup and saucer”, and the cup has been placed in the cup and saucer, then regardless of the state of the cup (such as standing or upside down), the task is determined to be completed, and the output result is 1; Example 4: If the subtask is “put the cup on the table into the cup and saucer”, and the cup and saucer has already placed an object, then the task is determined to be completed, and the output result is 1; Example 5: If the subtask is “put the cube on the table onto the shelf”, and the cube is placed on the second layer of the shelf, then the task is determined to be completed, and the output result is 1}.

[0153] The above description of the third prompt word is only an example, which is not limited here. Different prompt word templates and different prompt words can be preset according to different subtasks.

[0154] In another example, the robot can first obtain the subtext feature vector corresponding to the subtask performed at the last time, the image feature vector corresponding to the first scene image, the image feature vector corresponding to the scene image at the last time, and the prompt word feature vector corresponding to the third prompt word; then, through the visual language model, the subtask detection is performed based on the subtext feature vector corresponding to the subtask performed at the last time, the image feature vector corresponding to the first scene image, the image feature vector corresponding to the scene image at the last time, and the prompt word feature vector corresponding to the third prompt word to determine the subtask detection result. For details, refer to steps 401-403, which are not repeated here.

[0155] It can be understood that when the robot performs a complex long-range task, the task often needs to be decomposed into sub-tasks that need to be executed one by one. The streaming execution mode makes the success rate of the entire task affected by the success rate of each sub-task. At the same time, the errors and failures of sub-task execution will cause the accumulation of subsequent errors and the increase of subsequent task execution difficulty, and even cause the final task execution failure or incomplete task. Therefore, the robot performs loop detection on each executed sub-task. Only when the sub-task detection result is execution success, the next sub-task can be performed, which can improve the success rate of the robot in executing the entire complex long-range task.

[0156] In step 702, based on the motion control model, a sub-task completion flag of the sub-task executed at the last moment is determined.

[0157] In some examples, the robot can also determine the sub-task completion flag of the sub-task executed at the last moment based on the motion control model. The sub-task completion flag is used to indicate whether the sub-task execution is completed. For example, when the sub-task completion flag is 0, it indicates that the sub-task is not completed, and when the sub-task completion flag is 1, it indicates that the sub-task has been executed and completed.

[0158] In an alternative implementation, the embodiment of the present disclosure provides a specific implementation of determining a sub-task completion flag of a sub-task executed at the last moment based on a motion control model, comprising: Through the motion control model, based on the scene image at the last moment, the state of the robot body at the last moment and the sub-task executed at the last moment, the sub-task detection is performed to determine the sub-task completion flag of the sub-task executed at the last moment.

[0159] It can be understood that when the robot performs sub-task detection based on the scene image at the last moment, the state of the robot body at the last moment and the sub-task executed at the last moment through the motion control model, in addition to determining the action instruction, the sub-task completion flag of the sub-task executed at the last moment can also be determined. The scene image at the last moment is the second scene image, and the state of the robot body at the last moment is the second state of the robot body.

[0160] In some examples, the robot can first acquire the image feature vector corresponding to the scene image at the last moment, the state feature vector corresponding to the state of the robot body at the last moment and the sub-text feature vector corresponding to the sub-task executed at the last moment; and then through the motion control model, the image feature vector corresponding to the scene image at the last moment, the state feature vector corresponding to the state of the robot body at the last moment and the sub-text feature vector corresponding to the sub-task executed at the last moment are processed to determine the sub-task completion flag of the sub-task executed at the last moment. For details, please refer to steps 501-503.

[0161] At step 703, the robot determines the subtask switching signal corresponding to the subtask executed at the last time based on one or more of the subtask detection result of the subtask executed at the last time, the subtask completion flag of the subtask executed at the last time, and the state of the robot body at the last time.

[0162] In some examples, the robot can also determine the completion of the subtask executed at the last time based on the state of the robot body at the last time. If the state of the robot body at the last time meets the target state, it can be considered that the subtask executed at the last time has been completed. For example, the target state can be the state of the robot when the robot completes the subtask, which can be set in advance. For example, if the state of the robot body is the angle of each joint of the robot, the target angle corresponding to the completion of the subtask can be set in advance, and when the target angle is reached, it can be considered that the subtask executed at the last time has been completed. In addition, if the state of the robot body at the last time indicates that the robot has been in a stationary state, it can also be considered that the robot has completed the subtask executed at the last time.

[0163] In some implementations, when the subtask detection result of the subtask executed at the last time, the subtask completion flag of the subtask executed at the last time, and the state of the robot body at the last time all indicate that the subtask executed at the last time has been completed, the robot determines that the subtask switching signal corresponding to the subtask executed at the last time is 1, and the subtask needs to be switched. In other implementations, when the subtask detection result of the subtask executed at the last time and the subtask completion flag of the subtask executed at the last time both indicate that the subtask executed at the last time has been completed, the robot determines that the subtask switching signal corresponding to the subtask executed at the last time is 1. In other implementations, when the subtask detection result of the subtask executed at the last time indicates that the subtask executed at the last time has been completed, the robot determines that the subtask switching signal corresponding to the subtask executed at the last time is 1. The above are only examples, which can be set according to actual conditions.

[0164] In this embodiment, the robot can determine whether the subtask executed at the last time is completed by using any one of the subtask detection result of the subtask executed at the last time, the subtask completion flag of the subtask executed at the last time, or the state of the robot body at the last time, or a combination of multiple signals, so that the robustness of the determined subtask switching signal is stronger, which can improve the success rate of the robot in executing the entire complex long-range task.

[0165] FIG. 1B is a flowchart of the task planning process provided by an example embodiment of the present disclosure, which is described in combination with the robot structure shown in FIG. 7B . FIG. 7C .

[0166] Step 7011, at time t-n, the interactive terminal 13 of the robot can collect the second task instruction and send it to the first processor 11. The image acquisition device of the robot can collect the second scene image and send it to the first processor 11. The joint encoder of the robot obtains the state of the robot body at the last time (i.e. the second state), and sends the second state to the first processor 11 and the second processor 12.

[0167] Step 7012, the first processor 11 of the robot processes the second task instruction and the second scene image through the visual language model to determine the second subtask sequence corresponding to the second task instruction. Through the communication mode between the first processor 11 and the second processor 12, the first processor 11 sends the second subtask sequence to the second processor 12.

[0168] Step 7013, after the second processor 12 receives the second subtask sequence sent by the first processor 11, it processes the second subtask sequence for task planning, determines the subtask to be executed by the robot, and returns the subtask to the first processor 11. The subtask to be executed by the robot includes the subtask executed at the last time.

[0169] Step 7014, after the first processor 11 receives the subtask executed at the last time, it processes the second scene image, the second state and the subtask executed at the last time through the motion control model to determine the action instruction 0 executed by the robot. The first processor 11 sends the action instruction 0 to the controller in the robot, and the controller controls the execution end of the robot to execute the related action based on the action instruction, so as to realize the interaction with the surrounding environment. The execution end can be one or more devices of the robot, such as mechanical arm, mechanical hand, mechanical leg or mechanical foot.

[0170] Step 7015, after the first processor 11 receives the subtask executed at the last time, it detects the subtask executed at the last time to determine the subtask detection result and the subtask completion flag. Specifically, it detects the subtask through the visual language model to determine the subtask detection result, and detects the subtask through the motion control model to determine the subtask completion flag. The first processor 11 sends the subtask detection result and the subtask completion flag to the second processor 12.

[0171] Step 7016, the second processor 12 determines the subtask switching signal corresponding to the subtask executed at the last time based on one or more of the subtask detection result, the subtask completion flag and the second state.

[0172] Step 7017, the second processor 12 determines the second sub-task sub-sequence that is not completed in the second sub-task sequence corresponding to the last time according to the sub-task switching signal and the second sub-task sequence corresponding to the last time. Wherein, if the sub-task switching signal is 1, the action instruction 0 is emptied, and the emptying signal is sent to the controller, and the controller can control the robot to be in the target state (such as the static state) according to the emptying signal until the execution of the next action instruction, so as to avoid the robot repeatedly executing the action instruction 0 or the irregular small amplitude jitter after the execution of the action instruction 0.

[0173] Step 7018, after the interactive terminal 13 collects the first task instruction at t time, the first task instruction is sent to the first processor 11. The image acquisition device of the robot collects the first scene image and sends it to the first processor 11.

[0174] Step 7019, the first processor 11 processes the first task instruction and the first scene image based on the visual language model to determine the first sub-task sequence corresponding to the first task instruction. Through the communication mode between the first processor 11 and the second processor 12, the first processor 11 sends the first sub-task sequence to the second processor 12.

[0175] Step 7020, after the second processor 12 receives the first sub-task sequence sent by the first processor 11, The second processor 12 can process the task planning based on the first sub-task sequence and the second sub-task sub-sequence that is not completed at the last time to determine the target sub-task to be executed by the robot. The second processor 12 returns the target sub-task to the first processor 11.

[0176] Step 7021, after the first processor 11 receives the target sub-task, the first scene image, the first state and the target sub-task are processed based on the motion control model to determine the action instruction 1 to be executed by the robot. Wherein, the first state can be collected by the joint encoder and sent to the first processor 11 and the second processor 12. The first processor 11 sends the action instruction 1 to the controller in the robot, and the controller controls the execution end of the robot to execute related actions based on the action instruction, so as to realize the interaction with the surrounding environment. In addition, after the first processor 11 receives the target sub-task, the target sub-task can also be detected based on the visual language model to execute the subsequent steps, which will not be described here.

[0177] It should be noted that the collected scene image, the state of the robot and other information can be collected in real time, and the collection frequency is not limited.

[0178] It can be understood that if the structure of the robot is as shown in FIG. 1C The functions of the first processor 11 and the second processor 12 can be realized byFIG. 7C The processor 10 shown implements, and no longer detailed here.

[0179] It can also be understood that the execution sequence of the steps can be adjusted according to actual conditions. FIG. 7C The execution sequence of the steps can be adjusted according to actual conditions.

[0180] FIG. 1B is a flowchart of a robot control method provided by another exemplary embodiment of the present disclosure, which can be combined with FIG. 7C The robot structure and FIG. 1B The entire process of the robot control method is described.

[0181] As FIG. 7C The second task instruction is the instruction at time t-n (i.e., the last time), such as "clean the table", and the first task instruction is the instruction at time t (i.e., the current time), such as "leave the earphone behind".

[0182] At step 7101, after the robot interacts with the surrounding environment and the user, the interaction terminal 13 of the robot receives the second task instruction at time t-n, and sends the second task instruction to the first processor 11 of the robot. After the image acquisition device of the robot acquires the second scene image at time t-n, the second scene image is sent to the first processor 11. After the joint encoder of the robot acquires the second state (such as the joint angle) of the robot at time t-n, the second state is sent to the first processor 11 and the second processor 12.

[0183] At step 7102, the first processor 11 can perform vectorization processing on the second task instruction to determine the text feature vector at time t-n; and perform encoding processing on the second scene image based on the image encoder to determine the image feature vector at time t-n.

[0184] At step 7103, the first processor 11 processes the text feature vector and the image feature vector through the visual language model to determine the second subtask sequence corresponding to the second task instruction and the real-time voice instruction. The second subtask sequence exemplarily includes subtasks such as "fold the earphone", "fold the toy", and "fold the book", and the real-time voice instruction is, for example, "I will now tidy up the table". The first processor 11 sends the second subtask sequence to the second processor 12 of the robot, and feeds back the real-time voice instruction to the user through the interaction terminal 13.

[0185] At step 7104, the second processor 12 performs task planning processing on the second subtask sequence to realize subtask scheduling and determine the subtask to be executed by the robot, including the executed subtask at time t-n. The second processor 12 sends the executed subtask at time t-n to the first processor 11.

[0186] At step 7105, the first processor 11 encodes the second state based on the state encoder to determine the state feature vector at the t-n moment. The first processor 11 vectorizes the received instructions of the sub-tasks performed at the t-n moment to determine the sub-text feature vector. If the motion model is a diffusion model, the first processor 11 can also encode the noise signal based on the noise encoder to obtain the noise feature vector. The noise encoder can be a multi-layer perception machine. The first processor 11 processes the image feature vector, the state feature vector, the sub-text feature vector, and the noise feature vector through the motion control model to determine the action instruction 0. The first processor 11 sends the action instruction 0 to the controller in the robot, and the controller controls the execution end of the robot to perform relevant actions based on the action instruction, thereby realizing interaction with the surrounding environment.

[0187] At step 7106, after the first processor 11 receives the sub-tasks performed at the t-n moment, the first processor 11 detects the sub-tasks through the visual language model. Specifically, the first processor 11 detects the sub-tasks based on the sub-tasks performed at the t-n moment, the first scene image (the scene image collected at the t moment), and the second scene image to determine the sub-task detection result. Moreover, the first processor 11 detects the sub-tasks based on the second scene image, the second state, and the sub-tasks performed at the t-n moment through the motion control model to determine the sub-task completion flag of the sub-tasks performed at the t-n moment. The first processor 11 sends the sub-task detection result and the sub-task completion flag to the second processor 12.

[0188] At step 7107, the second processor 12 determines the sub-task switching signal corresponding to the sub-tasks performed at the t-n moment based on one or more of the sub-task detection result (not shown in the figure), the sub-task completion flag, and the second state of the robot body (not shown in the figure). The second processor 12 determines the second sub-task sub-sequence that has not been completed in the second sub-task sequence corresponding to the t-n moment based on the sub-task switching signal at the t-n moment and the second sub-task sequence corresponding to the t-n moment.

[0189] At step 7108, after the interaction terminal 13 receives the first task instruction at the t moment, the interaction terminal 13 sends the first task instruction to the first processor 11. After the image acquisition device of the robot collects the first scene image at the t moment, the image acquisition device sends the first scene image to the first processor 11. After the joint encoder of the robot collects the first state (such as the joint angle) of the robot at the t moment, the joint encoder sends the first state to the first processor 11 and the second processor 12.

[0190] Step 7109: The first processor 11 processes the first task instruction at time t and the first scene image at time t using a visual language model to determine the corresponding first sub-task sequence and real-time voice instruction. For specific implementation details, refer to the process for determining the second sub-task sequence. The first processor 11 sends the first sub-task sequence to the robot's second processor 12. The real-time voice instruction, for example, is "I will leave the headphones on the table," and the first processor 11 feeds this real-time voice instruction back to the user through the interactive terminal. The first sub-task sequence includes sub-tasks such as "place the headphones."

[0191] In step 7110, the second processor 12 determines a sorted combined subtask sequence based on the first subtask sequence and the second subtask subsequence that was not completed at time tn. The sorted combined subtask sequence includes tasks such as "placing headphones, putting away toys, and putting away books." After task planning processing, the target subtask to be executed by the robot is determined to be "placing headphones." The second processor 12 returns the target subtask to the first processor 11.

[0192] Step 7111: The first processor 11 processes the first scene image at time t, the first state at time t, and the target sub-task using a motion control model to determine the action instruction 1 to be executed by the robot. For a detailed implementation, refer to the process for determining action instruction 0. The first processor 11 sends action instruction 1 to the controller in the robot. The controller, based on the action instruction, controls the robot's execution end to perform relevant actions, thereby enabling interaction with the surrounding environment.

[0193] Understandably, after receiving the target subtask, the first processor can perform real-time loop closure detection on the target subtask based on a visual language model to obtain the subtask detection result. The first processor can also perform real-time subtask detection on the target subtask based on a motion control model to determine the subtask completion flag. The second processor, based on one or more of the subtask detection result, the subtask completion flag, and the robot's real-time state, determines the subtask switching signal corresponding to the target subtask. This allows for monitoring the completion status of the target subtask, preventing subtask execution failures that could reduce the task success rate. Specifically, if the subtask detection result indicates that the target subtask has failed, the robot can re-plan the task based on components such as the first processor, the second processor, and the interactive terminal, and even perform scene reconstruction to ensure the success rate of complex, long-term tasks. During re-planning or scene reconstruction, the robot can construct corresponding prompts to assist the visual language model in decomposing the subtask.

[0194] Understandably, if the robot structure is like... FIG. 8 As shown, then it can be FIG. 8 The functions of the first processor 11 and the second processor 12 described herein are both provided by Exemplary DeviceThe processor 10 shown implements, and will not be described in detail here.

[0195] It should also be understood that the order of execution of the steps in the method is not strictly limited FIG. 9 The order of execution of the steps in the method can be adjusted according to actual conditions.

[0196] In summary, in the robot control method provided by the present disclosure, the target subtask to be executed by the robot is determined by planning the first subtask sequence and the second subtask sequence corresponding to the last time, so that the robot can respond to the user's task instruction in real time and interact with the outside in real time during the execution of the task, thereby improving the interactive friendliness and task execution efficiency of the robot. In addition, the real-time first task instruction is responded to and the subtask is split through the upper visual language model, and after the subtask planning is completed, the action instruction is determined through the lower motion control model, so as to realize the focus on specific tasks through the upper and lower model structures, process the complex long-range task based on the upper and lower double models, and generate more accurate action instructions, improve the accuracy of the robot in understanding complex instructions, and thereby improve the success rate of the robot in executing complex long-range tasks.

[0197] FIG. 9 is a flowchart of a training method of a motion control model provided by an exemplary embodiment of the present disclosure. The present embodiment can be applied to a robot or an electronic device, such as FIG. 10 As shown, the method comprises the following steps 801-803: Step 801, obtaining a sample set, the sample set comprising: a single task instruction, a scene image, a robot body state, a label of an action instruction, and a label of a task completion flag.

[0198] The sample set can include a large amount of sample data, and each sample data includes a single task instruction, a scene image, a robot body state, a label of an action instruction, and a label of a task completion flag.

[0199] The label of the action instruction and the label of the task completion flag can be obtained by manual annotation, and the label can be used as the true value data during model training.

[0200] Step 802, using the motion control model to be trained to process the scene image, the robot body state, and the single task instruction, to determine a predicted action instruction and a predicted task completion flag.

[0201] It should be noted that step 802 can refer to the implementation of step 204 described above, and will not be described in detail here.

[0202] Step 803, training the motion control model to be trained based on the predicted action instruction, the predicted task completion flag, the label of the action instruction, and the label of the task completion flag, to obtain a target motion control model.

[0203] In practice, the first error can be determined based on the predicted action instruction and the label of the action instruction, and the first loss function is set, and the first loss is obtained from the first loss function and the first error. The second error is determined based on the predicted task completion flag and the label of the task completion flag, and the second loss function is set, and the second loss is obtained from the second loss function and the second error. Further, the first loss and the second loss are iteratively trained based on the motion control model. The obtained target motion control model is the trained motion control model.

[0204] In some embodiments, when training the motion control model, the image encoder and the state encoder can also be trained synchronously. The specific process is as follows: A sample set is obtained, which includes a single task instruction, a scene image, a robot body state, a label of an action instruction, and a label of a task completion flag.

[0205] Based on the image encoder, the scene image is encoded to determine an image feature vector; based on the state encoder, the state of the robot body is encoded to determine a state feature vector; and the single task instruction is vectorized to determine a text feature vector; Based on the motion control model to be trained, the image feature vector, the state feature vector, and the text feature vector are processed for action decision and task detection to determine a predicted action instruction and a predicted task completion flag; Based on the predicted action instruction, the predicted task completion flag, the label of the action instruction, and the label of the task completion flag, the image encoder, the state encoder, and the motion control model to be trained are trained to obtain a target motion control model.

[0206] Specifically, after obtaining the first loss and the second loss based on the predicted action instruction, the predicted task completion flag, the label of the action instruction, and the label of the task completion flag, the parameters in the image encoder, the parameters in the state encoder, and the parameters in the motion control model to be trained are adjusted at the same time by the first loss and the second loss.

[0207] It can be understood that if the motion control model is a diffusion model, a multi-layer perceptron for encoding noise signals can also be trained together.

[0208] In some embodiments, the visual language model can also be trained by a robot or an electronic device.

[0209] Based on the above embodiments, the robot can achieve three functions through the visual language model. First, the robot can understand and decompose complex task instructions into multiple sub-tasks through the language and image understanding capabilities of the visual language model. Second, the robot can obtain a reply statement for the task instruction through the visual language model, and feed back the reply statement to the user to enhance the interaction ability between the robot and the user. For example, the robot generates a voice instruction based on the reply statement and performs voice feedback to the outside based on the voice instruction. In another example, the robot can also display the reply statement on the display screen of the robot for the user to view. Third, the robot can also implement loop detection through the visual language model. Loop detection can be understood as detecting the completion of the sub-tasks. It can be understood that the robot can also implement other functions through the visual language model, which are not limited further here.

[0210] If the visual language model needs to implement the first function described above, the visual language model is trained using a first sample set. Specifically, it includes: obtaining a first sample set, the first sample set including task instructions, scene images, and real labels of sub-task sequences; processing the task instructions and the scene images through the visual language model to be trained to determine a predicted sub-task sequence; training the visual language model to be trained based on the predicted sub-task sequence and the real labels of the sub-task sequences to obtain a target visual language model.

[0211] Among them, the third error can be determined based on the predicted sub-task sequence and the real labels of the sub-task sequences, and the third loss function is set, and the third loss is obtained from the third loss function and the third error. The visual language model is iteratively trained using the third loss. The target visual language model is the trained visual language model.

[0212] If the visual language model needs to realize the above three functions, specifically, the visual language model is trained using a second sample set. The second sample set can include task instructions, subtask instructions, a first image, a second image, and target labels. The first image and the second image are scene images collected at different times for the same scene, and the collection time of the first image is earlier than that of the second image. In addition, the target labels are determined according to the output target of the model. For example, if the visual language model can perform task decomposition, the target labels include the true labels of the subtask sequence (such as the expected subtask sequence); if the visual language model can also provide a reply statement for the task instructions, the target labels can also include the true labels of the reply statement (such as the expected reply statement); if the visual language model can also realize loop detection, the target labels can also include the true labels of the subtask detection result (such as the expected subtask detection result). The visual language model can be trained to have the above three functions by adding adapters in the visual language model. Here, the specific structure of the visual language model is not limited and can be set according to actual conditions.

[0213] For example, the visual language model to be trained can include a model backbone, an adapter 1, and an adapter 2. The model backbone is used to realize general processing capability for text and images, and a pre-trained model backbone can be directly used. The pre-trained model backbone learns general processing capability from a large amount of unlabeled data (text, image-text, and other multi-modal data). The adapter 1 and the adapter 2 can use LoRA (Low-Rank Adaptation) adapters, which are not limited here. When training the visual language model, the model backbone is frozen, and only the parameters of the adapter 1 and the adapter 2 are fine-tuned. The task instructions, the first image, the true labels of the subtask sequence, and the true labels of the reply statement are used as sample data of task 1 to adjust the parameters of the adapter 1; the subtask instructions, the first image, the second image, and the true labels of the subtask detection result are used as sample data of task 2 to adjust the parameters of the adapter 2.

[0214] Specifically, sample data of the task 1 is extracted, the task instruction and the first image in the sample data of the task 1 are processed through the model backbone and the activated adapter 1 in the visual language model, and a predicted subtask sequence and a predicted reply statement are obtained. Sample data of the task 2 is extracted, the subtask instruction, the first image and the second image in the sample data of the task 2 are processed through the model backbone and the activated adapter 2 in the visual language model, and a predicted subtask detection result is obtained. A first task loss is determined based on the predicted subtask sequence and the real label of the subtask sequence, the predicted reply statement and the real label of the reply statement. A second task loss is determined based on the predicted subtask detection result and the real label of the subtask detection result. The first task loss and the second task loss are weighted and aggregated to obtain a total task loss, and the parameters of the adapter 1 and the adapter 2 are updated simultaneously using the total task loss. When the training stop condition is met, a target visual language model, i.e., a trained visual language model, is obtained. The training stop condition can be an iteration number or other training stop condition, which can be set according to actual needs. In this way, the target visual language model can realize three functions of decomposing a task corresponding to a task instruction, providing a reply statement for the task instruction, and realizing loop detection.

[0215] It can be understood that the structure and training method of the visual language model described above are only described as an example, and other model structures and training methods can also be used, which are not limited further herein.

[0216] FIG. 11 FIG. 11 FIG. 1 is a structural schematic diagram of a robot control device provided by an example embodiment of the present disclosure. Exemplary Robot The robot control device 900 shown includes: The first acquisition module 901 is configured to acquire a first task instruction, a first scene image and a first state of a robot body at a current time; The first processing module 902 is configured to process the first task instruction and the first scene image through a visual language model to determine a first subtask sequence corresponding to the first task instruction; The second processing module 903 is configured to perform task planning processing based on the first subtask sequence and a second subtask sequence corresponding to a previous time to determine a target subtask to be executed by the robot; wherein the robot has not completed at least one second subtask in the second subtask sequence corresponding to the previous time; The third processing module 904 is configured to process the first scene image, the first state and the target subtask through a motion control model to determine an action instruction to be executed by the robot; The first control module 905 is configured to control the robot to execute the action instruction.

[0217] FIG. 12 is a structural schematic diagram of a robot control device provided by another exemplary embodiment of the present disclosure.

[0218] In some optional implementations, the first processing module 902 includes: The first processing unit 1001 is configured to perform vectorization processing on the first task instruction, and determine a target text feature vector. The second processing unit 1002 is configured to perform encoding processing on the first scene image based on an image encoder, and determine an image feature vector. The third processing unit 1003 is configured to perform processing on the target text feature vector and the image feature vector by a visual language model, and determine a first subtask sequence corresponding to the first task instruction.

[0219] In some optional implementations, the third processing module 904 includes: The fourth processing unit 1004 is configured to perform encoding processing on the first state based on a state encoder, and determine a state feature vector. The fifth processing unit 1005 is configured to perform vectorization processing on the target subtask, and determine a subtext feature vector. The sixth processing unit 1006 is configured to perform processing on the image feature vector corresponding to the first scene image, the state feature vector, and the subtext feature vector by a motion control model, and determine an action instruction to be executed by the robot.

[0220] In some optional implementations, the robot control device 900 further includes: The determining module 906 is configured to determine a voice instruction based on the first task instruction and the first scene image by a visual language model. The second control module 907 is configured to control the robot to output a voice corresponding to the voice instruction based on the voice instruction.

[0221] In some optional implementations, the second subtask sequence corresponding to the previous moment is determined by the following method: determining a second task instruction and a second scene image of the previous moment; performing processing on the second task instruction and the second scene image by a visual language model, and determining a second subtask sequence corresponding to the second task instruction.

[0222] In some optional implementations, the second processing module 903 includes: The first determining unit 1007 is configured to determine a subtask switching signal corresponding to a subtask executed at the previous moment. The second determining unit 1008 is configured to determine a second subtask subsequence that is not completed in the second subtask sequence corresponding to the previous moment based on the subtask switching signal at the previous moment and the second subtask sequence corresponding to the previous moment. The seventh processing unit 1009 is configured to perform task planning processing based on the first subtask sequence and the second subtask subsequence that is not completed at the last time, and determine a target subtask to be executed by the robot.

[0223] In some optional implementations, the first determining unit 1007 includes: a first determining subunit configured to determine a subtask detection result of the subtask executed at the last time based on a visual language model; a second determining subunit configured to determine a subtask completion flag of the subtask executed at the last time based on a motion control model; and a third determining subunit configured to determine a subtask switching signal corresponding to the subtask executed at the last time based on one or more of the subtask detection result of the subtask executed at the last time, the subtask completion flag of the subtask executed at the last time, and a state of the robot body at the last time.

[0224] In some optional implementations, the first determining subunit is specifically configured to determine the subtask detection result by performing subtask detection on the subtask executed at the last time, the first scene image, and the scene image at the last time based on the visual language model.

[0225] In some optional implementations, the second determining subunit is specifically configured to determine the subtask completion flag of the subtask executed at the last time by performing subtask detection on the scene image at the last time, the state of the robot body at the last time, and the subtask executed at the last time based on the motion control model.

[0226] The beneficial technical effects of the exemplary embodiments of the device correspond to the beneficial technical effects of the exemplary robot control method described above, which will not be repeated here.

[0227] FIG. 12 FIG. 11 is a structural schematic diagram of a training device of a motion control model according to an exemplary embodiment of the present disclosure. FIG. 12 The training device 1100 of the motion control model shown includes: The second acquisition module 1101 is configured to acquire a sample set, and the sample set includes a single-task instruction, a scene image, a state of a robot body, a label of an action instruction, and a label of a task completion flag. The fourth processing module 1102 is configured to process the scene image, the state of the robot body, and the single-task instruction by using the motion control model to be trained, and determine a predicted action instruction and a predicted task completion flag. The training module 1103 is configured to train the motion control model to be trained based on the predicted action instruction, the predicted task completion flag, the label of the action instruction, and the label of the task completion flag, to obtain a target motion control model.

[0228] Corresponding beneficial technical effects of the device example embodiment can be referred to the corresponding beneficial technical effects of the training method of the motion control model example embodiment described above, which will not be repeated here.

[0229] Exemplary Computer Program Product and Computer-Readable Storage Medium ​ is a structural schematic diagram of a robot provided by another example embodiment of the present disclosure. As shown in ​ The robot 1200 includes at least one processor 1201 and a memory 1202.

[0230] The processor 1201 can be a central processing unit (CPU) or other forms of processing units having data processing capability and / or instruction execution capability, and can control other components in the electronic device 10 to perform desired functions. The visual language model and the motion control model (not shown in the figure) can be deployed in the processor 1201.

[0231] The memory 1202 can include one or more computer program products, which can include various forms of computer readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM), cache memory, and / or the like. The non-volatile memory may, for example, include read-only memory (ROM), hard disk, flash memory, and / or the like. One or more computer program instructions can be stored on the computer readable storage medium, and the processor 1201 can run the one or more computer program instructions to implement the robot control method of various embodiments of the present disclosure described above based on the deployed visual language model and motion control model, and / or the training method of the motion control model, and also to implement other desired functions.

[0232] In one example, the robot 1200 can further include an input device 1203 and an output device 1204, which are interconnected through a bus system and / or other forms of connection mechanism (not shown).

[0233] The input device 1203 can further include, for example, a microphone, a touch display screen, a physical button, and / or the like.

[0234] The output device 1204 can output various information to the outside, which can include, for example, a display screen, a speaker, an indicator light, a projection device, a communication network and a remote output device connected thereto, and / or the like.

[0235] Of course, in order to simplify, ​ Only some of the components in the robot 1200 related to the present disclosure are shown in the robot 1200, and components such as buses, input / output interfaces, and / or the like are omitted. In addition, the robot 1200 can further include any other appropriate components according to specific application cases.

[0236] ​ In addition to the above method and device, the embodiments of the present disclosure can also provide a computer program product comprising computer program instructions which, when executed by a processor, cause the processor to perform the steps of the robot control method of various embodiments of the present disclosure described in the above “Exemplary Method” section, and / or the training method of the motion control model.

[0237] The computer program product can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++, etc., and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server.

[0238] In addition, the embodiments of the present disclosure can also be a computer readable storage medium having stored thereon computer program instructions which, when executed by a processor, cause the processor to perform the steps of the robot control method of various embodiments of the present disclosure described in the above “Exemplary Method” section, and / or the training method of the motion control model.

[0239] The computer readable storage medium can take the form of one or more combinations of any of the following: a readable signal medium or a readable storage medium. The readable storage medium, for example, but not limited to, includes an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination thereof. More specific examples (non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0240] The above describes the basic principles of the present disclosure in combination with specific embodiments, but the advantages, benefits, effects, etc. mentioned in the present disclosure are only examples and are not limiting, and it cannot be considered that each embodiment of the present disclosure must have them. In addition, the above specific details are only for the purpose of example and for the purpose of understanding, and are not limiting, and the above details do not limit the present disclosure to the above specific details.

[0241] Those skilled in the art will appreciate that various adaptations and modifications of the application can be configured without departing from the scope and spirit of the application. Therefore, it is to be understood that, within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described herein.

Claims

1. A robot control method, comprising: Acquire the first task instruction, the first scene image, and the first state of the robot body at the current moment; The first task instruction and the first scene image are processed using a visual language model to determine the first sub-task sequence corresponding to the first task instruction. Based on the first sub-task sequence and the second sub-task sequence corresponding to the previous time step, task planning processing is performed to determine the target sub-task to be executed by the robot; wherein, the robot has not completed at least one second sub-task in the second sub-task sequence corresponding to the previous time step. The motion control model processes the first scene image, the first state, and the target sub-task to determine the action instructions to be executed by the robot. Control the robot to execute the action commands.

2. The method according to claim 1, wherein, The step of processing the first task instruction and the first scene image through a visual language model to determine the first sub-task sequence corresponding to the first task instruction includes: The first task instruction is vectorized to determine the target text feature vector; Based on the image encoder, the first scene image is encoded to determine the image feature vector; The visual language model is used to process the target text feature vector and the image feature vector to determine the first subtask sequence corresponding to the first task instruction.

3. The method according to claim 1, wherein, The process of processing the first scene image, the first state, and the target sub-task using a motion control model to determine the action instructions to be executed by the robot includes: Based on the state encoder, the first state is encoded to determine the state feature vector; The target subtask is vectorized to determine the subtext feature vector; The motion control model processes the image feature vector, the state feature vector, and the sub-text feature vector corresponding to the first scene image to determine the action command to be executed by the robot.

4. The method according to claim 1, wherein, Also includes: Based on the first task instruction and the first scene image, the visual language model is used to determine the voice instruction. Based on the voice command, the robot is controlled to output the voice corresponding to the voice command.

5. The method according to claim 1, wherein, The second subtask sequence corresponding to the previous time step is determined in the following ways: Determine the second task instruction and the second scene image from the previous moment; Using the visual language model, the second task instruction and the second scene image are processed to determine the second sub-task sequence corresponding to the second task instruction.

6. The method according to any one of claims 1-5, wherein, The task planning process based on the first sub-task sequence and the second sub-task sequence corresponding to the previous time step, to determine the target sub-task to be executed by the robot, includes: Determine the subtask switching signal corresponding to the subtask executed in the previous moment; Based on the subtask switching signal of the previous time and the second subtask sequence corresponding to the previous time, determine the unfinished second subtask subsequence in the second subtask sequence corresponding to the previous time. Based on the first sub-task sequence and the second sub-task sequence that was not completed in the previous time step, task planning is performed to determine the target sub-task to be executed by the robot.

7. The method according to claim 6, wherein, The determination of the subtask switching signal corresponding to the subtask executed in the previous moment includes: Based on the visual language model, determine the subtask detection result of the subtask executed in the previous time step; Based on the motion control model, determine the subtask completion flag of the subtask executed in the previous moment; Based on one or more of the subtask detection result of the subtask executed in the previous moment, the subtask completion flag of the subtask executed in the previous moment, and the state of the robot body in the previous moment, the subtask switching signal corresponding to the subtask executed in the previous moment is determined.

8. The method according to claim 7, wherein, The subtask detection result based on the visual language model, determining the subtask detection result of the subtask executed in the previous time step, includes: Using a visual language model, subtask detection is performed based on the subtask executed in the previous time step, the first scene image, and the scene image in the previous time step, and the subtask detection result is determined.

9. The method according to claim 7, wherein, The determination of the subtask completion flag for the subtask executed in the previous moment based on the motion control model includes: Using the motion control model, subtask detection is performed based on the scene image of the previous moment, the state of the robot body of the previous moment, and the subtask executed in the previous moment, to determine the subtask completion flag of the subtask executed in the previous moment.

10. A method for training a motion control model, comprising: Obtain a sample set, which includes: single task instructions, scene images, robot body states, labels for action instructions, and labels for task completion markers; Using the motion control model to be trained, the scene image, the robot body state, and the single task instruction are processed to determine the predicted action instruction and the predicted task completion flag; Based on the predicted action command, the predicted task completion flag, the label of the action command, and the label of the task completion flag, the motion control model to be trained is trained to obtain the target motion control model.

11. A robot control device, comprising: The first acquisition module is used to acquire the first task instruction, the first scene image, and the first state of the robot body at the current moment. The first processing module is used to process the first task instruction and the first scene image through a visual language model to determine the first sub-task sequence corresponding to the first task instruction. The second processing module is used to perform task planning processing based on the first sub-task sequence and the second sub-task sequence corresponding to the previous time step, and determine the target sub-task to be executed by the robot; wherein the robot has not completed at least one second sub-task in the second sub-task sequence corresponding to the previous time step. The third processing module is used to process the first scene image, the first state, and the target sub-task through a motion control model to determine the action instructions to be executed by the robot. The first control module is used to control the robot to execute the action commands.

12. A training device for a motion control model, comprising: The second acquisition module is used to acquire a sample set, which includes: single task instructions, scene images, robot body state, labels of action instructions, and labels of task completion flags; The fourth processing module is used to process the scene image, the robot body state and the single task instruction using the motion control model to be trained, and to determine the predicted action instruction and the predicted task completion flag. The training module is used to train the motion control model to be trained based on the predicted action command, the predicted task completion flag, the label of the action command, and the label of the task completion flag, so as to obtain the target motion control model.

13. A computer-readable storage medium storing a computer program, which, when executed by a processor, is used to implement the robot control method according to any one of claims 1-9, or to implement the training method for the motion control model according to claim 10.

14. A robot, the robot comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the robot control method according to any one of claims 1-9, or to implement the motion control model training method according to claim 10.

Images