Robot control method and apparatus

Abstract

Embodiments of the present application provide a robot control method and device, the method comprising: obtaining a task scene graph based on a camera preset by a robot; determining a target task and a target action prediction model corresponding to the target task based on the task scene graph, different tasks corresponding to different action prediction models; collecting a target image in a process in which the robot executes the target task based on the camera, and determining actual joint angle data of the robot; inputting the target image and the actual joint angle data into the target action prediction model to obtain predicted joint angle data; and controlling the robot to complete the target task based on the predicted joint angle data. The method in the embodiments of the present application can efficiently control the robot to execute a task.

Description

Technical Field

[0001] This application relates to the field of robotics, and in particular to a robot control method and apparatus. Background Technology

[0002] With the development of artificial intelligence, robots can perform some simple tasks, especially tasks such as grasping and manipulating objects.

[0003] Many methods of controlling robots to perform tasks rely on visual depth information. These methods typically require high-precision depth sensors and complex environmental modeling to locate and grasp objects during task execution. However, during task execution, the types of objects are numerous, their shapes are irregular, and their placement is varied. This results in the limited adaptability of this method of relying on depth information to perform tasks to environmental changes, which cannot meet the needs of robots to perform tasks. Summary of the Invention

[0004] This application provides a robot control method and apparatus for efficiently controlling a robot to perform tasks.

[0005] In a first aspect, embodiments of this application provide a robot control method, including:

[0006] The robot acquires a task scene map based on a pre-set camera, and determines the target task and the target action prediction model corresponding to the target task based on the task scene map. Different tasks correspond to different action prediction models.

[0007] Based on the target image captured by the camera during the robot's execution of the target task, and the actual joint angle data of the robot determined, the target image and the actual joint angle data are input into the target motion prediction model to obtain predicted joint angle data;

[0008] The robot is controlled to complete the target task based on the predicted joint angle data.

[0009] In one possible implementation, determining the target task and the target action prediction model corresponding to the target task based on the task scenario graph includes:

[0010] The task scene graph is input into a pre-trained task detection model to obtain the execution status of different tasks output by the pre-trained task detection model.

[0011] The task whose preset task conditions are successfully matched with the corresponding task execution status is identified as the target task, and the action prediction model corresponding to the target task is identified as the target action prediction model.

[0012] In one possible implementation, the target action prediction model includes a target image encoder, a target first encoder, and a target decoder, wherein the output of the target image encoder is connected to the input of the target first encoder, and the output of the target first encoder is connected to the input of the target decoder.

[0013] The step of inputting the target image and the actual joint angle data into the target motion prediction model to obtain predicted joint angle data includes:

[0014] The target image is input into the target image encoder, and the actual joint angle data is input into the target first encoder to obtain the predicted joint angle data output by the target decoder.

[0015] In one possible implementation, before acquiring target images of the robot performing the target task based on the camera, determining the robot's actual joint angle data, and inputting the target images and the actual joint angle data into the target motion prediction model to obtain predicted joint angle data, the method further includes:

[0016] The target tool for performing the target task is determined based on the target task.

[0017] Replace the robot's tool with the target tool.

[0018] In one possible implementation, before acquiring target images of the robot performing the target task based on the camera, determining the robot's actual joint angle data, and inputting the target images and the actual joint angle data into the target motion prediction model to obtain predicted joint angle data, the method further includes:

[0019] The system acquires training images of the robot for different tasks captured by the camera, actual joint angle data of the robot during training, and target joint angle data of the robot.

[0020] Based on the training images corresponding to each task, the actual joint angle data of the corresponding task, and the target joint angle data of the corresponding task, different initial motion prediction models are trained to determine the motion prediction models corresponding to different tasks.

[0021] In one possible implementation, acquiring the training images of the robot corresponding to the task captured by the camera, the actual joint angle data of the robot during training, and the target joint angle data of the robot when the robot performs different tasks includes:

[0022] When the robot is controlled by the control device to perform different tasks, the actual joint angle data of the robot during the training of different tasks is obtained, and the training images collected by the robot's camera for different tasks and the control pose information of the control device controlling the robot to perform different tasks are also obtained.

[0023] Based on inverse kinematics, the control pose information of the robot performing different tasks is converted into target joint angle data to determine the target joint angle data for different tasks.

[0024] In one possible implementation, training different initial motion prediction models based on training images corresponding to each task, actual joint angle data of the corresponding task, and target joint angle data of the corresponding task to determine the motion prediction model corresponding to different tasks includes:

[0025] The training images and actual joint angle data corresponding to each task are input into different initial motion prediction models to obtain the first training prediction joint angle data output by each initial motion prediction model for different tasks.

[0026] Based on the first training prediction joint angle data and the target joint angle data of the corresponding task, the initial motion prediction model is trained to determine the motion prediction model corresponding to different tasks.

[0027] In one possible implementation, the initial motion prediction model includes an initial first encoder and an initial first motion prediction model, wherein the output of the initial first encoder is connected to the input of the initial first motion prediction model.

[0028] The process of training different initial motion prediction models based on training images corresponding to each task, actual joint angle data of the corresponding task, and target joint angle data of the corresponding task to determine the motion prediction model corresponding to different tasks includes:

[0029] The actual joint angle data of each task and the target joint angle data of each task are input into the initial first encoder of different initial motion prediction models. The training images of each task and the actual joint angle data of each task are input into the initial first motion prediction models of different initial motion prediction models to obtain the motion style variables output by each initial first encoder for different tasks and the second training prediction joint angle data output by each initial first motion prediction model for different tasks.

[0030] Based on the motion style variables output by each initial first encoder for different tasks, the second training prediction joint angle data output by each initial first motion prediction model for different tasks, and the target joint angle data of different tasks, the initial motion prediction model for the corresponding task is trained to obtain the trained initial motion prediction model, which includes the trained initial first encoder and the trained initial first motion prediction model.

[0031] The initial first action prediction model trained for different tasks is determined as the action prediction model for the corresponding task.

[0032] In one possible implementation, the step of training an initial motion prediction model for a corresponding task based on the motion style variables output by each initial first encoder for different tasks, the second training prediction joint angle data output by each initial first motion prediction model for different tasks, and the target joint angle data for different tasks, to obtain the trained initial motion prediction model, includes:

[0033] Based on the action style variables output by each initial first encoder for different tasks, construct the first loss function corresponding to each task;

[0034] Based on the second training prediction joint angle data output by each initial first motion prediction model for different tasks and the target joint angle data of the corresponding task, a second loss function is constructed for each task.

[0035] The initial action prediction model for each task is trained based on the first loss function and the second loss function for each task, thus obtaining the trained initial action prediction model.

[0036] Secondly, embodiments of this application provide a robot control device, including:

[0037] The task determination module is used to acquire a task scene map based on the robot's preset camera, and determine the target task and the target action prediction model corresponding to the target task based on the task scene map. Different tasks correspond to different action prediction models.

[0038] The joint angle prediction module is used to acquire target images of the robot during the execution of the target task based on the camera, determine the actual joint angle data of the robot, and input the target image and the actual joint angle data into the target action prediction model to obtain predicted joint angle data.

[0039] The task execution module is used to control the robot to complete the target task based on the predicted joint angle data.

[0040] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0041] The memory stores computer-executed instructions;

[0042] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0043] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0044] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0045] The robot control method and apparatus provided in this application automatically identify the target task to be performed through the robot's own camera, select the corresponding target action prediction model according to the determined target task, and after determining the target action prediction model corresponding to the target task, acquire the target image of the task to be performed and the actual joint angle data of the robot during the execution of the target task. By inputting the target image and the actual joint angle data into the target action prediction model, the predicted joint angle data of the robot to perform the target task in the future can be predicted. Thus, the robot is controlled to complete the target task through the predicted joint angle data. This process predicts the predicted joint angle data of the robot in the future through the robot's visual data and motion data, enabling the robot to adjust its action strategy in real time, adapt to different scenarios during the execution of the task, and improve the reliability and efficiency of the task execution. Attached Figure Description

[0046] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0047] Figure 1 This is an exemplary embodiment illustrating an implementation scenario.

[0048] Figure 2 This is a flowchart illustrating an exemplary robot control method;

[0049] Figure 3 This is a flowchart illustrating a robot control method in another exemplary embodiment;

[0050] Figure 4 This is a flowchart illustrating a robot control method in another exemplary embodiment;

[0051] Figure 5 This is a schematic diagram illustrating the structure of a target action prediction model in an exemplary embodiment;

[0052] Figure 6 This is a flowchart illustrating a robot control method in another exemplary embodiment;

[0053] Figure 7 This is a flowchart illustrating a robot control method in another exemplary embodiment;

[0054] Figure 8 This is a schematic diagram illustrating the structure of an initial action prediction model in an exemplary embodiment;

[0055] Figure 9 This is a block diagram illustrating a robot control device in an exemplary embodiment;

[0056] Figure 10 A schematic diagram of the structure of the electronic device provided in this application.

[0057] The accompanying drawings have illustrated specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to specific embodiments. Detailed Implementation

[0058] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0059] Compared to traditional single-arm or wheeled robots, dual-arm robots offer greater operational flexibility and the ability to perform complex tasks, such as grasping, serving, and distributing food in the catering industry. This capability gives them enormous potential in practical applications such as automated catering services and food stall operations.

[0060] However, most current robot manipulation methods focus on simple tasks in static environments, such as grasping and moving a single object, while research on dynamic and diverse scenarios remains relatively limited. For example, in the catering industry, the variety, irregular shapes, and random placement of dishes present significant challenges for robot operation, including insufficient grasping accuracy and low motion planning efficiency. Therefore, how to effectively cope with dynamic environments within a framework of imitation learning and improve the operational efficiency and robustness of dual-arm robots in task execution is a crucial issue that urgently needs to be addressed.

[0061] In some scenarios, robots primarily rely on visual depth information for grasping and manipulation. This process typically requires high-precision depth sensors and complex environmental modeling to locate and grasp objects. However, the variety of objects, such as vegetables, is vast, with irregular shapes and diverse placement methods, limiting the adaptability of this approach to environmental changes. Furthermore, depth information in complex scenes is easily affected by ambient light, object reflections, and other factors, reducing the accuracy and robustness of grasping. In addition, to meet the demands of complex operations, this approach usually requires the design of sophisticated rules and parameter adjustments, which not only increases development difficulty but also places higher demands on real-time performance and computational resources.

[0062] The robot control method and apparatus provided in this application are intended to solve the above-mentioned technical problems in the prior art.

[0063] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0064] like Figure 1 This is an exemplary embodiment illustrating an implementation scenario, which includes a robot 100, a controller 200, and a camera 300.

[0065] The controller 200 is located inside the robot 100, and the camera 300 is located on the robot, such as on the robot's head.

[0066] The controller 200 communicates with the camera 300. The controller 200 can also control the operation of the robot 00, such as controlling the joint angle of the arm of the robot 100 to perform corresponding tasks.

[0067] The execution subject of the method in this embodiment is the controller 200. The controller 200 is equipped with a motion prediction model. The camera 200 can capture images and transmit the images to the controller 200. The controller 200 controls the robot 100 according to the transmitted images.

[0068] In some embodiments, the controller 200 acquires a task scene map based on the camera 300 preset by the robot 100, and determines the target task and the target action prediction model corresponding to the target task based on the task scene map. Different tasks correspond to different action prediction models. The controller 200 acquires target images of the robot during the execution of the target task based on the camera, and determines the actual joint angle data of the robot. The target images and actual joint angle data are input into the target action prediction model to obtain predicted joint angle data. The controller 200 controls the robot 100 to complete the target task based on the predicted joint angle data.

[0069] In some embodiments, the target task is a grasping task, such as grasping and plating dishes in a catering scenario.

[0070] Understandably, the robot control device can be set in Figure 1 In the controller 200, but as shown in this embodiment, Figure 1 The implementation environment shown is merely exemplary. In other embodiments, the robot control method can also be applied to other implementation environments, and the robot control device can also be set in other structures in other implementation environments. No specific limitations are made here.

[0071] In this embodiment, the controller 200 can be an edge environment or a cloud environment, such as a physical server, a server cluster, or a cloud server, which are devices with data processing capabilities. No specific restrictions are imposed here.

[0072] The device scheduling device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0073] Figure 2 This is a flowchart illustrating an exemplary embodiment of a robot control method, applied to... Figure 1 Controller 100 in, such as Figure 2 As shown, the method includes steps S201 to S203, which are described in detail below:

[0074] S201. Acquire a task scene map based on the robot's preset camera, and determine the target task and the target action prediction model corresponding to the target task based on the task scene map.

[0075] In some embodiments, before the robot performs a task, the robot's controller can determine the target task that the robot needs to perform by capturing a task scene image through a camera. The target task is a grasping task, such as serving food or picking up items, and there are no specific restrictions here.

[0076] In some embodiments, the camera captures a task scene image and transmits the task scene image to the controller.

[0077] The controller identifies the target task based on the task scenario diagram.

[0078] In some embodiments, different tasks correspond to different action prediction models. Once the target task is determined, the action prediction model corresponding to the target task can be determined.

[0079] In some embodiments, the controller has multiple motion prediction models internally.

[0080] In some embodiments, the target task can be determined using a task detection model.

[0081] In one embodiment, the controller inputs the task scene graph into a pre-trained task detection model to obtain the execution status of different tasks output by the pre-trained task detection model; the task whose preset task conditions are successfully matched with the corresponding task execution status is determined as the target task, and the action prediction model corresponding to the target task is determined as the target action prediction model.

[0082] The pre-trained task detection model can be the YOLO (You Only Look Once, real-time object detection) detection network or other detection networks; no specific restrictions are imposed here.

[0083] In some embodiments, the robot control method is applied to a catering scenario. The robot's camera captures images of the dishes on the dining table to obtain a task scenario map. Based on the task scenario map, the target task is determined. At this time, the target task may be the task of serving a certain dish.

[0084] S202. Based on the camera, collect target images during the robot's execution of the target task, determine the robot's actual joint angle data, and input the target images and actual joint angle data into the target motion prediction model to obtain predicted joint angle data.

[0085] In some embodiments, after determining the target task, the controller can call the target motion prediction model and, with the assistance of the robot's actual joint angle data and the target image captured by the camera, predict the joint angle data for the robot to perform the target task.

[0086] In some embodiments, the actual joint angle data are the joint angles of the robot in its current state, including the joint angle data corresponding to the spatial pose information of the robot's two arms and the joint angle data of the gripper.

[0087] The joint angle data corresponding to the spatial pose information of the robot's two arms includes 6 degrees of freedom (position and orientation) for each arm, and the joint angle data of the gripper includes 1 degree of freedom for each gripper corresponding to each arm, that is, the joint angles include 14 degrees of freedom.

[0088] Predicted joint angle data includes the robot's joint angles at a future moment or a sequence of joint angles over a certain period of time.

[0089] In some embodiments, the action prediction model can be one of Transformer (a deep learning model based on self-attention mechanism), CNN (convolutional neural network), LSTM (long short-term memory network), etc., without specific limitations.

[0090] In some embodiments, the action prediction model is a pre-trained model.

[0091] S203. Control the robot to complete the target task based on predicted joint angle data.

[0092] In some embodiments, after determining the predicted joint angle data, the robot's arms and grippers can be controlled based on the predicted joint angle data to complete the target task.

[0093] In some embodiments, the controller acquires the target image and actual joint angle data in real time, predicts the predicted joint angle data for the next moment, and controls the robot's arms and grippers based on the predicted joint angle data for the next moment.

[0094] In some embodiments, the predicted joint angle data controls the angles of the robot's arms and grippers to achieve the target task.

[0095] In one embodiment, the target task is to serve food, and the process of the target task may include the robot's right arm gripping the control plate, and the left arm picking up the food and placing it on the plate. The corresponding target image may include the shape of the food on the plate and the position of the food to be served. The predicted joint angle data is the data controlling the spatial pose, direction, and angle of the robot's left and right arms. This predicted joint angle data is used to control the robot's right arm to grip the control plate and the left arm to pick up the food and place it on the plate. When the predicted joint angle data controls the robot's left and right arms to complete the target task, different foods require different approaches from the left arm. The way the right arm completes the dish serving based on the predicted joint angle data is different. For example, if some dishes need to be shaped, the corresponding predicted joint angle data can control the robot to first place the grabbed dishes in the center of the plate, and then place the newly grabbed dishes around the dishes placed in the center of the plate, so as to meet the shape requirements of the dishes. Since different tasks correspond to different motion prediction models, the motion prediction model will predict the corresponding predicted joint angle data according to the needs of different tasks, so as to meet the execution requirements of the corresponding tasks. As mentioned above, the order of placing dishes is first in the center and then around the edges.

[0096] In some embodiments, the controller acquires a target image and actual joint angle data, predicts joint angle data for a future time period, and controls the robot's arms and grippers to perform the target task during that future time period using the predicted joint angle data. After the future time period has passed, the controller continues to acquire the target image and actual joint angle data and predict the predicted joint angle data for a future time period until the target task is completed.

[0097] In this embodiment, the robot automatically identifies the target task to be performed using its own camera, selects the corresponding target action prediction model based on the determined target task, and after determining the target action prediction model corresponding to the target task, acquires the target image of the task and the actual joint angle data of the robot during the execution of the target task. By inputting the target image and the actual joint angle data into the target action prediction model, the predicted joint angle data of the robot executing the target task at a future time can be predicted, thereby controlling the robot to complete the target task through the predicted joint angle data.

[0098] This process uses visual data and motion data to predict the robot's joint angles in the future, enabling the robot to adjust its motion strategy in real time to adapt to different scenarios during task execution, thereby improving the reliability and efficiency of task execution.

[0099] Figure 3 This is a flowchart illustrating a robot control method in another exemplary embodiment, such as... Figure 3 As shown, this method is one possible implementation of step S201, and is described in detail below:

[0100] S301. Input the task scene graph into the pre-trained task detection model to obtain the execution status of different tasks output by the pre-trained task detection model.

[0101] In some embodiments, the task scene diagram can be a scene diagram of the items that the robot is to grasp. For example, in the catering industry, the picture of the dishes on the table is the task scene diagram, or in the scene of moving items, the picture of the items to be moved is the task scene diagram.

[0102] The controller is equipped with a task detection model. By inputting the task scenario diagram into the task detection model, the execution status of different tasks can be obtained.

[0103] The execution status can be the location of items corresponding to one or more tasks, or the quantity of items corresponding to one or more tasks, etc. There are no specific restrictions here.

[0104] In some embodiments, the task detection model can be trained using historical task scene graphs and labels set on those graphs.

[0105] S302. The task whose preset task conditions are successfully matched with the execution status of the corresponding task is identified as the target task, and the action prediction model corresponding to the target task is identified as the target action prediction model.

[0106] In some embodiments, different tasks have corresponding pre-defined task conditions. When the execution status matches the task conditions, the task corresponding to the successfully matched execution status is determined as the target task.

[0107] If there are multiple target tasks, then each target task should be addressed accordingly. Figure 2 The operation of the embodiment shown.

[0108] In some embodiments, the task conditions may be that the item corresponding to a certain task is located in a certain area, or that the quantity of the item corresponding to a certain task is less than a preset value, etc., without specific limitations.

[0109] In some embodiments, for the catering industry, the content of the task scenario diagram is different dishes on the table, and the execution status is the quantity of a certain dish. One dish corresponds to one task, and the execution status of different tasks corresponds to the quantity of different dishes. The preset task condition for each task is that the quantity of the corresponding dish is less than a preset value. When the quantity of the corresponding dish in the execution status of a task satisfies the condition that the corresponding dish is less than a preset value, then the task corresponding to that dish is the target task.

[0110] It is understandable that the preset values ​​in the task conditions for different tasks can be the same or different.

[0111] If there are 3 dishes: shredded potatoes, bean sprouts, and green beans, and these 3 dishes correspond to 3 tasks, the execution status of the 3 tasks is as follows: 2 plates of shredded potatoes, 3 plates of bean sprouts, and 1 plate of green beans. The task conditions for the 3 tasks are that the number of shredded potatoes is less than 2 plates, the number of bean sprouts is less than 2 plates, and the number of green beans is less than 1 plate. Thus, the successful match is 1 plate of green beans, which is less than 2 plates. Therefore, the task corresponding to green beans is the target task, and the target task is to serve green beans.

[0112] Understandably, the target task of picking green beans corresponds to a target action prediction model for picking green beans, and the green bean picking operation can be performed through this target action prediction model.

[0113] In some embodiments, the target task of serving green beans may include the action flow of clamping a plate, scooping green beans, and placing a plate, all of which are controlled by predicted joint angle data obtained by the target action prediction model.

[0114] In some embodiments, after a target task is completed, the operation of acquiring a task scene map based on the robot's preset camera is triggered again to identify a new target task.

[0115] In this embodiment, the robot interacts with the task scene image captured by the task detection model camera, which can automatically identify the target task and determine the nature of the target task, thereby improving the robot's autonomy and efficiency in performing tasks.

[0116] Figure 4 This is a flowchart illustrating a robot control method in another exemplary embodiment, such as... Figure 4 As shown, this method is one possible implementation of step S202, and is described in detail below:

[0117] S401. Input the target image into the target image encoder and input the actual joint angle data into the target first encoder to obtain the predicted joint angle data output by the target decoder.

[0118] In some embodiments, such as Figure 5 As shown, the target action prediction model includes a target image encoder, a target first encoder, and a target decoder. The output of the target image encoder is connected to the input of the target first encoder, and the output of the target first encoder is connected to the input of the target decoder.

[0119] Understandably, the structure of other trained action prediction models may include an image encoder, a first encoder, and a decoder, with the output of the image encoder connected to the input of the first encoder, and the output of the first encoder connected to the input of the decoder.

[0120] In some embodiments, the target image encoder can be one of DinoV2 (self-distillation framework), MAE (mask autoencoder), VGG (building a deep network by stacking 3x3 convolutional layers), etc., without specific limitations.

[0121] The target first encoder can be one of the structures such as Transformer encoder, CNN encoder, LSTM encoder, etc., and the target decoder can be one of the structures such as Transformer decoder, CNN decoder, LSTM decoder, etc., without specific restrictions.

[0122] In some embodiments, a target image is input to a target image encoder, and actual joint angle data is input to a target first encoder. The data encoded by the target image encoder for the target image and the data encoded by the target first encoder are input to a target decoder, thereby the target decoder outputs predicted joint angle data.

[0123] In this embodiment, the target motion prediction model sets a target image encoder for encoding images and a target first encoder for encoding text data. Thus, by inputting the target image into the target image encoder and the actual joint angle data into the target first encoder, corresponding encoding is performed for different types of data, so that the target decoder outputs accurate predicted joint angle data.

[0124] Figure 6 This is a flowchart illustrating a robot control method in another exemplary embodiment, such as... Figure 6 As shown, this method proposes a way to switch tools corresponding to the target task, which is described in detail below:

[0125] S601. Determine the target tools for executing the target task based on the target task.

[0126] In some embodiments, different tasks may correspond to different tools. For example, in the catering industry, different dishes correspond to different serving tools. Shredded potatoes correspond to a gripper, while green beans correspond to a gripper. In the transportation industry, grippers are used for different tasks.

[0127] The tools used for different tasks can be the same or different.

[0128] S602, Replace the robot's tool with the target tool.

[0129] In some embodiments, a converter may be provided at the end of the robot arm. The converter has multiple connection interfaces, with one interface corresponding to one tool. After the target tool is determined, the converter is controlled to rotate to the connection interface corresponding to the target tool, and the target task can then be achieved through the target tool.

[0130] In some embodiments, the predicted joint angle data includes the degrees of freedom of the target tool.

[0131] In some embodiments, a tool switching model may also exist. After determining the target tool, the tool switching model outputs control parameters for switching to the target tool based on the target tool. These control parameters are used to control the robot to switch to the target tool.

[0132] The tool switching model can include a preset switching process. Different tools correspond to different switching processes. Each step in the switching process includes corresponding control parameters. Thus, through this tool, the control parameters of the switching process to switch to the target tool can be determined in the tool switching model, thereby realizing the switching of the target tool.

[0133] Figure 7 This is a flowchart illustrating a robot control method in another exemplary embodiment, such as... Figure 7As shown, this method proposes a training approach for the action prediction model, which is detailed below:

[0134] S701. Acquire training images of the robot, actual joint angle data of the robot during training, and target joint angle data of the robot, captured by the camera when the robot performs different tasks.

[0135] In some embodiments, data on the robot performing different tasks is acquired, and action prediction models corresponding to different tasks are trained using the data on the robot performing different tasks.

[0136] In some embodiments, when controlling the robot to perform different tasks through a control device, the actual joint angle data of the robot when performing different tasks is obtained, and the training images collected by the robot's camera for different tasks and the control pose information of the control device controlling the robot to perform different tasks are obtained; the control pose information of the robot performing different tasks is converted into target joint angle data based on inverse kinematics to determine the target joint angle data for different tasks.

[0137] In some embodiments, the control device may be a control handle, which outputs control pose information to control the robot to perform tasks.

[0138] The control device can control the six degrees of freedom of the robot arm and is equipped with buttons to control the opening and closing of the robot's end effector gripper, i.e., one degree of freedom of the gripper. In this way, the control posture information of the control device can be converted into target joint angle data of 14 degrees of freedom.

[0139] The control pose information is the control information output by the handle. This control pose information acts on the robot and can be converted into target joint angle data. This target joint angle data is the target joint angle data that the handle wants the robot to achieve.

[0140] In some embodiments, the robot can be controlled by manual operation of control devices to perform different tasks.

[0141] In some embodiments, an individual can wear a VR (virtual reality) headset to observe the robot's perspective and control the robot to perform different tasks based on the robot's perspective through a control device, thereby improving the reliability of the control pose information.

[0142] In one embodiment, a task is a food serving task. The task process includes the robot's right arm picking up a plate, the left arm picking up the corresponding food and placing it on the plate, and then the right arm placing the plate with the food on the dining table in front. The robot is controlled by the control device to realize the task process corresponding to this task, and the actual joint angle data, training images and target joint angle data of the robot in the task process are obtained.

[0143] In some embodiments, when the control device controls the robot to perform different tasks, it can acquire the actual joint angle data of the robot performing different tasks, as well as the training images collected by the robot when performing different tasks. The training images are images collected by the robot's camera when the control device controls the robot to perform different tasks. It can also perform inverse kinematics calculations based on the control pose information of the control device to obtain target joint angle data. In this way, the actual joint angle data, training images, and target joint angle data corresponding to different tasks can be obtained.

[0144] S702. Based on the training images corresponding to each task, the actual joint angle data of the corresponding task, and the target joint angle data of the corresponding task, different initial motion prediction models are trained to determine the motion prediction models corresponding to different tasks.

[0145] The initial motion prediction model was trained using actual joint angle data, training images, and target joint angle data corresponding to different tasks, and the motion prediction model corresponding to different tasks was determined.

[0146] In some embodiments, the training images and actual joint angle data corresponding to each task are input into different initial motion prediction models to obtain the first training predicted joint angle data output by each initial motion prediction model for different tasks; the corresponding initial motion prediction model is trained based on the first training predicted joint angle data output by each initial motion prediction model for different tasks and the target joint angle data of the corresponding task to determine the motion prediction model corresponding to different tasks.

[0147] The structure of the initial motion prediction model can be found in [reference]. Figure 5 The structure shown uses target joint angle data as labels. The parameters of each initial motion prediction model are adjusted based on the target joint angle data and the first training predicted joint angle data, so that the first training predicted joint angle data is closer to the target joint angle data, thereby improving the reliability of model training.

[0148] For a specific task, the training image and actual joint angle data corresponding to the task are input into the initial motion prediction model to obtain the first training predicted joint angle data output by the initial motion prediction model for the task. The initial motion prediction model is then trained using the first training predicted joint angle data and the target joint angle data for the task to obtain the motion prediction model corresponding to the task.

[0149] The target joint angle data can be the training image of the initial motion prediction model, the target joint angle data at the time corresponding to the actual joint angle data during training, or it can be the training image of the initial motion prediction model, the sequence of target joint angle data for a time period after the time corresponding to the actual joint angle data during training. The first training prediction joint angle data is also the sequence of predicted joint angle data for a time period after the training image of the initial motion prediction model, the time corresponding to the actual joint angle data during training. The motion prediction model obtained in this way predicts the sequence of predicted joint angle data for a time period after the target image.

[0150] In other embodiments, such as Figure 8 As shown, the initial motion prediction model includes an initial first encoder and an initial first motion prediction model, with the output of the initial first encoder connected to the input of the initial first motion prediction model.

[0151] The structure of the initial first action prediction model can be referenced. Figure 5 As shown in one embodiment, the initial first action prediction model includes an initial image encoder, an initial encoder, and an initial decoder. After training the initial action prediction model, the initial image encoder, the initial encoder, and the initial decoder correspond to the target image encoder, the target first encoder, and the target decoder, respectively.

[0152] At this point, the actual joint angle data of training for each task and the target joint angle data of the corresponding task are input into the initial first encoder in different initial motion prediction models, and the training images and actual joint angle data of training for each task are input into the initial first motion prediction models in different initial motion prediction models, so as to obtain the motion style variables output by each initial first encoder for different tasks and the second training prediction joint angle data output by each initial first motion prediction model for different tasks.

[0153] The training images corresponding to each task and the actual joint angle data of the corresponding task are input into the initial first motion prediction model in different initial motion prediction models. This can be done by inputting the training images corresponding to each task into the initial image encoder and the actual joint angle data of the training into the initial encoder. The parameters output by the initial image encoder for the training images are simultaneously input into the initial encoder. The data output by the initial encoder based on the parameters output by the initial image encoder for the training images and the actual joint angle data of the training are input into the initial decoder. The initial decoder outputs the second training prediction joint angle data.

[0154] Based on the motion style variables output by each initial first encoder for different tasks, the second training prediction joint angle data output by each initial first motion prediction model for different tasks, and the target joint angle data for different tasks, the initial motion prediction model for the corresponding task is trained to obtain the trained initial motion prediction model, which includes the trained initial first encoder and the trained initial first motion prediction model; the initial first motion prediction model trained for different tasks is determined as the motion prediction model for the corresponding task.

[0155] For a specific task, the actual joint angle data of the training for that task and the target joint angle data of the corresponding task are input into the initial first encoder in the initial motion prediction model to obtain the motion style variable output by the initial first encoder; the training image and actual joint angle data of the corresponding task are input into the initial first motion prediction model in the initial motion prediction model, and the initial first motion prediction model also receives the motion style variable to obtain the second training prediction joint angle data.

[0156] At this point, the target joint angle data is used as the label, and the standard normal distribution is used as the label to train the initial motion prediction model, thereby obtaining the initial motion prediction model after training for the task. The initial first motion prediction model in the initial motion prediction model after training for the task is determined as the motion prediction model for the corresponding task.

[0157] In some embodiments, a first loss function corresponding to each task is constructed based on the action style variables output by each initial first encoder for different tasks; a second loss function corresponding to each task is constructed based on the second training prediction joint angle data output by each initial first action prediction model for different tasks and the target joint angle data of the corresponding task; and an initial action prediction model for the corresponding task is trained based on the first loss function and the second loss function for each task to obtain the trained initial action prediction model.

[0158] The first loss function can be a regularization loss function, and the second loss function can include one or more of the reconstruction loss function and the balancing loss function.

[0159] In some embodiments, in the first loss function, the action style variable is used as the predicted value. The initial output of the first encoder is a 64-dimensional vector, i.e., the action style variable. The first 32 dimensions represent the mean of the normal distribution, and the last 32 dimensions represent the variance of the normal distribution. Then, a 32-dimensional vector is generated by calculating the normal distribution probability. This 32-dimensional vector is called the actual content of the action style variable. The first loss function is obtained by applying a loss function between the action style variable and the standard normal distribution. The second loss function is obtained by applying a loss function based on the training predicted joint angle data and the target joint angle data. By weighting the first loss function and the second loss function, the parameters in the initial action prediction model are adjusted together until the initial action prediction model converges, thus obtaining the trained initial action prediction model.

[0160] In this embodiment, an initial first encoder is introduced to calculate motion style variables on the actual joint angle data and target joint angle data. These motion style variables guide the training direction of the initial motion prediction model, resulting in a more accurate motion prediction model that can more accurately predict the joint angle data.

[0161] In this embodiment, when the robot is controlled by the control device to perform different tasks, images from a first-person perspective (i.e., training images) are collected, along with control pose information and actual joint angle data. The control pose information is used to calculate the target joint angle data based on inverse kinematics. The target joint angle data, actual joint angle data, and training images are used to train the initial motion prediction model through multimodal perception input, thereby improving the accuracy of the motion prediction model training and reducing the dependence on high-cost hardware. The predicted joint angle data obtained subsequently by the motion prediction model can ensure the stability and smoothness of the robot operation.

[0162] Experiments show that by implementing the above-mentioned robot control method to perform food serving tasks in a food serving scenario, a 90% success rate can be achieved. The robot can accurately complete a series of actions, including picking up plates, serving dishes, and placing plates. This provides a brand-new technical solution for the automation and intelligence of catering services. Of course, this food serving scenario is illustrative, and the robot control method can also enable the robot to perform tasks in other scenarios.

[0163] Figure 9 This is a block diagram illustrating an exemplary embodiment of a robot control device 900, which includes:

[0164] The task determination module 910 is used to acquire a task scene map based on the robot's preset camera, and determine the target task and the target action prediction model corresponding to the target task based on the task scene map. Different tasks correspond to different action prediction models.

[0165] The joint angle prediction module 930 is used to acquire target images of the robot during the execution of the target task based on the camera, determine the actual joint angle data of the robot, and input the target image and actual joint angle data into the target motion prediction model to obtain the predicted joint angle data.

[0166] The task execution module 950 is used to control the robot to complete the target task based on predicted joint angle data.

[0167] In one possible implementation, the task determination module includes:

[0168] The execution status determination unit is used to input the task scene graph into the pre-trained task detection model and obtain the execution status of different tasks output by the pre-trained task detection model.

[0169] The task determination unit is used to determine the task that successfully matches the preset task conditions of each task with the execution status of the corresponding task as the target task, and to determine the action prediction model corresponding to the target task as the target action prediction model.

[0170] In one possible implementation, the target motion prediction model includes a target image encoder, a first target encoder, and a target decoder. The output of the target image encoder is connected to the input of the first target encoder, and the output of the first target encoder is connected to the input of the target decoder. The joint angle prediction module includes:

[0171] The joint angle prediction unit is used to input the target image into the target image encoder and input the actual joint angle data into the target first encoder to obtain the predicted joint angle data output by the target decoder.

[0172] In one possible implementation, the robot control device further includes:

[0173] The target tool determination module is used to determine the target tool for performing the target task based on the target task.

[0174] The tool conversion module is used to change the robot's tool to the target tool.

[0175] In one possible implementation, the robot control device further includes:

[0176] The data acquisition module is used to acquire training images of the corresponding tasks, actual joint angle data of the robot during training, and target joint angle data of the robot, captured by the camera when the robot performs different tasks.

[0177] The training module is used to train different initial motion prediction models based on the training images corresponding to each task, the actual joint angle data of the corresponding task, and the target joint angle data of the corresponding task, so as to determine the motion prediction model corresponding to different tasks.

[0178] In one possible implementation, the data acquisition module includes:

[0179] The first data acquisition unit is used to acquire the actual joint angle data of the robot when it performs different tasks under the control of the control device, and to acquire the training images collected by the robot's camera for different tasks and the control pose information of the control device controlling the robot to perform different tasks.

[0180] The data conversion unit is used to convert the control pose information of the robot performing different tasks into target joint angle data based on inverse kinematics, so as to determine the target joint angle data for different tasks.

[0181] In one possible implementation, the training module includes:

[0182] The first data input unit is used to input the training images and actual joint angle data corresponding to each task into different initial motion prediction models to obtain the first training prediction joint angle data output by each initial motion prediction model for different tasks.

[0183] The first training unit is used to train the corresponding initial motion prediction model based on the first training prediction joint angle data output by each initial motion prediction model for different tasks and the target joint angle data of the corresponding task, so as to determine the motion prediction model corresponding to different tasks.

[0184] In one possible implementation, the initial action prediction model includes an initial first encoder and an initial first action prediction model, with the output of the initial first encoder connected to the input of the initial first action prediction model; the training module includes:

[0185] The second data input unit is used to input the actual joint angle data of training for each task and the target joint angle data of the corresponding task into the initial first encoder in different initial motion prediction models, and to input the training image and the actual joint angle data of training for each task into the initial first motion prediction model in different initial motion prediction models, so as to obtain the motion style variables output by each initial first encoder for different tasks and the second training prediction joint angle data output by each initial first motion prediction model for different tasks.

[0186] The second training unit is used to train the initial motion prediction model for the corresponding task based on the motion style variables output by each initial first encoder for different tasks, the second training prediction joint angle data output by each initial first motion prediction model for different tasks, and the target joint angle data for different tasks, so as to obtain the trained initial motion prediction model. The trained initial motion prediction model includes the trained initial first encoder and the trained initial first motion prediction model.

[0187] The model acquisition unit is used to determine the initial first action prediction model after training for different tasks as the action prediction model for the corresponding task.

[0188] In one possible implementation, the second training unit includes:

[0189] The first loss function calculation module is used to construct the first loss function for each task based on the action style variables output by each initial first encoder for different tasks.

[0190] The second loss function calculation module is used to construct the second loss function for each task based on the second training prediction joint angle data output by each initial first action prediction model for different tasks and the target joint angle data of the corresponding task.

[0191] The training module is used to train the initial action prediction model for each task based on the first loss function and the second loss function for each task, thus obtaining the trained initial action prediction model.

[0192] The robot control device provided in this embodiment can be used to execute the above-described robot control method. Its implementation principle and technical effect are similar, and will not be described again here.

[0193] Figure 10 A schematic diagram of the structure of the electronic device provided in this application. Figure 10 As shown, the electronic device 100 provided in this embodiment includes at least one processor 1001 and a memory 1002. Optionally, the device 100 further includes a communication component 1003. The processor 1001, memory 1002, and communication component 1003 are connected via a bus 1004.

[0194] In a specific implementation, at least one processor 1001 executes computer execution instructions stored in memory 1002, causing at least one processor 1001 to perform the above-described method.

[0195] The specific implementation process of processor 1001 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0196] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0197] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0198] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0199] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0200] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0201] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0202] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0203] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0204] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0205] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0206] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0207] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0208] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A robot control method, characterized in that, include: The robot acquires a task scene map based on a pre-set camera, and determines the target task and the target action prediction model corresponding to the target task based on the task scene map. Different tasks correspond to different action prediction models. The system acquires training images of the robot for different tasks captured by the camera, actual joint angle data of the robot during training, and target joint angle data of the robot. Based on the training images corresponding to each task, the actual joint angle data of the corresponding task, and the target joint angle data of the corresponding task, different initial motion prediction models are trained to determine the motion prediction models corresponding to different tasks. Based on the target images captured by the camera during the robot's execution of the target task, and the actual joint angle data of the robot determined, the target images and the actual joint angle data are input into the target motion prediction model to obtain predicted joint angle data; wherein, the predicted joint angle data includes the degrees of freedom of the target tool performing the target task; The robot is controlled to complete the target task based on the predicted joint angle data; The initial motion prediction model includes an initial first encoder and an initial first motion prediction model, wherein the output of the initial first encoder is connected to the input of the initial first motion prediction model. The process of training different initial motion prediction models based on training images corresponding to each task, actual joint angle data of the corresponding task, and target joint angle data of the corresponding task to determine the motion prediction model corresponding to different tasks includes: The actual joint angle data of each task and the target joint angle data of each task are input into the initial first encoder of different initial motion prediction models. The training images of each task and the actual joint angle data of each task are input into the initial first motion prediction models of different initial motion prediction models to obtain the motion style variables output by each initial first encoder for different tasks and the second training prediction joint angle data output by each initial first motion prediction model for different tasks. Based on the motion style variables output by each initial first encoder for different tasks, the second training prediction joint angle data output by each initial first motion prediction model for different tasks, and the target joint angle data of different tasks, the initial motion prediction model for the corresponding task is trained to obtain the trained initial motion prediction model, which includes the trained initial first encoder and the trained initial first motion prediction model. The initial first action prediction model trained for different tasks is determined as the action prediction model for the corresponding task.

2. The method according to claim 1, characterized in that, The method for determining the target task and the corresponding target action prediction model based on the task scene graph includes: The task scene graph is input into a pre-trained task detection model to obtain the execution status of different tasks output by the pre-trained task detection model. The task whose preset task conditions are successfully matched with the corresponding task execution status is identified as the target task, and the action prediction model corresponding to the target task is identified as the target action prediction model.

3. The method according to claim 1, characterized in that, The target action prediction model includes a target image encoder, a target first encoder, and a target decoder. The output of the target image encoder is connected to the input of the target first encoder, and the output of the target first encoder is connected to the input of the target decoder. The step of inputting the target image and the actual joint angle data into the target motion prediction model to obtain predicted joint angle data includes: The target image is input into the target image encoder, and the actual joint angle data is input into the target first encoder to obtain the predicted joint angle data output by the target decoder.

4. The method according to claim 1, characterized in that, Before acquiring target images of the robot performing the target task based on the camera, determining the robot's actual joint angle data, and inputting the target images and the actual joint angle data into the target motion prediction model to obtain predicted joint angle data, the method further includes: The target tool for performing the target task is determined based on the target task. Replace the robot's tool with the target tool.

5. The method according to claim 1, characterized in that, The acquisition of training images for the corresponding tasks captured by the camera, actual joint angle data of the robot during training, and target joint angle data of the robot when the robot performs different tasks includes: When the robot is controlled by the control device to perform different tasks, the actual joint angle data of the robot during the training of different tasks is obtained, and the training images collected by the robot's camera for different tasks and the control pose information of the control device controlling the robot to perform different tasks are also obtained. Based on inverse kinematics, the control pose information of the robot performing different tasks is converted into target joint angle data to determine the target joint angle data for different tasks.

6. The method according to claim 1, characterized in that, The process of training different initial motion prediction models based on training images corresponding to each task, actual joint angle data of the corresponding task, and target joint angle data of the corresponding task to determine the motion prediction model corresponding to different tasks includes: The training images and actual joint angle data corresponding to each task are input into different initial motion prediction models to obtain the first training prediction joint angle data output by each initial motion prediction model for different tasks. Based on the first training prediction joint angle data and the target joint angle data of the corresponding task, the initial motion prediction model is trained to determine the motion prediction model corresponding to different tasks.

7. The method according to claim 1, characterized in that, The initial motion prediction model for the corresponding task is trained based on the motion style variables output by each initial first encoder for different tasks, the second training prediction joint angle data output by each initial first motion prediction model for different tasks, and the target joint angle data for different tasks, resulting in the trained initial motion prediction model, including: Based on the action style variables output by each initial first encoder for different tasks, construct the first loss function corresponding to each task; Based on the second training prediction joint angle data output by each initial first motion prediction model for different tasks and the target joint angle data of the corresponding task, a second loss function is constructed for each task. The initial action prediction model for each task is trained based on the first loss function and the second loss function for each task, thus obtaining the trained initial action prediction model.

8. A robot control device, characterized in that, include: The task determination module is used to acquire a task scene map based on the robot's preset camera, and determine the target task and the target action prediction model corresponding to the target task based on the task scene map. Different tasks correspond to different action prediction models. The data acquisition module is used to acquire training images of the corresponding tasks captured by the camera, actual joint angle data of the robot during training, and target joint angle data of the robot when the robot performs different tasks. The training module is used to train different initial motion prediction models based on the training images corresponding to each task, the actual joint angle data of the corresponding task, and the target joint angle data of the corresponding task, so as to determine the motion prediction model corresponding to different tasks. The joint angle prediction module is used to acquire target images of the robot during the execution of the target task based on the camera, determine the actual joint angle data of the robot, and input the target images and the actual joint angle data into the target motion prediction model to obtain predicted joint angle data; wherein, the predicted joint angle data includes the degrees of freedom of the target tool performing the target task; The task execution module is used to control the robot to complete the target task based on the predicted joint angle data; The initial action prediction model includes an initial first encoder and an initial first action prediction model, with the output of the initial first encoder connected to the input of the initial first action prediction model; the training module includes: The second data input unit is used to input the actual joint angle data of training for each task and the target joint angle data of the corresponding task into the initial first encoder in different initial motion prediction models, and to input the training image and the actual joint angle data of training for each task into the initial first motion prediction model in different initial motion prediction models, so as to obtain the motion style variables output by each initial first encoder for different tasks and the second training prediction joint angle data output by each initial first motion prediction model for different tasks. The second training unit is used to train the initial motion prediction model for the corresponding task based on the motion style variables output by each initial first encoder for different tasks, the second training prediction joint angle data output by each initial first motion prediction model for different tasks, and the target joint angle data for different tasks, so as to obtain the trained initial motion prediction model. The trained initial motion prediction model includes the trained initial first encoder and the trained initial first motion prediction model. The model acquisition unit is used to determine the initial first action prediction model after training for different tasks as the action prediction model for the corresponding task.

Images