Methods for training motion expert models, robot control methods, and electronic devices

Abstract

This application provides a method for training motion expert models, a robot control method, and an electronic device, relating to the fields of artificial intelligence and robotics. The method for training motion expert models can train a first motion expert model (for predicting the robot's motion direction information when performing a task) and a second motion expert model (for predicting the robot's motion detail information when performing a task) based on different training data, making the training processes of the first and second motion expert models hierarchical. Furthermore, the second motion expert model can be trained using partial training data from the first motion expert model, allowing the first motion expert model to provide directional guidance for generating executable motion sequences for the second motion expert model. This improves the accuracy of the visual language motion model in generating executable motion sequences, thereby enhancing the accuracy of the robot in performing operational tasks.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and robotics, and in particular to a method for training motion expert models, a robot control method, and an electronic device. Background Technology

[0002] Vision-language-action models (VLA) are a core technological framework for embodied AI. By integrating visual perception, language understanding, and action generation capabilities, VLA enables intelligent agents (such as robots) to perform complex tasks in the physical world. VLA can generate executable action sequences based on visual input and verbal instructions, thereby driving robots to complete tasks such as grasping and moving.

[0003] Currently, the process by which visual language action models generate executable action sequences based on visual input and language instructions suffers from inaccuracies, thus affecting the accuracy of robots in performing tasks. Summary of the Invention

[0004] This application provides a method for training a motion expert model, a robot control method, and an electronic device, which can improve the accuracy of the visual language motion model in generating executable motion sequences based on visual input and language instructions, thereby improving the accuracy of the robot in performing operational tasks.

[0005] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0006] Firstly, embodiments of this application provide a method for training an action expert model, which can be applied to electronic devices. Specifically, the electronic device can acquire first action sequence samples, second action sequence samples, third action sequence samples, fourth action sequence samples, and multimodal features corresponding to a task to be processed. The multimodal features are determined based on image data associated with the task to be processed and natural language instructions. A first reference action direction feature is determined based on the first and fourth action sequence samples. The second action sequence samples and the multimodal features are input into a first network model, and the fourth action sequence samples, the first reference action direction feature, and the multimodal features are input into a second network model. The first predicted action sequence output by the first network model and the second predicted action sequence output by the second network model are then acquired. The first network model is used for pre-training... The system measures the motion direction information of the robot when performing the task to be processed, and the second network model is used to predict the action details of the robot when performing the task to be processed. The parameters of the first network model are adjusted according to the first action sequence sample, the first predicted action sequence, and the first loss function until the first predicted action sequence output by the first network model makes the first loss function converge. The parameters of the second network model are adjusted according to the third action sequence sample, the second predicted action sequence, and the second loss function until the second predicted action sequence output by the second network model makes the second loss function converge. The first network model when the first loss function converges is used as the first action expert model, and the second network model when the second loss function converges is used as the second action expert model.

[0007] In this way, electronic devices can train a first motion expert model (MAM) for predicting the robot's motion direction information when performing a task, and a second MAM for predicting the robot's motion detail information when performing a task, based on different training data. This makes the training processes of the first and second MAM models hierarchical. Furthermore, the second MAM model can be trained using some of the training data from the first MAM model. This allows the first MAM model to provide directional guidance for the second MAM model to generate executable motion sequences. As a result, the accuracy of the visual-language motion model in generating executable motion sequences based on visual input and language instructions can be improved, thereby enhancing the accuracy of the robot in performing operational tasks.

[0008] In conjunction with the first aspect, in one possible implementation, obtaining the first action sequence sample may include:

[0009] Obtain a sample action sequence, which includes N action segments. Each of the N action segments corresponds to an action associated with the task to be processed. The sample action sequence occupies T time steps, where N is an integer greater than 0 and T is greater than 0. Select n1 action segments from the N action segments as the first action sequence sample. The interval between two adjacent action segments in the n1 action segments is k time steps, where n1 is less than N and k is greater than 0.

[0010] Here, a sample action sequence can be understood as a pre-collected set of continuous actions arranged in chronological or logical order, and a first action sequence sample can be understood as a coarse-grained set of actions collected from this set of continuous actions. In some examples, the sample action sequence may include a pre-collected set of continuous actions of the robot completing a grasping task, and the first action sequence sample may include multiple actions of the robot completing the grasping task, with time step intervals between these multiple actions.

[0011] In other words, the sample action sequence is a real action sequence, and the first action sequence sample is a coarse-grained action truth sequence. Thus, by selecting the first action sequence sample from the real action sequence, the accuracy of data acquisition can be improved, which in turn helps to improve the reliability and accuracy of the first action expert model training.

[0012] In conjunction with the first aspect, in one possible implementation, obtaining the second action sequence sample may include:

[0013] Acquire first noise data, the time steps occupied by the first noise data are the same as the time steps occupied by the first action sequence sample; determine the second action sequence sample based on the first action sequence sample and the first noise data.

[0014] In this embodiment, the first noise data is coarse-grained noise data, and the time span (i.e., the time steps occupied) of the first noise data is the same as the time span of the first action sequence sample. It can be understood that the second action sequence sample is a coarse-grained action sequence carrying noise.

[0015] In this way, coarse-grained motion sequences carrying noise can be obtained as training data for the first motion expert model, enabling the first motion expert model to distinguish between coarse-grained motion sequences and noise data. This reduces or avoids the influence of noise on the first motion expert model when predicting the motion direction information of the robot when performing the task to be processed, thereby improving the robustness of the first motion expert model.

[0016] In conjunction with the first aspect, in one possible implementation, the first action sequence sample, the first predicted action sequence, and the first loss function can satisfy the following relationship:

[0017]

[0018] Among them, L coarse Denotes the first loss function. Represents the first action sequence sample, π coarse This represents the first network model. This represents the second action sequence sample. This represents the first predicted action sequence, and MSE represents the mean squared error.

[0019] In this way, the model parameters of the first action expert model can be adjusted based on the difference between the actual coarse-grained action sequence and the predicted coarse-grained action sequence, which helps to improve the accuracy of the training of the first action expert model.

[0020] In conjunction with the first aspect, in one possible implementation, obtaining the third action sequence sample as described above may include:

[0021] Select n2 action segments from N action segments as the third action sequence sample. The n2 action segments are consecutive action segments and n2 is less than N.

[0022] In this application embodiment, the third action sequence sample can be understood as a fine-grained action set collected from the above-mentioned continuous action set. In some examples, the sample action sequence may include a pre-collected continuous action set of the robot completing the grasping task. The third action sequence sample may include multiple actions of the robot completing the grasping task, and these multiple actions are continuous actions.

[0023] In other words, the sample action sequence is the real action sequence, and the third action sequence sample is the ground truth sequence of fine-grained actions. Thus, by selecting the third action sequence sample from the real action sequence, the accuracy of data acquisition can be improved, which in turn helps to improve the reliability and accuracy of the training of the first action expert model.

[0024] In conjunction with the first aspect, in one possible implementation, obtaining the fourth action sequence sample as described above may include:

[0025] Acquire the second noise data, the time step occupied by the second noise data is the same as the time step occupied by the third action sequence sample; determine the fourth action sequence sample based on the third action sequence sample and the second noise data.

[0026] In this embodiment, the second noise data is fine-grained noise data, and the time span (i.e., the time steps occupied) of the second noise data is the same as the time span of the third action sequence sample. It can be understood that the fourth action sequence sample is a fine-grained action sequence carrying noise.

[0027] In this way, noisy, fine-grained motion sequences can be obtained as training data for the second motion expert model, enabling the second motion expert model to distinguish between fine-grained motion sequences and noisy data. This reduces or avoids the influence of noise on the second motion expert model when predicting the motion details of the robot performing the task, thereby improving the robustness of the second motion expert model.

[0028] In conjunction with the first aspect, in one possible implementation, the third action sequence sample, the second predicted action sequence, and the second loss function can satisfy the following relationship:

[0029]

[0030] Among them, L fine This represents the second loss function. Represents the third action sequence sample, π fine This represents the second network model. This represents the fourth action sequence sample, and AlignFeature represents the weighted encoded feature. This represents the second predicted action sequence.

[0031] In this way, the model parameters of the second action expert model can be adjusted based on the difference between the actual fine-grained action sequence and the predicted fine-grained action sequence, which helps to improve the accuracy of model training.

[0032] In conjunction with the first aspect, in one possible implementation, the determination of the first reference action direction feature based on the first action sequence sample and the fourth action sequence sample may include:

[0033] The first action sequence sample is encoded to obtain the first action encoding feature; the first weight encoding feature is determined based on the first action encoding feature and the fourth action sequence sample; the first weight encoding feature is concatenated with the fourth action sequence sample to obtain the first reference action direction feature.

[0034] By encoding and weighting the first action sequence, the noise carried in the first predicted action sequence can be avoided from affecting the training process of the second action expert model, thereby improving the stability of the training of the second action expert model.

[0035] In conjunction with the first aspect, in one possible implementation, the second action sequence sample carries first noise data, the fourth action sequence sample carries second noise data, and the time step occupied by the first noise data is greater than the time step occupied by the second noise data.

[0036] In other words, the time steps occupied by coarse-grained noise data are greater than the time steps occupied by fine-grained noise data.

[0037] Secondly, embodiments of this application provide a robot control method applicable to electronic devices. The method includes: the electronic device acquiring third noise data, fourth noise data, and multimodal features corresponding to a task to be processed, the multimodal features being determined based on image data associated with the task and natural language instructions; inputting the third noise data and multimodal features into a first motion expert model, and acquiring a first predicted motion sequence output by the first motion expert model, the first motion expert model being used to predict the motion direction information of the robot when performing the task; determining a second reference motion direction feature based on the first predicted motion sequence and the fourth noise data; inputting the fourth noise data, multimodal features, and the second reference motion direction feature into a second motion expert model, and acquiring a second predicted motion sequence output by the second motion expert model, the second motion expert model being used to predict the motion detail information of the robot when performing the task; and controlling the robot to perform the task based on the second predicted motion sequence.

[0038] In this way, the electronic device can first generate motion direction information of the robot when performing the task through the first motion expert model, and then generate motion detail information that matches the motion direction information through the second motion expert model. Then, the electronic device can control the robot to perform the task based on the output of the second motion expert model. This can provide directional guidance and refined motion guidance to the robot, thereby improving the accuracy of the robot in performing the task.

[0039] In conjunction with the second aspect, in one possible implementation, the determination of the second reference action direction feature based on the first predicted action sequence and the fourth noise data may include:

[0040] The first predicted action sequence is encoded to obtain the second action encoding feature; the second weight encoding feature is determined based on the second action encoding feature and the fourth noise data; the second weight encoding feature and the fourth noise data are concatenated to obtain the second reference action direction feature.

[0041] By encoding and weighting the first predicted action sequence, the noise carried in the first predicted action sequence can be avoided from affecting the reasoning process of the second action expert model, thereby improving the accuracy of the second action expert model's reasoning.

[0042] In conjunction with the second aspect, in one possible implementation, the acquisition of multimodal features corresponding to the task to be processed may include:

[0043] Acquire image data and natural language instructions associated with the task to be processed; acquire image features corresponding to the task to be processed based on the image data, and acquire semantic features corresponding to the task to be processed based on the natural language instructions; acquire multimodal features corresponding to the task to be processed based on the image features and semantic features.

[0044] In conjunction with the second aspect, in one possible implementation, the time step occupied by the third noise data is greater than the time step occupied by the fourth noise data; that is, the time step occupied by the coarse-grained noise data is greater than the time step occupied by the fine-grained noise data.

[0045] Thirdly, embodiments of this application provide an electronic device including a memory and a processor; wherein the memory is used to store a set of computer instructions; when the processor executes the set of computer instructions, the electronic device performs the method described in the first or second aspect above.

[0046] Fourthly, embodiments of this application provide a computer-readable storage medium including computer instructions; when the computer instructions are executed on an electronic device, the electronic device causes the electronic device to perform the method described in the first or second aspect above.

[0047] Fifthly, embodiments of this application provide a computer program product, which includes a computer program that, when executed by a processor, implements the methods described in the first or second aspect above. Attached Figure Description

[0048] Figure 1 This is a scene diagram of an AI system provided in an embodiment of this application.

[0049] Figure 2 An example flowchart of a method for training an action expert model provided in an embodiment of this application.

[0050] Figure 3 This is a schematic diagram illustrating the principle of obtaining a first action sequence sample and a third action sequence sample, provided in an embodiment of this application.

[0051] Figure 4 This is an example flowchart of a robot control method provided in an embodiment of this application.

[0052] Figure 5 This is a structural example diagram of a device for training an action expert model provided in an embodiment of this application.

[0053] Figure 6 This is a structural example diagram of a robot control device provided in an embodiment of this application.

[0054] Figure 7This is a structural block diagram of an electronic device for executing a robot control method or a method for training a motion expert model, provided in an embodiment of this application. Detailed Implementation

[0055] The terms "first," "second," and "third," etc., used in this application specification, claims, and the aforementioned drawings are used to distinguish different objects, not to limit a specific order.

[0056] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0057] To ensure clarity and brevity in the description of the following embodiments, a brief introduction to the relevant technologies is given first.

[0058] Visual-language-action (VLAA) models are multimodal models that combine visual information, language understanding, and action generation. By learning the mapping between visual images and natural language commands, VLAA models can generate executable action sequences, thereby controlling robots to complete real-world tasks.

[0059] Currently, visual language action models have wide applications in humanoid robots, autonomous vehicles, medical and industrial robots, precision agriculture, and augmented reality navigation. For example, in precision agriculture, visual language action models can be used to control robots to complete tasks such as picking and retrieving based on visual observation and natural language commands; in autonomous driving scenarios, visual language action models can be used to implement human-like driving strategies.

[0060] However, current visual language action models typically rely on a visual language model (VLM) as a multimodal feature extractor to extract multimodal features. Based on these multimodal features, coarse-grained actions (such as moving an arm to a target area) and fine-grained actions (such as adjusting finger posture) are generated in the same action segment. This can easily lead to insufficient long-range consistency or inaccurate local precision, thus affecting the accuracy of the robot in performing its tasks.

[0061] To at least address the aforementioned problems, embodiments of this application provide a method for training motion expert models. This method trains a first motion expert model (MA) for predicting the motion direction information of a robot performing a task, and a second MA for predicting the motion detail information of a robot performing a task, based on different training data. This results in a hierarchical training process for the first and second MAs. Furthermore, by using partial training data from the first MA to train the second MA, the first MA can provide directional guidance for generating executable motion sequences. This improves the accuracy of the visual-language motion model in generating executable motion sequences based on visual input and verbal instructions, thereby enhancing the accuracy of the robot in performing operational tasks.

[0062] The following is a brief description of the application environment of the method for training action expert models provided in the embodiments of this application.

[0063] Please see Figure 1 , Figure 1 This is a schematic diagram of a scenario for an AI system provided in an embodiment of this application. Figure 1 As shown, the AI ​​system 100 may include an electronic device 110, which may be a robot, vehicle, drone, or other device used to perform target tasks. Taking a robot as an example, the electronic device 110 can be used to perform target tasks such as sweeping, cooking, walking the dog, watering flowers, serving food, or playing Go, or to perform target tasks such as welding, assembly, and handling.

[0064] In one example, an image processing model 111 and an action model 112 may be deployed on an electronic device 110.

[0065] Furthermore, in one example, an image acquisition device 113 may be provided on the electronic device 110 for acquiring images around the electronic device 110, such as color images (e.g., RGB images).

[0066] In one example, electronic device 110 can be used to execute the method for training an action expert model provided in the embodiments of this application. Specifically, during the execution of a target task by electronic device 110, electronic device 110 can acquire RGB images through image acquisition device 113 and obtain RGB image features based on the RGB images. Electronic device 110 can input the RGB image features into image processing model 111, which can understand and extract features from the RGB image features and output intermediate features, which are used to characterize the information in the RGB image features. Further, electronic device 110 can input the intermediate features, depth image features, and noisy action features into action model 112 to obtain the predicted action features output by action model 112. Here, the depth image features correspond to the RGB image features in time. The predicted action features can be used to characterize the motion trajectory of electronic device 110 in future time periods. For example, when electronic device 110 is a robot, the predicted action features can be used to characterize the motion trajectory of the robot's arm end, head end, and / or leg end.

[0067] In one example, electronic device 110 can be controlled to perform a target task based on predicted motion features. For example, when electronic device 110 is a robot and the predicted motion features are used to characterize the motion trajectory of the robot's end effector, the robot can control the movement of the robot's end effector based on the predicted motion features.

[0068] In other examples, at least one of the image processing model 111 and the action model 112 can be deployed on a computer device, which can be a server or a terminal device, such as a mobile phone, laptop, or desktop computer. For example, image processing model 111 can be deployed on a computer device, and action model 112 can be deployed on an electronic device 110. The computer device can input RGB image features into image processing model 111 to obtain intermediate features output by image processing model 111, and send the intermediate features to electronic device 110. Electronic device 110 can input the intermediate features, depth image features, and noisy action features into action model 112 to obtain predicted action features output by action model 112. As another example, action model 112 can be deployed on a computer device, and image processing model 111 can be deployed on electronic device 110. Electronic device 110 can input RGB image features into image processing model 111 to obtain intermediate features output by image processing model 111, and send the intermediate features to the computer device. The computer device can input intermediate features, depth image features, and noisy motion features into the motion model 112 to obtain the predicted motion features output by the motion model 112, and then send the predicted motion features to the electronic device 110. Alternatively, if both the image processing model 111 and the motion model 112 are deployed on the computer device, the computer device can input RGB image features into the image processing model 111 to obtain the intermediate features output by the image processing model 111, input the intermediate features, depth image features, and noisy motion features into the motion model 112 to obtain the predicted motion features output by the motion model 112, and then send the predicted motion features to the electronic device 110.

[0069] In one example, electronic device 110 is a vehicle, and the task the vehicle needs to perform is to travel to a target location. RGB image features can include features corresponding to the RGB image captured by the vehicle at its current location. Predicted motion features can characterize the vehicle's trajectory over a future time period, and the vehicle can be controlled to move from its current location to the target location based on these predicted motion features. Alternatively, electronic device 110 is a dual-arm robot, and the task the dual-arm robot needs to perform is to grasp a target object. RGB image features can include features corresponding to the RGB image captured by the dual-arm robot at its current location (or the current moment). Predicted motion features can characterize the trajectory of the robot's arm ends over a future time period, and the dual-arm robot can be controlled to move its arms to grasp the target object based on these predicted motion features.

[0070] For example, the image acquisition device 113 may include a monocular camera, a binocular camera, or other types of cameras.

[0071] Alternatively, the method for training the action expert model provided in the embodiments of this application can also be executed by the server.

[0072] It should be understood that the above application scenario examples are only shown to facilitate understanding of the spirit and principles of this application, and the embodiments of this application are not limited thereto. Rather, the embodiments of this application can be applied to any applicable scenario.

[0073] The method for training an expert model of motion provided in the embodiments of this application will be described below with reference to the accompanying drawings.

[0074] Please see Figure 2 , Figure 2 This is an example flowchart illustrating a method for training a motion expert model, provided as an embodiment of this application. This method for training a motion expert model can be applied to… Figure 1 The AI ​​system shown can be trained by a terminal device or a server. Here, the method of training the action expert model provided in this embodiment is described using a server as an example.

[0075] like Figure 2 As shown, the method for training an action expert model provided in this embodiment includes the following steps S201 to S205.

[0076] S201. The server obtains the first action sequence sample, the second action sequence sample, the third action sequence sample, the fourth action sequence sample, and the multimodal features corresponding to the task to be processed.

[0077] In this embodiment, the first action sequence sample, the second action sequence sample, the third action sequence sample, the fourth action sequence sample, and the multimodal features can be collectively referred to as training data. The first action sequence sample, the second action sequence sample, and the multimodal features can be used to train a first network model; the second action sequence sample, the third action sequence sample, and the multimodal features can be used to train a second network model. It should be noted that the first action sequence sample can also be used as training data for the second network model.

[0078] The first action sequence sample represents the truth sequence of coarse-grained actions, the second action sequence sample represents the truth sequence of coarse-grained actions carrying first noise data, the third action sequence sample represents the truth sequence of fine-grained actions, and the fourth action sequence sample represents the truth sequence of fine-grained actions carrying second noise data. The first noise data is different from the second noise data. The methods for obtaining the first to fourth action sequence samples will be described in detail below.

[0079] In one possible implementation, real action sequences can be pre-set or collected as sample action sequences, and then a first action sequence sample and a third action sequence sample can be obtained from the sample action sequences. The sample action sequence can be understood as a pre-collected set of continuous actions arranged in chronological or logical order, where each action represents a body state of the robot, such as the joint angle of a robot arm. The first action sequence sample can be understood as a coarse-grained set of actions collected from this continuous action set, where there is a time span between adjacent actions. The third action sequence sample can be understood as a fine-grained set of actions collected from this continuous action set, where adjacent actions can be consecutive.

[0080] As an example, the pre-collected sample action sequence can include N action segments. Each of the N action segments corresponds to an action associated with the task to be processed. The sample action sequence occupies T time steps, meaning the length of the entire trajectory of the sample action sequence is T. Each action segment in the sample action sequence occupies at least one time step, where N is a positive integer and T is greater than 0. Based on this, n1 action segments can be selected from the N action segments as the first action sequence sample, and n2 action segments can be selected from the N action segments as the third action sequence sample. Among these, the interval between any two adjacent action segments in the n1 action segments is k time steps, where n1 is greater than 0 and less than N, and k is greater than 0; the n2 action segments are consecutive action segments, where n2 is greater than 0 and less than N.

[0081] In one specific implementation, please refer to Figure 3 , Figure 3 This diagram illustrates the principle of obtaining a first action sequence sample and a third action sequence sample, as provided in an embodiment of this application. Figure 3 As shown, assume the pre-collected sample action sequences include action fragment 1, action fragment 2, action fragment 3, action fragment 4, ..., action fragment N. Action fragment 1 corresponds to action 1 associated with the task to be processed, action fragment 2 corresponds to action 2 associated with the task to be processed, action fragment 3 corresponds to action 3 associated with the task to be processed, action fragment 4 corresponds to action 4 associated with the task to be processed, ..., action fragment N corresponds to action n associated with the task to be processed. Action fragment 1 occupies a time step of 1, action fragment 2 occupies a time step of 1, action fragment 3 occupies a time step of 1, action fragment 4 occupies a time step of 2, and action fragment N occupies a time step of Tt. The time step occupied by the action associated with the task to be processed is the same as the time step occupied by its corresponding action fragment.

[0082] Optionally, action fragment 2, action fragment 4, ..., and action fragment N can be selected as the first action sequence sample from action fragment 1, action fragment 2, action fragment 3, action fragment 4, ..., and action fragment N, where N is an even number. Optionally, action fragment 1, action fragment 2, and action fragment 3 can be selected as the third action sequence sample from action fragment 1, action fragment 2, action fragment 3, action fragment 4, ..., and action fragment N.

[0083] It should be understood that n1 and n2 above are only examples and are not used to limit the way the first action sequence sample and the third action sequence sample are selected from the sample action sequence.

[0084] In some other implementations, the sample action sequence can also be set as a. 1:T At this point, we can start from a 1:T Selected action sequences As the first action sequence sample, where t represents a certain time, and the value of t ranges from (0, T), a t L represents the robot action at time t. c Indicates the length of the coarse-grained action segment; optional, L c The value range can be up to 60 future time steps. Similarly, the range from a... 1:T Selected action sequences As a third action sequence sample, L f L represents the length of a fine-grained action segment. f < <L c For example, L f The value range can be up to 10 future time steps.

[0085] In other words, the sample action sequences are real action sequences, the first action sequence sample is a coarse-grained ground truth sequence of actions, and the third action sequence sample is a fine-grained ground truth sequence of actions. In this way, by selecting the first action sequence sample and the third action sequence sample from real action sequences, the accuracy of data acquisition can be improved, which in turn helps to improve the reliability and accuracy of training the first action expert model and the second action expert model.

[0086] In one possible implementation, first noise data can be acquired, and then a second action sequence sample can be determined based on the first action sequence sample and the first noise data. The first noise data can be at least one of sensor noise or environmental interference. The time steps occupied by the first noise data are the same as the time steps occupied by the first action sequence sample; that is, the first noise data is coarse-grained noise data. Therefore, the second action sequence sample can be understood as a coarse-grained action sequence carrying noise.

[0087] In some examples, the first action sequence sample and the first noise data can be concatenated into a high-dimensional vector, which can then be used as the second action sequence sample. Alternatively, other methods can be used to determine the second action sequence sample based on the first action sequence sample and the first noise data. For instance, quality-aware multimodal fusion (QMF) can be used to fuse the first action sequence sample and the first noise data to obtain the second action sequence sample. The specific implementation process will not be detailed here.

[0088] Furthermore, it should be understood that the training process of the first network model can be iterated multiple times, and the initial noise data can be different in each iteration. For example, the initial noise data in later iterations can be stronger than the initial noise data in earlier iterations. In other words, as the iteration process progresses, the signal strength of the initial noise data can be gradually increased to gradually map the noise samples into the action sequence, which helps the model gradually learn to identify the noise mixed in the action sequence.

[0089] Similarly, second noise data can be acquired, and then a fourth action sequence sample can be determined based on the third action sequence sample and the second noise data. The second noise data can be at least one of sensor noise or environmental interference, or other types of noise; the specific type is not limited. The time steps occupied by the second noise data are the same as those occupied by the third action sequence sample. That is, the second noise data is fine-grained noise data, so it can be understood that the third action sequence sample is a fine-grained action sequence carrying noise.

[0090] The method for obtaining the fourth action sequence sample is similar to that for obtaining the second action sequence sample, and will not be described in detail here.

[0091] Furthermore, the training process of the second network model can be iterated multiple times, and the second noise data can be different in each iteration. For example, the second noise data in later iterations can be stronger than the second noise data in earlier iterations. In other words, as the iteration process progresses, the signal strength of the second noise data can be gradually increased to gradually map the noise samples into the action sequence, which helps the model gradually learn to identify the noise mixed in the action sequence.

[0092] By using noisy, coarse-grained motion sequences as training data for the first motion expert model, the model can distinguish between coarse-grained motion sequences and noisy data. This reduces or avoids the impact of noise on the first motion expert model's prediction of the robot's motion direction information when performing a task, thereby improving its robustness. Similarly, by using noisy, fine-grained motion sequences as training data for the second motion expert model, the model can distinguish between fine-grained motion sequences and noisy data. This reduces or avoids the impact of noise on the second motion expert model's prediction of the robot's motion details when performing a task, thereby improving its robustness.

[0093] Furthermore, the second action sequence sample carries first noise data, and the fourth action sequence sample carries second noise data. The time steps occupied by the first noise data are greater than the time steps occupied by the second noise data. In other words, the time steps occupied by the coarse-grained noise data are greater than the time steps occupied by the fine-grained noise data.

[0094] In this embodiment, the multimodal features are determined based on image data associated with the task to be processed and natural language instructions.

[0095] The "task to be processed" refers to the task to be executed by the robot under control. The specific task type is not limited; for example, the task to be processed could be a grasping task. The image data associated with the task to be processed can be understood as visual image data acquired during the process of controlling the robot to execute the task. For example, the image data associated with the task to be processed could include RGB images acquired through the head camera, left wrist camera, and right wrist camera of the humanoid robot. The natural language commands associated with the task to be processed can be understood as voice data acquired during the process of controlling the robot to execute the task. For example, the natural language commands associated with the task to be processed could include voice data when the user controls the robot to execute the task.

[0096] In one possible implementation, image data and natural language instructions associated with the task to be processed can be acquired first; then, image features corresponding to the task to be processed can be acquired based on the image data, and semantic features corresponding to the task to be processed can be acquired based on the natural language instructions; finally, multimodal features corresponding to the task to be processed can be acquired based on the image features and semantic features.

[0097] As an example, a visual language model can be used to extract visual local features (such as the position of objects and spatial relationships in the image) layer by layer from image data associated with the task to be processed through a hierarchical Transformer architecture, and semantic features of instructions (such as actions and target objects in the instructions) layer by layer from natural language instructions associated with the task to be processed. Then, multimodal features are generated during the computation of the attention mechanism. For the specific generation process, please refer to relevant technologies, which will not be elaborated here.

[0098] S202, The server determines the first reference action direction feature based on the first action sequence sample and the fourth action sequence sample.

[0099] In this embodiment, the first reference motion direction feature can be used as training data for the second network model. The first reference motion direction feature is used to provide directional guidance for the second network model to generate fine-grained motion detail information, so as to avoid the second network model generating motion detail information in the wrong direction, thereby improving the accuracy of predicting the motion detail information of the robot when performing the task to be processed by the second network model (i.e., the second motion expert model).

[0100] Optionally, the first action sequence sample may contain noise. To avoid the disturbance introduced by this noise affecting the stability of the second network model training, as one implementation method, the server can encode the first action sequence sample (e.g., linear encoding) to obtain the first action encoding feature. Further, the server can determine the first weight encoding feature based on the first action encoding feature and the fourth action sequence sample, and then concatenate the first weight encoding feature with the fourth action sequence sample to obtain the first reference action direction feature.

[0101] By encoding and weighting the first action sequence, the noise carried in the first predicted action sequence can be avoided from affecting the training process of the second action expert model, thereby improving the stability of the training of the second action expert model.

[0102] S203. The server inputs the second action sequence sample and multimodal features into the first network model, and inputs the fourth action sequence sample, the first reference action direction feature and multimodal features into the second network model, and obtains the first predicted action sequence output by the first network model, and obtains the second predicted action sequence output by the second network model.

[0103] The first network model predicts the robot's motion direction when performing the task, while the second network model predicts the robot's detailed motion information. Therefore, the first predicted motion sequence represents the motion direction predicted by the first network model, and the second predicted motion sequence represents the detailed motion information predicted by the second network model.

[0104] In one possible implementation, the network structures of the first network model and the second network model can be identical, and the specific network structures are not limited. For example, both the first and second network models can include multiple convolutional units, and the number of these units is the same, wherein the multiple convolutional units in the first network model correspond one-to-one with the multiple convolutional units in the second network model. The specific structure of the convolutional units in either the first or second network model is not limited. For example, a convolutional unit can consist of a single convolutional layer, a convolutional layer and a response function, or multiple Transformer blocks.

[0105] Furthermore, the first network model and the second network model can be action models based on reinforcement learning (RL), such as action generators based on diffusion models, action decoders based on Transformers, or hierarchical controller architectures, and the specific types are not limited.

[0106] S204. The server adjusts the parameters of the first network model based on the first action sequence sample, the first predicted action sequence, and the first loss function until the first predicted action sequence output by the first network model causes the first loss function to converge. The server also adjusts the parameters of the second network model based on the third action sequence sample, the second predicted action sequence, and the second loss function until the second predicted action sequence output by the second network model causes the second loss function to converge.

[0107] The parameters of the first network model may include weights and / or activation values, and the parameters of the second network model may include weights and / or activation values, etc., without specific limitations. In some examples, adjusting the parameters of the first network model may refer to updating the weights of the convolutional units of the first network model; similarly, adjusting the parameters of the second network model may refer to updating the weights of the convolutional units of the second network model.

[0108] In one possible implementation, the first action sequence sample, the first predicted action sequence, and the first loss function can satisfy the following relationship:

[0109]

[0110] Among them, L coarse Denotes the first loss function. Represents the first action sequence sample, π coarse This represents the first network model. This represents the second action sequence sample. This represents the first predicted action sequence, and MSE represents the mean squared error.

[0111] In this way, the model parameters of the first action expert model can be adjusted based on the difference between the actual coarse-grained action sequence and the predicted coarse-grained action sequence, which helps to improve the accuracy of model training.

[0112] In addition, it should be noted that during the process of adjusting the parameters of the first network model, the output of the first network model (i.e., the first predicted action sequence) will be supervised using the first action sequence sample to help the first network model learn to generate action sequences that are the same as or similar to the first action sequence sample.

[0113] In one possible implementation, the third action sequence sample, the second predicted action sequence, and the second loss function can satisfy the following relationship:

[0114]

[0115] Among them, L fine This represents the second loss function. Represents the third action sequence sample, π fine This represents the second network model. This represents the fourth action sequence sample, and AlignFeature represents the weighted encoded feature. This represents the second predicted action sequence.

[0116] In this way, the model parameters of the second action expert model can be adjusted based on the difference between the actual fine-grained action sequence and the predicted fine-grained action sequence, which helps to improve the accuracy of the training of the second action expert model.

[0117] S205. The server uses the first network model when the first loss function converges as the first action expert model, and uses the second network model when the second loss function converges as the second action expert model.

[0118] Specifically, when the first predicted action sequence output by the first network model is close to the first action sequence sample, the first loss function converges, and the first network model can be used as the first action expert model; when the second predicted action sequence output by the second network model is close to the second action sequence sample, the second loss function converges, and the second network model can be used as the second action expert model.

[0119] It should be understood that the first motion expert model trained has the ability to predict the motion direction information when the robot performs a task, and the second motion expert model trained has the ability to predict the motion detail information when the robot performs a task. This application's implementation trains both the first and second motion expert models simultaneously, and provides motion direction guidance for the training process of the second motion expert model using a coarse-grained motion truth sequence. On the one hand, this improves both the accuracy of motion direction prediction and the accuracy of motion detail prediction; on the other hand, it allows for the adjustment of fine-grained motion detail information using the coarse-grained motion truth sequence, so that the fine-grained motion detail information carries both motion direction information and motion detail information determined based on that motion direction information. This results in more accurate fine-grained motion detail information, thereby improving the precision of robot control.

[0120] Furthermore, compared to existing technologies that directly generate coarse-grained and fine-grained action sequences based on multimodal information that integrates visual image information and natural language commands, the embodiments of this application can gradually generate action sequences for controlling the robot to perform tasks in a coarse-grained to fine-grained manner, making the entire control process hierarchical, thereby helping to improve the efficiency and robustness of robot control.

[0121] In summary, the method for training motion expert models provided in this application trains a first motion expert model for predicting the motion direction information of the robot when performing a task, and a second motion expert model for predicting the motion detail information of the robot when performing a task, based on different training data. This makes the training processes of the first and second motion expert models hierarchical. Furthermore, by using part of the training data of the first motion expert model to train the second motion expert model, the first motion expert model can provide directional guidance for the second motion expert model to generate executable motion sequences. In this way, the accuracy of the visual-language motion model in generating executable motion sequences based on visual input and language instructions can be improved, thereby improving the accuracy of the robot in performing operational tasks.

[0122] Please see Figure 4 , Figure 4This is an example flowchart of a robot control method provided in an embodiment of this application. The robot control method can be executed by an electronic device, which can be a terminal and / or a server. Here, we will illustrate the method by example, with a server executing the robot control method provided in this embodiment. Figure 4 As shown, the robot control method may include the following steps S301 to S305.

[0123] S301, The server acquires the third noise data, the fourth noise data, and the multimodal features corresponding to the task to be processed.

[0124] The multimodal features are determined based on image data associated with the task to be processed and natural language instructions. The specific determination process can be found in the description of S201 in the aforementioned embodiments, and will not be repeated here.

[0125] In this embodiment, the type and acquisition method of the third noise data can be the same as those of the aforementioned first noise data, and will not be described in detail. Optionally, the content of the third noise data can be different from that of the first noise data. Similarly, the type and acquisition method of the fourth noise data can be the same as those of the second noise data, but the content of the fourth noise data can be different from that of the second noise data.

[0126] The time step occupied by the third noise data is longer than the time step occupied by the fourth noise data.

[0127] S302. The server inputs the third noise data and multimodal features into the first action expert model and obtains the first predicted action sequence output by the first action expert model.

[0128] Among them, the first motion expert model is based on the aforementioned Figure 2 The process of iteratively training the first network model to obtain the first action expert sub-model, as shown in the method described above, can be referred to the relevant descriptions above and will not be repeated here.

[0129] The first motion expert model is used to predict the robot's motion direction information when performing the task. Specifically, the first motion expert model can estimate the robot's motion path when performing the task based on multimodal features and third-party noise data, and then generate a first predicted motion sequence.

[0130] S303, The server determines the second reference action direction features based on the first predicted action sequence and the fourth noise data.

[0131] In this embodiment, the second reference motion direction feature can be used to provide directional guidance for the second motion expert model to predict the motion details of the robot when performing the task to be processed, so as to improve the accuracy of motion details determination. The determination process of the second reference motion direction feature is similar to the determination process of the first reference motion direction feature in the previous embodiment.

[0132] Specifically, the server can encode the first predicted action sequence to obtain the second action encoding feature; further, the server can determine the second weight encoding feature based on the second action encoding feature and the fourth noise data, and then concatenate the second weight encoding feature with the fourth noise data to obtain the second reference action direction feature.

[0133] S304. The server inputs the fourth noise data, multimodal features, and second reference action direction features into the second action expert model, and obtains the second predicted action sequence output by the second action expert model.

[0134] Among them, the second action expert model is based on the aforementioned Figure 2 The process of iteratively training the second network model to obtain the second action expert sub-model, as shown in the method described above, can be referred to the relevant descriptions above and will not be repeated here.

[0135] The second motion expert model is used to predict the detailed motion information of the robot when performing the task. Specifically, the second motion expert model can estimate the details of the robot's motion (such as the details of local motions) when performing the task based on multimodal features, fourth noise data, and second reference motion direction features, and then generate a second predicted motion sequence.

[0136] Furthermore, it should be understood that since the second motion expert model also refers to the directional guidance information from the second reference motion direction feature when predicting the motion details information of the robot when performing the task to be processed, the motion details information is motion details information carrying motion direction information. Therefore, when controlling the robot to perform the task to be processed, the robot can be controlled to perform the task to be processed only according to the second predicted motion sequence, corresponding to S305 below.

[0137] S305, The server controls the robot to execute the task to be processed based on the second predicted action sequence.

[0138] In summary, the robot control method provided in this application first generates motion direction information of the robot when performing a task through a first motion expert model, then generates motion detail information that matches the motion direction information through a second motion expert model, and then controls the robot to perform the task based on the output of the second motion expert model. This can provide directional guidance and refined motion guidance to the robot, thereby improving the accuracy of the robot in performing the task.

[0139] It is understood that, in order to implement the functions in the above embodiments, the server includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, based on the units and method steps described in conjunction with the embodiments disclosed in this application, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application scenario and design constraints of the technical solution.

[0140] Please see Figure 5 , Figure 5 This is a structural example diagram of a device for training an action expert model, provided as an embodiment of this application. Figure 5 As shown, the apparatus 400 for training an action expert model includes: a training data acquisition module 410, an action direction feature acquisition module 420, an input module 430, a model parameter adjustment module 440, and an acquisition module 450.

[0141] The training data acquisition module 410 is used to acquire first action sequence samples, second action sequence samples, third action sequence samples, fourth action sequence samples, and multimodal features corresponding to the task to be processed. The multimodal features are determined based on image data associated with the task to be processed and natural language instructions.

[0142] In one possible implementation, the training data acquisition module 410 can be used to acquire sample action sequences, which include N action segments. Each of the N action segments corresponds to an action associated with the task to be processed. The sample action sequence occupies T time steps, where N is an integer greater than 0 and T is greater than 0. From the N action segments, n1 action segments are selected as the first action sequence samples. The interval between two adjacent action segments in the n1 action segments is k time steps, where n1 is less than N and k is greater than 0.

[0143] In one possible implementation, the training data acquisition module 410 can be used to acquire first noise data, the time steps occupied by the first noise data being the same as the time steps occupied by the first action sequence sample; and to determine the second action sequence sample based on the first action sequence sample and the first noise data.

[0144] In one possible implementation, the first action sequence sample, the first predicted action sequence, and the first loss function satisfy:

[0145]

[0146] Among them, L coarse Denotes the first loss function. Represents the first action sequence sample, π coarse This represents the first network model. This represents the second action sequence sample. This represents the first predicted action sequence, and MSE represents the mean squared error.

[0147] In one possible implementation, the training data acquisition module 410 can be used to select n2 action segments from N action segments as third action sequence samples, where the n2 action segments are consecutive action segments and n2 is less than N.

[0148] In one possible implementation, the training data acquisition module 410 can be used to acquire second noise data, the time steps occupied by the second noise data being the same as the time steps occupied by the third action sequence sample; and to determine the fourth action sequence sample based on the third action sequence sample and the second noise data.

[0149] In one possible implementation, the third action sequence sample, the second predicted action sequence, and the second loss function satisfy:

[0150]

[0151] Among them, L fine This represents the second loss function. Represents the third action sequence sample, π fine This represents the second network model. This represents the fourth action sequence sample, and AlignFeature represents the weighted encoded feature. This represents the second predicted action sequence.

[0152] The action direction feature acquisition module 420 is used to determine the first reference action direction feature based on the first action sequence sample and the fourth action sequence sample.

[0153] In one possible implementation, the action direction feature acquisition module 420 can be used to encode the first action sequence sample to obtain the first action encoding feature; determine the first weight encoding feature based on the first action encoding feature and the fourth action sequence sample; and concatenate the first weight encoding feature with the fourth action sequence sample to obtain the first reference action direction feature.

[0154] In one possible implementation, the second action sequence sample carries first noise data, and the fourth action sequence sample carries second noise data, wherein the time step occupied by the first noise data is greater than the time step occupied by the second noise data.

[0155] The input module 430 is used to input the second action sequence sample and multimodal features into the first network model, and to input the fourth action sequence sample, the first reference action direction feature and multimodal features into the second network model, and to obtain the first predicted action sequence output by the first network model and the second predicted action sequence output by the second network model. The first network model is used to predict the motion direction information of the robot when performing the task to be processed, and the second network model is used to predict the action detail information of the robot when performing the task to be processed.

[0156] The model parameter adjustment module 440 is used to adjust the parameters of the first network model according to the first action sequence sample, the first predicted action sequence and the first loss function until the first predicted action sequence output by the first network model makes the first loss function converge; and to adjust the parameters of the second network model according to the third action sequence sample, the second predicted action sequence and the second loss function until the second predicted action sequence output by the second network model makes the second loss function converge.

[0157] The acquisition module 450 is used to take the first network model when the first loss function converges as the first action expert model, and the second network model when the second loss function converges as the second action expert model.

[0158] It should be understood that the specific implementations of the training data acquisition module 410, the action direction feature acquisition module 420, the input module 430, the model parameter adjustment module 440, and the acquisition module 450 in the above embodiments can refer to the above. Figure 2 or Figure 3 To avoid repetition, the descriptions of the methods shown will not be repeated.

[0159] The apparatus for training motion expert models provided in this application trains a first motion expert model for predicting the motion direction information of a robot performing a task, and a second motion expert model for predicting the motion detail information of a robot performing a task, based on different training data. This makes the training processes of the first and second motion expert models hierarchical. Furthermore, by using part of the training data of the first motion expert model to train the second motion expert model, the first motion expert model can provide directional guidance for the second motion expert model to generate executable motion sequences. In this way, the accuracy of the visual-language motion model in generating executable motion sequences based on visual input and language instructions can be improved, thereby improving the accuracy of the robot in performing operational tasks.

[0160] Please see Figure 6 , Figure 6 This is a structural example diagram of a robot control device provided in an embodiment of this application. Figure 6 As shown, the robot control device 500 includes: a data acquisition module 510, a first acquisition module 520, a determination module 530, a second acquisition module 540, and a control module 550.

[0161] The data acquisition module 510 is used to acquire third noise data, fourth noise data, and multimodal features corresponding to the task to be processed. The multimodal features are determined based on the image data associated with the task to be processed and natural language instructions.

[0162] In one possible implementation, the data acquisition module 510 can be used to acquire image data and natural language instructions associated with the task to be processed; acquire image features corresponding to the task to be processed based on the image data, and acquire semantic features corresponding to the task to be processed based on the natural language instructions; and acquire multimodal features corresponding to the task to be processed based on the image features and semantic features.

[0163] In one possible implementation, the time step occupied by the third noise data is longer than the time step occupied by the fourth noise data.

[0164] The first acquisition module 520 is used to input the third noise data and multimodal features into the first motion expert model and acquire the first predicted motion sequence output by the first motion expert model. The first motion expert model is used to predict the motion direction information of the robot when performing the task to be processed.

[0165] The determination module 530 is used to determine the second reference action direction features based on the first predicted action sequence and the fourth noise data.

[0166] In one possible implementation, the determining module 530 can be used to encode the first predicted action sequence to obtain a second action encoding feature; determine a second weight encoding feature based on the second action encoding feature and the fourth noise data; and concatenate the second weight encoding feature with the fourth noise data to obtain a second reference action direction feature.

[0167] The second acquisition module 540 is used to input the fourth noise data, multimodal features and second reference action direction features into the second action expert model, and acquire the second predicted action sequence output by the second action expert model. The second action expert model is used to predict the action details of the robot when performing the task to be processed.

[0168] The control module 550 is used to control the robot to perform the task to be processed based on the second predicted action sequence.

[0169] It should be understood that the specific implementations of the data acquisition module 510, the first acquisition module 520, the determination module 530, the second acquisition module 540, and the control module 550 in the above embodiments can refer to the above description. Figure 4 To avoid repetition, the descriptions of the methods shown will not be repeated.

[0170] Please see Figure 7 , Figure 7 This is a structural block diagram of an electronic device for executing a robot control method or training a motion expert model, provided as an embodiment of this application. Figure 7 As shown, the electronic device 600 may specifically be a server, a task execution device, a control device for the task execution device, a server that interacts with the task execution device, or other devices.

[0171] Reference Figure 7 The electronic device 600 includes a processing component 610, which further includes one or more processors, and memory resources represented by memory 620 for storing instructions executable by the processing component 610, such as application programs. The application programs stored in memory 620 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 610 is configured to execute instructions to perform the aforementioned robot control methods or methods for training motion expert models.

[0172] Electronic device 600 may also include a power supply component configured to perform power management of electronic device 600, a wired or wireless network interface configured to connect electronic device 600 to a network, and an input / output (I / O) interface. Electronic device 600 may operate based on an operating system stored in memory 620, such as Windows Server™, MacOSX™, Unix™, Linux™, FreeBSD™, or similar.

[0173] Furthermore, this application also provides a non-transitory computer-readable storage medium, which, when the instructions in the storage medium are executed by the processor of the aforementioned electronic device 600, enables the aforementioned electronic device 600 to execute a robot control method or a method for training a motion expert model.

[0174] In addition, this application also provides a computer program product, which includes a computer program. When the computer program is executed by the processor of a computer device, it enables the computer device to execute the robot control method or the method for training motion expert models provided in any of the above embodiments.

[0175] All of the above-mentioned optional technical solutions can be combined in any way to form optional embodiments of this application, and will not be described in detail here.

[0176] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0177] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0178] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0179] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0180] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0181] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program checksums, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0182] It should be noted that in the description of this application, the terms "first," "second," "third," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0183] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0184] In the various embodiments of this application, unless otherwise specified or logically conflicting, the terminology and / or descriptions between different embodiments are consistent and can be referenced mutually. Technical features in different embodiments can be combined to form new embodiments based on their inherent logical relationships. The various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application. The order of the process numbers does not imply the order of execution; the execution order of each process should be determined by its function and inherent logic.

[0185] The above description is merely a specific implementation of this embodiment, but the protection scope of this embodiment is not limited thereto. Any changes or substitutions within the technical scope disclosed in this embodiment should be covered within the protection scope of this embodiment. Therefore, the protection scope of this embodiment should be determined by the protection scope of the claims.

Claims

1. A method of training a motion specialist model, the method comprising: The method includes: A first action sequence sample, a second action sequence sample, a third action sequence sample, a fourth action sequence sample, and multimodal features corresponding to the task to be processed are obtained. The first action sequence sample is a ground truth sequence of coarse-grained actions, the second action sequence sample is a coarse-grained action sequence carrying noise, the third action sequence sample is a ground truth sequence of fine-grained actions, and the fourth action sequence sample is a fine-grained action sequence carrying noise. The multimodal features are determined based on image data associated with the task to be processed and natural language instructions. The first reference action direction feature is determined based on the first action sequence sample and the fourth action sequence sample. The second action sequence sample and the multimodal features are input into the first network model, and the fourth action sequence sample, the first reference action direction feature and the multimodal features are input into the second network model. The first predicted action sequence output by the first network model and the second predicted action sequence output by the second network model are obtained. The first network model is used to predict the motion direction information of the robot when performing the task to be processed, and the second network model is used to predict the action detail information of the robot when performing the task to be processed. The parameters of the first network model are adjusted according to the first action sequence sample, the first predicted action sequence, and the first loss function until the first predicted action sequence output by the first network model causes the first loss function to converge. The parameters of the second network model are adjusted according to the third action sequence sample, the second predicted action sequence, and the second loss function until the second predicted action sequence output by the second network model causes the second loss function to converge. The first network model when the first loss function converges is used as the first action expert model, and the second network model when the second loss function converges is used as the second action expert model.

2. The method of claim 1, wherein, The process of obtaining the first action sequence sample includes: Obtain a sample action sequence, which includes N action segments. Each of the N action segments corresponds to an action associated with the task to be processed. The sample action sequence occupies T time steps, where N is an integer greater than 0 and T is greater than 0. From the N action segments, n1 action segments are selected as the first action sequence sample. The interval between two adjacent action segments in the n1 action segments is k time steps, where n1 is less than N and k is greater than 0.

3. The method of claim 2, wherein, Obtain a sample of the second action sequence, including: Acquire first noise data, the time steps occupied by the first noise data being the same as the time steps occupied by the first action sequence sample; The second action sequence sample is determined based on the first action sequence sample and the first noise data.

4. The method according to claim 3, characterized in that, The first action sequence sample, the first predicted action sequence, and the first loss function satisfy: ； Among them, the Denotes the first loss function, the Represents the first action sequence sample, the Represents the first network model, the This represents the second action sequence sample, the Represents the first predicted action sequence, the This represents the mean square error.

5. The method according to any one of claims 2-4, characterized in that, Obtain third action sequence samples, including: From the N action segments, n2 action segments are selected as the third action sequence sample, where the n2 action segments are consecutive action segments and n2 is less than N.

6. The method according to claim 5, characterized in that, Obtain the fourth action sequence sample, including: Acquire second noise data, the time steps occupied by the second noise data being the same as the time steps occupied by the third action sequence sample; The fourth action sequence sample is determined based on the third action sequence sample and the second noise data.

7. The method according to claim 6, characterized in that, The third action sequence sample, the second predicted action sequence, and the second loss function satisfy: ； Among them, the Denotes the second loss function, the This represents the third action sequence sample, the This represents the second network model, the This represents the fourth action sequence sample, the The weighted encoding features are represented by the following. This represents the second predicted action sequence.

8. The method according to claim 1, characterized in that, The step of determining the first reference action direction feature based on the first action sequence sample and the fourth action sequence sample includes: The first action sequence sample is encoded to obtain the first action encoding feature; The first weighted encoding feature is determined based on the first action encoding feature and the fourth action sequence sample; The first weighted encoding feature is concatenated with the fourth action sequence sample to obtain the first reference action direction feature.

9. The method according to claim 1, characterized in that, The second action sequence sample carries first noise data, and the fourth action sequence sample carries second noise data, wherein the time step occupied by the first noise data is greater than the time step occupied by the second noise data.

10. A robot control method, characterized in that, The method includes: Acquire third noise data, fourth noise data, and multimodal features corresponding to the task to be processed. The multimodal features are determined based on image data associated with the task to be processed and natural language instructions. The third noise data and the multimodal features are input into the first motion expert model as described in any one of claims 1-9, and the first predicted motion sequence output by the first motion expert model is obtained. The first motion expert model is used to predict the motion direction information of the robot when it performs the task to be processed. The second reference action direction feature is determined based on the first predicted action sequence and the fourth noise data; The fourth noise data, the multimodal features, and the second reference motion direction features are input into the second motion expert model as described in any one of claims 1-9, and the second predicted motion sequence output by the second motion expert model is obtained. The second motion expert model is used to predict the motion details of the robot when it performs the task to be processed. The robot is controlled to perform the task to be processed based on the second predicted action sequence.

11. The method according to claim 10, characterized in that, The step of determining the second reference action direction feature based on the first predicted action sequence and the fourth noise data includes: The first predicted action sequence is encoded to obtain the second action encoding feature; The second weighted coding feature is determined based on the second action coding feature and the fourth noise data; The second weighted encoding feature is concatenated with the fourth noise data to obtain the second reference action direction feature.

12. The method according to claim 10 or 11, characterized in that, Obtain multimodal features corresponding to the task to be processed, including: Acquire image data and natural language instructions associated with the task to be processed; Based on the image data, image features corresponding to the task to be processed are obtained, and based on the natural language instructions, semantic features corresponding to the task to be processed are obtained; Based on the image features and the semantic features, obtain the multimodal features corresponding to the task to be processed.

13. The method according to claim 12, characterized in that, The time step occupied by the third noise data is greater than the time step occupied by the fourth noise data.

14. An electronic device, characterized in that, It includes a memory and a processor, the memory being used to store a set of computer instructions; when the processor executes the set of computer instructions, it causes the electronic device to perform the method as described in any one of claims 1-9 or 10-13.

15. A computer-readable storage medium, characterized in that, Includes computer instructions; when the computer instructions are executed on a computing device, they cause the computing device to perform the method as described in any one of claims 1-9 or 10-13.

16. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 13.

Images