Robot action generation method, model training method, and related device

Abstract

Embodiments of the present application provide a robot action generation method, a model training method and related equipment. The robot action generation method comprises: receiving task input information corresponding to a robot action generation task; generating condition information based on the task input information; generating motion latent feature information using an action diffusion model based on the condition information; generating a continuous action token sequence using an action autoregressive model based on the motion latent feature information; and decoding the continuous action tokens in the continuous action token sequence to obtain robot action information corresponding to a continuous action segment. This method uses continuous action tokens representing multiple frames of robot pose data as the generation and decoding unit, and generates subsequent continuous action tokens in combination with motion latent feature information and already generated continuous action tokens, which can reduce the number of reasoning rounds for action generation and enable already generated continuous action tokens to participate in the generation of the remaining continuous action tokens.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to robot motion generation methods, model training methods, and related equipment. Background Technology

[0002] With the development of robotics and artificial intelligence technologies, robots can generate actions based on visual, linguistic, and state information to achieve autonomous movement and human-robot interaction. However, existing robot action generation methods are inefficient when generating a continuous sequence of actions, making it difficult to meet the real-time action generation requirements of robots.

[0003] Based on this, embodiments of this application provide a robot motion generation method, a model training method, and related equipment to improve related technologies. Summary of the Invention

[0004] The purpose of this application is to provide a robot motion generation method, a model training method, and related equipment to improve the efficiency of robot motion generation.

[0005] In a first aspect, embodiments of this application provide a robot motion generation method, comprising: receiving task input information corresponding to a robot motion generation task; generating condition information based on the task input information; generating motion latent feature information using a motion diffusion model based on the condition information, wherein the motion latent feature information is used to characterize motion semantics related to the robot motion generation task; generating a continuous action lexical sequence using an action autoregressive model based on the motion latent feature information, wherein, except for the first continuous action lexical, the remaining continuous action lexicals to be generated are all generated based on the motion latent feature information and at least some of the already generated continuous action lexicals, and each continuous action lexical corresponds to a continuous action segment composed of multiple temporally continuous robot posture data; and decoding the continuous action lexicals in the continuous action lexical sequence to obtain robot motion information corresponding to the continuous action segment.

[0006] Secondly, embodiments of this application provide a method for training a robot action generation model. The robot action generation model includes an action diffusion model and an action autoregressive model. The method includes: receiving a first training sample from a first training dataset, the first training sample including first training input information and first training supervision information; generating first training condition information based on the first training input information; inputting the first training condition information, unconditional training information, and training diffusion time step information into the action diffusion model; performing inference using the action diffusion model to obtain first training motion latent feature information; generating first training prediction information based on the first training motion latent feature information; determining a first training loss based on the first training prediction information and the first training supervision information; and training the action diffusion model based on the first training loss; and receiving a second training sample from a second training dataset, the second training sample including second training input information and a training continuous action lexical sequence corresponding to the second training input information. Each of the training continuous action lexes corresponds to a continuous action segment composed of multiple temporally continuous robot posture data. Second training condition information is generated based on the second training input information. The second training condition information, unconditional training information, and training diffusion time step information are input into the action diffusion model. The action diffusion model is used for inference to obtain second training motion latent feature information. Based on the second training motion latent feature information, the action autoregressive model is used to generate a predicted continuous action lexe sequence. A second training loss is determined based on the predicted continuous action lexe sequence and the training continuous action lexe sequence. The action diffusion model and the action autoregressive model are trained based on the second training loss. During training based on the second training loss, the model parameters of the action diffusion model are updated according to a first learning rate scheduling strategy, and the model parameters of the action autoregressive model are updated according to a second learning rate scheduling strategy. The maximum learning rate corresponding to the action diffusion model is less than the maximum learning rate corresponding to the action autoregressive model.

[0007] Thirdly, embodiments of this application provide a computer device, the computer device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of any of the above methods.

[0008] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the above methods.

[0009] Fifthly, embodiments of this application provide a computer program product, the computer program product including a computer program, which, when executed by a processor, implements the steps of any of the above methods. Attached Figure Description

[0010] The embodiments of this application are further described below with reference to the accompanying drawings and specific implementation details.

[0011] Figure 1 This is a flowchart illustrating a robot motion generation method provided in an embodiment of this application.

[0012] Figure 2 This is a schematic diagram of the structure of a robot motion generation model provided in an embodiment of this application.

[0013] Figure 3 This is a flowchart illustrating a robot motion generation model training method provided in an embodiment of this application.

[0014] Figure 4 This is a structural block diagram of a robot provided in an embodiment of this application.

[0015] Figure 5 This is a structural block diagram of a computer device provided in an embodiment of this application. Detailed Implementation

[0016] See Figure 1 , Figure 1 This is a flowchart illustrating a robot motion generation method provided in an embodiment of this application.

[0017] To improve the efficiency of robot motion generation, this application provides a robot motion generation method, including steps S101 to S105.

[0018] Step S101: Receive the task input information corresponding to the robot motion generation task.

[0019] Step S102: Generate condition information based on the task input information.

[0020] Step S103: Based on the conditional information, generate motion latent feature information using the motion diffusion model. The motion latent feature information is used to characterize the motion semantics related to the robot motion generation task.

[0021] Step S104: Based on the motion latent feature information, a continuous action word sequence is generated using an action autoregressive model. In the continuous action word sequence, except for the first continuous action word, the remaining continuous action words to be generated are all generated based on the motion latent feature information and at least some of the already generated continuous action words. Each continuous action word corresponds to a continuous action segment composed of multiple temporally continuous robot posture data.

[0022] Step S105: Decode the continuous action words in the continuous action word sequence to obtain the robot action information of the corresponding continuous action segment.

[0023] Robot motion generation tasks refer to processing tasks that require generating continuous robot movements. Task input information refers to information corresponding to the robot motion generation task, used to describe or define the motion generation target and serve as the basis for motion generation, such as "wave your right hand forward." Conditional information can be information formed based on the task input information that can be processed by the motion diffusion model, used to guide the motion diffusion model to generate latent motion feature information that matches the robot motion generation task.

[0024] Action diffusion models can be generative models that generate latent motion feature information based on conditional information. Latent motion feature information can be an abstract semantic representation of motion related to the robot action generation task, extracted by the action diffusion model from the conditional information. Latent motion feature information provides the semantic basis needed to generate robot actions for the action autoregressive model, but it is not directly equivalent to the robot action information that can be obtained through decoding.

[0025] An action autoregressive model can be a model that generates consecutive action words sequentially over time. When generating subsequent consecutive action words, the action autoregressive model uses both latent motion feature information and already generated consecutive action words in the sequence, so that the already generated consecutive action words form the generation context for subsequent consecutive action words.

[0026] Each continuous action lexical can be used to represent a continuous action segment. A single continuous action lexical does not represent robot posture data at only one moment, but rather represents a continuous action segment composed of multiple temporally consecutive robot posture data. A sequence of continuous action lexical ...

[0027] See Figure 2 , Figure 2This is a schematic diagram of the structure of a robot motion generation model provided in an embodiment of this application. The robot motion generation model may include a motion encoding module, a visual encoding module, a text encoding processing module, an embedding model, a motion diffusion model, a motion autoregressive model, and a motion decoding module. The outputs of the motion encoding module, the visual encoding module, and the text encoding processing module are respectively connected to the embedding model. The output of the embedding model is connected to the motion diffusion model, the output of the motion diffusion model is connected to the motion autoregressive model, and the output of the motion autoregressive model is connected to the motion decoding module. The aforementioned connections represent the data transmission relationships between the corresponding modules. The modules can be deployed on the same computing device or on multiple computing devices capable of data interaction.

[0028] In an example illustrating model hierarchy and data dimensions, the outputs of the action encoding module, visual encoding module, and text encoding processing module can all be converted into 512-dimensional features through a linear projection layer, and the embedding model can output 512-dimensional conditional features. The action diffusion model can include eight sequentially connected denoising network layers, each of which can include eight attention heads; the action autoregressive model can include eight sequentially connected masked self-attention layers, each of which can include eight attention heads, and the hidden feature dimension can be 512. For a task expected to generate nine consecutive action words, the motion latent feature information can adopt a 9×512 data shape, and the word output layer of the action autoregressive model can output approximately 15,000 consecutive action words in the consecutive action word vocabulary, along with the probabilities corresponding to sequence end markers. The aforementioned number of layers, number of attention heads, and feature dimensions are used to provide a feasible configuration; other values ​​can be adopted based on computational resources, action duration, and vocabulary size.

[0029] The motion decoding module can convert continuous motion terms into multi-frame robot posture data within a corresponding continuous motion segment based on the correspondence between continuous motion terms and robot posture data. When continuous motion terms correspond to relative motion, the motion decoding module can use the robot's current posture or the end posture of the previous continuous motion segment as the anchor posture and accumulate the relative joint changes recorded by the continuous motion terms in chronological order. When continuous motion terms correspond to absolute posture, the motion decoding module can directly read the robot posture data arranged in chronological order.

[0030] In this embodiment, by generating conditional information based on task input information and utilizing an action diffusion model to generate latent motion feature information representing the motion semantics related to the robot's action generation task based on the conditional information, task semantic basis can be provided for the generation of continuous action lexics. By using an action autoregressive model to generate the remaining continuous action lexics based on the latent motion feature information and at least some of the already generated continuous action lexics, the already generated continuous action lexics can participate in the generation of subsequent continuous action lexics. By decoding each continuous action lexicographical unit, robot action information of the corresponding continuous action fragment can be obtained.

[0031] This embodiment addresses the problem of the large number of generation rounds required for generating robot posture data frame by frame in existing methods. It adopts the concept of using continuous action words corresponding to multiple frames of temporally continuous robot posture data as the generation and decoding units, and combines motion potential feature information and the generated continuous action words to generate the remaining continuous action words. This reduces the number of generation rounds required to generate robot motion information containing a set number of robot posture data, and makes the continuous action words form a continuous action word sequence according to the generation order.

[0032] In some embodiments, the task input information may include robot posture information, visual information, and task instruction information, wherein the visual information may include image information and / or video information acquired by the robot.

[0033] In some embodiments, generating conditional information based on the task input information may include: encoding the robot posture information, the visual information, and the task instruction information respectively to obtain robot posture encoding information, visual encoding information, and task instruction encoding information; and generating the conditional information using an embedding model based on the robot posture encoding information, the visual encoding information, and the task instruction encoding information.

[0034] Robot pose information can represent the robot's pose in one or more frames at the start of motion generation. Task instruction information can be textual information describing the robot's action requirements. Embedding models can transform different types of encoded information into a feature space that can be processed by motion diffusion models, thus forming conditional information.

[0035] The embedding model can perform feature projection on robot posture encoding information, visual encoding information, and task instruction encoding information respectively, so that the three types of encoded information have the same or fusionable feature dimensions, and then fuse them using at least one of the following methods: concatenation, summation, attention fusion, or adaptive normalization. In one implementation, the embedding model includes a linear projection layer corresponding to the three types of encoded information and a conditional fusion layer. The conditional fusion layer concatenates the projected robot posture features, visual features, and task instruction features in a set order and outputs a conditional feature sequence, i.e., conditional information.

[0036] In this embodiment, by retaining information from the robot's initial posture, visual scene, and task semantics respectively, and then mapping and fusing the three types of encoded information into conditional information by the embedding model, the action diffusion model can simultaneously receive constraints from the ontology state, external environment, and action requirements in the same conditional feature space.

[0037] This embodiment addresses the problem that a single type of input cannot fully represent the conditions for generating robot actions. It adopts a method that uses robot posture information, visual information, and task instruction information to collaboratively form conditional information, thereby improving the completeness of the conditional information in expressing the starting point of the action, the scene, and the semantics of the task.

[0038] In some embodiments, encoding the robot posture information, the visual information, and the task instruction information respectively to obtain robot posture encoding information, visual encoding information, and task instruction encoding information may include: using an action encoding module to convert the robot posture information into a robot posture tensor based on a continuous action lexicon, and using the robot posture tensor as the robot posture encoding information; using a visual encoding module to convert the visual information into a visual tensor, and using the visual tensor as the visual encoding information; segmenting the task instruction information into words, and converting the segmentation results into a task instruction tensor based on a text lexicon, and using the task instruction tensor as the task instruction encoding information.

[0039] The action encoding module can be a processing module that converts robot posture information into robot posture tensors based on a continuous action lexicon. The continuous action lexicon can record continuous action lexicons, robot posture data, and the correspondence between them. Tensors can be numerical data structures organized according to a set dimension. The visual encoding module can convert image information and / or video information into visual tensors. Word segmentation can divide task instruction information into characters, words, or semantic units composed of multiple characters; the text lexicon can record the correspondence between word segmentation results and numerical identifiers.

[0040] The action encoding module matches continuous pose segments from the robot's pose information with continuous action words from a continuous action word lexicon, and forms a robot pose tensor using the matched word identifiers or word features. The visual encoding module employs a variational autoencoder-based encoding network to compress image frames in pixel space into a visual latent representation; image information can be processed as video information containing only one frame, and video information can be processed as a sequence of image frames arranged according to acquisition time. The text encoding processing module first segments the task instruction information, then queries the text lexicon, converts each segmentation result into a numerical identifier, and arranges them according to the original text order to obtain the task instruction tensor.

[0041] In this embodiment, by adopting an encoding path that is compatible with the data format for robot posture information, visual information, and task instruction information, the temporal relationship of action segments, the spatial or spatiotemporal information of visual frames, and the semantic order of text units can be preserved respectively, and a structured input can be provided for the embedding model.

[0042] This embodiment addresses the issue of inconsistent original data structures and scales across different modalities by employing a combination of action lexical matching, visual feature encoding, and text segmentation mapping to transform the input of multimodal tasks into a tensor representation that can be further fused.

[0043] In some embodiments, generating motion potential feature information based on the conditional information using an action diffusion model may include: using the action diffusion model, based on the conditional information and unconditional information, performing inference at multiple diffusion time steps corresponding to the diffusion time step information to obtain the motion potential feature information, wherein the unconditional information is a feature unrelated to a specific robot action generation task, and the diffusion time step information is used to identify the current inference stage of the action diffusion model.

[0044] Unconditional information can be features unrelated to the specific robot action generation task, such as features generated by an embedding model from a given empty input, or set features with the same data shape as conditional information. Diffusion timestep information can be used to identify the current inference stage of the action diffusion model, but does not represent the actual time when the robot action occurs. Each diffusion timestep corresponds to a current intermediate motion representation, which is the data processed by the action diffusion model in the corresponding inference stage. The action diffusion model processes the current intermediate motion representation, conditional or unconditional information, and diffusion timestep information to obtain a prediction result used to update the current intermediate motion representation.

[0045] The motion diffusion model can include a time-step embedding layer, a conditional fusion layer, multiple denoising network layers, and an output projection layer. The time-step embedding layer converts diffusion time-step information into time-step features. The conditional fusion layer feeds conditional or unconditional information into the denoising network layers. Multiple denoising network layers progressively update intermediate motion representations. The output projection layer outputs the prediction result corresponding to the current diffusion time step. The denoising network layers can be attention-based or have downsampling and upsampling paths.

[0046] In one exemplary implementation, the action diffusion model can employ noise prediction for inference. The action diffusion model first generates or receives the intermediate motion representation corresponding to the termination diffusion time step T. Intermediate motion indicates It can be sampled from a given noise distribution. For a diffusion time step t executed sequentially from T to 1, the action diffusion model is based on the current intermediate motion representation. Conditional noise prediction results are obtained by combining conditional information and diffusion time step information. The action diffusion model is based on the current intermediate motion representation. By combining unconditional information and diffusion time step information, we can obtain unconditional noise prediction results. .

[0047] You can follow Determine the guided noise prediction results Where s represents the conditional guidance intensity. Conditional noise prediction results. With unconditional noise prediction results The differences between them can characterize the guiding direction of conditional information on the current intermediate motion representation. The motion diffusion model is based on the current intermediate motion representation. Guided noise prediction results And the update parameters corresponding to the diffusion time step t, and the next intermediate motion representation is obtained according to the diffusion sampling relationship corresponding to the noise prediction method. After completing the update corresponding to diffusion time step 1, the obtained intermediate motion representation will be... As potential feature information of motion.

[0048] The aforementioned noise prediction method is one possible implementation. In other implementations, the motion diffusion model can use the noiseless motion latent representation, the change in the current intermediate motion representation, or the score information as the prediction object, and update the intermediate motion representation using the diffusion sampling relationship corresponding to the respective prediction object. All prediction methods update the intermediate motion representation step by step according to multiple diffusion time steps to obtain the motion latent feature information.

[0049] In this embodiment, the unconditional noise prediction result is used to characterize the prediction result when there are no specific robot motion generation task constraints, and the difference between the conditional noise prediction result and the unconditional noise prediction result is used to characterize the guiding component generated by the conditional information. By combining the guiding component with the unconditional noise prediction result to form the guiding noise prediction result, the intermediate motion representation update direction of each diffusion time step can be adjusted using the conditional information, and the motion potential feature information related to the robot motion generation task can be obtained after multiple diffusion time steps.

[0050] This embodiment addresses the problem that it is difficult to gradually correct the latent motion representation in a single model processing. It adopts a method in which conditional prediction results, unconditional prediction results, and diffusion time step information jointly participate in multiple diffusion time step updates, thereby gradually forming latent motion feature information related to the robot motion generation task.

[0051] In some embodiments, the step of generating a continuous action lexical sequence using an autoregressive motion model based on the latent motion feature information, and the step of decoding the continuous action lexical sequences to obtain robot motion information for corresponding continuous action segments, may include: generating a first continuous action lexical based on the latent motion feature information using the autoregressive motion model, and adding the first continuous action lexical to the continuous action lexical sequence; for the i-th continuous action lexical to be generated, generating it using the autoregressive motion model based on the latent motion feature information and the previous 1, 2, ..., or i-1 continuous action lexicals of the i-th continuous action lexical, and adding the generated i-th continuous action lexical to the continuous action lexical sequence in the order of generation, where i is an integer greater than 1; wherein, for continuous action lexical other than the first continuous action lexical, the process of generating the continuous action lexical is at least partially overlapping in time with the process of decoding the corresponding previous continuous action lexical to obtain robot motion information for the corresponding continuous action segment.

[0052] The first consecutive action lexical can be the first consecutive action lexical generated in the consecutive action lexical sequence. The i-th consecutive action lexical can be the second, third, ..., or last consecutive action lexical generated in the order of generation following the first consecutive action lexical. At least partial temporal overlap means that the generation time interval of the i-th consecutive action lexical and the decoding time interval of the (i-1)-th consecutive action lexical can intersect. The end of the consecutive action lexical sequence generation can be triggered by a sequence end marker, setting the number of lexical units, setting the action duration, or a task end condition. The 1, 2, ..., or (i-1)-th consecutive action lexical preceding the i-th consecutive action lexical can represent historical lexical prefixes formed by the 1, 2, ..., or (i-1)-th consecutive action lexical generated before the i-th consecutive action lexical. The action autoregressive model can select the corresponding number of historical lexical prefixes based on the corresponding generation position.

[0053] An action autoregressive model may include a sequentially connected layer of continuous action lexical embeddings, a positional encoding layer, at least one masked self-attention layer, a motion latent feature fusion layer, and a lexical output layer.

[0054] The continuous action lexical embedding layer is used to convert the generated continuous action lexicals into continuous action lexical features, and the positional encoding layer is used to form the generation position information of continuous action lexicals in the continuous action lexical sequence. The masked self-attention layer is used to restrict the historical lexical prefixes that the corresponding continuous action lexical can use according to the generation position, so that the i-th continuous action lexical is generated only based on the corresponding historical lexical prefixes in the previous 1, 2, ..., or i-1 continuous action lexicals.

[0055] The motion latent feature fusion layer fuses the continuous action lexical features corresponding to historical lexical prefixes with the motion latent feature information. The motion latent feature information can be input into the action autoregressive model via cross-attention, or it can be concatenated as a prefix feature with the continuous action lexical features. The lexical output layer determines the output probability of the continuous action lexical to be generated based on the fusion result.

[0056] Specifically, when generating the first continuous action lexical, the action autoregressive model can take the latent motion feature information and the set sequence start identifier as input, and determine the first continuous action lexical using one of the following methods: maximum probability selection, sampling of the first k candidates, or sampling with a set probability distribution. When generating the i-th continuous action lexical, the action autoregressive model can take the latent motion feature information and the corresponding historical lexical prefixes formed by the 1, 2, ..., or i-1 consecutive action lexicals preceding the i-th continuous action lexical as input.

[0057] As the action autoregressive model continues to generate subsequent consecutive action words, the action decoding module can process the previously generated consecutive action words in parallel. After the consecutive action word sequence is generated, the action decoding module decodes the last consecutive action word so that each consecutive action word in the sequence forms corresponding robot action information.

[0058] In this embodiment, by enabling the action autoregressive model to generate word-by-word based on the potential motion feature information and the already generated continuous action words, and by ensuring that the generation of subsequent continuous action words and the decoding of the previous continuous action words overlap at least partially in time, a pipelined processing procedure for generation and decoding can be formed while maintaining the sequence dependency relationship.

[0059] This embodiment addresses the problem of long output waiting time caused by waiting for the complete continuous action word sequence to be generated before unified decoding. It adopts a method that combines the step-by-step generation of continuous action words with the streaming decoding of prior continuous action words, thereby reducing the serial waiting time and enabling the robot to start executing the generated continuous action segments earlier.

[0060] In some embodiments, the lexical output space of the action autoregressive model can be limited by a continuous action lexical dictionary. Each continuous action lexical generated by the action autoregressive model in each round corresponds to a continuous action lexical in the continuous action lexical dictionary, and each continuous action lexical is added to the continuous action lexical sequence in the order of generation.

[0061] In some embodiments, the continuous action lexicon may include basic continuous action lexicons and compound continuous action lexicons. The basic continuous action lexicons may correspond to a continuous action segment composed of a predetermined number of temporally consecutive robot posture data. The compound continuous action lexicons may be obtained by combining robot posture data corresponding to at least two basic continuous action lexicons in chronological order.

[0062] The lexical output space can be the set of candidate continuous action lexical units that the action autoregressive model can output. Basic continuous action lexical units can correspond to continuous action segments composed of a set number of temporally continuous robot posture data. Compound continuous action lexical units can correspond to longer continuous action segments obtained by combining robot posture data corresponding to at least two basic continuous action lexical units in temporal order.

[0063] The continuous action lexicon can record the lexicon identifier, corresponding robot posture data, and lexicon representation for continuous action segment matching for each basic continuous action lexicon and compound continuous action lexicon. The lexicon output layer of the action autoregressive model can output the probability corresponding to each lexicon identifier in the continuous action lexicon, so that the output of each action autoregressive model can be mapped to multi-frame robot posture data through the continuous action lexicon.

[0064] In one implementation, each frame of robot pose data can include 78 dimensions of pose data, with 72 dimensions representing body pose and 6 dimensions representing hand pose. Hand pose can be represented using 6-dimensional data after dimensionality reduction via principal component analysis. Continuous action lexes can record the relative motion between adjacent robot poses, or they can record the absolute joint states corresponding to each robot pose.

[0065] When the action autoregressive model generates a sequence of continuous action lexical units, it prioritizes matching compound continuous action lexical units that cover robot posture data across multiple consecutive frames and whose lexical distance meets the matching threshold. If no suitable compound continuous action lexical units exist, it then matches basic continuous action lexical units. The action decoding module can read the robot posture data corresponding to each continuous action lexical unit in the continuous action lexical unit dictionary and reconstruct it according to the order in the continuous action lexical unit sequence. When there are overlapping postures or transitional robot posture data between adjacent continuous action lexical units, it can remove duplicate frames or retain transitional frames according to the connection relationship recorded in the dictionary.

[0066] In this embodiment, by limiting the candidate output of the action autoregressive model with a continuous action lexicon and making the basic continuous action lexicon and the compound continuous action lexicon correspond to continuous action segments of different lengths, the output of the action autoregressive model can establish a definite mapping with the decodeable robot posture data, and the generation unit with different time granularity can be selected according to the action content.

[0067] This embodiment addresses the problem of balancing the efficiency of short action expression and long action generation when using fixed-length continuous action lexical units. It adopts a method in which basic continuous action lexical units and compound continuous action lexical units work together to support the output space, thereby achieving both fine-grained action expression and rapid generation of longer action segments.

[0068] In some embodiments, the continuous action lexicon can be constructed as follows: receiving multiple robot posture data from a lexicon dataset, the lexicon dataset including first robot posture data obtained by redirecting human motion posture data to robot joints, and second robot posture data acquired by the robot; converting the frame rate of the multiple robot posture data to a target frame rate, and dividing each robot posture data after frame rate conversion into multiple candidate basic continuous action lexicons according to a set number of frames; performing elimination and merging on the candidate basic continuous action lexicons to obtain and store basic continuous action lexicons; performing concatenation and merging on at least some of the basic continuous action lexicons while retaining the basic continuous action lexicons to obtain composite continuous action lexicons; and constructing the continuous action lexicon based on the basic continuous action lexicons and the composite continuous action lexicons.

[0069] The vocabulary dataset can be a collection of multiple robot posture data used to construct a continuous action lexicon. Redirection can be the mapping of human joint motion in human motion posture data to robot joints, ensuring the mapping results satisfy the robot's joint structure and range of motion. The target frame rate can be a uniform frame rate used during the construction of the continuous action lexicon. The target frame rate can be determined based on at least one of the following: robot control frequency, original sampling frequency of robot posture data, temporal resolution requirements of continuous action segments, input requirements of the motion encoding module, and a set number of frames corresponding to the continuous action lexicon. Candidate basic continuous action lexicons can be continuous action segments obtained from the robot posture data according to a set number of frames and awaiting screening. Elimination and merging can be the process of reducing action content repetition for candidate basic continuous action lexicons, while concatenation and merging can be the process of forming compound continuous action lexicons based on basic continuous action lexicons.

[0070] Human motion posture data can be sourced from publicly available human motion posture datasets such as Human3.6M, and the robot performing the redirection can be, for example, a humanoid robot with corresponding joints. During redirection, reinforcement learning can be used to establish a mapping between human joints and robot joints, obtaining first robot posture data that conforms to robot joint constraints in a robot simulation environment. This first robot posture data can be checked through 3D reconstruction and manually adjusted. Second robot posture data can be obtained by the robot performing and collecting corresponding actions, and can undergo data cleaning, action annotation, and semantic action filtering. The two types of robot posture data can be combined into a vocabulary dataset according to a set ratio.

[0071] When constructing a continuous action lexicon, robot posture data from different sources can first be converted to a target frame rate through resampling, and then divided into candidate basic continuous action lexicons according to a set number of frames in chronological order. After eliminating and merging the candidate basic continuous action lexicons, basic continuous action lexicons representing the corresponding action content can be obtained and stored. Based on retaining the basic continuous action lexicons, at least some of the basic continuous action lexicons can be concatenated and merged to obtain composite continuous action lexicons corresponding to longer continuous action segments. By assigning lexicon identifiers to the basic continuous action lexicons and composite continuous action lexicons respectively and recording the corresponding robot posture data, a continuous action lexicon containing both types of continuous action lexicons can be constructed.

[0072] In this embodiment, by using first robot posture data obtained by redirecting human motion posture data and second robot posture data actually collected by the robot together to construct a continuous action lexicon, the first robot posture data can provide robot posture data corresponding to different human actions, and the second robot posture data can provide robot posture data obtained by the robot's actual execution. By converting robot posture data from different sources to a target frame rate, candidate basic continuous action lexicons can have a consistent time scale. By eliminating and merging candidate basic continuous action lexicons that reduce repetition of action content, and concatenating composite continuous action lexicons while retaining basic continuous action lexicons, the continuous action lexicon can simultaneously retain both basic and composite continuous action lexicons.

[0073] This embodiment addresses the issues of limited action coverage in single-source robot posture data and repetition in candidate action segments by employing a combination of multi-source robot posture data, frame rate normalization, elimination merging, and splicing / merging. This expands the action coverage of the continuous action lexicon and provides action representations of different durations through basic and compound continuous action lexicons.

[0074] In some embodiments, the elimination and merging of the candidate basic continuous action lexical units to obtain basic continuous action lexical units may include: for any two candidate basic continuous action lexical units, determining a body posture distance based on robot posture data in the two candidate basic continuous action lexical units, and determining a hand posture distance based on robot posture data in the two candidate basic continuous action lexical units; performing a weighted summation of the body posture distance and the hand posture distance according to set body posture weights and hand posture weights respectively to obtain the lexical distance between the two candidate basic continuous action lexical units; clustering multiple candidate basic continuous action lexical units based on the lexical distance, and determining the number of lexical units to be selected for each cluster category based on the number of candidate basic continuous action lexical units contained in each cluster category; filtering the candidate basic continuous action lexical units contained in each cluster category based on the number of lexical units to be selected for each cluster category, the source and time sequence of robot posture data in the candidate basic continuous action lexical units contained in each cluster category, to obtain the basic continuous action lexical units.

[0075] Body pose distance is a distance determined based on the body pose data at corresponding moments in two candidate basic continuous action words, used to characterize the difference in body pose between the two candidate basic continuous action words. Hand pose distance is a distance determined based on the hand pose data at corresponding moments in two candidate basic continuous action words, used to characterize the difference in hand pose between the two candidate basic continuous action words. For two candidate basic continuous action words containing the same number of frames, the body pose vector distance and hand pose vector distance for each corresponding frame can be calculated separately, and then the body pose distance and hand pose distance can be obtained by summing or averaging. This can be done according to... Determine the word distance D, where, Indicates body posture distance. Indicates the distance of hand gestures. Indicates body posture weights, Indicates the hand gesture weights. Body posture distance and hand posture distance can be measured using Euclidean distance, average joint distance, or other distances that can characterize posture differences.

[0076] For any two candidate basic continuous action lexical units, the body pose distance and hand pose distance can be determined based on the body pose vector in the robot pose data and the hand pose vector in the robot pose data, respectively, and then... Determine the word distance D, where, Indicates body posture distance. This represents the hand pose distance. The body pose distance and hand pose distance can be represented by the Euclidean distance between the corresponding pose vectors, the average joint distance, or other distances that characterize pose differences. 0.7 and 0.3 are used to provide an example of body pose weights and hand pose weights; both can be set to other values ​​depending on the robot structure and the task.

[0077] Elimination and merging can be achieved by first forming multiple clusters based on a set clustering distance threshold, and then determining the number of words to be selected based on the number of candidate basic continuous action words contained in each cluster. If the clustering results contain multiple long-tailed categories that each contain only one candidate basic continuous action word, these long-tailed categories can be grouped into a combined category, and the weight of the combined category can be increased when calculating the number of words to be selected. Specifically, a depth-first search can be used to prioritize candidate basic continuous action words from the same robot posture data that are temporally continuous; alternatively, dynamic programming or sorting by the length of consecutive segments can be used to achieve the same priority retention relationship.

[0078] In this embodiment, body posture distance and hand posture distance are determined based on robot posture data from two candidate basic continuous action lexical units, respectively. A weighted sum is then performed according to body posture weights and hand posture weights to obtain lexical distances that simultaneously reflect differences in body posture and hand posture. Clustering is performed based on lexical distances, and the number of lexical units selected is determined according to the number of candidate basic continuous action lexical units contained in each cluster. This allows for the allocation of a retention quantity of basic continuous action lexical units for different cluster categories. By combining the lexical selection quantity with the source and temporal sequence of the robot posture data for filtering, basic continuous action lexical units with source and temporal correlations can be retained while reducing duplicate candidate basic continuous action lexical units.

[0079] This embodiment addresses the problem that it is difficult to fully evaluate the action differences between candidate basic continuous action words by using only the posture difference of a single part. It adopts a weighted method of body posture distance and hand posture distance, word distance-based clustering, and source and temporal sequence filtering, thereby improving the representativeness of basic continuous action words for different action content and preserving temporal sequence association information for subsequent splicing.

[0080] In some embodiments, the step of concatenating and merging at least a portion of the basic continuous action words to obtain composite continuous action words may include: in response to the existence of robot posture data corresponding to basic continuous action words originating from the same robot trajectory data and the robot posture data corresponding to basic continuous action words being temporally continuous, determining a concatenation priority parameter for each basic continuous action word based on the number of candidate basic continuous action words included in the cluster category to which each basic continuous action word belongs; determining a concatenation starting word according to the concatenation priority parameter in descending order; and, starting from the concatenation starting word, concatenating the robot posture data corresponding to each basic continuous action word. Based on the source and temporal sequence of the robot posture data, at least one candidate word sequence is determined. Each candidate word sequence consists of multiple basic continuous action words originating from the same robot posture data and in temporal sequence. From the candidate word sequence, a target word sequence containing the most basic continuous action words is determined and concatenated in chronological order to obtain a composite continuous action word. In response to any basic continuous action word being used to generate the composite continuous action word, the concatenation priority parameter of any basic continuous action word is reduced. Based on the updated concatenation priority parameters of each basic continuous action word, the starting word for concatenation is re-determined, and concatenation continues.

[0081] Robot trajectory data can be a sequence of robot posture data collected chronologically during robot action execution. Multiple basic continuous action words from the same robot trajectory data can correspond to different temporally continuous data segments within the same robot posture data sequence. The concatenation priority parameter is a parameter used to determine the order in which basic continuous action words are selected as the starting concatenation word. The concatenation priority parameter can be determined based on the number of candidate basic continuous action words contained in the cluster category to which the corresponding basic continuous action word belongs. The candidate word sequence can be a word sequence consisting of multiple basic continuous action words from the same robot posture data and in temporal continuity, starting with the concatenation starting word. The target word sequence can be the candidate word sequence containing the largest number of basic continuous action words in at least one candidate word sequence. Decreasing the concatenation priority parameter can reduce the selection order of basic continuous action words already used to generate compound continuous action words in subsequent concatenations.

[0082] For a compound continuous action word formed by concatenating x basic continuous action words, the target generation quantity num in a generation batch can be determined by num = -a × ln(x) + b, where x is the number of basic continuous action words involved in the concatenation, and a and b are parameters used to adjust the number of compound continuous action words of different lengths. As x increases, the target generation quantity num decreases. In one example, when a = 30 and b = 100, the target generation quantities of 2-compound continuous action words and 3-compound continuous action words are approximately 79 and 67, respectively.

[0083] When performing concatenation and merging, the number of candidate basic continuous action words contained in the cluster category to which the basic continuous action word belongs can be used as the initial concatenation priority parameter for that basic continuous action word. After determining the starting word for concatenation according to the concatenation priority parameter in descending order, at least one candidate word sequence can be determined starting from the starting word based on the source and temporal sequence of the robot posture data. The candidate word sequence containing the most basic continuous action words is then determined as the target word sequence. Concatenating the target word sequence in chronological order yields the composite continuous action word. After any basic continuous action word is used to generate a composite continuous action word, the corresponding concatenation priority parameter can be decreased by 1, and the next starting word for concatenation can be determined based on the updated concatenation priority parameter.

[0084] In this embodiment, the splicing priority parameter is determined based on the number of candidate basic continuous action lexical units contained in the cluster category, and the splicing starting lexical unit can be determined according to the splicing priority parameter; by forming a candidate lexical unit sequence based on the source and temporal sequence of robot posture data, and selecting the target lexical unit sequence containing the most basic continuous action lexical units, the longer lexical unit sequence with original temporal relationship can be used to generate compound continuous action lexical units first; by reducing the corresponding splicing priority parameter after the basic continuous action lexical units are used, the selection order of subsequent splicing starting lexical units can be adjusted.

[0085] This embodiment addresses the problem of difficulty in determining the selection order and splicing length of basic continuous action words during the generation of compound continuous action words. It adopts a combination of splicing priority parameters, homologous temporal candidate word sequences, the longest target word sequence, and post-use parameter updates to generate compound continuous action words based on the original temporal relationship of robot posture data, and allows different basic continuous action words to participate in multiple splicing rounds.

[0086] In some embodiments, the step of concatenating and merging at least a portion of the basic continuous action lexical units to obtain composite continuous action lexical units may include: for the basic continuous action lexical units to be concatenated, in response to the absence of any basic continuous action lexical units whose corresponding robot posture data originates from the same robot trajectory data or whose corresponding robot posture data are not sequentially discontinuous, determining the lexical distance between each basic continuous action lexical unit and the basic continuous action lexical unit to be concatenated from basic continuous action lexical units in clusters other than the cluster to which the basic continuous action lexical unit to be concatenated belongs, according to the lexical distance determination method, and selecting the... Basic continuous action lexical units with a lexical distance less than or equal to a set distance threshold are used as subsequent basic continuous action lexical units. Interpolation processing is performed on the robot posture data corresponding to the basic continuous action lexical unit to be concatenated and the subsequent basic continuous action lexical unit to obtain transitional robot posture data between the continuous action segment corresponding to the basic continuous action lexical unit to be concatenated and the continuous action segment corresponding to the subsequent basic continuous action lexical unit. Based on the robot posture data corresponding to the basic continuous action lexical unit to be concatenated, the transitional robot posture data, and the robot posture data corresponding to the subsequent basic continuous action lexical unit, concatenation is performed to obtain a composite continuous action lexical unit.

[0087] The basic continuous action word to be concatenated can be the current basic continuous action word to be concatenated with the subsequent basic continuous action word. The distance threshold can be the upper limit of the distance used to determine whether two basic continuous action words in different clusters can be connected across clusters. The transition robot posture data can be one or more intermediate postures located between the end posture of the basic continuous action word to be concatenated and the starting posture of the subsequent basic continuous action word. Ending the concatenation and merging can stop searching for subsequent basic continuous action words for the basic continuous action word to be concatenated and retain the currently formed concatenation result.

[0088] When there are no subsequent basic continuous action words of the same origin and with a sequential order, basic continuous action words whose word distance from the basic continuous action word to be concatenated is not greater than a set distance threshold can be selected from other cluster categories as subsequent basic continuous action words. After selection, the end pose of the basic continuous action word to be concatenated and the starting pose of the subsequent basic continuous action word can be interpolated to obtain transitional robot pose data. Interpolation can be performed in joint angle space, or the corresponding pose can be transformed to Lie algebra representation space before interpolation, and then transformed back to robot joint pose through exponential mapping. When there are no candidate words that meet the set distance threshold, the current concatenation can be terminated to reduce the situation where action segments with excessively different poses are forcibly combined into composite continuous action words.

[0089] In this embodiment, by filtering subsequent words across categories based on word distance when the original temporal continuity relationship is missing, and supplementing the transition robot posture data obtained by interpolation between two action segments, a posture continuity connection path can be established for cross-category action segments; the splicing can be stopped when there are no candidate words that meet the distance condition, thus restricting discontinuous action combinations from entering the continuous action word word book.

[0090] This embodiment addresses the problem that relying solely on time segments from the same source may not result in a sufficient number of compound continuous action terms. It employs a combination of cross-category selection under distance constraints, pose interpolation, and failure termination to increase the number of compound continuous action terms that can be formed and improve pose transitions at splicing boundaries.

[0091] In one example, the vocabulary dataset includes first and second robot pose data in a 1:1 ratio. All robot pose data is converted to 30 frames per second, and each robot pose data point after frame rate conversion is divided into 4-frame segments, resulting in candidate basic continuous action words of size 4×78. For any two candidate basic continuous action words, body pose distance and hand pose distance are determined based on their body pose data and hand pose data, respectively. For example, if the body pose distance between two candidate basic continuous action words is 0.20, the hand pose distance is 0.50, and the body pose weight and hand pose weight are 0.7 and 0.3 respectively, then the word distance between them is 0.7×0.20 + 0.3×0.50 = 0.29. With a clustering distance threshold of 0.35, the two candidate basic continuous action words are grouped into the same cluster. Clustering and elimination merging are performed on multiple candidate basic continuous action words based on word distance, keeping the number of basic continuous action words obtained and stored to approximately 10,000.

[0092] In the aforementioned example, the number of candidate basic continuous action lexical units contained in the five cluster categories are 6, 3, 1, 1, and 1, respectively. After merging the last three long-tail categories into a combined category, the number of candidate basic continuous action lexical units contained in the three categories are 6, 3, and 3, respectively. Setting the category adjustment coefficient of the combined category to 2, we obtain the effective numbers 6, 3, and 6 for determining the number of lexical units to be selected, corresponding to a ratio of 2:1:2. When determining the number of lexical units to be selected for each cluster category, we can combine the corresponding ratio with the actual number of candidate basic continuous action lexical units contained in each cluster category, ensuring that the number of lexical units selected does not exceed the actual number of candidate basic continuous action lexical units contained in the corresponding cluster category. For example, if the target total selection is 5 basic continuous action lexical units, we can select 2, 1, and 2 candidate basic continuous action lexical units from the aforementioned three categories respectively, according to a ratio of 2:1:2. When filtering according to the number of corresponding word elements, priority is given to retaining candidate basic continuous action word elements that are formed by adjacent data segments in the same robot posture data and are continuous in time, thus obtaining basic continuous action word elements.

[0093] While retaining the aforementioned basic continuous action lexical units, at least some of these units are concatenated and merged. Specifically, the number of candidate basic continuous action lexical units contained in the cluster category to which each basic continuous action lexical unit belongs is used as the initial concatenation priority parameter for that unit. The starting concatenation lexical unit is then determined in descending order of these priority parameters. Starting with the starting concatenation lexical unit, at least one candidate lexical unit sequence is determined based on the source and temporal sequence of the robot posture data corresponding to each basic continuous action lexical unit. The candidate lexical unit sequence containing the largest number of basic continuous action lexical units is then determined as the target lexical unit sequence. The basic continuous action lexical units in the target lexical unit sequence are concatenated in chronological order to obtain composite continuous action lexical units. After any basic continuous action lexical unit is used to generate a composite continuous action lexical unit, the concatenation priority parameter of that unit is decreased by 1. The starting concatenation lexical unit is then re-determined based on the updated concatenation priority parameters of each basic continuous action lexical unit, and the concatenation process continues.

[0094] In the aforementioned concatenation process, in response to the absence of a subsequent basic continuous action lexical originating from the same robot trajectory data as the basic continuous action lexical to be concatenated and whose corresponding robot posture data are sequentially continuous, basic continuous action lexicals with a lexical distance not greater than a set distance threshold are selected from other cluster categories as subsequent basic continuous action lexicals. The end-effector robot posture data of the basic continuous action lexical to be concatenated and the beginning-effector robot posture data of the subsequent basic continuous action lexical are interpolated to obtain transition robot posture data between the two continuous action segments. Then, based on the robot posture data corresponding to the basic continuous action lexical to be concatenated, the transition robot posture data, and the robot posture data corresponding to the subsequent basic continuous action lexical, concatenation is performed to obtain a composite continuous action lexical.

[0095] For a compound continuous action word formed by concatenating x basic continuous action words, the target generation quantity num in a generation batch is determined according to num = -a × ln(x) + b. For example, when a = 30 and b = 100, the target generation quantities of 2-compound continuous action words and 3-compound continuous action words are approximately 79 and 67, respectively. By executing multiple generation batches, compound continuous action words are added to retain approximately 10,000 basic continuous action words, resulting in a total of approximately 15,000 basic and compound continuous action words. A continuous action word lexicon is then constructed based on the basic and compound continuous action words. After construction, the action encoding module uses the longest matching method to convert the robot posture data into a sequence of continuous action words. The action decoding module, based on the same continuous action word lexicon, restores the continuous action words to robot posture data arranged in chronological order.

[0096] See Figure 3 , Figure 3 This is a flowchart illustrating a robot motion generation model training method provided in an embodiment of this application.

[0097] This application also provides a robot motion generation model training method, wherein the robot motion generation model includes a motion diffusion model and a motion autoregression model, and the robot motion generation model training method includes steps S201 to S207.

[0098] Step S201: Receive a first training sample from the first training dataset, wherein the first training sample includes first training input information and first training supervision information.

[0099] Step S202: Generate first training condition information based on the first training input information, input the first training condition information, training unconditional information and training diffusion time step information into the action diffusion model, and use the action diffusion model to perform inference to obtain the first training motion potential feature information.

[0100] Step S203: Generate first training prediction information based on the first training motion potential feature information, determine a first training loss based on the first training prediction information and the first training supervision information, and train the action diffusion model based on the first training loss.

[0101] Step S204: Receive the second training sample from the second training dataset. The second training sample includes the second training input information and the training continuous action word sequence corresponding to the second training input information. Each training continuous action word corresponds to a continuous action segment composed of multiple temporally continuous robot posture data.

[0102] Step S205: Generate second training condition information based on the second training input information, input the second training condition information, training unconditional information and training diffusion time step information into the action diffusion model, and use the action diffusion model to perform inference to obtain the second training motion potential feature information.

[0103] Step S206: Based on the second training motion latent feature information, generate a predicted continuous action lexical sequence using the action autoregressive model.

[0104] Step S207: Determine a second training loss based on the predicted continuous action lexical sequence and the training continuous action lexical sequence, and train the action diffusion model and the action autoregressive model based on the second training loss.

[0105] During the training process based on the second training loss, the model parameters of the action diffusion model are updated according to the first learning rate scheduling strategy, and the model parameters of the action autoregressive model are updated according to the second learning rate scheduling strategy. The maximum learning rate corresponding to the action diffusion model is less than the maximum learning rate corresponding to the action autoregressive model.

[0106] The first training sample can be used to train the action diffusion model, and the first training supervision information can be used to evaluate the first training prediction information generated from the first training motion latent feature information. The second training sample can be used to train the action diffusion model and the action autoregressive model, and the training continuous action word sequence can serve as the supervision target for each word position. The first training loss and the second training loss respectively characterize the difference between the first training prediction information and the first training supervision information, and the difference between the predicted continuous action words and the trained continuous action words.

[0107] The robot motion generation model includes a motion diffusion model and a motion autoregressive model. The training process can include a first training phase and a second training phase. The first training phase mainly trains the motion diffusion model's ability to form potential motion feature information based on first training conditions. The second training phase updates the motion diffusion model and the motion autoregressive model based on a second training loss. The motion diffusion model uses a smaller maximum learning rate for adaptation. After reaching the set training progress, the motion diffusion model continues forward inference but stops updating parameters, while the motion autoregressive model continues to update its parameters based on the second training loss. The two training phases can be executed sequentially, and the parameters of the motion diffusion model after the first training phase can be used as the initial model parameters for the second training phase.

[0108] See Figure 2 In the first training phase, the output of the action diffusion model can be connected to the visual decoding model that generates the first training prediction information, and the first training loss can be determined based on the first training prediction information and the first training supervision information. In the second training phase, the output of the action diffusion model can be connected to the action autoregressive model, and the output of the action autoregressive model can be connected to the word output layer. For the j-th word position, the first to (j-1)-th training continuous action words and the second training motion latent feature information can be used as inputs, and the j-th training continuous action word can be used as the supervision target; the first word position can use the sequence start identifier and the second training motion latent feature information as inputs.

[0109] The second training loss can be the continuous action lexicon-level cross-entropy loss. When the training continuous action lexicon sequence includes N training continuous action lexicons, it can be calculated according to... Determine the second training loss ,in, This indicates that the action autoregressive model has the target word at the j-th lexical position as the supervision target. Determined prediction probabilities. The model parameters of the action diffusion model and the action autoregressive model are updated based on the second training loss.

[0110] In this embodiment, a supervisory relationship is established between the first training input information, the first training motion latent feature information, and the first training supervision information using the first training samples. This allows the action diffusion model to be trained to generate the first training motion latent feature information. A supervisory relationship is established between the second training motion latent feature information, historical training continuous action words, and the continuous action words to be predicted using the second training samples. Based on the second training loss, the action diffusion model and the action autoregressive model are trained, enabling both models to jointly adapt to the continuous action word prediction task.

[0111] This embodiment addresses the issue that the input-output formats and training objectives of the action diffusion model and the action autoregressive model are different. It adopts a phased training approach using two types of training samples and two types of training losses, thereby enabling the robot action generation model to form a complete generation capability from task input conditions to potential motion feature information, and then to continuous action word sequences.

[0112] In some embodiments, during training based on the second training loss, the model parameters of the action diffusion model can be updated according to a first learning rate scheduling strategy, and the model parameters of the action autoregressive model can be updated according to a second learning rate scheduling strategy, wherein the maximum learning rate corresponding to the action diffusion model can be less than the maximum learning rate corresponding to the action autoregressive model.

[0113] In some embodiments, the robot motion generation model training method may further include: in response to the training progress reaching a set training progress, stopping the updating of the model parameters of the motion diffusion model, and continuing to update the model parameters of the motion autoregressive model.

[0114] The first and second learning rate scheduling strategies can respectively specify how the learning rate of the action spread model and the action autoregressive model changes with the training progress. The maximum learning rate can be the upper limit of the learning rate that the corresponding model is allowed to reach during training. Stopping the updating of the model parameters of the action spread model can mean that the action spread model continues to perform forward inference, but the backpropagation results no longer change the model parameters of the action spread model.

[0115] The first and second learning rate scheduling strategies can each include at least one of the following: warm-up, constant learning rate, segmented decay, or cosine annealing. The cosine annealing stage can be performed according to... Determine the current learning rate lr, where, and These represent the minimum learning rate and the maximum learning rate, respectively. and These represent the current scheduling progress and the total scheduling progress, respectively. The maximum learning rate of the action diffusion model is less than that of the action autoregressive model, which allows the action diffusion model to undergo a smaller-scale adaptation in the second training phase.

[0116] Training progress can be determined based on the number of training steps completed, the number of training rounds completed, or the proportion of the number of training samples processed relative to the total training volume. Once the training progress reaches the set target, the model parameters of the motion diffusion model can be set to a non-updateable state; the motion diffusion model will still output the second training motion latent feature information based on the second training condition information, and the motion autoregressive model will continue to update its model parameters based on the second training loss.

[0117] In this embodiment, by setting a small maximum learning rate for the action diffusion model and stopping updates in the later stages of training, the impact of the second training loss on the already formed potential motion representation ability can be limited; by setting a large maximum learning rate for the action autoregressive model and continuing to update it, the action autoregressive model can be made to focus on adapting to the continuous action lexical prediction task.

[0118] This embodiment addresses the issue of different training maturity and adaptation targets of the two models by employing differentiated learning rate scheduling and time-sharing parameter updates. This approach maintains the existing representational capabilities of the action diffusion model while improving the adaptation of the action autoregressive model to continuous action terms of the robot.

[0119] In some embodiments, the first training dataset may include first training image information and / or first training video information and first training text information that have a corresponding relationship, and the first training input information may include the first training image information and / or the first training video information and the first training text information.

[0120] In some embodiments, generating the first training prediction information based on the first training motion latent feature information may include: generating predicted image information and / or predicted video information using a visual decoding model based on the first training motion latent feature information, and using the predicted image information and / or the predicted video information as the first training prediction information; the first training supervision information includes target image information and / or target video information corresponding to the first training input information.

[0121] The corresponding relationship can be that the first training text information is used to describe the changes in objects, scenes, human poses, or human movements in the corresponding first training image information or first training video information. The visual decoding model can convert the first training motion latent feature information into predicted image information or predicted video information in pixel space. The target image information and target video information can be visual supervision data used for comparison with the corresponding prediction results.

[0122] The first training image information and the first training text information can, for example, come from the Large-scale Artificial Intelligence Open Network (LAION) dataset or the Conceptual Captions 12M (CC12M) dataset. The first training video information and the first training text information can, for example, come from the InternVid video-text dataset, or other image-text or video-text data with corresponding relationships can be used. The first training image information can be processed as a single frame of video, and the first training video information can be processed as a stack of image frames arranged in chronological order. In the reconstruction-based training method, the input visual information itself can serve as the target visual information; in the conditional prediction-based training method, the input visual information can include the starting frame or the reference frame, and the target visual information can include subsequent image frames or the target video.

[0123] In one feasible path for the first training phase, the visual coding module can employ a variational autoencoder network to convert target image or video information into a latent visual representation of the target. The training process can sample training diffusion time steps t from 1 to T, and sample noise e, according to... Generate the noisy latent representation corresponding to time step t ,in, This represents the accumulated signal retention parameter corresponding to time step t. The action diffusion model can be based on... The first training condition information, the training uncondition information, and the training diffusion time step information are used to predict the noise, resulting in the predicted noise e′. Based on the predicted noise e′, the potential feature information of the first training motion is determined. Visual decoding models can This is converted into predicted image information or predicted video information. To train the unconditional inference branch, the first training condition information can be replaced with the training unconditional information according to a set replacement probability.

[0124] The first training loss can be determined based on the pixel-wise difference between the predicted image information and the target image information, and / or the pixel-wise difference between the predicted video information and the corresponding image frames in the target video information. Let M be the total number of target pixels or target video pixels, and let m-th target pixel value and predicted pixel value be denoted as... and , can be according to Determine the first training loss Alternatively, absolute value loss, latent representation reconstruction loss, or a weighted combination of the aforementioned losses can be used. In the first training phase, the model parameters of the visual encoding module, visual decoding model, and text encoding processing module can be fixed, while the model parameters of the embedding model and action diffusion model are updated; alternatively, at least some of the model parameters in the aforementioned modules can be jointly updated.

[0125] In this embodiment, by forming the first training input information with images and / or videos and their corresponding text, and by using a visual decoding model to restore the first training motion potential feature information to a space that can be compared with the target visual information, a backpropagable association can be formed between visual content, text semantics, motion potential feature information and visual supervision.

[0126] This embodiment addresses the problem of training difficulties when there is a lack of direct action lexical supervision for motion latent feature information. It adopts the correspondence between images and / or videos and text, as well as visual reconstruction supervision to train the motion diffusion model, thereby using multimodal visual data to establish a mapping between task conditions and latent motion representations.

[0127] In some embodiments, the first training dataset may include multiple first training samples, wherein the first training input information in at least a portion of the first training samples may include human motion image information and / or human motion video information, and the proportion of the at least a portion of the first training samples in the first training dataset may be greater than or equal to a set proportion.

[0128] The first training input information in at least a portion of the first training samples may include human motion image information and / or human motion video information, which may refer to samples in the human motion image information and / or human motion video information that contain human posture or human posture changing over time. The set proportion may be the minimum proportion set in the training data configuration for the aforementioned first training samples.

[0129] Whether the first training input information includes human motion image information and / or human motion video information can be determined through manual annotation, existing data labels, or human motion recognition results. Then, the first training samples can be filtered or sampled according to a set ratio. When building training batches, stratified sampling can also be used to ensure that the proportion of the first training samples containing human motion image information and / or human motion video information is not less than a set ratio.

[0130] In this embodiment, by setting a minimum proportion of the first training samples in the first training dataset, which include human motion image information and / or human motion video information, the motion diffusion model can continuously receive visual supervision containing human posture and motion changes while learning the correspondence between general image and text or video text.

[0131] This embodiment addresses the problem that an excessive number of static scene samples in the first training data may weaken motion representation learning. It adopts a method that ensures the first training samples include human motion image information and / or human motion video information with the lowest possible proportion, thereby improving the content correlation between the first training dataset and the robot motion generation task.

[0132] In some embodiments, the second training dataset may include first robot posture data obtained by redirecting human motion posture data to robot joints, and second robot posture data obtained by the robot.

[0133] In some embodiments, the robot motion generation model training method may further include: determining training robot posture information based on the first robot posture data and / or the second robot posture data, and using the training robot posture information as at least a part of the second training input information; and using an action encoding module to convert the first robot posture data and / or the second robot posture data into a training continuous action lexicon sequence based on a continuous action lexicon vocabulary.

[0134] The first robot posture data can be robot posture data formed by redirecting human motion posture data to robot joints. The second robot posture data can be robot posture data collected by the robot performing actions. Training robot posture information can characterize the robot's posture state before performing the target action in the second training sample. Training continuous action lexical sequences can be lexical sequences obtained by the action encoding module encoding the robot posture data as the supervision target based on a continuous action lexical vocabulary.

[0135] The first type of robot posture data can be inspected through 3D reconstruction and manually adjusted, while the second type of robot posture data can be cleaned, labeled with actions, and filtered using semantic actions. These two types of robot posture data can be combined into a second training dataset according to a set ratio. The second training dataset can also include visual information and task instruction information corresponding to the robot posture data, ensuring that the second training input information has fields corresponding to the task input information during the robot action generation stage.

[0136] When the same robot pose data is used to generate training robot pose information and training continuous action lexical sequences, it can be divided into input segments and supervision segments according to time sequence. The input segment can be used to determine the training robot pose information before the robot performs the target action, while the supervision segment can be used to generate the training continuous action lexical sequence using the action encoding module. The training robot pose information and the training continuous action lexical sequence thus originate from the same action sample and have a temporal relationship, but the training robot pose information does not include all future action segments that are the prediction target. In another implementation, the training robot pose information may only include one frame of robot pose data at the start of the supervision segment.

[0137] In this embodiment, by simultaneously using the first robot posture data obtained from redirection and the second robot posture data collected by the robot, and by forming the second training input information from the prior posture in the same action sample and forming the training continuous action lexical sequence from the subsequent posture, the input state and the supervised action can maintain a temporal correspondence and reduce the leakage of the supervised target to the input.

[0138] This embodiment addresses the problem of insufficient action coverage or robot realism in training data from a single source by using two types of robot posture data to form a second training dataset and performing temporally ordered input supervision partitioning, thereby taking into account action diversity, robot motion characteristics, and the causal order of training data.

[0139] In one example, each first training sample in the first training dataset includes a video clip and first training text information describing the video content. The video clip includes 16 frames of images. The first training dataset includes 300 first training samples, of which 210 first training samples contain video clips with human motion in their first training input information, accounting for 70%, which is not less than the exemplary set ratio of 2 / 3. The visual encoding module converts the target video into a target visual latent representation, samples training diffusion time steps t from 1 to 50, and adds noise to the target visual latent representation. The action diffusion model determines the first training motion latent feature information based on the noisy latent representation, the first training conditional information, the training unconditional information, and the training diffusion time step information. The visual decoding model decodes the first training motion latent feature information into 16 frames of predicted video. The pixel difference between the predicted video and the corresponding image frames in the target video is used as the first training loss, and the embedding model and action diffusion model are updated according to the first training loss. To train the unconditional inference branch, 10% of the first training conditional information can be replaced with training unconditional information, and the remaining 90% of the first training samples use the first training conditional information.

[0140] In the second training phase of the aforementioned example, the ratio of the first robot pose data to the second robot pose data in the second training dataset is approximately 1:2. A single robot pose data set comprises 36 frames of robot pose data collected at 30 frames per second. Frames 1 to 4 serve as input segments to determine the training robot pose information; frames 5 to 36 serve as supervision segments. The action encoding module, based on a continuous action lexicon, converts the 32-frame supervision segment into 8 basic continuous action lexicons, or into a combination of basic continuous action lexicons and compound continuous action lexicons that can cover the same 32 frames. The action autoregressive model uses a sequence start identifier at the first lexicon position and uses the training continuous action lexicons preceding the corresponding position as supervision prefixes at each subsequent lexicon position to obtain the predicted probability of each lexicon position, and determines the second training loss accordingly.

[0141] In the parameter update process of the aforementioned example, the action diffusion model uses the maximum learning rate. The cosine annealing strategy; the action autoregressive model lowers the learning rate from [previous value] within the first 10% of the total training progress. linearly increase to Then, a cosine annealing strategy is used to reduce the learning rate. An adaptive moment estimation optimizer is used for training, with a batch size of 32. When the training progress reaches 75%, the model parameters of the action diffusion model are stopped from being updated, but the action diffusion model continues to perform forward inference and provide the second training motion latent feature information to the action autoregressive model; the model parameters of the action autoregressive model continue to be updated during the remaining 25% of the training process.

[0142] In some embodiments, the action autoregressive model may include low-rank adaptation parameters.

[0143] In some embodiments, the robot motion generation model training method may further include: receiving post-training data corresponding to a target post-training action, the post-training data including post-training input information and robot posture data corresponding to the target post-training action; converting the robot posture data corresponding to the target post-training action into a post-training continuous action lexicon sequence based on a continuous action lexicon vocabulary, the post-training continuous action lexicon sequence including multiple post-training continuous action lexicons; generating post-training conditional information based on the post-training input information, inputting the post-training conditional information, the training unconditional information, and the training diffusion time step information into the motion diffusion model, and using the motion diffusion model for inference to obtain post-training motion latent feature information; and based on the post-training motion latent feature information... The autoregressive action model is used to generate a first post-training predicted continuous action lexical. For subsequent post-training continuous action lexicals in the post-training continuous action lexical sequence, based on the post-training motion latent feature information and the post-training continuous action lexicals preceding the corresponding subsequent post-training continuous action lexical, the autoregressive action model is used to generate corresponding post-training predicted continuous action lexicals. The post-training loss is determined based on the post-training predicted continuous action lexical and the corresponding post-training continuous action lexical. The robot motion generation model is post-trained based on the post-training loss. During the post-training process, the model parameters of the motion diffusion model and the model parameters of the autoregressive action model, except for the low-rank adaptation parameter, are frozen, and the low-rank adaptation parameter is updated based on the post-training loss.

[0144] The target network layer can be the network layer selected in the action autoregressive model to set the low-rank adaptation parameters. The low-rank adaptation parameters can be trainable parameters appended to the original weight matrix of the target network layer and represented by two low-rank matrices. Post-training data can be data corresponding to the target post-training actions and used to supplement the training. Freezing model parameters means that the model parameters continue to participate in the forward computation, but the backpropagation results do not change the corresponding model parameters.

[0145] The target network layer can include at least one of the following: a query projection layer, a key projection layer, a value projection layer, an output projection layer, or a feedforward network layer within the attention layer. For the original weight matrix W of the target network layer, two trainable low-rank matrices A and B can be set, and the weight matrix W′ used for the forward computation during post-training is determined according to W′ = W + B × A, where the rank of the product of matrix B and matrix A is less than the dimension of the original weight matrix W. During post-training, the original weight matrix W is fixed, and only the low-rank matrices A and B are updated based on the post-training loss.

[0146] Post-training data can include post-training input information and the target post-training action corresponding to the post-training input information. A continuous action lexicon can convert the robot pose data corresponding to the target post-training action into a post-training continuous action lexicon sequence; if the continuous action lexicon is expanded as described later, the conversion can be performed based on the expanded continuous action lexicon. The action diffusion model can generate post-training motion latent feature information based on the post-training input information, and the action autoregressive model can use the post-training motion latent feature information and supervised lexicon prefixes to determine the post-training prediction probability of each lexicon position. The post-training loss can use the same lexicon-level cross-entropy loss as the second training loss. During post-training, the model parameters of the action diffusion model and the model parameters of the action autoregressive model, except for the low-rank adaptation parameters, remain frozen; the backpropagation results only update the low-rank adaptation parameters.

[0147] In this embodiment, by converting the robot posture data corresponding to the target post-training action into a post-training continuous action lexicon sequence based on a continuous action lexicon vocabulary, and generating post-training prediction continuous action lexicons based on the latent feature information of post-training motion and the corresponding historical post-training continuous action lexicons, a supervisory relationship between the target post-training action and the post-training prediction result can be established. By freezing the model parameters of the action diffusion model and the model parameters in the action autoregressive model except for the low-rank adaptation parameters, the parameter changes generated by post-training can be restricted to the low-rank adaptation parameters.

[0148] This embodiment addresses the problem that full-parameter post-training may alter the robot's existing motion generation capabilities and result in a large amount of parameter updates. It employs a method of freezing the original model parameters of the motion diffusion model and the motion autoregressive model, and updating low-rank adaptation parameters based on the post-training loss. This allows the robot's motion generation model to adapt to the target post-trained motion and limits the range of parameter updates during post-training.

[0149] In some embodiments, converting the robot posture data corresponding to the target post-training action into a post-training continuous action lexicon sequence based on a continuous action lexicon vocabulary may include: performing action segment matching on the robot posture data corresponding to the target post-training action based on continuous action lexicons in the continuous action lexicon vocabulary; determining the action length corresponding to the target post-training action segment in response to the presence of a target post-training action segment in the robot posture data corresponding to the target post-training action that is not matched to an existing continuous action lexicon in the continuous action lexicon vocabulary; and determining the action length corresponding to the target post-training action segment in response to the action length being no greater than the continuous action lexicon. The vocabulary already contains the maximum action length corresponding to the continuous action lexicon. New continuous action lexicons are generated based on the target post-training action fragment. In response to the action length corresponding to the target post-training action fragment being greater than the maximum action length, the target post-training action fragment is divided according to the maximum action length, and new continuous action lexicons are generated based on the divided action sub-fragments. The new continuous action lexicons are added to the continuous action lexicon vocabulary to obtain an expanded continuous action lexicon vocabulary. Based on the expanded continuous action lexicon vocabulary, the robot posture data corresponding to the target post-training action is converted into the post-training continuous action lexicon sequence.

[0150] The target post-training action can be a robot action selected for supplementary training. Action fragment matching involves comparing the continuous robot posture data corresponding to the target post-training action with the robot posture data corresponding to existing continuous action lexicons in the continuous action lexicon, and determining the action fragment that can be represented by the existing continuous action lexicons. The target post-training action fragment can be an action fragment that has not been matched with an existing continuous action lexicon. The maximum action length can be the maximum number of posture frames or the maximum duration of the continuous action fragment corresponding to an existing continuous action lexicon. The newly added continuous action lexicon can be a new lexicon built based on unmatched action fragments or segmented action sub-fragments.

[0151] Post-target training actions can include specific robot actions selected for supplementary training. The robot posture data corresponding to the post-target training actions can be obtained through robot acquisition and then manually labeled and adjusted for posture. When expanding the continuous action lexicon, it can start from the beginning position of the robot posture data corresponding to the post-target training actions and match it sequentially with the robot posture data corresponding to existing continuous action lexicons. Action segments that can be matched with existing continuous action lexicons continue to use the corresponding existing continuous action lexicons; action segments that cannot be matched with existing continuous action lexicons are used to generate new continuous action lexicons.

[0152] When the length of the post-target training action segment is no greater than the maximum action length, a new continuous action lexicon can be generated based on the complete post-target training action segment. When the length of the post-target training action segment is greater than the maximum action length, it can be divided into multiple action sub-segments according to the maximum action length in chronological order, and new continuous action lexicons can be generated for each sub-segment. Each new continuous action lexicon can record the corresponding lexicon identifier, robot posture data, and lexicon representation used for lexicon matching in the continuous action lexicon dictionary.

[0153] To enable newly added continuous action lexics to participate in the output when the original parameters of the action autoregressive model are frozen, a fixed lexical representation can be used to limit the lexical output space. In one implementation, the action encoding module generates a fixed lexical representation based on the robot posture data corresponding to the newly added continuous action lexics; the lexical output layer of the action autoregressive model outputs action features, and determines the output probability of each continuous action lexicographical unit by the similarity between the action features and the fixed lexical representations in the continuous action lexicographical unit vocabulary. When a new continuous action lexicographical unit is added to the continuous action lexicographical unit vocabulary, only the fixed lexical representation used for similarity calculation is added; the original parameters of the action autoregressive model that need to be updated during post-training are not added.

[0154] In another implementation, the lexical output layer of the action autoregressive model can pre-set multiple reserved lexical positions. During lexical expansion, a correspondence is established between newly added consecutive action lexical positions and the unused reserved lexical positions. During post-training, the original output parameters corresponding to the reserved lexical positions are fixed, and the action features generated by the action autoregressive model are adjusted through low-rank adaptation parameters so that the corresponding action features can be mapped to the newly added consecutive action lexical positions. Fixed lexical representation and reserved lexical positions are two examples of adapting lexical expansion to the freezing of the original model parameters.

[0155] In this embodiment, by first using existing continuous action lexical units to perform action segment matching on the robot posture data corresponding to the target post-training action, existing matching continuous action lexical units can be reused. By generating new continuous action lexical units only for unmatched action segments and using the maximum action length corresponding to the existing continuous action lexical units as the dividing criterion, the action length corresponding to the new continuous action lexical units can be controlled. By adding the new continuous action lexical units to the continuous action lexical unit vocabulary and re-transforming the robot posture data corresponding to the target post-training action, a post-training continuous action lexical unit sequence containing the new continuous action lexical units can be obtained.

[0156] This embodiment addresses the problem that existing continuous action lexicons cannot fully represent the target post-training actions. It employs existing lexicon matching, unmatched action segment recognition, generation of new lexicons based on the maximum action length, and re-conversion based on the expanded lexicon to expand the types of robot actions that the continuous action lexicon can represent and form a post-training supervision sequence for the target post-training actions.

[0157] In one example, the maximum action length corresponding to existing continuous action lexicons in the continuous action lexicon is 24 frames. The target post-training action corresponds to 180 frames of robot pose data. After action segment matching, one 18-frame unmatched action segment and one 61-frame unmatched action segment are obtained. One new continuous action lexicon is generated based on the 18-frame unmatched action segment; the 61-frame unmatched action segment is divided into three action sub-segments of 24, 24, and 13 frames, and three new continuous action lexicons are generated. The four new continuous action lexicons and their fixed lexicon representations are added to the continuous action lexicon lexicon. The action decoding module can then read the 18, 24, 24, and 13 frames of robot pose data based on the corresponding lexicon identifiers.

[0158] In the aforementioned example, the existing continuous action lexicon contains 15,000 continuous action lexicons. After adding four new continuous action lexicons, the expanded continuous action lexicon contains 15,004 continuous action lexicons. Post-training input information includes task instruction information corresponding to the target post-training action, initial robot posture information, and visual information. The embedding model and action diffusion model generate latent motion feature information for post-training. The action autoregressive model generates predicted probabilities for each lexicon position based on the latent motion feature information and the supervision lexicon prefix, and determines the lexicon-level cross-entropy loss based on the post-training continuous action lexicons. Low-rank adaptation parameters with a rank of 8 are set in the query projection layer and value projection layer of the action autoregressive model. All model parameters of the action diffusion model and the original model parameters of the action autoregressive model remain unchanged; only the low-rank adaptation parameters are updated. After training, the action autoregressive model can select new continuous action lexicons in the output space defined by the expanded continuous action lexicon. The action decoding module restores the new continuous action lexicons to the robot posture data corresponding to the target post-training action.

[0159] See Figure 4 , Figure 4 This is a structural block diagram of a robot provided in an embodiment of this application.

[0160] This application also provides a robot, which includes a control module for executing the steps of any of the robot motion generation methods described above.

[0161] See Figure 5 , Figure 5This is a structural block diagram of a computer device provided in an embodiment of this application.

[0162] This application also provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of any of the above methods.

[0163] The embodiments of this application do not limit the computer device, such as a local computer device, a cloud computer device, a distributed computer device, etc.

[0164] The computer device may include: a memory 110, a processor 120, and a communication interface 130. The memory 110, the processor 120, and the communication interface 130 are connected through internal connection paths.

[0165] The memory 110 is used to store computer programs, which in some implementations may include code for implementing the methods of the embodiments of this application.

[0166] The processor 120 is used to execute the computer program stored in the memory 110 to control the communication interface 130 to receive input data and information, and output operation results and other data.

[0167] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the above methods.

[0168] This application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of any of the above methods.

[0169] It is understood that in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of this application.

[0170] It is understood that the various implementation methods described in this application can be implemented individually or in combination, and this application does not limit them.

[0171] Unless otherwise stated, all technical and scientific terms used in this application have the same meaning as would be understood by one of ordinary skill in the art. The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The singular forms “a,” “an,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0172] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution.

[0173] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the embodiments described above can be referred to the corresponding processes in other embodiments, and will not be repeated here.

[0174] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and other division methods may exist in actual implementation. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections can be through some interfaces; indirect couplings or communication connections between apparatuses or units can be electrical, mechanical, or other forms.

[0175] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for generating robot actions, characterized in that, include: Receive task input information corresponding to the robot motion generation task; Generate condition information based on the task input information; Based on the conditional information, motion latent feature information is generated using an action diffusion model. The motion latent feature information is used to characterize the motion semantics related to the robot action generation task. Based on the motion latent feature information, a continuous action word sequence is generated using an action autoregressive model. In the continuous action word sequence, except for the first continuous action word, the remaining continuous action words to be generated are all generated based on the motion latent feature information and at least some of the already generated continuous action words. Each of the continuous action words corresponds to a continuous action segment composed of multiple temporally continuous robot posture data. Decode the continuous action words in the continuous action word sequence to obtain the robot action information of the corresponding continuous action segments.

2. The robot motion generation method according to claim 1, characterized in that, The task input information includes robot posture information, visual information, and task instruction information, wherein the visual information includes image information and / or video information collected by the robot; The generation of condition information based on the task input information includes: The robot posture information, the visual information, and the task instruction information are encoded respectively to obtain robot posture encoding information, visual encoding information, and task instruction encoding information; The conditional information is generated using an embedding model based on the robot posture encoding information, the visual encoding information, and the task instruction encoding information.

3. The robot motion generation method according to claim 2, characterized in that, The process of encoding the robot posture information, the visual information, and the task instruction information respectively to obtain robot posture encoding information, visual encoding information, and task instruction encoding information includes: Using an action encoding module, the robot posture information is converted into a robot posture tensor based on a continuous action lexicon, and the robot posture tensor is used as the robot posture encoding information. The visual information is converted into a visual tensor using a visual encoding module, and the visual tensor is used as the visual encoded information. The task instruction information is segmented into words, and the segmentation results are converted into a task instruction tensor based on the text dictionary. The task instruction tensor is then used as the task instruction encoding information.

4. The robot motion generation method according to claim 1, characterized in that, The step of generating motion latent feature information using an action diffusion model based on the conditional information includes: Using the motion diffusion model, based on the conditional information and unconditional information, reasoning is performed at multiple diffusion time steps corresponding to the diffusion time step information to obtain the motion potential feature information. The unconditional information is a feature that is not related to a specific robot motion generation task, and the diffusion time step information is used to identify the current reasoning stage of the motion diffusion model.

5. The robot motion generation method according to claim 1, characterized in that, The step of generating a continuous action lexical sequence using an autoregressive motion model based on the motion latent feature information, and decoding the continuous action lexical segments in the continuous action lexical sequence to obtain robot motion information corresponding to the continuous action fragments, includes: Based on the motion latent feature information, the first continuous action word is generated using the action autoregressive model, and the first continuous action word is added to the continuous action word sequence; For the i-th continuous action word to be generated, the action autoregressive model is used to generate it based on the motion latent feature information and the 1, 2, ..., or i-1 consecutive action words preceding the i-th continuous action word. The generated i-th continuous action word is then added to the continuous action word sequence in the order of generation, where i is an integer greater than 1. The process of generating the continuous action word other than the 1-th continuous action word overlaps at least partially in time with the process of decoding the corresponding preceding continuous action word to obtain the robot action information of the corresponding continuous action segment.

6. The robot motion generation method according to claim 1, characterized in that, The lexical output space of the action autoregressive model is limited by the continuous action lexical dictionary; The continuous action lexicon includes basic continuous action lexicons and compound continuous action lexicons. The basic continuous action lexicons correspond to continuous action segments composed of a set number of temporally continuous robot posture data. The compound continuous action lexicons are obtained by combining robot posture data corresponding to at least two basic continuous action lexicons in chronological order.

7. The robot motion generation method according to claim 6, characterized in that, The continuous action word metavocabulary is constructed in the following way: Receive multiple robot pose data from a vocabulary dataset, the vocabulary dataset including first robot pose data obtained by redirecting human motion pose data to robot joints, and second robot pose data obtained by collecting data from the robot. The frame rate of the multiple robot posture data is converted into a target frame rate, and the robot posture data after frame rate conversion is divided into multiple candidate basic continuous action words according to a set number of frames. Elimination and merging are performed on the candidate basic continuous action lexical units to obtain and store the basic continuous action lexical units; While retaining the basic continuous action lexical units, at least some of the basic continuous action lexical units are spliced ​​and merged to obtain compound continuous action lexical units; The continuous action lexicon is constructed based on the basic continuous action lexicon and the compound continuous action lexicon.

8. The robot motion generation method according to claim 7, characterized in that, The elimination and merging of the candidate basic continuous action lexical units to obtain basic continuous action lexical units includes: For any two candidate basic continuous action words, determine the body posture distance based on the robot posture data in the two candidate basic continuous action words, and determine the hand posture distance based on the robot posture data in the two candidate basic continuous action words. The body posture distance and the hand posture distance are weighted and summed according to the set body posture weight and hand posture weight respectively to obtain the word distance between two candidate basic continuous action words. Based on the lexical distance, multiple candidate basic continuous action lexical units are clustered, and the number of lexical units selected for each cluster category is determined based on the number of candidate basic continuous action lexical units contained in each cluster category. Based on the number of lexical selections corresponding to each cluster category, and the source and timing of robot posture data in the candidate basic continuous action lexicals contained in each cluster category, the candidate basic continuous action lexicals contained in each cluster category are filtered to obtain the basic continuous action lexicals.

9. The robot motion generation method according to claim 8, characterized in that, The process of concatenating and merging at least a portion of the basic continuous action lexical units to obtain compound continuous action lexical units includes: In response to the existence of robot posture data corresponding to basic continuous action words originating from the same robot trajectory data and robot posture data corresponding to basic continuous action words being temporally continuous, the splicing priority parameters of each basic continuous action word are determined based on the number of candidate basic continuous action words contained in the cluster category to which each basic continuous action word belongs. The starting word for splicing is determined according to the splicing priority parameters from largest to smallest; Starting from the concatenation start word, based on the source and timing of the robot posture data corresponding to each of the basic continuous action words, at least one candidate word sequence is determined. Each candidate word sequence is composed of multiple basic continuous action words that originate from the same robot posture data and are in a continuous timing sequence. The target word sequence containing the most basic continuous action words is determined from the candidate word sequence, and then concatenated in chronological order to obtain the compound continuous action word. In response to any basic continuous action lexical being used to generate the compound continuous action lexical, the concatenation priority parameter of the basic continuous action lexical is reduced; Based on the updated concatenation priority parameters of each basic continuous action lexical, the concatenation starting lexical is re-determined, and the concatenation continues.

10. The robot motion generation method according to claim 8, characterized in that, The process of concatenating and merging at least a portion of the basic continuous action lexical units to obtain compound continuous action lexical units includes: For the basic continuous action words to be concatenated, in response to the absence of robot posture data corresponding to the basic continuous action words originating from the same robot trajectory data or the discontinuous timing of the robot posture data corresponding to the basic continuous action words, the word distance between each basic continuous action word and the basic continuous action words to be concatenated is determined from the basic continuous action words in other cluster categories outside the cluster category to which the basic continuous action words to be concatenated belong, according to the word distance determination method, and basic continuous action words with word distance less than or equal to a set distance threshold are selected as subsequent basic continuous action words; Interpolation processing is performed on the robot posture data corresponding to the basic continuous action lexical to be spliced ​​and the subsequent basic continuous action lexical to obtain the transition robot posture data between the continuous action segment corresponding to the basic continuous action lexical to be spliced ​​and the continuous action segment corresponding to the subsequent basic continuous action lexical. The robot posture data corresponding to the basic continuous action lexical to be concatenated, the transition robot posture data, and the robot posture data corresponding to the subsequent basic continuous action lexical are concatenated to obtain a composite continuous action lexical.

11. A method for training a robot motion generation model, characterized in that, The robot motion generation model includes a motion diffusion model and a motion autoregressive model, and the training method for the robot motion generation model includes: Receive a first training sample from a first training dataset, wherein the first training sample includes first training input information and first training supervision information; First training condition information is generated based on the first training input information. The first training condition information, training uncondition information and training diffusion time step information are input into the action diffusion model. The action diffusion model is used to perform inference to obtain the first training motion potential feature information. First training prediction information is generated based on the first training motion potential feature information, and a first training loss is determined based on the first training prediction information and the first training supervision information. The action diffusion model is then trained based on the first training loss. Receive a second training sample from the second training dataset. The second training sample includes second training input information and a training continuous action word sequence corresponding to the second training input information. Each training continuous action word corresponds to a continuous action segment composed of multiple temporally continuous robot posture data. Based on the second training input information, second training condition information is generated. The second training condition information, training uncondition information, and training diffusion time step information are input into the action diffusion model. The action diffusion model is used to perform inference to obtain the second training motion potential feature information. Based on the second training motion latent feature information, the predicted continuous action word sequence is generated using the action autoregressive model; A second training loss is determined based on the predicted continuous action word sequence and the trained continuous action word sequence, and the action diffusion model and the action autoregressive model are trained based on the second training loss. During the training process based on the second training loss, the model parameters of the action diffusion model are updated according to the first learning rate scheduling strategy, and the model parameters of the action autoregressive model are updated according to the second learning rate scheduling strategy. The maximum learning rate corresponding to the action diffusion model is less than the maximum learning rate corresponding to the action autoregressive model.

12. The robot motion generation model training method according to claim 11, characterized in that, The first training dataset includes multiple first training samples, wherein at least some of the first training samples contain first training input information including human motion image information and / or human motion video information, and the proportion of the at least some of the first training samples in the first training dataset is greater than or equal to a set proportion.

13. The robot motion generation model training method according to claim 11, characterized in that, The action autoregressive model includes low-rank adaptation parameters; The robot motion generation model training method also includes: Receive post-training data corresponding to the target post-training action, wherein the post-training data includes post-training input information and robot posture data corresponding to the target post-training action; Based on the continuous action lexicon, the robot posture data corresponding to the target post-training action is converted into a post-training continuous action lexicon sequence, which includes multiple post-training continuous action lexicons. Based on the post-training input information, post-training condition information is generated. The post-training condition information, the unconditional training information, and the training diffusion time step information are input into the action diffusion model. The action diffusion model is used to perform inference to obtain the post-training motion potential feature information. Based on the post-training motion latent feature information, the first post-training predicted continuous action word is generated using the action autoregressive model. For subsequent post-training continuous action words in the post-training continuous action word sequence, the corresponding post-training predicted continuous action word is generated using the action autoregressive model based on the post-training motion latent feature information and the post-training continuous action words before the corresponding subsequent post-training continuous action words. The post-training loss is determined based on the post-training predicted continuous action lexical and the corresponding post-training continuous action lexical. The robot motion generation model is post-trained based on the post-training loss, wherein, during the post-training process, the model parameters of the motion diffusion model and the model parameters of the motion autoregressive model, except for the low-rank adaptation parameters, are frozen, and the low-rank adaptation parameters are updated based on the post-training loss.

14. The robot motion generation model training method according to claim 13, characterized in that, The step of converting the robot pose data corresponding to the target post-training action into a post-training continuous action lexicon sequence based on a continuous action lexicon ... Based on the continuous action lexicon in the continuous action lexicon, the robot posture data corresponding to the target post-training action is matched with action segments. In response to the presence of a target post-training action fragment in the robot posture data corresponding to the target post-training action that is not matched to an existing continuous action lexicon in the continuous action lexicon, the action length corresponding to the target post-training action fragment is determined. In response to the condition that the action length corresponding to the post-target training action fragment is not greater than the maximum action length corresponding to the existing continuous action lexicon in the continuous action lexicon vocabulary, a new continuous action lexicon is generated based on the post-target training action fragment. In response to the target being trained, the action length of the action segment being greater than the maximum action length, the target being trained, the action segment is divided according to the maximum action length, and new continuous action lexical units are generated based on the divided action sub-segments. The newly added continuous action lexicon is added to the continuous action lexicon vocabulary to obtain the expanded continuous action lexicon vocabulary; Based on the expanded continuous action lexicon, the robot posture data corresponding to the target post-training action is converted into the post-training continuous action lexicon sequence.

15. A computer device, characterized in that, The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the method according to any one of claims 1 to 14.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1 to 14.

17. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1 to 14.

Images