Action generation method and device based on multi-modal pre-training, robot and medium

Abstract

The invention relates to the technical field of artificial intelligence, can be applied to the field of medical health and the field of financial transaction, discloses an action generation method and device based on multi-modal pre-training, a robot and a medium, and is applied to a high-frequency action generation scene of an intelligent surgical robot or an intelligent customer service and wealth management scene. The method comprises the following steps: acquiring a language instruction, a visual image and robot body sensing data; performing multi-modal feature alignment and cross-modal feature fusion on the language instruction, the visual image and the robot body sensing data through a pre-trained visual language model to generate a fused joint feature vector; generating a target continuous control instruction based on the fused joint features through a pre-trained target action model by adopting a flow matching technology; and generating continuous actions of the robot based on the target continuous control instruction. According to the method, the robot action generation efficiency and the cross-platform adaptability are improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology and can be applied to the fields of medical health and financial transactions, and in particular to a method, device, robot and medium for motion generation based on multimodal pre-training. Background Art

[0002] In the fields of robot learning and embodied intelligence, achieving universal and transferable intelligent control has always been a core challenge. Robotic motion generation methods can be applied to healthcare and financial transactions. For example, they can be used to generate high-frequency motions for intelligent surgical robots, or in intelligent customer service and wealth management scenarios.

[0003] Existing robot motion generation methods mainly face three key bottlenecks: First, the diversity of robot hardware platforms leads to significant differences in motion space and perception modalities. Existing models are usually designed for a single form, making it difficult to achieve cross-platform knowledge transfer. Each time a new robot is deployed, it needs to be retrained, which greatly increases development costs. Secondly, complex and dexterous tasks require high-frequency and continuous motion control, and existing methods based on autoregressive discretization are limited by the frame-by-frame prediction mechanism and cannot meet real-time requirements; although the diffusion model can generate continuous motion, it lacks deep integration with the vision-language model, resulting in insufficient synergy between semantic understanding and motion generation, making the motion generation efficiency low. Finally, the problem of data efficiency is particularly prominent: the cost of collecting real robot data is extremely high, and the cross-platform data is highly heterogeneous. Traditional pre-training methods find it difficult to effectively integrate multi-source data, resulting in the need for a large amount of labeled data for fine-tuning the model on new tasks or new forms. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to propose a method, device, robot and medium for motion generation based on multimodal pre-training to improve the efficiency and cross-platform adaptability of robot motion generation.

[0005] In order to solve the above technical problems, the embodiment of the present application provides an action generation method based on multimodal pre-training, comprising:

[0006] Obtain language instructions, visual images, and robot proprioception data;

[0007] Performing multimodal feature alignment and cross-modal feature fusion on the language instructions, the visual image, and the robot proprioception data through a pre-trained visual language model to generate a fused joint feature vector;

[0008] Generate target continuous control instructions based on the fused joint features by using a stream matching technology through a pre-trained target action model;

[0009] Continuous actions of the robot are generated based on the target continuous control instructions.

[0010] In order to solve the above technical problems, the embodiment of the present application provides an action generation device based on multimodal pre-training, comprising:

[0011] Data acquisition module, used to obtain language instructions, visual images and robot proprioception data;

[0012] a feature fusion module, configured to perform multimodal feature alignment and cross-modal feature fusion on the language instructions, the visual image, and the robot proprioception data using a pre-trained visual language model to generate a fused joint feature vector;

[0013] An instruction generation module is used to generate target continuous control instructions based on the fused joint features by using a stream matching technology through a pre-trained target action model;

[0014] The action generation module is used to generate continuous actions of the robot based on the target continuous control instructions.

[0015] To solve the above technical problems, a technical solution adopted by the present invention is: to provide a robot, including one or more processors; a memory for storing one or more programs, so that the one or more processors can implement any one of the above-mentioned motion generation methods based on multimodal pre-training.

[0016] In order to solve the above technical problems, a technical solution adopted by the present invention is: a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements any one of the above-mentioned action generation methods based on multimodal pre-training.

[0017] The embodiments of the present invention provide a method, device, robot and medium for motion generation based on multimodal pre-training. The method comprises: obtaining language instructions, visual images and robot proprioception data; performing multimodal feature alignment and cross-modal feature fusion on the language instructions, visual images and robot proprioception data through a pre-trained visual language model to generate a fused joint feature vector; using a pre-trained target motion model to generate a target continuous control instruction based on the fused joint feature using a flow matching technique; and generating a continuous motion of the robot based on the target continuous control instruction. The embodiments of the present invention achieve cross-platform high-frequency continuous motion generation through multimodal feature alignment and flow matching technology, solving the control migration difficulty problem caused by the heterogeneity of robot hardware. At the same time, the cross-platform pre-training mechanism effectively improves data utilization efficiency, thereby improving the efficiency of robot motion generation and cross-platform adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the solutions in this application, a brief introduction will be given below to the drawings required for use in the description of the embodiments of this application. Obviously, the drawings described below are some embodiments of this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0019] Figure 1 1 is a schematic diagram of an application environment of a method for generating motion based on multimodal pre-training in one embodiment of the present invention;

[0020] Figure 2 This is a flowchart of the implementation process of the action generation method based on multimodal pre-training provided in an embodiment of the present application;

[0021] Figure 3 yes Figure 2 A schematic flow chart of a specific implementation of step S2;

[0022] Figure 4 yes Figure 3 A schematic flow chart of a specific implementation of step S21;

[0023] Figure 5 yes Figure 2 A schematic flow chart of a specific implementation of step S3;

[0024] Figure 6 yes Figure 2 A schematic flow chart of a specific implementation method before step S2;

[0025] Figure 7 yes Figure 2 A schematic flow chart of a specific implementation method before step S3;

[0026] Figure 8 Schematic diagram of a motion generation device based on multimodal pre-training provided in an embodiment of the present application;

[0027] Figure 9 Schematic diagram of the robot provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] Unless otherwise defined, all technical and scientific terms used herein have the same meanings as commonly understood by those skilled in the art to which this application belongs. The terms used in the specification of the application are for the purpose of describing specific embodiments only and are not intended to limit this application. The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of this application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0029] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0030] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0031] The present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0032] It should be noted that the motion generation method based on multimodal pre-training provided in the embodiments of the present application is generally performed by a robot. Accordingly, the motion generation device based on multimodal pre-training is generally configured in the robot.

[0033] The action generation method based on multimodal pre-training provided by the embodiment of the present invention can be applied in Figure 1 In an application environment, the client communicates with the robot through the network. The server can receive voice commands from the client and obtain visual images and robot proprioception data; then generate continuous robot actions based on the language commands, visual images, and robot proprioception data.

[0034] The action generation method based on multimodal pre-training provided in the embodiment of the present application can be applied to high-frequency action generation scenarios of intelligent surgical robots, or to intelligent customer service and wealth management scenarios.

[0035] In existing technologies, the field of robotics control has long faced difficulties in cross-platform migration due to the diversity of hardware platforms. Robots of different morphologies have different motion spaces and perception modalities, and traditional methods require retraining models for specific hardware. The generation of high-frequency continuous motion is limited by the autoregressive prediction mechanism, which cannot meet the real-time requirements of complex tasks. Multimodal data fusion is inefficient, and the acquisition of real robot data is expensive and highly heterogeneous, which restricts the model's adaptability to new scenarios. For example, when a dual-arm collaborative robot performs precision assembly tasks, it needs to simultaneously process visual environment information, language commands, and body joint states. Existing methods make it difficult to achieve efficient coordination of multi-source data, resulting in delayed motion generation and trajectory jitter.

[0036] To address the above issues, a unified feature representation space needs to be constructed to eliminate hardware differences in order to address cross-platform migration challenges; to address high-frequency control requirements, a breakthrough in discrete prediction mechanisms is needed to achieve continuous trajectory generation; and to address data efficiency bottlenecks, a multimodal pre-training framework needs to be developed to improve feature generalization capabilities. By analyzing the modal complementarity in robot control, it is found that language instructions provide task semantics, visual images capture environmental states, and proprioception reflects real-time physical constraints. Unifying the encoding of the three can enhance state perception integrity, but the problem of inconsistent feature scales needs to be addressed. A continuous action generation model based on flow matching theory can bypass the discretization step and directly output high-frequency control signals, but it needs to be deeply coupled with multimodal features to ensure consistency in action semantics. Therefore, this application proposes an action generation method based on multimodal pre-training, including: obtaining language instructions, visual images, and robot proprioception data; aligning the multimodal data with features and fusing cross-modal features to generate a joint feature vector through a pre-trained visual language model; generating high-frequency continuous control instructions based on the joint features using flow matching technology through a target action model; and generating continuous robot actions based on the control instructions. This application realizes the cross-platform migration capability of robot motion generation, adapting to robots of different forms without retraining the model; improving the real-time and smoothness of high-frequency continuous motion generation to meet the requirements of precision operation tasks; enhancing the efficiency of multimodal data fusion and reducing the cost of real robot data collection and annotation.

[0037] See also Figure 2 , Figure 2 A specific implementation of the action generation method based on multimodal pre-training is shown.

[0038] It should be noted that the method of the present invention is not limited to the method of Figure 2 The process sequence shown is limited to the following steps:

[0039] S1: Acquire language instructions, visual images, and robot proprioception data.

[0040] Specifically, language instructions include natural language task descriptions (e.g., "fold clothes"). Visual data includes RGB or depth images. Robot proprioception data includes joint angles, end-effector pose, and sensor data (e.g., IMU, torque).

[0041] S2: Perform multimodal feature alignment and cross-modal feature fusion on the language instructions, the visual image, and the robot proprioception data through a pre-trained visual language model to generate a fused joint feature vector.

[0042] Specifically, the features of language instructions, visual images and robot body perception data are extracted respectively through a pre-trained visual language model to obtain multimodal features, and the multimodal features are feature aligned to obtain aligned multimodal features; the aligned multimodal features are feature spliced ​​to generate spliced ​​features; the spliced ​​features are nonlinearly fused through a multi-layer perceptron to obtain a fused joint feature vector.

[0043] A pre-trained visual language model is a deep learning model trained through cross-modal contrastive learning. It can be implemented using structures such as CLIP and ALBEF. Its purpose is to establish a joint representation space for language and visual modalities. The pre-trained visual language model includes a pre-trained visual encoder, language model, multi-layer perception, and feature fusion model.

[0044] See also Figure 3 , Figure 3 A specific implementation of step S2 is shown, which is described in detail as follows:

[0045] S21: extracting features of the language instructions, the visual images, and the robot proprioception data respectively through the pre-trained visual language model to obtain multimodal features, and aligning the multimodal features to obtain aligned multimodal features.

[0046] S22: performing feature splicing on the aligned multimodal features to generate spliced ​​features.

[0047] S23: Perform nonlinear fusion on the concatenated features through a multi-layer perceptron to obtain a fused joint feature vector.

[0048] Specifically, during the feature extraction phase, the visual encoder uses a convolutional neural network to extract spatial semantic information from the image, such as object shape and positional relationships. The language model uses a Transformer architecture to parse the deep semantics of the instruction, such as action goals and operation sequence. Proper perception data is encoded into physical state vectors, such as joint angles and end-point velocity, using a fully connected network. Subsequently, features from each modality are linearly projected into a shared embedding space, and cross-modal alignment is achieved by minimizing the contrastive loss function, bringing vectors of the same semantic concept closer together across different modalities. A feature concatenation operation integrates these aligned vectors along the channel dimension to form a high-dimensional joint feature. For example, 512-dimensional image features, 256-dimensional language features, and 128-dimensional perceptual features are concatenated into an 896-dimensional vector. Finally, a multi-layer perceptron performs a nonlinear transformation on this vector using two fully connected layers, such as using a ReLU activation function and batch normalization layers, to eliminate redundant information and enhance cross-modal correlations. The output is a fused feature vector of 1024 dimensions. Furthermore, the concatenated features can be fused using a cross-attention mechanism to produce a fused joint feature vector. This application introduces a shared embedding space mapping and nonlinear fusion mechanism. For example, by using contrastive learning to align heterogeneous features into a unified space, and then using a multi-layer perceptron for high-level interaction, it effectively addresses the semantic gap problem caused by different encoding methods for cross-modal data. Compared to shallow alignment methods that only use linear projections, this application can establish more sophisticated cross-modal associations, such as locating objects described by language instructions in visual features, while combining ontological perception data to determine accessibility.

[0049] Among them, multimodal feature alignment refers to mapping the feature vectors of different modalities to the same metric space. Specifically, this can be achieved using a linear projection matrix or an attention mechanism to eliminate the distribution differences between different data sources. Feature splicing refers to connecting the feature vectors of different modalities in the channel dimension. Specifically, this can be achieved by splicing operations along the feature dimension to preserve the original information of each modality. The multilayer perceptron refers to a neural network module composed of fully connected layers and nonlinear activation functions. Specifically, it can be implemented using the ReLU activation function with a residual connection structure. Its function is to achieve nonlinear interaction of cross-modal features.

[0050] See also Figure 4 , Figure 4 A specific implementation of step S21 is shown, which is described in detail as follows:

[0051] S211: Extract features from the visual image through a pre-trained visual encoder to obtain image features.

[0052] S212: Encode the language instruction using a pre-trained language model to generate semantic features.

[0053] S213: Encode the robot proprioception data through a pre-trained multi-layer perceptron to generate perception features.

[0054] S214: Mapping the image features, semantic features, and perceptual features to a shared embedding space to perform feature alignment on the image features, semantic features, and perceptual feature mappings to obtain the aligned multimodal features.

[0055] Specifically, the visual encoder first performs a convolution operation on the input RGB image to extract visual features that include object shape and spatial relationships, for example, using the fifth residual block of ResNet-50 to output a 2048-dimensional feature vector. The language model performs word segmentation and position encoding on natural language instructions, and then generates context-sensitive semantic vectors through a self-attention mechanism. A multi-layer perceptron normalizes the joint angle and torque signals in the robot's proprioceptive data, and then gradually increases the dimensionality through fully connected layers, for example, encoding 7-dimensional proprioceptive data into 256-dimensional features. Finally, a shared linear projection matrix is ​​used to uniformly map the three types of features to a 256-dimensional space. Within this space, the cosine similarity between the modalities is calculated for alignment, eliminating feature distribution shifts caused by sensor differences. This application effectively solves the feature alignment problem of multimodal robot perception data caused by differences in sensor type, data dimension, and sampling frequency, and achieves a deep fusion of visual, language, and proprioceptive information in cross-platform control tasks. This method enables robots of different morphologies to share a unified feature representation space. For example, the semantic features acquired by a humanoid robot can be directly matched with the visual features of an industrial robotic arm, significantly improving the effectiveness of cross-platform transfer learning.

[0056] Among them, the pre-trained visual encoder refers to a feature extraction network trained based on a large-scale image dataset. Specifically, it can be implemented using the ResNet-50 or Vision Transformer structure. Its function is to extract spatial feature representations with high-level semantic information from visual images. The pre-trained language model refers to a semantic encoder trained based on a text corpus. Specifically, it can be implemented using the BERT or RoBERTa structure. Its function is to convert natural language instructions into structured semantic vectors. The pre-trained multi-layer perceptron refers to an encoding network composed of fully connected layers and nonlinear activation functions. Specifically, it can be implemented using a three-layer 512-dimensional hidden layer structure. Its function is to encode low-dimensional time series data such as robot proprioception into high-dimensional feature vectors. The shared embedding space refers to a feature projection space with a unified dimension. Specifically, it can be implemented by using a linear transformation matrix to map the features of each modality to a 256-dimensional space. Its function is to establish semantic associations by forcing different modalities within the unified space.

[0057] S3: Generate target continuous control instructions based on the fused joint features using a stream matching technique through a pre-trained target action model.

[0058] Flow matching technology constructs continuous motion trajectories based on probabilistic flow fields, avoiding the traditional frame-discretization method. Conditional neural differential equations can be used to model time-continuous control signals. The target motion model includes a pre-trained visual language model and a flow matching model.

[0059] See also Figure 5 , Figure 5 A specific implementation of step S3 is shown, which is described in detail as follows:

[0060] S31: Obtain the robot form identifier.

[0061] S32: Generate a continuous action trajectory based on the fused joint features using a conditional probability flow field.

[0062] S33: Adjusting the input and output dimensions of the target motion model according to the robot morphology identifier to obtain an adjusted target motion model.

[0063] S34: Converting the continuous motion trajectory into the target continuous control instruction through the adjusted target motion model.

[0064] Among them, the robot morphology identification refers to the structured code used to characterize the robot hardware configuration, which can be implemented by a combination of hash values ​​of the number of joint degrees of freedom, the type of end effector, and the topology of the kinematic chain. It is used to provide the model with metadata of the robot morphology to support the dynamic adjustment of the input and output dimensions. Among them, the conditional probability flow field refers to a continuous motion trajectory generation module built based on flow matching technology, which can be implemented by a vector field modeling method controlled by ordinary differential equations. By using multimodal joint features as initial conditions to drive the trajectory evolution process, a smooth and continuous action sequence is generated. Among them, adjusting the input and output dimensions refers to dynamically configuring the neural network structure parameters according to the morphology identification. Specifically, it can be implemented by a pluggable dimensional adaptation layer. The corresponding fully connected layer weight matrix is ​​mapped through a preset robot morphology configuration table, so that a single model can be compatible with the control instruction generation requirements of robots with different degrees of freedom.

[0065] Specifically, after obtaining the robot morphology identification, the multimodal joint features are mapped into a high-dimensional continuous motion trajectory through the conditional probability flow field. The trajectory has differential continuity in the time dimension. Subsequently, based on the joint degrees of freedom and actuator type parameters recorded in the morphology identification, the target motion model automatically loads the corresponding input and output adaptation layer to convert the continuous trajectory into high-frequency control instructions executable by the specific robot platform. This dynamic dimensionality adjustment mechanism allows the model to adapt to the 14-degree-of-freedom control requirements of the dual-arm collaborative robot or the 6-degree-of-freedom control requirements of the mobile operating platform without retraining. At the same time, the continuous probability modeling capability of the flow matching technology ensures the generation frequency of 50 control instructions per second. This application solves the problem of motion space differences caused by the diversity of robot hardware, so that the same pre-trained model can be directly applied to robot platforms of different forms without retraining. At the same time, the continuous motion trajectory generated by the flow matching technology meets the high-frequency control requirements and overcomes the real-time shortcomings of the traditional discretization method.

[0066] S4: Generate continuous actions of the robot based on the target continuous control instructions.

[0067] Specifically, the robot executes target continuous control instructions to generate continuous movements of the robot. The movements are high-frequency continuous movements and can be applied to different application scenarios.

[0068] See also Figure 6 , Figure 6 A specific implementation method before step S2 is shown, which is described in detail as follows:

[0069] S2A: Obtain cross-platform pre-training data and pre-process the cross-platform pre-training data to generate pre-processed data. The cross-platform pre-training data includes image data, initial language instructions, robot state data, and action labels.

[0070] S2B: Extracting image features from the image data through a visual encoder.

[0071] S2C: The initial language instructions are encoded into initial semantic features through a language encoder, and the robot state data is converted into initial perceptual features by a multi-layer perceptron.

[0072] S2D: A contrast loss function is used to perform pairwise contrast loss calculation based on the image features, the initial semantic features, and the initial perceptual features to generate a contrast loss value.

[0073] S2E: Adjusting parameters of the visual language model based on the contrast loss value, and adjusting the weight of the contrast loss function, so as to retrain the visual language model and generate the pre-trained visual language model.

[0074] Specifically, during the cross-platform data collection phase, a heterogeneous dataset can be obtained, including videos of dual-arm collaborative robot operations, mobile robot navigation images, and robotic arm assembly task videos. During preprocessing, RGB images from different viewpoints are uniformly scaled to a 224×224 resolution, joint angle data with different degrees of freedom are normalized to the range [-1, 1], and the timestamps of language commands and action sequences are aligned using a sliding window. The visual encoder can use a ResNet-50 network to extract image features, the language encoder uses the BERT model to generate semantic features, and the perceptual encoder uses a three-layer fully connected network to process the body state data. During the contrastive learning phase, when calculating the similarity matrix of image-semantic feature pairs, a temperature coefficient of 0.07 can be set. Similarity scores for positive and negative sample pairs are obtained through matrix multiplication. During weight adjustment, the combined weights of the contrastive loss function are optimized through backpropagation. For example, the initial weights for the image-semantic contrastive loss are set to 0.4, the image-perceptual contrast weight is set to 0.3, and the semantic-perceptual contrast weight is set to 0.3. These weights are automatically adjusted during training. This application effectively solves the problem of feature space fragmentation caused by cross-platform data heterogeneity, making the pre-trained model compatible with the sensor input formats of different robots. Through the multimodal cross-contrast learning mechanism, the alignment accuracy of visual, language, and proprioceptive features in the shared space is significantly improved, so that only a small amount of labeled data is needed to achieve the target performance when fine-tuning new tasks. The dynamic weight adjustment mechanism avoids the subjectivity of manually setting loss weights, enabling the model to adapt to the differences in modal importance in different task scenarios, such as automatically enhancing the contrast strength of image modalities in tasks requiring high-precision visual positioning.

[0075] Among them, cross-platform pre-training data refers to a heterogeneous data set from different robot platforms. Specifically, it can be implemented by using sensor data from various forms including biped robots, robotic arms, and wheeled mobile robots. It is used to solve the problem of insufficient model generalization ability caused by single platform data. Preprocessing refers to the format unification and normalization of the original multi-source data. Specifically, it can be implemented by time series alignment, image resolution standardization, and joint angle normalization methods to eliminate the structural differences between data from different platforms. The contrast loss function refers to the loss function that calculates the similarity between different modal features. Specifically, it can be implemented by the InfoNCE loss function with cosine similarity and temperature coefficient adjustment, forcing different modal features to be implicitly aligned in a shared space. Weight adjustment refers to the dynamic balance of the contribution ratio of different modal contrast losses. Specifically, it can be implemented by using learnable attention weight parameters to avoid a single modal feature dominating the training process.

[0076] See also Figure 7 , Figure 7 A specific implementation method before step S3 is shown, which is described in detail as follows:

[0077] S3A: Obtain an initial robot shape identification and a continuous action sequence with a timestamp, and obtain an initial fused joint feature output by the pre-trained visual language model.

[0078] S3B: Constructing an initial conditional probability flow field, and adjusting the output dimension of the initial conditional probability flow field based on the initial robot form identifier to obtain an adjusted conditional probability flow field.

[0079] S3C: Generate initial continuous actions based on the initial fused joint features through the adjusted conditional probability flow field.

[0080] S3D: Calculate a flow matching loss value based on the initial connection action and the continuous action sequence, and train a flow matching model based on the flow matching loss value to obtain a target flow matching model.

[0081] S3E: Constructing an initial action model based on the target stream matching model and the pre-trained visual language model, and fine-tuning the initial action model to generate the target action model.

[0082] Specifically, the initial robot morphology identifier is input into the conditional probability flow field to dynamically adjust the output dimensionality. For example, the 14-dimensional action space of a dual-arm system is dimensionally matched with the 7-dimensional action space of a single manipulator, enabling the same model to adapt to the control requirements of robots with different degrees of freedom. A continuous action sequence with timestamps is used as a supervisory signal in the flow matching loss calculation. By minimizing the cumulative error in the time dimension between the generated and true trajectories, for example, using a time-weighted mean squared error function, the temporal coherence of the joint motions during the action generation process is ensured. The initial fused joint features output by a pre-trained visual language model serve as conditional input to drive the flow matching model, directly translating the semantic understanding results into motion control commands, such as converting an object grasping intention into a continuous motion trajectory of the manipulator end-point. The model fine-tuning phase utilizes transfer learning methods based on semantic action information, such as generating action templates by parsing task instructions, allowing for rapid adaptation to new tasks using a small amount of real-world data.

[0083] The initial robot morphology identifier refers to an encoding vector used to identify the robot's physical structural features. Specifically, it can be generated using a combination of parameters such as joint degrees of freedom, link length, and end-effector type. Its purpose is to provide a unified conditional input interface for robots of different morphologies. A timestamped continuous motion sequence refers to a sequence of robot joint angles or end-effector poses recorded at a fixed sampling frequency, such as six-dimensional pose data at 50 frames per second. Its purpose is to provide a time-continuous motion trajectory supervision signal for flow matching training. The conditional probability flow field refers to a vector field defined in state space. Specifically, it can be parameterized using a neural network and is used to describe the probability flow mapping relationship from the initial state to the target state. Its purpose is to map high-dimensional features into continuous motion trajectories. The flow matching loss value is a loss function calculated by comparing the distribution difference between the generated motion trajectory and the true trajectory. Specifically, it can be implemented using the Wasserstein distance or KL divergence. Its purpose is to optimize the motion generation continuity of the flow matching model.

[0084] Furthermore, step 3E includes: constructing an initial action model based on the target flow matching model and the pre-trained visual language model, and obtaining robot task data; parsing the robot task data into subtask sequences, and mapping the subtask sequences to preset action templates to obtain semantic action information; fine-tuning the initial action model based on the semantic action information to obtain the target action model.

[0085] The target flow matching model refers to a neural network architecture that generates continuous motion trajectories through flow matching technology. This can be achieved using a dynamic system modeling approach based on conditional probabilistic flow fields. Its purpose is to convert high-level semantic features into continuous control instructions in physical space. Robot task data refers to the interaction records between task objectives and environmental states. It can be collected through instructional programming or natural language descriptions and serves as the raw input for task decomposition. Subtask sequences refer to chains of atomic operations formed through structured parsing of complex tasks. These can be generated using semantic segmentation algorithms or grammatical parse trees to achieve hierarchical decoupling of task objectives. Preset action templates refer to a library of predefined standardized action semantic units, stored as parameterized motion primitives or symbolic instruction sets, used to map abstract tasks into executable control logic. Semantic action information refers to a structured representation that integrates task semantics and motion parameters. It can be encoded as a vector sequence containing joint space trajectories and timing constraints, enabling precise alignment between task intent and underlying control.

[0086] Specifically, the initial action model forms a cross-modal action generation framework by fusing the action generation capabilities of the stream matching model with the semantic understanding capabilities of the visual language model. The semantic parsing module decomposes robot task data into a sequence of subtasks with temporal relationships. For example, the "assembly part" task is broken down into atomic operations such as "grasping a screw," "locating a screw hole," and "screwing and tightening." Each subtask is mapped into standardized semantic units by querying a library of preset action templates. For example, "grasping a screw" is mapped into a parameterized action primitive that includes end-user posture adjustment and gripping force control. The resulting semantic action information retains the high-level semantic features of the task while also including control parameters related to the robot's morphology, forming a structured supervisory signal that adapts to different hardware platforms. During the model fine-tuning phase, the semantic action information serves as a training objective to drive the update of the initial action model parameters, enabling rapid adaptation of the pre-trained knowledge to the kinematic constraints of the new robot. For example, the control strategy can be adapted to the different joint degrees of freedom of a six-axis robot arm and a quadruped robot, enabling cross-platform migration without re-collecting the full action data.

[0087] The motion generation method based on multimodal pre-training provided in this application can be applied to the medical and health fields and the financial transaction fields. In the medical and health field, it can be applied to the surgical motion generation scenarios of intelligent surgical robots, personalized rehabilitation training system scenarios and multimodal medical imaging diagnosis assistance scenarios. Intelligent surgical robots can have high-frequency motion generation and cross-morphological adaptation capabilities, supporting different surgical scenarios (such as minimally invasive surgery, orthopedic positioning). The intelligent surgical robot generates accurate instrument motion trajectories based on preoperative images (CT / MRI) and real-time endoscopic images; dynamically adjusts the parameters of the robotic arm (such as force and speed) to adapt to the anatomical structures of different patients. Smooth and continuous motion is achieved through stream matching technology to reduce the risk of intraoperative jitter; a single model is adapted to multiple surgical robot platforms (such as the Da Vinci system and domestic robotic arms). In intelligent customer service and wealth management in the field of financial transactions, the multimodal interaction capabilities of vision, language and proprioception are combined to improve the natural interaction level of service robots.

[0088] In an embodiment of the present application, language instructions, visual images, and robot proprioception data are obtained; multimodal feature alignment and cross-modal feature fusion are performed on the language instructions, visual images, and robot proprioception data using a pre-trained visual language model to generate a fused joint feature vector; a stream matching technique is used to generate target continuous control instructions based on the fused joint features using a pre-trained target action model; and continuous robot actions are generated based on the target continuous control instructions. This embodiment of the present invention achieves cross-platform high-frequency continuous action generation through multimodal feature alignment and stream matching techniques, solving the control migration difficulties caused by the heterogeneity of robot hardware. At the same time, a cross-platform pre-training mechanism effectively improves data utilization efficiency, thereby improving the efficiency of robot action generation and cross-platform adaptability.

[0089] Please refer to Figure 8 , as a response to the above Figure 2 The present application provides an embodiment of a motion generation device based on multimodal pre-training, which is similar to the embodiment of the present invention. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various intelligent robots.

[0090] like Figure 8 As shown, the motion generation device based on multimodal pre-training of this embodiment includes: a data acquisition module 51, a feature fusion module 52, an instruction generation module 53 and an action generation module 54, wherein:

[0091] A data acquisition module 51 is used to acquire language instructions, visual images and robot proprioception data;

[0092] a feature fusion module 52 for performing multimodal feature alignment and cross-modal feature fusion on the language instruction, the visual image, and the robot proprioception data using a pre-trained visual language model to generate a fused joint feature vector;

[0093] An instruction generation module 53 is configured to generate target continuous control instructions based on the fused joint features by using a pre-trained target action model and a flow matching technology;

[0094] The action generation module 54 is used to generate continuous actions of the robot based on the target continuous control instructions.

[0095] Furthermore, the feature fusion module 52 includes:

[0096] a feature alignment unit, configured to extract features of the language instruction, the visual image, and the robot proprioception data respectively through the pre-trained visual language model to obtain multimodal features, and perform feature alignment on the multimodal features to obtain aligned multimodal features;

[0097] A feature splicing unit, configured to perform feature splicing on the aligned multimodal features to generate spliced ​​features;

[0098] The nonlinear fusion unit is used to perform nonlinear fusion on the concatenated features through a multi-layer perceptron to obtain a fused joint feature vector.

[0099] Furthermore, the feature alignment unit includes:

[0100] A first feature generation unit is configured to extract features from the visual image using a pre-trained visual encoder to obtain image features;

[0101] A second feature generation unit is used to encode the language instruction through a pre-trained language model to generate semantic features;

[0102] A third feature generation unit is used to encode the robot proprioception data through a pre-trained multi-layer perceptron to generate perception features;

[0103] A feature mapping unit is used to map the image features, semantic features and perceptual features to a shared embedding space to perform feature alignment on the image features, semantic features and perceptual feature mappings to obtain the aligned multimodal features.

[0104] Furthermore, the instruction generation module 53 includes:

[0105] A form identification obtaining unit, used to obtain a form identification of the robot;

[0106] a continuous action trajectory generating unit, configured to generate a continuous action trajectory based on the fused joint features using a conditional probability flow field;

[0107] a dimension adjustment unit, configured to adjust the input and output dimensions of the target motion model according to the robot morphology identifier to obtain an adjusted target motion model;

[0108] An instruction conversion unit is used to convert the continuous action trajectory into the target continuous control instruction through the adjusted target action model.

[0109] Furthermore, the feature fusion module 52 further includes:

[0110] a preprocessing unit, configured to obtain cross-platform pre-training data and pre-process the cross-platform pre-training data to generate pre-processed data, wherein the cross-platform pre-training data includes image data, initial language instructions, robot state data, and action labels;

[0111] An image feature extraction unit, configured to extract image features from the image data using a visual encoder;

[0112] a semantic feature encoding unit, configured to encode the initial language instruction into initial semantic features through a language encoder, and convert the robot state data into initial perceptual features through a multi-layer perceptron;

[0113] a contrast loss value calculation unit, configured to perform pairwise contrast loss calculation based on the image features, the initial semantic features, and the initial perceptual features using a contrast loss function to generate a contrast loss value;

[0114] A pre-trained visual language model generating unit is used to adjust the parameters of the visual language model based on the contrast loss value and adjust the weight of the contrast loss function to re-train the visual language model and generate the pre-trained visual language model.

[0115] Furthermore, the instruction generation module 53 further includes:

[0116] An action sequence acquisition unit, configured to acquire an initial robot morphology identifier and a continuous action sequence with a timestamp, and to acquire an initial fused joint feature output by the pre-trained visual language model;

[0117] a conditional probability flow field adjustment unit, configured to construct an initial conditional probability flow field and adjust an output dimension of the initial conditional probability flow field based on the initial robot form identifier to obtain an adjusted conditional probability flow field;

[0118] an initial continuous action generating unit, configured to generate an initial continuous action based on the initial fused joint feature through the adjusted conditional probability flow field;

[0119] a flow matching loss value calculation unit, configured to calculate a flow matching loss value based on the initial connection action and the continuous action sequence, and train a flow matching model based on the flow matching loss value to obtain a target flow matching model;

[0120] A model fine-tuning unit is used to construct an initial action model based on the target stream matching model and the pre-trained visual language model, and to perform model fine-tuning on the initial action model to generate the target action model.

[0121] Furthermore, the model fine-tuning unit includes:

[0122] A task data acquisition unit, configured to construct an initial action model based on the target stream matching model and the pre-trained visual language model, and acquire robot task data;

[0123] a semantic action information generating unit, configured to parse the robot task data into a subtask sequence, and map the subtask sequence to a preset action template to obtain semantic action information;

[0124] The target action model generating unit is used to fine-tune the initial action model based on the semantic action information to obtain the target action model.

[0125] To solve the above technical problems, the present application also provides a robot. Figure 9 , Figure 9 This is the basic structural diagram of the robot in this embodiment.

[0126] The robot 6 includes a memory 61, a processor 62, and a network interface 63 that are interconnected through a system bus. It should be noted that Figure 9 The figure shows only a robot 6 having three components: a memory 61, a processor 62, and a network interface 63. However, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead. It should be understood by those skilled in the art that a robot is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), an embedded device, etc.

[0127] The memory 61 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, magnetic disk, optical disk, etc. In some embodiments, the memory 61 may be an internal storage unit of the robot 6, such as the hard disk or internal memory of the robot 6. In other embodiments, the memory 61 may also be an external storage device of the robot 6, such as a plug-in hard disk equipped on the robot 6, a smart media card (SMC), a secure digital (SD) card, a flash memory card, etc. Of course, the memory 61 may also include both the internal storage unit of the robot 6 and its external storage device. In this embodiment, the memory 61 is generally used to store the operating system and various application software installed on the robot 6, such as the program code of the multimodal pre-training-based motion generation method. In addition, the memory 61 may also be used to temporarily store various types of data that have been output or are about to be output.

[0128] In some embodiments, the processor 62 can be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 62 is generally used to control the overall operation of the robot 6. In this embodiment, the processor 62 is used to execute program code stored in the memory 61 or process data, such as executing the program code of the above-mentioned multimodal pre-training-based motion generation method to implement various embodiments of the multimodal pre-training-based motion generation method.

[0129] The network interface 63 may include a wireless network interface or a wired network interface. The network interface 63 is generally used to establish a communication connection between the robot 6 and other electronic devices.

[0130] The present application also provides another embodiment, namely, providing a computer-readable storage medium, which stores a computer program, and the computer program can be executed by at least one processor to enable the at least one processor to perform the steps of the above-mentioned action generation method based on multimodal pre-training.

[0131] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods of each embodiment of the present application.

[0132] Obviously, the embodiments described above are only some of the embodiments of the present application, rather than all of the embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions described in the aforementioned specific embodiments, or to make equivalent replacements for some of the technical features therein. Any equivalent structure made using the contents of the present application specification and the accompanying drawings, directly or indirectly used in other related technical fields, is also within the scope of patent protection of the present application.

Claims

1. A motion generation method based on multimodal pre-training, characterized in that: include: Obtain language instructions, visual images, and robot proprioception data; Performing multimodal feature alignment and cross-modal feature fusion on the language instructions, the visual image, and the robot proprioception data through a pre-trained visual language model to generate a fused joint feature vector; Generate target continuous control instructions based on the fused joint features by using a stream matching technology through a pre-trained target action model; Continuous actions of the robot are generated based on the target continuous control instructions.

2. The action generation method based on multimodal pre-training according to claim 1 is characterized in that: The method of performing multimodal feature alignment and cross-modal feature fusion on the language instruction, the visual image, and the robot proprioception data through a pre-trained visual language model to generate a fused joint feature vector includes: Extracting features of the language instruction, the visual image, and the robot proprioception data respectively through the pre-trained visual language model to obtain multimodal features, and aligning the multimodal features to obtain aligned multimodal features; Performing feature splicing on the aligned multimodal features to generate spliced ​​features; The concatenated features are nonlinearly fused through a multi-layer perceptron to obtain a fused joint feature vector.

3. The action generation method based on multimodal pre-training according to claim 2 is characterized in that: The pre-trained visual language model is used to extract features of the language instruction, the visual image, and the robot proprioception data to obtain multimodal features, and the multimodal features are aligned to obtain aligned multimodal features, including: Extracting features from the visual image using a pre-trained visual encoder to obtain image features; Encoding the language instructions through a pre-trained language model to generate semantic features; Encoding the robot proprioception data through a pre-trained multi-layer perceptron to generate perception features; The image features, semantic features, and perceptual features are mapped to a shared embedding space to perform feature alignment on the image features, semantic features, and perceptual feature mappings to obtain the aligned multimodal features.

4. The action generation method based on multimodal pre-training according to claim 1, characterized in that: The pre-trained target action model uses a stream matching technology to generate target continuous control instructions based on the fused joint features, including: Get the robot's form identifier; Using a conditional probability flow field to generate a continuous action trajectory based on the fused joint features; Adjusting the input and output dimensions of the target motion model according to the robot morphology identifier to obtain an adjusted target motion model; The continuous motion trajectory is converted into the target continuous control instruction through the adjusted target motion model.

5. The method for motion generation based on multimodal pre-training according to any one of claims 1 to 4, characterized in that: Before performing multimodal feature alignment and cross-modal feature fusion on the language instructions, the visual image, and the robot proprioception data using a pre-trained visual language model to generate a fused joint feature vector, the method further includes: Obtaining cross-platform pre-training data and pre-processing the cross-platform pre-training data to generate pre-processed data, wherein the cross-platform pre-training data includes image data, initial language instructions, robot state data, and action labels; extracting image features from the image data using a visual encoder; Encoding the initial language instruction into initial semantic features through a language encoder, and converting the robot state data into initial perceptual features through a multi-layer perceptron; Performing pairwise contrast loss calculation based on the image features, the initial semantic features, and the initial perceptual features using a contrast loss function to generate a contrast loss value; The parameters of the visual language model are adjusted based on the contrast loss value, and the weight of the contrast loss function is adjusted to retrain the visual language model and generate the pre-trained visual language model.

6. The method for generating motion based on multimodal pre-training according to claim 5, characterized in that: Before generating target continuous control instructions based on the fused joint features using a stream matching technology through a pre-trained target action model, the method further includes: Obtaining an initial robot morphology identifier and a time-stamped continuous action sequence, and obtaining an initial fused joint feature output by the pre-trained visual language model; Constructing an initial conditional probability flow field, and adjusting the output dimension of the initial conditional probability flow field based on the initial robot morphology identifier to obtain an adjusted conditional probability flow field; generating an initial continuous action based on the initial fused joint feature through the adjusted conditional probability flow field; Calculating a flow matching loss value based on the initial connection action and the continuous action sequence, and training a flow matching model based on the flow matching loss value to obtain a target flow matching model; An initial action model is constructed based on the target flow matching model and the pre-trained visual language model, and the initial action model is fine-tuned to generate the target action model.

7. The method for motion generation based on multimodal pre-training according to claim 6, characterized in that: The step of constructing an initial action model based on the target stream matching model and the pre-trained visual language model, and fine-tuning the initial action model to generate the target action model includes: Building an initial action model based on the target stream matching model and the pre-trained visual language model, and acquiring robot task data; Parsing the robot task data into subtask sequences, and mapping the subtask sequences to preset action templates to obtain semantic action information; The initial action model is fine-tuned based on the semantic action information to obtain the target action model.

8. An action generation device based on multimodal pre-training, characterized in that: include: Data acquisition module, used to obtain language instructions, visual images and robot proprioception data; a feature fusion module, configured to perform multimodal feature alignment and cross-modal feature fusion on the language instructions, the visual image, and the robot proprioception data using a pre-trained visual language model to generate a fused joint feature vector; An instruction generation module is used to generate target continuous control instructions based on the fused joint features by using a stream matching technology through a pre-trained target action model; The action generation module is used to generate continuous actions of the robot based on the target continuous control instructions.

9. A robot, characterized in that: The invention comprises a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the action generation method based on multimodal pre-training as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the action generation method based on multimodal pre-training according to any one of claims 1 to 7.

Images