Motion information generation method, electronic device, and storage medium

Abstract

Embodiments of the present application provide a kind of action information generation method, electronic equipment and storage medium, the method includes: the multimodal information and target haptic query information that are generated in advance are input to target multimodal trunk model and obtain uniform haptic latent variable and conditional characteristic information;Unified haptic latent variable is input to target rough haptic predictor and obtain rough predicted haptic latent variable, rough predicted haptic latent variable is input to target fine haptic predictor and obtain fine predicted haptic latent variable, and conditional characteristic information is input to target action prediction model and obtain predicted action information;Fine predicted haptic latent variable, predicted action information and the robot state information and high-frequency haptic image information obtained are input to target action haptic controller and obtain residual action information, predicted action information and residual action information are fused and obtain target action information.This scheme can change haptic signal from passive observation into dynamic interaction signal that can be actively understood, predicted and used for control.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method for generating motion information, an electronic device, and a storage medium. Background Technology

[0002] With the rapid development of vision-language-motion models and robot fundamental model technologies, multimodal fusion-based general robot manipulation has become an important development direction in the field of intelligent robotics. Currently, using RGB images, language commands, and robot body states as input, low-frequency motion blocks are generated through multimodal encoding, and operations are performed by actuators such as robotic arms and grippers. This approach performs well in tasks with minimal contact, such as grasping, moving, and placing. However, in scenarios with rich contact, such as insertion, assembly, and wiping, the success or failure of the task depends on local, instantaneous, fine-grained physical interactions such as collisions, slippage, jamming, and misalignment of holes and slots. These states are mostly located in local contact areas, are easily occluded, and cannot be reliably and timely perceived by relying solely on vision.

[0003] Related technologies use tactile or force feedback to make reactive corrections to robot movements. The aforementioned methods passively introduce tactile signals, which can lead to insufficient tactile semantic understanding of the generated motion information, making it unable to respond to changes in the interaction state in a timely manner and resulting in low motion adjustment accuracy. Summary of the Invention

[0004] This application provides a method for generating motion information, an electronic device, and a storage medium. It addresses the problems in related technologies where the generated motion information suffers from insufficient tactile semantic understanding, inability to respond promptly to changes in interactive states, and low motion adjustment accuracy. It enables unified encoding of tactile signals and multimodal contexts, transforming tactile signals from passive observation into dynamic interactive signals that can be actively understood, predicted, and used for control. This improves the response speed to changes in robot interactive states in rich contact scenarios and also enhances the accuracy of robot motion adjustment.

[0005] In a first aspect, embodiments of this application provide a method for generating action information, the method comprising: The pre-generated multimodal information and target tactile query information are input into the trained target multimodal backbone model to obtain unified tactile latent variables and conditional feature information; The unified tactile latent variable is input into the trained target coarse tactile predictor to obtain coarse predicted tactile latent variable, the coarse predicted tactile latent variable is input into the trained target fine tactile predictor to obtain fine predicted tactile latent variable, and the conditional feature information is input into the trained target action prediction model to obtain predicted action information. The fine predicted tactile latent variables, the predicted action information, and the acquired robot state information and high-frequency tactile image information are input into the trained target action tactile controller to obtain residual action information. The predicted action information and the residual action information are then fused to obtain target action information.

[0006] Secondly, embodiments of this application also provide an action information generation device, including: The multimodal processing module is configured to input pre-generated multimodal information and target tactile query information into the trained target multimodal backbone model to obtain unified tactile latent variables and conditional feature information; The first action prediction module is configured to input the unified tactile latent variable into the trained target coarse tactile predictor to obtain coarse predicted tactile latent variable, input the coarse predicted tactile latent variable into the trained target fine tactile predictor to obtain fine predicted tactile latent variable, and input the conditional feature information into the trained target action prediction model to obtain predicted action information. The second motion prediction module is configured to input the fine predicted tactile latent variable, the predicted motion information, and the acquired robot state information and high-frequency tactile image information into the trained target motion tactile controller to obtain residual motion information. The fusion module is configured to fuse the predicted action information and the residual action information to obtain the target action information.

[0007] Thirdly, embodiments of this application also provide an electronic device, the device comprising: One or more processors; Memory, used to store one or more computer programs; When one or more computer programs are executed by one or more processors, an electronic device implements an action information generation method as described in any of the first aspects.

[0008] Fourthly, embodiments of this application also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements an action information generation method as described in any of the first aspects.

[0009] As described above, by setting target tactile query information, a multimodal aligned tactile latent space can be actively constructed, transforming tactile feedback from a passive auxiliary input into an active interactive prior, achieving a deep fusion of tactile semantic understanding and action generation. Through a two-level tactile prediction mechanism from coarse to fine, high-precision prediction of future contact states can be achieved, providing reliable prior information for feedforward control. A target action tactile controller enables the synergy between low-frequency action planning and high-frequency residual correction, balancing task planning and instantaneous contact response. This scheme achieves unified encoding of tactile signals and multimodal context, transforming tactile signals from passive observation into dynamic interactive signals that can be actively understood, predicted, and used for control. This improves the response speed to changes in robot interaction states in contact-rich scenarios and also enhances the accuracy of robot action adjustments. Attached Figure Description

[0010] Figure 1 A flowchart illustrating an action information generation method provided in this application embodiment; Figure 2 A flowchart illustrating a specific implementation process for training a raw tactile encoder to obtain a target tactile encoder, as provided in this application embodiment; Figure 3 A schematic diagram of the system architecture for the application action information generation method provided in the embodiments of this application; Figure 4 A flowchart illustrating a specific implementation process for joint training of multiple processing modules is provided in this application embodiment; Figure 5 A flowchart illustrating a specific implementation process for training an original motion haptic controller to obtain a target motion haptic controller, provided as an embodiment of this application; Figure 6 A structural block diagram of an action information generation device provided in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0011] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and not for limiting the invention. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention and not the entire structure.

[0012] It should be noted that, due to space limitations, this application specification does not exhaustively list all possible implementation methods. Those skilled in the art should be able to conceive after reading this application specification that, as long as the technical features do not contradict each other, any combination of technical features can constitute an optional implementation method.

[0013] Currently, VLA (Vision-Language-Action Model) and basic robot models are widely used in general robot operation tasks. These methods typically acquire RGB images, verbal commands, and the robot's own state as input, then generate low-frequency action blocks via a visual language model or multimodal encoder, which are then executed by a robotic arm, gripper, or dexterous hand. In routine tasks with low contact levels, such as grasping, moving, and placing, VLA has demonstrated good performance. However, in contact-intensive operation scenarios such as insertion, assembly, wiping, and posture adjustment, the performance often depends on local, instantaneous, and fine-grained physical interaction states. These physical interaction states include, but are not limited to, initial contact establishment, local collisions, edge scraping, slippage, jamming, slot alignment deviations, surface height changes, and relative posture changes after external force disturbances. These physical interaction states typically arise in local contact areas between the gripper and the object, or between the object and the environment, and are easily obscured by the gripper, the object, or the robot itself. Relying solely on visual perception makes it difficult to obtain this type of state information in a timely and reliable manner. Current VLA architectures typically employ an action block mechanism, which outputs a sequence of actions for a predetermined duration in a single step to balance inference efficiency and action smoothness. This strategy is well-suited for tasks with low contact intensity, but in contact-intensive tasks, it can easily lead to low-frequency semi-open-loop or open-loop execution patterns: when collisions, slippage, or jamming occur during the execution of a single action block, the system struggles to correct them in a timely manner, and errors accumulate continuously during subsequent action executions.

[0014] The inventors have discovered that the relevant technical solutions can be categorized as follows: The first category consists of VLA basic models and action block strategies. This type of approach typically employs pre-trained visual language models, diffusion strategies, flow matching strategies, or action head networks to map visual information, language commands, and ontological states into robot action blocks. While this approach provides basic execution capabilities for general robot operations, its perceptual inputs are primarily visual information and ontological states, without adequately modeling local contact states.

[0015] The second category is tactile / force fusion robot manipulation schemes. These schemes introduce physical interaction signals such as optical tactile feedback, array pressure, force / torque, and joint torque into VLA models, diffusion strategies, or imitation learning strategies. For example, tactile images, tactile embedding vectors, or force sequences are used as additional observation terms, concatenated with visual and linguistic features, or tactile features are selectively introduced when contact occurs. This second category can alleviate the lack of direct physical interaction perception in pure vision schemes, but most schemes still belong to a passive fusion mode: tactile information is only used as supplementary background information alongside vision; the model does not actively determine the contact stage corresponding to the tactile signal, the reliability of the tactile modality compared to the visual modality, or the guiding significance of tactile changes for subsequent action execution. Furthermore, pre-trained VLA models are usually not trained based on tactile data distribution; directly concatenating high-dimensional visual tactile images easily creates a modal gap. For high-information-density images output by optical tactile sensors, directly inputting them into the model significantly increases inference overhead and easily leads to strong binding of the model to the appearance, resolution, lighting conditions, and colloidal texture parameters of specific tactile sensors, limiting generalization.

[0016] The third category comprises predictive modeling and tactile world modeling solutions. These solutions explicitly learn intermediate sub-objectives, future visual / tactile states, implicit semantic alignment, or contact dynamics characteristics within the VLA model or robot strategy, enabling the model to acquire structured task priors. Other methods align tactile embedding vectors to the visual-linguistic semantic space or predict future tactile images and latent tactile variables to model the evolution of local contact states. While this third category helps the model learn the changing trends of future contact states, existing solutions often focus on the prediction process itself or offline world model deduction, without unified optimization with contact semantic understanding, language-conditional action generation, and online high-frequency action correction. This makes it difficult to translate predicted information into timely, stable, and constrained error-correcting actions during actual execution.

[0017] The fourth category is a fast-slow control scheme that combines slow high-level strategies with fast low-level feedback control. This type of scheme combines low-frequency visual language reasoning or diffusion action blocks with high-frequency tactile / force feedback control. It improves the control response bandwidth in contact-intensive tasks by performing closed-loop corrections to the executed actions through fast control branches. This fourth type of scheme can improve the response lag problem caused by action block mechanisms, but most schemes still rely on current tactile or force feedback for reactive correction, usually adjusting the action only after abnormal contact occurs, lacking feedforward control priors provided by future tactile predictions. If optimization is only achieved by shortening the action block length or using time integration, it is also easy to cause problems such as action instability, sensitivity to hyperparameters, overly smooth response, or stuttering before grasping.

[0018] As can be seen from the above, in the relevant technical solutions, tactile input is only used as a passive auxiliary word, lacking an active semantic understanding mechanism. Its tactile thought chain or contact stage reasoning is disconnected from future tactile prediction. Tactile / force feedback is mostly used for reactive correction, lacking feedforward control priors provided by future tactile prediction. Predictive world models, VLA action generation, and high-frequency closed-loop control have not yet formed a unified technical framework. Based on this, the embodiments of this application aim to provide a method for generating action information, an electronic device, and a storage medium to solve the problems of insufficient tactile semantic understanding, inability to respond promptly to changes in interactive states, and low action adjustment accuracy in the action information generated in the relevant technologies.

[0019] The embodiments of the present invention will be described in detail below.

[0020] Figure 1 A flowchart of an action information generation method provided in an embodiment of this application is shown below. Figure 1 As shown, the motion information generation method includes, but is not limited to, the following steps S101-S103: Step S101: Input the pre-generated multimodal information and target tactile query information into the trained target multimodal backbone model to obtain unified tactile latent variables and conditional feature information.

[0021] The multimodal information can be a set of feature vectors that can be processed by the model after encoding raw data from different modalities such as vision, text, and touch. Specifically, it can include visual word sequences, text word sequences, and touch word sequences. The target tactile query information can be a set of learned embedding vectors after training convergence. Specifically, it can be in the form of word sequences. Through an attention mechanism, task-related tactile interaction information is actively retrieved and extracted from the multimodal features to achieve active tactile perception. Optionally, the target tactile query information can include multiple tactile query word sequences. For example, the number can be... ,in, The target haptic query sequence can be a haptic prediction window, and the dimension of each haptic query term sequence can be consistent with the term embedding dimension of the target multimodal backbone model, for example, 2048 dimensions. Of course, the number, dimension, and access position of the target haptic query information sequence can be adjusted according to the needs of the actual application scenario. It can be inserted into the input layer, intermediate layer, policy head, etc. of the multimodal backbone model, or it can be implemented through cross-attention mechanisms, adapters, low-rank adaptation, gating fusion, or other parameter-efficient structures; this application does not impose any limitations on these. The target multimodal backbone model can be a pre-trained multimodal fusion and inference foundational network, built based on an open-source vision-language model architecture, used to achieve cross-modal information alignment, semantic inference, and feature output. The unified haptic latent variable can be a compact feature vector extracted from the target haptic query information and mapped to a low-dimensional space. It carries semantic, modal reliability judgment, and action guidance information at the current contact stage and can be regarded as the core representation of haptics as an active prior. Conditional feature information can be a general multimodal feature sequence that integrates visual, linguistic, and tactile priors and outputs. Specifically, it can be in the form of a word sequence, serving as a conditional input for the generation of subsequent actions, carrying task objectives and environmental state information.

[0022] In one embodiment, the multimodal information includes a visual lexical sequence, a textual lexical sequence, and a tactile lexical sequence. The generation process of the visual lexical sequence, textual lexical sequence, and tactile lexical sequence is as follows: Acquire visual image information, task instruction information, and low-frequency tactile image information; Visual image information is input into the trained target visual encoder to obtain a visual word sequence, task instruction information is input into the trained target text encoder to obtain a text word sequence, and low-frequency tactile image information is input into the trained target tactile encoder to obtain a tactile word sequence.

[0023] The visual image information can be scene image data acquired by the robot's wrist camera, first-person view camera, top camera, or other external cameras, such as the robot's workspace, target object, gripper, and environment, used to perceive the relative position of the target object, environmental layout, and robot end effector. The task instruction information can be a description of the operation objective given by the user in natural language, used to clarify the task objective, target object, and expected result, such as "pick up the USB and insert it into the interface," "grab the wiping board and wipe the whiteboard," or "grab the gear and assemble it onto the gear shaft." Low-frequency tactile image information can be acquired by one or more tactile / visual-tactile sensors mounted on the robot's end effector at a low sampling frequency (e.g., 2Hz), serving as the tactile input for the multimodal backbone model. In contrast, the high-frequency tactile image information, acquired at a high sampling frequency (e.g., 30Hz), is subsequently input to the target motion tactile controller. For example, taking a robot equipped with a two-finger gripper, visual-tactile sensors are installed on the tips of the two fingers. Each visual-tactile sensor outputs three-channel tactile images of depth deformation, first tangential shear deformation, and second tangential shear deformation. By using different sampling frequencies, corresponding low-frequency and high-frequency tactile image information can be acquired. It should be noted that the low-frequency tactile image data uses a lower frequency to adapt to the multimodal fusion computation of the multimodal backbone model, ensuring the efficiency of model inference. The high-frequency tactile images provide instantaneous physical interaction perception for the target motion tactile controller, compensating for the low-frequency response lag of the multimodal backbone model, so as to support the target motion tactile controller to output high-frequency residual motion information, which is adapted to the robot's actual high-frequency execution actions. Thus, visual image information provides global semantics and spatial positioning, task instruction information clarifies the task objective, and low-frequency tactile image information provides direct physical interaction perception. The three complement each other to comprehensively characterize the state of the operation scene, providing a complete raw data foundation for subsequent multimodal fusion. A target visual encoder can be a trained, convergent image feature extraction network, such as employing a ViT (Vision Transformer) architecture, to encode image blocks into ordered sequences of visual lexical units. The visual lexical sequence can be the image feature representation output by the target visual encoder, where each lexical unit corresponds to the semantic and spatial features of a local region of the image, and their ordered arrangement represents the information of the entire image. A target text encoder can be a trained, convergent text feature extraction network, such as employing the encoding layer of a pre-trained language model to map segmented text into a sequence of text lexical units. The text lexical sequence can be the language feature representation output by the target text encoder, where each lexical unit corresponds to the semantic features of a segmented text, and their ordered arrangement represents the semantics of the entire task instruction. A target haptic encoder can be a trained, convergent haptic feature compression network, used to map high-dimensional haptic images into low-dimensional, compact haptic latent variables.Optionally, the haptic encoder can employ a variational masked autoencoder, which may include a Transformer encoder and a Transformer decoder. The Transformer encoder can output a haptic term sequence, and the Transformer decoder can be used for training reconstruction and visualization. Of course, the haptic encoder can also employ convolutional networks, ordinary autoencoders, diffusing autoencoders, state-space models, contrastive learning encoders, or other structures that can map haptic observations to latent variables; this application does not impose any limitations on these. The haptic term sequence can be a compact feature representation output by the target haptic encoder, preserving core physical information such as contact intensity, deformation, and shear slip, with a dimensionality far lower than the original image. Thus, by extracting features from visual, textual, and haptic data separately using independent encoders, a standardized representation of information from different modalities is achieved, laying a consistent feature foundation for subsequent multimodal fusion.

[0024] In one embodiment, a specific implementation process for training a raw tactile encoder to obtain a target tactile encoder is described. The raw tactile encoder can be a tactile feature compression network to be trained with randomly initialized parameters, while the target tactile encoder can be a converged tactile feature compression network. (Please refer to...) Figure 2 This is a flowchart illustrating a specific implementation process for training a target tactile encoder from an original tactile encoder, as provided in an embodiment of this application. Figure 2 As shown, the specific implementation process of training the original tactile encoder to obtain the target tactile encoder includes, but is not limited to, the following steps S201-S203: Step S201: Obtain the first tactile sample data, and perform random occlusion processing on the first tactile sample data to obtain occluded image block data and visible image block data; The first tactile sample data can be a tactile image dataset used for pre-training the tactile encoder. The source of the first tactile sample data is explained below using a specific robot in a real-world application scenario: The robot can be equipped with a robotic arm, with an end effector installed at the end of the arm. The end effector can be a mechanical gripper, a dexterous hand, or similar device with tactile sensing capabilities. Taking a mechanical gripper as an example, a visual-tactile sensor is installed on the gripper. Each visual-tactile sensor records three channels of surface deformation signals, including depth deformation, first tangential shear deformation, and second tangential shear deformation. Before collecting the first tactile sample data, different task categories are pre-set according to actual needs. For example, in one specific implementation, there are four task categories: picking and adjusting, picking and wiping, picking and inserting, and picking and assembling. Multiple sub-tasks are set for different task types, which can include: adjusting the posture of a pencil and placing it in a cup, adjusting the angle of a tubular object and placing it in a shelf, wiping a whiteboard, wiping the surface of a vase, inserting a USB drive, inserting a two-pronged plug, assembling a large gear, and assembling a small gear, etc. For each sub-task, tactile images output by the visual-tactile sensor can be acquired based on a preset sampling frequency under two types of motion trajectories. The first type of motion trajectory is the normal expert demonstration trajectory, which can be the standard motion trajectory corresponding to the completion of the sub-task. It is used to learn stable operating behavior. Taking USB insertion as an example, the standard motion trajectory can be that the mechanical gripper centers and holds the middle of the USB plug, the robotic arm moves the USB to the front of the interface, adjusts it to be coaxial with the interface and the end face of the USB plug makes smooth contact with the interface shell, and the robotic arm smoothly advances along the axis until the plug is fully inserted. The second type is the disturbance recovery trajectory. This trajectory can be the action trajectory that corrects and recovers after receiving a disturbance and continues to execute the task during the completion of a sub-task according to the standard action trajectory. It is used to learn corrective behaviors for recovery from abnormal contact states. The disturbance can include changes in the initial angle of the object, changes in the height of the wiping surface, changes in the slot direction, changes in the gear shaft direction, manual pushing, changes in platform posture, or relative offset after contact. Taking USB insertion as an example, the installation height of the USB interface is adjusted to have a vertical deviation from the initial planned position. Correspondingly, the disturbance recovery trajectory can be that the mechanical gripper centers the middle of the USB plug, the robotic arm moves the USB to the front of the initially set interface, and if it cannot be adjusted to be coaxial with the interface and the end face of the USB plug makes smooth contact with the interface shell, the height of the end of the robotic arm is readjusted to complete the interface alignment, and the robotic arm smoothly advances along the axis until the plug is fully inserted. For mechanical grippers with visual and tactile sensors on both gripping arms, the three-channel tactile images of the left and right fingertips can be synchronized in time to form an original tactile image sequence. In this three-channel tactile image, the three channels represent depth deformation, first tangential shear deformation, and second tangential shear deformation, respectively. Each original tactile image can be stored with 16-bit precision, and during training, it is normalized from the uint16 range to an interval. Then linearly mapped to This process is used to obtain the final first tactile sample data. Randomly occluding the first tactile sample data can be done by replacing a portion of the image with a mask at a preset ratio (e.g., 25%). The occluded image patch data can be randomly occluded tactile image regions, which are the targets that the decoder needs to infer and reconstruct. Therefore, the image patch data consists of unoccluded tactile image regions that retain their original information, serving as the input for the tactile encoder to extract features.

[0025] Step S202: Input the visible image block data into the original tactile encoder to obtain the first tactile latent variable, and input the first tactile latent variable into the preset decoder to obtain the reconstructed tactile image data. The reconstructed tactile image data includes the first reconstructed image block data corresponding to the occluded image block data and the second reconstructed image block data corresponding to the visible image block data.

[0026] Optionally, the original haptic encoder adopts a ViT-Small structure, containing 12 Transformer layers, 8 attention heads, and 192-dimensional hidden layers. The original haptic encoder is used to perform a low-dimensional (e.g., 32-dimensional) mapping transformation on the input high-dimensional visible image patch data to obtain the first haptic latent variable. The first haptic latent variable can be a low-dimensional feature vector obtained by the original haptic encoder through feature extraction based on the visible image patch data. The pre-defined decoder can be a haptic decoding network paired with the original haptic encoder, used for image reconstruction based on the first haptic latent variable. It is used only during the training phase and can be removed during the inference phase. The reconstructed haptic image data can be the image data reconstructed by the pre-defined decoder based on the visible image patch data represented by the first haptic latent variable. Its aim is to reconstruct the complete haptic image corresponding to the first haptic sample data, so as to simultaneously predict both the visible image patch portion and the occluded image patch portion. The first reconstructed image patch data can be the portion of the reconstructed haptic image data corresponding to the original occluded region, which is the missing region content generated by the pre-defined decoder based on the first haptic latent variable. The second reconstructed image block data can be a portion of the reconstructed tactile image data corresponding to the original visible area, which is the visible area content generated by the preset decoder based on the first tactile latent variable inference.

[0027] Step S203: Calculate the target loss value based on the occluded image patch data, visible image patch data, first reconstructed image patch data, and second reconstructed image patch data using the target loss function. Then, iteratively update the parameters of the original tactile encoder and the preset decoder based on the target loss value until the loss converges to obtain the trained target tactile encoder.

[0028] The target loss function can be an optimization objective function that measures the difference between the reconstructed tactile image and the real tactile image. Specifically, it can be composed of a weighted average of occluded region reconstruction loss, visible region reconstruction loss, and a KL divergence regularization term. The occluded region reconstruction loss measures the pixel-level error between the occluded image patch data and the first reconstructed image patch data, and can specifically employ L1 loss, L2 loss, etc. The visible region reconstruction loss measures the pixel-level error between the visible image patch data and the second reconstructed image patch data, and can specifically employ L1 loss, L2 loss, etc. The KL divergence regularization term is used to regularize the probability distribution corresponding to the first tactile latent variable, making it approximate a preset standard normal prior distribution, and can specifically employ the KL divergence algorithm. For example, the calculation formula for the target loss function is as follows:

[0029] in, It can be the target loss value. It can be the occlusion region reconstruction loss value, used to measure the pixel-level error between the occluded image patch data and the first reconstructed image patch data. It can be the visible region reconstruction loss value, used to measure the pixel-level error between the visible image patch data and the second reconstructed image patch data. The weighting coefficients for the visible region reconstruction loss are: The weighting coefficients for the reconstruction loss of the occluded region. These are the weighting coefficients for the KL divergence regularization term, used to adjust the constraint strength of the regularization term. The KL divergence operation is used to measure the degree of difference between two probability distributions. This is the first tactile sample data. As the first latent variable of touch, Given the first tactile sample data x as input, what is the probability distribution that the first tactile latent variable z follows? It follows a multivariate standard normal distribution, where, It is a vector with zero mean. It is the unit covariance matrix.

[0030] Furthermore, after calculating the target loss value, the parameters of the original haptic encoder and the preset decoder are updated based on the gradient of the target loss value through backpropagation. This process continues until the loss no longer decreases after several consecutive rounds, or until a preset number of training steps is reached. At this point, training stops, and the encoder parameters are saved as the trained target haptic encoder. It should be noted that the preset decoder is only used during the training phase and can be removed during the inference phase.

[0031] As described in steps S201-S203 above, the encoder-decoder self-encoding architecture, through the reconstruction task, enables the encoder to learn compact latent variables with high information density. This ensures that the latent variables can completely retain the core physical information related to contact, improving the representational ability and generalization of the latent variables. This provides a basic representation for subsequent tactile understanding, future tactile prediction, and action correction, and reduces the computational overhead of directly processing high-dimensional tactile images in the multimodal backbone model. Furthermore, by randomly occluding the first tactile sample data, the preset original tactile encoder cannot directly read the pixel information of the occluded area. Therefore, the subsequent preset decoder must infer the deformation distribution of the occluded area based on the contact deformation features of the visible area. This allows the original tactile encoder to learn core physical logic such as the spatial distribution of contact pressure and the law of shear deformation transmission. Its output first tactile latent variable can represent the physical essence of contact, rather than simply memorizing surface details of the image.

[0032] Step S102: Input the unified tactile latent variable into the trained target coarse tactile predictor to obtain coarse predicted tactile latent variable, input the coarse predicted tactile latent variable into the trained target fine tactile predictor to obtain fine predicted tactile latent variable, and input the conditional feature information into the trained target action prediction model to obtain predicted action information.

[0033] The target coarse tactile predictor can be a lightweight prediction network that has been trained to convergence, used to output the global trend of future tactile changes (e.g., contact enhancement, contact weakening, contact transfer, increased risk of slippage, etc.). Optionally, the target coarse tactile predictor can adopt a two-layer MLP (Multi-Layer Perceptron), where the vector dimension changes during data processing from "2048 dimensions of input" to "512 dimensions in the middle" to "32 dimensions of output", and is activated using the GELU activation function. Of course, the target coarse tactile predictor can also be implemented using linear layers, Transformer, RNN (Recurrent Neural Network), or state-space models. The coarse predicted tactile latent variables can be the low-detail future tactile representation output by the target coarse tactile predictor, retaining only the overall evolution trend of the contact state, and forming the basis for fine-grained prediction. The target fine tactile predictor can be a generative prediction network that has been trained to convergence, used to supplement local contact details (e.g., edge collision, local deformation, etc.) on the basis of coarse prediction, to achieve high-precision future tactile prediction. Optionally, the target fine haptic predictor can adopt a Diffusion Transformer architecture, which contains 8 Transformer basic blocks, a hidden dimension of 512, and 8 attention heads. Of course, the target fine haptic predictor can also be implemented using a flow matching model, a diffusion model, a temporal convolutional network, or other generative prediction models, which are not limited in this application. The fine predictive haptic latent variable can be a high-detail future haptic representation output by the target fine haptic predictor, fully characterizing the instantaneous contact state at a future moment. It should be noted that the fine predictive haptic latent variable is a deduced output of the contact state at a future moment, corresponding to multiple future contact states within the prediction time window, providing a clear action adjustment lead for the target motion haptic controller. For example, in a USB insertion task, the fine predictive haptic latent variable can predict in advance that the plug edge will collide with the interface chamfer at the next moment. The fine predictive haptic latent variable can output the residual motion information of pose adjustment in advance without waiting for the actual collision to occur and for the haptic feedback to change before correction, thus completing the centering correction before the collision occurs and avoiding contact impact and jamming from the source. The target action prediction model can be a convergent action generation network, implemented based on flow matching or diffusion strategies, responsible for outputting low-frequency action blocks as a task-level planning benchmark. The input to this target action prediction model is conditional feature information, which is generated by the target multimodal backbone model based on visual image information, task instruction information, and low-frequency tactile image information. Corresponding to the aforementioned low-frequency tactile image information, the predicted action information can be a low-frequency action sequence output by the target action prediction model, specifically in the form of a word sequence, corresponding to control instructions such as end-effector pose, joint control parameters, velocity, force-controlled target, gripper opening / closing amount, or combinations thereof, at multiple future time steps.It should be noted that the predicted motion information aims to provide executable, continuous motion trajectories to characterize the complete task chain, providing a macroscopic motion path reference for the subsequent decision-making of the target motion haptic controller. The target motion haptic controller needs to output high-frequency residual motion information to match the inherent operating rhythm of the robot servo system; its input is high-frequency haptic image information to ensure instantaneous constraints on changes in contact state. Therefore, the low-frequency predicted motion information output by the target motion prediction model can provide a reference for the high-frequency residual motion information output by the subsequent target motion haptic controller, ensuring that the final motion conforms to the task objective.

[0034] Step S103: Input the fine predicted tactile latent variables, predicted action information, and the acquired robot state information and high-frequency tactile image information into the trained target action tactile controller to obtain residual action information, and fuse the predicted action information and residual action information to obtain target action information.

[0035] The robot state information can be real-time perceived data of the robot body, including end-effector pose, joint angles, gripper opening, and motion speed, used to anchor the current physical configuration. High-frequency tactile image information can be real-time tactile data acquired at high frequencies, used to capture abnormal contacts such as instantaneous collisions, slippage, and jamming, providing closed-loop feedback signals. For example, the low-frequency tactile image information used to generate the aforementioned multimodal information can have a sampling frequency of 2Hz, while the sampling frequency of the high-frequency tactile image information can be 30Hz. The target motion tactile controller can be a lightweight high-frequency control network that has been trained and converged, responsible for combining future tactile feedforward, real-time tactile feedback, and robot state to output a small residual correction to the baseline motion, achieving closed-loop high-frequency control. Optionally, the target motion tactile controller can use a 6-layer lightweight Transformer with a hidden dimension of 192 and 3 attention heads. Of course, the target motion tactile controller can also be implemented using MLP, RNN, temporal convolutional networks, diffusion controllers, impedance controllers, force-position hybrid controllers, or learning residual controllers, etc., which are not limited in this application. Residual motion information can be a sequence of motion corrections output by the target motion haptic controller, specifically in the form of a word sequence. This represents the adjustment amount that needs to be superimposed on the baseline motion, with limited amplitude to ensure motion stability. Target motion information can be the final high-frequency motion sequence executed, also in the form of a word sequence. This sequence balances task-level planning and real-time contact correction, providing guidance for subsequent motion execution by the robot's actuators.

[0036] In one embodiment, the specific formula for fusing the predicted action information and the residual action information is as follows:

[0037] in, For target action information, To predict action information, For residual motion information, The hyperbolic tangent activation function is used to limit the residual motion information, mapping any value of residual to... Within the interval, This is a scaling factor used to map the normalized range of the tanh() output to the actual maximum allowable correction magnitude range. This is the gating factor, used to determine the effectiveness of overall control residual correction.

[0038] Furthermore, the generated target motion information can be applied to the robot execution module, enabling the robot execution module to receive the corrected final motion and drive the robotic arm, gripper, dexterous hand, or other end effector to execute.

[0039] As can be seen from steps S101-S103 above, by setting target tactile query information, a multimodal aligned tactile latent space can be actively constructed, transforming tactile feedback from a passive auxiliary input to an active interactive prior, achieving a deep fusion of tactile semantic understanding and action generation. Through a two-level tactile prediction mechanism from coarse to fine, high-precision prediction of future contact states can be achieved, providing reliable prior information for feedforward control. The target action tactile controller achieves the synergy between low-frequency action planning and high-frequency residual correction, balancing task planning and instantaneous contact response. This scheme enables unified encoding of tactile signals and multimodal context, transforming tactile signals from passive observation into dynamic interactive signals that can be actively understood, predicted, and used for control. This improves the response speed to changes in robot interaction states in contact-rich scenarios and also enhances the accuracy of robot action adjustment.

[0040] Figure 3 A schematic diagram of the system architecture for the application action information generation method provided in the embodiments of this application, as shown below. Figure 3As shown, the target visual encoder 301, target text encoder 302, and target tactile encoder 303 are configured in parallel. The target visual encoder 301 receives visual image information and outputs a visual word sequence; the target text encoder 302 receives task instruction information and outputs a text word sequence; and the target tactile encoder 303 receives low-frequency tactile image information and outputs a tactile word sequence. The target multimodal backbone model 304 receives the visual word sequence, text word sequence, tactile word sequence, and target tactile query information, and outputs unified tactile latent variables and conditional feature information. The target coarse tactile predictor 305 receives the unified tactile latent variables and outputs coarse predicted tactile latent variables. The target fine tactile predictor 306 receives the coarse predicted tactile latent variables and outputs fine predicted tactile latent variables. The target action prediction model 307 receives conditional feature information and outputs predicted action information. The target motion tactile controller 308 is used to receive fine predicted tactile latent variables, predicted motion information, robot state information and high-frequency tactile image information, and output residual motion information. This residual motion information is used to fuse with the predicted motion information to obtain target motion information.

[0041] Figure 4 This application provides a flowchart illustrating a specific implementation process for jointly training multiple processing modules, including a target visual encoder, a target text encoder, a target multimodal backbone model, a target coarse tactile predictor, a target fine tactile predictor, and a target action prediction model. While jointly training the multiple processing modules, the specific parameters of the target tactile query information can be updated synchronously. It should be noted that the aforementioned target tactile encoder is trained independently; during joint training, the parameters of the target tactile encoder need to be frozen. Figure 4 As shown, the joint training process for the target tactile query information, target visual encoder, target text encoder, target multimodal backbone model, target coarse tactile predictor, target fine tactile predictor, and target action prediction model includes, but is not limited to, the following steps S401-S405: Step S401: Obtain first visual sample data, first task instruction sample data, second tactile sample data, and initial tactile query information to be learned.

[0042] The first visual sample data can be image data collected by the robot's wrist camera, first-person view camera, top camera, or other external cameras for the joint training process. It records the complete execution process of various contact-rich tasks, such as picking and adjusting, picking and wiping, picking and inserting, and picking and assembling, in different task scenarios. Specifically, it involves the visual morphology and spatial relationship of the manipulated object, target workstation, end effector, and contact process, providing visual observation information of the work scene. The first task instruction sample data can be text instructions in natural language form for the joint training process, such as "pick up the USB and insert the interface," "grab the wiping board and wipe the whiteboard," and "grab the gear and assemble it onto the gear shaft." It is accompanied by instruction text with various sentence structures or expressions to cover diverse language expression habits in real-world scenarios, conveying the semantic goals and execution requirements of the task. The second tactile sample data can be a tactile image dataset for the joint training process. The initial tactile query information can be a randomly initialized sequence of query terms used to extract tactile-related information from multimodal features.

[0043] Step S402: Input the first visual sample data into the original visual encoder to obtain the first visual sample word sequence, input the first task instruction sample data into the original text encoder to obtain the first text sample word sequence, and input the second tactile sample data into the target tactile encoder to obtain the first tactile sample word sequence.

[0044] The original visual encoder can be a visual feature compression network to be trained with randomly initialized parameters, the original text encoder can be a text feature compression network to be trained with randomly initialized parameters, and the target tactile encoder can be obtained by performing the aforementioned steps S201-S203. The first visual sample word sequence can be a visual feature sequence obtained by encoding the first visual sample data through the original visual encoder. The first text sample word sequence can be a text feature sequence obtained by encoding the first task instruction sample data through the original text encoder. The first tactile sample word sequence can be a tactile feature sequence obtained by encoding the second tactile sample data through the trained target tactile encoder.

[0045] Step S403: Input the first visual sample word sequence, the first text sample word sequence, the first tactile sample word sequence, and the initial tactile query information into the original multimodal backbone model to obtain the second tactile latent variable and the first conditional feature information.

[0046] The second tactile latent variable can be a low-dimensional feature vector obtained by the original multimodal backbone model from the feature extraction of the first visual sample word sequence, the first text sample word sequence, and the first tactile sample word sequence based on the initial tactile query information. This vector is used for subsequent semantic supervision and prediction loss calculation. The first conditional feature information can be a general multimodal feature sequence output by the original multimodal backbone model after fusing the first visual sample word sequence, the first text sample word sequence, and the first tactile sample word sequence. Specifically, it can be in the form of a word sequence and serves as the conditional input for the subsequent original action prediction model.

[0047] Step S404: Determine the first loss value based on the second tactile latent variable and the set language decoder; determine the second loss value based on the first conditional feature information and the original action prediction model; and determine the third and fourth loss values ​​based on the second tactile latent variable, the original coarse tactile predictor, and the original fine tactile predictor.

[0048] The language decoder can be a lightweight language model (e.g., a language model with approximately 0.6B parameters) used during the training phase to convert the second tactile latent variable into a natural language semantic description and provide tactile interaction semantic supervision. It does not require deployment during the actual inference phase. The first loss value can be the tactile interaction semantic supervision loss, used to measure the difference between the generated semantic description and the labeled semantics, and to constrain the semantic interpretability of the tactile latent variable. The second loss value can be the action generation loss, used to measure the difference between the predicted action and the labeled action, and to optimize the model's action planning ability. The third loss value can be the coarse-grained tactile prediction loss, used to measure the difference between the coarse prediction result and the true future tactile latent variable, and to optimize the global trend prediction ability of the coarse tactile predictor. The fourth loss value can be the fine-grained tactile prediction loss, used to measure the difference between the fine prediction result and the true future tactile latent variable, and to optimize the local detail prediction accuracy of the fine tactile predictor.

[0049] In one embodiment, the specific implementation process for determining the first loss value based on the second tactile latent variable and the set language decoder is as follows: Obtain the annotated interactive semantic text; The second tactile latent variable is input into the set language decoder to obtain tactile interactive semantic text; The first loss value is obtained by calculating the first loss function based on the tactile interactive semantic text and the labeled interactive semantic text.

[0050] The labeled interactive semantic text can take one or more forms, such as manually labeled natural language text corresponding to the tactile state at each time step, structured tags, stage tags, risk tags, or action guidance tags. Its content covers three dimensions: interaction stage analysis, modality dependency analysis, and action guidance analysis. Specifically, interaction stage analysis is used to determine states such as relaxed, stable grip, and dynamic contact. Relaxation indicates that no physical contact has occurred; stable grip indicates a relatively stable physical interaction; dynamic contact indicates rapidly changing contact conditions, potentially involving collisions, slippage, insertion, or alignment. Modality dependency analysis is used to determine contact intensity, force changes, slippage risk, visual observability, and the magnitude of tactile changes, and to determine which modality—visual or tactile—is more reliable in the current state. Action guidance analysis is used to describe the current operational state, potential risks, which modality should dominate the decision-making, and the appropriate strategies: maintaining, reducing contact, adjusting posture, continuing to advance, avoiding, or resuming the action. Because the annotated interactive semantic text covers three dimensions—interaction stage analysis, modality dependency analysis, and action guidance analysis—it enables the unified tactile latent variable to carry multi-dimensional semantic information, providing a semantic basis for subsequent dynamic adjustment of modality weights and guidance of action generation. Furthermore, the tactile interactive semantic text can be the semantic understanding output of the language decoder for the current contact state corresponding to the second tactile latent variable. Optionally, the first loss function can be the cross-entropy loss function. Specifically, this involves word-level alignment of the tactile interactive semantic text and the annotated interactive semantic text, calculating the cross-entropy error of each positional word prediction, summing and averaging over all positions to obtain the first loss value. By using cross-entropy loss to optimize text generation accuracy, the semantic accuracy of the tactile latent variable can be effectively constrained, improving the interpretability and semantic depth of the tactile representation.

[0051] In one embodiment, the specific implementation process for determining the second loss value based on the first conditional feature information and the original action prediction model is as follows: Obtain information on low-frequency actions; The first conditional feature information is input into the original action prediction model to obtain the first predicted action information; The second loss value is obtained by calculating the second loss function based on the first predicted action information and the labeled low-frequency action information.

[0052] The low-frequency action information can be real action data obtained from normal expert demonstration trajectories. The first predicted action information can be a predicted action sequence generated by the original action prediction model based on multimodal conditional features. After the original action prediction model is trained and the target action prediction model is obtained, the target action prediction model outputs low-frequency action blocks, i.e., predicted action information. The second loss function can be a flow matching loss function, specifically, it can calculate the error between the conditional vector field corresponding to the first predicted action information and the vector field corresponding to the labeled low-frequency action information to obtain the second loss value. By adopting a flow matching loss that matches the action prediction model, the accuracy of action generation can be efficiently optimized, allowing the model to quickly learn expert-level operation action trajectories.

[0053] In one embodiment, the specific implementation process for determining the third and fourth loss values ​​based on the second tactile latent variable, the original coarse tactile predictor, and the original fine tactile predictor is as follows: Obtain labeled tactile latent variables; The second tactile latent variable is input into the original coarse tactile predictor to obtain the first coarse predicted tactile latent variable, and the first coarse predicted tactile latent variable is input into the original fine tactile predictor to obtain the first fine predicted tactile latent variable; The third loss value is obtained by calculating the third loss function based on the first coarse-predicted tactile latent variable and the labeled tactile latent variable, and the fourth loss value is obtained by calculating the fourth loss function based on the first fine-predicted tactile latent variable and the labeled tactile latent variable.

[0054] The labeled tactile latent variables can be latent variables obtained by encoding real tactile images of future moments using a trained target tactile encoder. Specifically, tactile images from all moments in the demonstration dataset can be pre-input into the target tactile encoder to obtain and store the corresponding tactile latent variables. During joint training of multiple processing modules, latent variables from future moments after the current moment are read according to the time index and used as labeled tactile latent variables. The demonstration dataset can be a collection of tactile images obtained by synchronously sampling low-frequency data from visual-tactile sensors under normal expert demonstration trajectories. The first coarse-predicted tactile latent variable can be a coarse-grained representation of future tactile sensations output by the original coarse tactile predictor, containing only global contact trend information. The global contact trend information can describe only large-scale features such as the overall direction of change of the contact state, the overall intensity magnitude, and the overall force bias, without including fine information such as the spatial distribution of the contact area and the local deformation gradient. For example, taking the USB insertion task as an example, the specific representation of the global contact trend information is as follows: the depth deformation shows a continuous increasing trend, corresponding to the plug continuously advancing along the axial direction and the contact force gradually increasing; there is a tangential shearing trend to the left in the horizontal direction, corresponding to the plug's initial posture being generally biased to the right, and subsequent contact will be continuously subjected to a reaction force to the left; the contact area shows a uniform expansion trend, without sudden local abrupt changes. The first fine-grained predictive tactile latent variable can be the future tactile fine-grained representation output by the original fine-grained tactile predictor after supplementing details on the basis of the coarse prediction, containing complete contact state information. The complete contact state information can include all fine-grained physical information such as the spatial morphology of the entire contact area, the depth deformation at each spatial location, the shear deformation distribution in two tangential directions, the location of local stress concentration, and the local distribution of slippage trends. For example, taking a USB insertion task as an example, the specific representation of the complete contact state information is as follows: Reconstructing the deformation distribution of the tactile image at a future moment: the depth deformation is greatest in the lower right corner of the plug and smallest in the upper right corner, showing an overall asymmetrical distribution with a downward bias; Clearly defining the local characteristics of tangential deformation: shear deformation in the first direction is concentrated at the left edge of the plug, and shear deformation in the second direction is concentrated at the bottom edge of the plug, corresponding to the narrow contact band shape where the interface chamfer contacts the plug edge. Optionally, the third loss function can be the L1 loss function, specifically, calculating the absolute error between the first coarse-predicted tactile latent variable and the labeled tactile latent variable element by element, and taking the average to obtain the third loss value. Optionally, the fourth loss function can be the flow matching loss function, specifically, calculating the error between the conditional vector field corresponding to the first coarse-predicted tactile latent variable and the vector field corresponding to the labeled tactile latent variable to obtain the fourth loss value. It should be noted that coarse prediction uses L1 loss to ensure rapid convergence of the global trend, while fine prediction uses flow matching loss to ensure the accuracy of local detail generation.

[0055] Step S405: Based on the first loss value, the second loss value, the third loss value, and the fourth loss value, perform joint parameter iterative updates on the initial tactile query information, the original visual encoder, the original text encoder, the original multimodal backbone model, the original coarse tactile predictor, the original fine tactile predictor, and the original action prediction model until the loss converges to obtain the target tactile query information, the trained target visual encoder, the target text encoder, the target multimodal backbone model, the target coarse tactile predictor, the target fine tactile predictor, and the target action prediction model.

[0056] The first, second, third, and fourth loss values ​​can simultaneously incorporate four types of losses: semantic supervision, action generation, coarse prediction, and fine prediction. This enables multi-task optimization, allowing the unified tactile latent variable to simultaneously carry three types of information: semantic understanding, future prediction, and action guidance, thus unifying tactile understanding, prediction, and action generation. The total loss value is obtained by weighting and summing the first, second, third, and fourth loss values ​​according to preset weights. The gradients of all trainable parameters are calculated based on this total loss value, and the parameters of the initial tactile query information, the original visual encoder, the original text encoder, the original multimodal backbone model, the original coarse tactile predictor, the original fine tactile predictor, and the original action prediction model are updated synchronously until the total loss no longer decreases after several consecutive rounds, or when a preset number of training steps is reached, at which point training stops. Optionally, the joint training phase can primarily employ the normal expert demonstration trajectory mentioned above to learn a stable motion generation distribution and tactile prediction prior. In the subsequent training phase of the target motion tactile controller, both the normal expert demonstration trajectory and the perturbation recovery trajectory can be used simultaneously. The normal expert demonstration trajectory is used to maintain the reference motion distribution, while the perturbation recovery trajectory is used to learn the residual correction behavior after abnormal contact.

[0057] As can be seen from steps S401-S405 above, through multi-task loss collaborative optimization, tactile semantic understanding, future contact prediction and action generation are simultaneously realized under a unified feature framework, constructing an active tactile prior that combines semantics and prediction, ensuring the interpretability of tactile representation, and providing a reliable feedforward basis for subsequent high-frequency control.

[0058] Figure 5 This document provides a flowchart illustrating the specific implementation process of training a raw motion haptic controller to obtain a target motion haptic controller, as provided in an embodiment of this application. It should be noted that during the training process of the raw motion haptic controller, the parameters of the already trained target haptic query information, target visual encoder, target text encoder, target multimodal backbone model, target coarse haptic predictor, target fine haptic predictor, and target motion prediction model need to be frozen to ensure that only the raw motion haptic controller is trained, thus preventing the training process of the raw motion haptic controller from disrupting the learned motion generation distribution and haptic priors. Figure 5 As shown, the training process of the target motion haptic controller includes, but is not limited to, the following steps S501-S506: Step S501: Obtain second visual sample data, second task instruction sample data, and third tactile sample data.

[0059] The second visual sample data can be image data acquired by the robot's wrist camera, first-view camera, top camera, or other external cameras for the training process of the target motion haptic controller. The second task instruction sample data can be text instructions in natural language form for the training process of the target motion haptic controller. The third haptic sample data can be a dataset of haptic images for the training process of the target motion haptic controller.

[0060] Step S502: Input the second visual sample data into the target visual encoder to obtain the second visual sample word sequence, input the second task instruction sample data into the target text encoder to obtain the second text sample word sequence, and input the third tactile sample data into the target tactile encoder to obtain the second tactile sample word sequence.

[0061] The target visual encoder, target text encoder, and the target multimodal backbone model, target coarse tactile predictor, target fine tactile predictor, and target motion prediction model mentioned in subsequent steps can be obtained by executing steps S401-S405. The second visual sample word sequence can be a visual feature sequence obtained by encoding the second visual sample data through the target visual encoder. The second text sample word sequence can be a text feature sequence obtained by encoding the second task instruction sample data through the target text encoder. The second tactile sample word sequence can be a tactile feature sequence obtained by encoding the third tactile sample data through the target tactile encoder. The second visual sample data and the third tactile sample data can be obtained by simultaneously acquiring data from a preset camera and a visual-tactile sensor under the normal expert demonstration trajectory and the perturbation recovery trajectory mentioned in the foregoing embodiments, respectively. The normal expert demonstration trajectory is used to maintain the reference motion distribution, and the perturbation recovery trajectory is used to learn the behavior of returning to the correct execution process from an abnormal state. The target motion tactile controller trained by combining the aforementioned two types of trajectory data can provide constrained residual motion correction, enabling the robot to adjust its actions in advance when unstable contact trends are predicted, and to recover quickly after external perturbations or abnormal contact occur.

[0062] Step S503: Input the second visual sample word sequence, the second text sample word sequence, the second tactile sample word sequence, and the target tactile query information into the target multimodal backbone model to obtain the third tactile latent variable and the second conditional feature information.

[0063] The third tactile latent variable can be a low-dimensional feature vector obtained by the target multimodal backbone model from the feature extraction of the second visual sample word sequence, the second text sample word sequence, and the second tactile sample word sequence based on the target tactile query information, and is used for subsequent tactile prediction. The second conditional feature information can be a general multimodal feature sequence output by the target multimodal backbone model after fusing the second visual sample word sequence, the second text sample word sequence, and the second tactile sample word sequence, and is used for subsequent action prediction.

[0064] Step S504: Input the third tactile latent variable into the target coarse tactile predictor to obtain the second coarse predicted tactile latent variable, input the second coarse predicted tactile latent variable into the target fine tactile predictor to obtain the second fine predicted tactile latent variable, and input the second conditional feature information into the target action prediction model to obtain the second predicted action information.

[0065] The second coarse-predicted tactile latent variable can be the future coarse-predicted tactile result output by the target coarse-predictor. The second fine-predicted tactile latent variable can be the future fine-predicted tactile result output by the target fine-predictor, serving as the feedforward input to the motion haptic controller. The second predicted motion information can be the low-frequency reference motion block output by the target motion prediction model, serving as the correction reference for the motion haptic controller.

[0066] Step S505: Input the second fine predicted tactile latent variable, the second predicted action information, and the obtained robot sample state information and high-frequency tactile sample image information into the original action tactile controller to obtain the first residual action information.

[0067] The robot sample state information can be the body state data of the robot used for training, including end-effector pose, joint angles, gripper opening, and motion speed, used to anchor the current physical configuration. High-frequency tactile sample image information can be real-time tactile image data acquired at high frequencies, used to provide instantaneous contact feedback and capture abnormal contacts such as collisions, slippage, and jamming. The original motion tactile controller can be a parameter-initialized control network to be trained; for example, the original motion tactile controller can be a 6-layer lightweight Transformer with a hidden dimension of 192 and 3 attention heads. The first residual motion information can be a sequence of residual corrections obtained by the original motion tactile controller through nonlinear fusion and bias calculation of the input second fine-predicted tactile latent variable, second predicted motion information, second fine-predicted tactile latent variable, and second predicted motion information, representing the small adjustment amount that needs to be superimposed on the predicted motion.

[0068] Step S506: Calculate the fifth loss value using the fifth loss function based on the first residual action information and the obtained labeled high-frequency action information. Iterate and update the parameters of the original motion haptic controller based on the fifth loss value until the loss converges to obtain the trained target motion haptic controller.

[0069] The labeled high-frequency motion information can be expert real motion sequences corresponding to high-frequency sampling moments. These expert real motion sequences can be real robot control command sequences synchronously collected and recorded at a high-frequency sampling rate matching the robot servo system. Taking a USB insertion task as an example, the expert real motion sequence can be ground truth motion data sampled according to the stable gripping phase, alignment and approach phase, chamfering and insertion phase, and insertion into position phase. Specifically, it covers multi-dimensional control commands such as end-effector pose (3D position + 3D attitude), joint control quantities, gripper opening and closing amounts, movement speed, and force control target values ​​corresponding to each high-frequency sampling moment. Optionally, the fifth loss function can be an L1 loss function, specifically calculated element-wise as the absolute error between the first residual motion information and the labeled high-frequency motion information, and the average value is taken to obtain the fifth loss value. Based on this fifth loss value, the gradients of all trainable parameters in the original motion haptic controller are calculated, and the parameters of the original motion haptic controller are updated synchronously until the fifth loss value no longer decreases after several consecutive rounds, or when a preset number of training steps is reached, training stops.

[0070] As can be seen from steps S501-S506 above, by freezing the upstream module and adopting the training method of residual supervision, the stability of the baseline action and tactile prior can be guaranteed, while reducing the learning difficulty of the controller and improving training efficiency and convergence effect.

[0071] Figure 6 This is a structural block diagram of an action information generation device provided in an embodiment of this application. The device is configured to execute the action information generation method provided in the above embodiment, and possesses corresponding functional modules and beneficial effects for executing the method. For example... Figure 6 As shown, the device specifically includes: The multimodal processing module 601 is configured to input pre-generated multimodal information and target tactile query information into the trained target multimodal backbone model to obtain unified tactile latent variables and conditional feature information; The first action prediction module 602 is configured to input a unified tactile latent variable into a trained target coarse tactile predictor to obtain coarse predicted tactile latent variables, input the coarse predicted tactile latent variables into a trained target fine tactile predictor to obtain fine predicted tactile latent variables, and input conditional feature information into a trained target action prediction model to obtain predicted action information. The second motion prediction module 603 is configured to input the fine predicted tactile latent variables, predicted motion information, and acquired robot state information and high-frequency tactile image information into the trained target motion tactile controller to obtain residual motion information. The fusion module 604 is configured to fuse the predicted action information and the residual action information to obtain the target action information.

[0072] As described above, by setting target tactile query information, a multimodal aligned tactile latent space can be actively constructed, transforming tactile feedback from a passive auxiliary input into an active interactive prior, achieving a deep fusion of tactile semantic understanding and action generation. Through a two-level tactile prediction mechanism from coarse to fine, high-precision prediction of future contact states can be achieved, providing reliable prior information for feedforward control. A target action tactile controller enables the coordination of low-frequency action planning and high-frequency residual correction, balancing task planning and instantaneous contact response. This scheme achieves unified encoding of tactile signals and multimodal context, transforming tactile signals from passive observation into dynamic interactive signals that can be actively understood, predicted, and used for control. This improves the response speed to changes in robot interaction states in contact-rich scenarios and also enhances the accuracy of robot action adjustments.

[0073] In one possible embodiment, the multimodal information includes visual lexical sequences, textual lexical sequences, and tactile lexical sequences; the multimodal processing module 601 is further configured to: Acquire visual image information, task instruction information, and low-frequency tactile image information; Visual image information is input into the trained target visual encoder to obtain a visual word sequence, task instruction information is input into the trained target text encoder to obtain a text word sequence, and low-frequency tactile image information is input into the trained target tactile encoder to obtain a tactile word sequence.

[0074] In one possible embodiment, a first training module is also included, configured as follows: Acquire the first tactile sample data, and perform random occlusion processing on the first tactile sample data to obtain occluded image block data and visible image block data; The visible image block data is input into the original tactile encoder to obtain the first tactile latent variable. The first tactile latent variable is input into the preset decoder to obtain the reconstructed tactile image data. The reconstructed tactile image data includes the first reconstructed image block data corresponding to the occluded image block data and the second reconstructed image block data corresponding to the visible image block data. The target loss value is calculated based on the occluded image patch data, the visible image patch data, the first reconstructed image patch data, and the second reconstructed image patch data. The parameters of the original tactile encoder and the preset decoder are iteratively updated based on the target loss value until the loss converges to obtain the trained target tactile encoder.

[0075] In one possible embodiment, a second training module is also included, configured as follows: Acquire first visual sample data, first task instruction sample data, second tactile sample data, and initial tactile query information to be learned; The first visual sample data is input into the original visual encoder to obtain the first visual sample word sequence, the first task instruction sample data is input into the original text encoder to obtain the first text sample word sequence, and the second tactile sample data is input into the target tactile encoder to obtain the first tactile sample word sequence. The first visual sample word sequence, the first text sample word sequence, the first tactile sample word sequence, and the initial tactile query information are input into the original multimodal backbone model to obtain the second tactile latent variable and the first conditional feature information; The first loss value is determined based on the second tactile latent variable and the set language decoder; the second loss value is determined based on the first conditional feature information and the original action prediction model; and the third and fourth loss values ​​are determined based on the second tactile latent variable, the original coarse tactile predictor, and the original fine tactile predictor. Based on the first, second, third, and fourth loss values, the initial tactile query information, the original visual encoder, the original text encoder, the original multimodal backbone model, the original coarse tactile predictor, the original fine tactile predictor, and the original action prediction model are jointly updated iteratively until the loss converges to obtain the target tactile query information, the trained target visual encoder, the target text encoder, the target multimodal backbone model, the target coarse tactile predictor, the target fine tactile predictor, and the target action prediction model.

[0076] In one possible embodiment, the second training module is specifically configured as follows: Obtain the annotated interactive semantic text; The second tactile latent variable is input into the set language decoder to obtain tactile interactive semantic text; The first loss value is obtained by calculating the first loss function based on the tactile interactive semantic text and the labeled interactive semantic text.

[0077] In one possible embodiment, the second training module is further configured as follows: Obtain information on low-frequency actions; The first conditional feature information is input into the original action prediction model to obtain the first predicted action information; The second loss value is obtained by calculating the second loss function based on the first predicted action information and the labeled low-frequency action information.

[0078] In one possible embodiment, the second training module is further configured as follows: Obtain labeled tactile latent variables; The second tactile latent variable is input into the original coarse tactile predictor to obtain the first coarse predicted tactile latent variable, and the first coarse predicted tactile latent variable is input into the original fine tactile predictor to obtain the first fine predicted tactile latent variable; The third loss value is obtained by calculating the third loss function based on the first coarse-predicted tactile latent variable and the labeled tactile latent variable, and the fourth loss value is obtained by calculating the fourth loss function based on the first fine-predicted tactile latent variable and the labeled tactile latent variable.

[0079] In one possible embodiment, a third training module is also included, configured as follows: Acquire second visual sample data, second task instruction sample data, and third tactile sample data; The second visual sample data is input into the target visual encoder to obtain the second visual sample word sequence, the second task instruction sample data is input into the target text encoder to obtain the second text sample word sequence, and the third tactile sample data is input into the target tactile encoder to obtain the second tactile sample word sequence. The second visual sample word sequence, the second text sample word sequence, the second tactile sample word sequence, and the target tactile query information are input into the target multimodal backbone model to obtain the third tactile latent variable and the second conditional feature information; The third tactile latent variable is input into the target coarse tactile predictor to obtain the second coarse predicted tactile latent variable. The second coarse predicted tactile latent variable is input into the target fine tactile predictor to obtain the second fine predicted tactile latent variable. The second conditional feature information is input into the target action prediction model to obtain the second predicted action information. The second fine-predicted tactile latent variable, the second predicted motion information, and the acquired robot sample state information and high-frequency tactile sample image information are input into the original motion tactile controller to obtain the first residual motion information; The fifth loss value is calculated based on the first residual action information and the acquired labeled high-frequency action information. The parameters of the original motion haptic controller are iteratively updated based on the fifth loss value until the loss converges to obtain the trained target motion haptic controller.

[0080] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, such as... Figure 7As shown, the device includes a processor 701, a memory 702, an input device 703, and an output device 704; the number of processors 701 in the device can be one or more. Figure 7 Taking a processor 701 as an example; the processor 701, memory 702, input device 703, and output device 704 in the device can be connected via a bus or other means. Figure 7 Taking a bus connection as an example, the memory 702, as a computer-readable storage medium, can be configured to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the action information generation method in this embodiment. The processor 701 executes various functional applications and data processing of the device by running the software programs, instructions, and modules stored in the memory 702, thereby implementing the aforementioned action information generation method. The input device 703 can be configured to receive input digital or character information and generate key signal inputs related to user settings and function control of the device. The output device 704 may include a display screen or other display device.

[0081] The aforementioned electronic device includes a motion information generation device, which can be used to execute any motion information generation method and has corresponding functions and beneficial effects.

[0082] This application also provides a computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program performs relevant operations in the action information generation method provided in any embodiment of this application and has corresponding functions and beneficial effects.

[0083] Those skilled in the art will understand that embodiments of this application may be provided as methods, systems, or computer program products.

[0084] Therefore, this application may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, produce implementations of the flowchart... Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0085] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory. Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0086] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0087] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0088] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. A method for generating motion information, characterized in that, The method includes: The pre-generated multimodal information and target tactile query information are input into the trained target multimodal backbone model to obtain unified tactile latent variables and conditional feature information; The unified tactile latent variable is input into the trained target coarse tactile predictor to obtain coarse predicted tactile latent variable, the coarse predicted tactile latent variable is input into the trained target fine tactile predictor to obtain fine predicted tactile latent variable, and the conditional feature information is input into the trained target action prediction model to obtain predicted action information. The fine predicted tactile latent variables, the predicted action information, and the acquired robot state information and high-frequency tactile image information are input into the trained target action tactile controller to obtain residual action information. The predicted action information and the residual action information are then fused to obtain target action information.

2. The motion information generation method according to claim 1, characterized in that, The multimodal information includes visual lexical sequences, textual lexical sequences, and tactile lexical sequences; The generation process of the visual lexical sequence, the textual lexical sequence, and the tactile lexical sequence is as follows: Acquire visual image information, task instruction information, and low-frequency tactile image information; The visual image information is input into the trained target visual encoder to obtain a visual word sequence, the task instruction information is input into the trained target text encoder to obtain a text word sequence, and the low-frequency tactile image information is input into the trained target tactile encoder to obtain a tactile word sequence.

3. The motion information generation method according to claim 2, characterized in that, The training process of the target tactile encoder is as follows: First tactile sample data is acquired, and the first tactile sample data is randomly occluded to obtain occluded image block data and visible image block data; The visible image block data is input into the original tactile encoder to obtain the first tactile latent variable, and the first tactile latent variable is input into the preset decoder to obtain the reconstructed tactile image data. The reconstructed tactile image data includes the first reconstructed image block data corresponding to the occluded image block data and the second reconstructed image block data corresponding to the visible image block data. The target loss value is calculated based on the occluded image patch data, the visible image patch data, the first reconstructed image patch data, and the second reconstructed image patch data. The parameters of the original tactile encoder and the preset decoder are iteratively updated based on the target loss value until the loss converges to obtain the trained target tactile encoder.

4. The motion information generation method according to claim 2, characterized in that, The joint training process of the target tactile query information, target visual encoder, target text encoder, target multimodal backbone model, target coarse tactile predictor, target fine tactile predictor, and target action prediction model is as follows: Acquire first visual sample data, first task instruction sample data, second tactile sample data, and initial tactile query information to be learned; The first visual sample data is input into the original visual encoder to obtain the first visual sample word sequence, the first task instruction sample data is input into the original text encoder to obtain the first text sample word sequence, and the second tactile sample data is input into the target tactile encoder to obtain the first tactile sample word sequence. The first visual sample word sequence, the first text sample word sequence, the first tactile sample word sequence, and the initial tactile query information are input into the original multimodal backbone model to obtain the second tactile latent variable and the first conditional feature information; A first loss value is determined based on the second tactile latent variable and the set language decoder; a second loss value is determined based on the first conditional feature information and the original action prediction model; and a third and fourth loss values ​​are determined based on the second tactile latent variable, the original coarse tactile predictor, and the original fine tactile predictor. Based on the first loss value, the second loss value, the third loss value, and the fourth loss value, the initial tactile query information, the original visual encoder, the original text encoder, the original multimodal backbone model, the original coarse tactile predictor, the original fine tactile predictor, and the original action prediction model are jointly updated with iterative parameters until the loss converges to obtain the target tactile query information, the trained target visual encoder, the target text encoder, the target multimodal backbone model, the target coarse tactile predictor, the target fine tactile predictor, and the target action prediction model.

5. The motion information generation method according to claim 4, characterized in that, The determination of the first loss value based on the second tactile latent variable and the set language decoder includes: Obtain the annotated interactive semantic text; The second tactile latent variable is input into the set language decoder to obtain tactile interactive semantic text; A first loss value is obtained by calculating a first loss function based on the tactile interactive semantic text and the labeled interactive semantic text.

6. The motion information generation method according to claim 4, characterized in that, The step of determining the second loss value based on the first conditional feature information and the original action prediction model includes: Obtain information on low-frequency actions; The first conditional feature information is input into the original action prediction model to obtain the first predicted action information; The second loss value is obtained by calculating the second loss function based on the first predicted action information and the labeled low-frequency action information.

7. The motion information generation method according to claim 4, characterized in that, The determination of the third and fourth loss values ​​based on the second tactile latent variable, the original coarse tactile predictor, and the original fine tactile predictor includes: Obtain labeled tactile latent variables; The second tactile latent variable is input into the original coarse tactile predictor to obtain the first coarse predicted tactile latent variable, and the first coarse predicted tactile latent variable is input into the original fine tactile predictor to obtain the first fine predicted tactile latent variable; A third loss value is obtained by performing a third loss function based on the first coarse-predicted tactile latent variable and the labeled tactile latent variable, and a fourth loss value is obtained by performing a fourth loss function based on the first fine-predicted tactile latent variable and the labeled tactile latent variable.

8. The motion information generation method according to claim 4, characterized in that, The training process for the target motion haptic controller is as follows: Acquire second visual sample data, second task instruction sample data, and third tactile sample data; The second visual sample data is input into the target visual encoder to obtain a second visual sample word sequence, the second task instruction sample data is input into the target text encoder to obtain a second text sample word sequence, and the third tactile sample data is input into the target tactile encoder to obtain a second tactile sample word sequence; The second visual sample word sequence, the second text sample word sequence, the second tactile sample word sequence, and the target tactile query information are input into the target multimodal backbone model to obtain the third tactile latent variable and the second conditional feature information; The third tactile latent variable is input into the target coarse tactile predictor to obtain the second coarse predicted tactile latent variable, the second coarse predicted tactile latent variable is input into the target fine tactile predictor to obtain the second fine predicted tactile latent variable, and the second conditional feature information is input into the target action prediction model to obtain the second predicted action information. The second fine-predicted tactile latent variable, the second predicted motion information, and the acquired robot sample state information and high-frequency tactile sample image information are input into the original motion tactile controller to obtain the first residual motion information; The fifth loss value is calculated based on the first residual motion information and the acquired labeled high-frequency motion information. The parameters of the original motion haptic controller are iteratively updated based on the fifth loss value until the loss converges to obtain the trained target motion haptic controller.

9. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more computer programs; When the one or more computer programs are executed by the one or more processors, the electronic device implements the motion information generation method as described in any one of claims 1-8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the motion information generation method as described in any one of claims 1-8.

Images