Action prediction method, model training method and device for robot operation

Abstract

The application discloses a robot operation-oriented action prediction method, a model training method and equipment. The model training method comprises the following steps: obtaining a historical data set; the historical data set comprises a color vision sequence, a piezoresistive electronic skin tactile pressure map sequence, a robot action sequence and a real sample; based on the historical data set, a to-be-processed action is used to predict a world model, and future multi-modal feature information is predicted; wherein the future multi-modal feature information comprises future vision features, future tactile features and future action features; based on the future multi-modal feature information and the real sample, a total loss function is constructed; wherein the total loss function comprises a multi-modal flow matching loss and a tactile structured constraint loss, and the tactile structured constraint loss is determined by the real sample and the future tactile features; based on the total loss function, network parameters of the action prediction world model are updated until convergence, and the action prediction world model is obtained.

Description

Technical Field

[0001] This application relates to the fields of robot intelligent perception and robot motion planning technology, and more specifically, to a motion prediction method, model training method and device for robot operation. Background Technology

[0002] With the rapid development of autonomous operation technology for intelligent robots, time-series environment prediction schemes based on world models are widely used in interactive scenarios such as robot grasping and object manipulation.

[0003] In related technologies, most motion prediction world models for robots use color image visual sequences as the only input information, and infer environmental changes and robotic arm movement trajectories solely based on visual images.

[0004] However, visual images can only present the appearance and spatial position information of objects in two dimensions. They cannot characterize the physical interaction state such as pressure distribution and force deformation at the interface between the robot and the object. This results in a discrepancy between the temporal environmental change patterns derived from visual inference and the actual contact mechanics evolution process, leading to poor accuracy of the robot's motion trajectory. Summary of the Invention

[0005] In view of this, this application provides a motion prediction method, model training method and device for robot operation, in order to solve the problem that the discrepancy between the temporal environmental change law derived from vision and the actual contact mechanics evolution process leads to poor accuracy of the robotic arm's motion trajectory.

[0006] In a first aspect, this application provides a method for training a motion prediction world model for robot operations. The method includes: acquiring a historical dataset; wherein the historical dataset includes color visual sequences, piezoresistive electronic skin tactile pressure map sequences, robot action sequences, and real samples; based on the historical dataset, using the motion prediction world model to be processed, predicting future multimodal feature information; wherein the future multimodal feature information includes future visual features, future tactile features, and future action features; constructing a total loss function based on the future multimodal feature information and real samples; wherein the total loss function includes multimodal flow matching loss and tactile structured constraint loss, the tactile structured constraint loss being determined by real samples and future tactile features; and updating the network parameters of the motion prediction world model based on the total loss function until convergence, thus obtaining the motion prediction world model.

[0007] The method for training a motion prediction world model for robot operation provided in this application uses historical datasets, including color visual sequences, piezoresistive electronic skin tactile pressure map sequences, robot motion sequences, and real samples, as input to the model. Based on the historical datasets, the method drives the motion prediction world model to predict future multimodal feature information, including future visual features, future tactile features, and future motion features. Then, using the future multimodal feature information and real samples, a total loss function is constructed, consisting of a multimodal flow matching loss and a tactile structured constraint loss determined based on real samples and future tactile features. The network parameters of the motion prediction world model are iteratively updated according to the total loss function until convergence. During the data input stage, piezoresistive electronic skin tactile pressure map sequences are incorporated to characterize the pressure distribution and physical interaction state of the contact interface between the robot and the object. This corrects the deviation between the temporal environmental change patterns derived solely from color visual sequences and the actual contact mechanics evolution process. The trained motion prediction world model can output motion-related content that conforms to real mechanical laws, improving the accuracy of the robot arm's motion trajectory.

[0008] In one optional implementation, the action prediction world model includes a visual encoding branch, a tactile encoding branch, and an action encoding branch, all of which share the same transformer backbone network. Based on historical datasets, the action prediction world model is used to predict future multimodal feature information, including: determining visual terms corresponding to a color visual sequence using the visual encoding branch; determining tactile terms corresponding to a piezoresistive electronic skin tactile pressure map sequence using the tactile encoding branch; determining action terms corresponding to a robot action sequence using the action encoding branch; and predicting future visual features, future tactile features, and future action sequences using the transformer backbone network based on visual terms, tactile terms, and action terms.

[0009] In one optional implementation, the real samples include real future visual features, real future tactile features, and real future action sequences. Based on the future multimodal feature information and the real samples, a total loss function is constructed, including: adding Gaussian noise to the real future visual features, real future tactile features, and real future action sequences to obtain noisy visual samples, noisy tactile samples, and noisy action samples; constructing a multimodal flow matching loss based on the noisy visual samples, noisy tactile samples, noisy action samples, and future multimodal feature information; determining a clean tactile stress map estimator based on the noisy tactile samples and future tactile features; determining a tactile structured constraint loss based on the clean tactile stress map estimator; and performing a weighted fusion process on the multimodal flow matching loss and the tactile structured constraint loss to obtain the total loss function.

[0010] In one possible implementation, the tactile structured constraint loss is determined based on the clean tactile stress map estimator, including: calculating the tactile contact structure loss based on the spatial distribution characteristics of the clean tactile stress map estimator; calculating the tactile temporal smoothing loss based on the clean tactile stress map estimator in continuous time series and the real future tactile features; and weightedly fusing the tactile contact structure loss and the tactile temporal smoothing loss to obtain the tactile structured constraint loss.

[0011] In one possible implementation, a multimodal flow matching loss is constructed based on noisy visual samples, noisy tactile samples, noisy action samples, and future multimodal feature information. This includes: determining the visual flow matching loss based on noisy visual samples and future visual features; determining the tactile flow matching loss based on noisy tactile samples and future tactile features; determining the action flow matching loss based on noisy action samples and future action features; and performing a weighted fusion process on the visual flow matching loss, tactile flow matching loss, and action flow matching loss to obtain the multimodal flow matching loss.

[0012] In one possible implementation, based on the robot action sequence, action words corresponding to the robot action sequence are determined using action coding branches, including: using action coding branches to map the robot action sequence to a motion canvas of a preset dimension to obtain action vectors; performing channel-wise quantile normalization on the action vectors to constrain the action values ​​corresponding to the action vectors to a preset interval to obtain target action vector encoding; and determining action words based on the target action vector encoding.

[0013] In one possible implementation, the clean tactile stress map estimator is determined based on noisy tactile samples and future tactile features, including: fitting and solving the temporal evolution offset of tactile modalities based on noisy tactile samples and future tactile features; and performing noise reduction correction on the noisy tactile samples according to the tactile temporal evolution offset to obtain the clean tactile stress map estimator.

[0014] In one possible implementation, the tactile temporal smoothing loss is calculated based on the clean tactile pressure map estimate and the true future tactile features in a continuous time series. This includes: for each time index in the continuous time series, performing a difference operation on the clean tactile pressure map estimate at the current time and the clean tactile pressure map estimate at the next time to obtain the predicted tactile temporal difference features; for each time index in the continuous time series, performing a difference operation on the true future tactile features at the current time and the true future tactile features at the next time to obtain the true tactile temporal difference features; and determining the tactile temporal smoothing loss based on the predicted tactile temporal difference features and the true tactile temporal difference features.

[0015] Secondly, this application provides a motion prediction method for robot operation. The method includes: acquiring context sequence data of the robot in the current environment; wherein the context sequence data includes: a color visual sequence, a piezoresistive electronic skin tactile pressure map sequence, and a robot motion sequence; based on the context sequence data, using a motion prediction world model, performing a determination operation for a future motion sequence; wherein the determination operation includes: predicting future visual features based on the color visual sequence; predicting future tactile features based on the future visual features and the context sequence data; determining the future motion sequence based on preset language conditions, the context sequence data, the future visual features, and the future tactile features; and controlling the robot to execute the future motion sequence.

[0016] Thirdly, this application provides a training device for a motion prediction world model for robot operation. The device includes: an acquisition module for acquiring a historical dataset, wherein the historical dataset includes color visual sequences, piezoresistive electronic skin tactile pressure map sequences, robot action sequences, and real samples; a prediction module for predicting future multimodal feature information based on the historical dataset and using the motion prediction world model to be processed, wherein the future multimodal feature information includes future visual features, future tactile features, and future action features; a construction module for constructing a total loss function based on the future multimodal feature information and real samples, wherein the total loss function includes a multimodal flow matching loss and a tactile structured constraint loss, the tactile structured constraint loss being determined by real samples and future tactile features; and an update module for updating the network parameters of the motion prediction world model based on the total loss function until convergence, thus obtaining the motion prediction world model.

[0017] Fourthly, this application provides a computer device, including: a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the robot operation-oriented motion prediction world model training method of the first aspect or any corresponding embodiment thereof and / or the robot operation-oriented motion prediction method of the second aspect or any corresponding embodiment thereof.

[0018] Fifthly, this application provides a computer-readable storage medium storing computer instructions for causing a computer to execute the robot-oriented motion prediction world model training method of the first aspect or any corresponding embodiment thereof and / or the robot-oriented motion prediction method of the second aspect or any corresponding embodiment thereof.

[0019] Sixthly, this application provides a computer program product, including computer instructions for causing a computer to execute the robot operation-oriented motion prediction world model training method of the first aspect or any corresponding embodiment thereof and / or the robot operation-oriented motion prediction method of the second aspect or any corresponding embodiment thereof. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating a method for training a motion prediction world model for robot operations according to an embodiment of this application. Figure 2 This is a flowchart illustrating another method for training a motion prediction world model for robot operations according to an embodiment of this application. Figure 3 This is a flowchart illustrating another method for training a motion prediction world model for robot operation according to an embodiment of this application. Figure 4 This is a flowchart illustrating a motion prediction method for robot operation according to an embodiment of this application; Figure 5 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of this application. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0023] According to an embodiment of this application, an embodiment of a method for training a motion prediction world model for robot operations is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0024] This embodiment provides a method for training a motion prediction world model for robot operations, which can be used for motion prediction world models. Figure 1 This is a flowchart of a method for training a motion prediction world model for robot operations according to an embodiment of this application, such as... Figure 1 As shown, the process includes the following steps: Step S101: Obtain historical datasets; wherein, historical datasets include color visual sequences, piezoresistive electronic skin tactile pressure map sequences, robot action sequences, and real samples.

[0025] Historical datasets can be understood as collections of historical data used for model training. These historical datasets may include color visual sequences, piezoresistive electronic skin tactile pressure map sequences, robot action sequences, and real samples.

[0026] A color visual sequence can refer to a set of time-series color image frames continuously acquired by a robot's onboard camera. Specifically, the color visual sequence can be used to characterize the time-series data of continuous changes in scene space and object position and shape during operation.

[0027] Piezoresistive electronic skin tactile pressure map sequence can indicate the time sequence of frame-by-frame two-dimensional pressure distribution map collected by the piezoresistive sensor array of the mechanical fingertip, and is used to characterize the pressure magnitude and deformation state of the contact area at various points on the contact surface.

[0028] A robot motion sequence can be a set of continuous timing control instructions that indicate the joint angles, end-effector poses, and gripping openings of the robotic arm, used for the motion actions that the robot performs at each moment.

[0029] Real samples can indicate the true time series data of multimodal observations at future moments, specifically including future visual features, future tactile features, and future action sequences.

[0030] Specifically, three types of aligned temporal data are collected: visual scene, fingertip tactile force, and robot's own motion. All of these are aligned frame by frame according to the same timestamp to form temporally matched multimodal sample segments. At the same time, real observation data of future moments corresponding to each temporal segment are collected as real samples to form a historical dataset.

[0031] In a feasible scenario, the robot is equipped with a six-axis collaborative robotic arm, with the robotic arm covered by a piezoresistive electronic skin array, and a color high-definition time-series camera on its body. The robot repeatedly performs tasks such as grasping, pressing, sliding, and placing a water cup. The camera captures color images of the scene at a frame rate of 20Hz to generate a color visual sequence. The electronic skin simultaneously collects two-dimensional pressure distribution maps of the fingertips to generate a piezoresistive electronic skin tactile pressure map sequence. The motion controller records the joint point commands of each frame in real time to generate a robot motion sequence. Each segment of 80 frames of historical time-series data corresponds to the actual captured images, tactile force, and motion trajectory of the next 30 frames at future moments as real samples. All data is tagged with a unified timestamp and aligned. After removing abnormal segments with severe shaking and invalid empty touch, the training set and validation set are divided at a ratio of 9:1 and stored on a solid-state drive to form a historical dataset.

[0032] In one possible implementation, the timing segment of a single robot operation is denoted by its original time step index as follows: At each time step, observation data of three modalities—visual, tactile, and motor—are collected simultaneously. The standardized definitions of each modality are as follows: The data represents visual observations at time t. Let be the tensor of the piezoresistive electronic skin tactile pressure map at time t. All are tactile tensor dimensions; Let t be the robot's original motion vector; It should be noted that v represents vision, τ represents tactile sensation, and a represents action.

[0033] In a preferred embodiment of this application, the mechanical fingertip is equipped with two independent piezoresistive tactile sensor arrays, each array having a resolution of [resolution missing]. Therefore, the dimension of the tactile tensor is fixed as follows: ; .

[0034] Natural language commands and text descriptions of action segments input for robot operation tasks are denoted as follows: The text commands are input into the TextEncoder to complete the encoding, resulting in a language condition vector used to constrain the model's predictions. .

[0035] The core prediction objective of the motion prediction world model in this application is not merely to output future actions, but to simultaneously extrapolate visual latent variables, tactile pressure maps, and robot action sequences in the continuous time domain of the future, collectively denoted as: , , Wherein, parameter K represents the future prediction time domain length uniformly set by the model; For future visual latent variable prediction, For future haptic prediction and Predicting future action sequences.

[0036] Step S102: Based on the historical dataset, use the action prediction world model to predict future multimodal feature information; wherein, the future multimodal feature information includes future visual features, future tactile features and future action features.

[0037] Future visual features can indicate temporal features composed of visual latent variables. They are low-dimensional representations of visual images after being compressed by an encoder, and are used to characterize the evolution of future scene images. Future tactile features can indicate the temporal characteristics of future frame-by-frame tactile pressure maps output by the model, corresponding to the future changes in pressure distribution on the contact surface of the robotic arm; Future motion characteristics can indicate the timing characteristics of the future continuous motion commands of the robotic arm output by the model planning.

[0038] Specifically, after determining the historical dataset, it can be input into the action prediction world model to predict future multimodal feature information. The future multimodal feature information can be the output of the action prediction world model.

[0039] In one possible example, a preprocessing step is required before the actual computation. This involves first segmenting the raw time-series data of the robot's operation into several semantic action segments. The original frame index range corresponding to a single semantic action segment is denoted as... ; in, and Let represent the starting and ending original frame indices of the i-th segment, respectively. Assume the original acquisition frame rate is... The target visual frame rate used for visual latent extraction is Then the original frame step size used for sampling the visual frame is: ;in, This is the original frame step size.

[0040] For the i-th semantic action segment, the visual frame indices sampled within the semantic action segment satisfy: ;in, For visual sampling sequence number, The original frame step size for visual sampling. The original frame position, The total number of original visual frames obtained by sampling the i-th segment.

[0041] This application employs a Video Variational Autoencoder (VideoVAE) to compress the visual temporal dimension, defining a temporal compression ratio κ. In a preferred embodiment, a Wan-type Video Variational Autoencoder is used, with a fixed temporal compression ratio. Based on this compression ratio, calculate the total number of frames of visual latent variables output after temporal compression of the i-th semantic action segment: ;in, The total number of frames represents the visual latent variable.

[0042] To achieve temporal alignment between the piezoresistive electronic skin tactile pressure map sequence and the robot motion sequence, a tactile sampling alignment number is set. Action sampling alignment number .

[0043] Under a specific configuration ;in, Number of haptic alignment samples.

[0044] Based on the alignment rules mentioned above, single semantic action segments are normalized into historical data training segments of a unified dimension. The standardized definitions of each modality tensor are as follows: Visual latent variable tensor: ;in, These represent the number of channels, height, and width of the visual latent, respectively.

[0045] Piezoresistive electronic skin tactile pressure map sequence tensor: ; Robot action sequence tensor: .

[0046] Among them, the piezoresistive electronic skin tactile pressure map sequence is resampled or linearly interpolated to unify the tactile timing length. The robot's action sequence is standardized in length by truncating or padding with zeros. .

[0047] To avoid interference from zero-padding, missing sensor channels, and blank background areas where the robotic arm does not contact the model, binary mask tensors are generated simultaneously, namely, action effective mask and tactile effective mask.

[0048] Valid action mask: ; Valid haptic mask: .

[0049] The value 1 in the mask represents that the data at that position is valid interactive data and can participate in the error statistics of multimodal flow matching loss and haptic structured constraint loss; the value 0 represents the padding value, invalid background, or missing channel, which is directly masked in the loss calculation stage and does not participate in the error accumulation calculation.

[0050] Step S103: Construct a total loss function based on future multimodal feature information and real samples; wherein, the total loss function includes multimodal flow matching loss and tactile structured constraint loss, and the tactile structured constraint loss is determined by real samples and future tactile features.

[0051] Multimodal flow matching loss can be understood as a weighted fusion loss obtained by fitting the predicted features of the three modalities (visual, tactile, and action) to the corresponding ground truth values. It is used to constrain the overall multimodal temporal evolution law to be consistent with the changes in the real environment.

[0052] The tactile structured constraint loss can be understood as a constraint loss designed only for tactile modalities. It is calculated based on the true tactile values ​​in real samples and the future tactile features output by the model, and is used to correct tactile prediction distortion, temporal jumps, etc.

[0053] Specifically, after determining the future multimodal feature information and real samples, a total loss function can be further constructed.

[0054] As an example, the future visual features, future tactile features, and future action features are compared one by one with the corresponding ground truth data in the real samples. The visual flow matching loss, tactile flow matching loss, and action flow matching loss are calculated modally. The three losses are then weighted and summed to obtain the multimodal flow matching loss. The tactile ground truth part in the real samples is retrieved and compared with the future tactile features output by the model. First, the tactile temporal evolution offset is fitted to complete the denoising of the noisy tactile samples. The clean tactile pressure map estimate is solved. Then, the tactile contact structure loss and tactile temporal smoothing loss are calculated separately. The two losses are weighted and fused to generate the tactile structured constraint loss. Finally, a fixed weight coefficient is set, and the multimodal flow matching loss and the tactile structured constraint loss are weighted and summed to generate the total loss function.

[0055] Step S104: Based on the total loss function, update the network parameters of the action prediction world model until convergence, and obtain the action prediction world model.

[0056] The total loss value is backpropagated to all network layers of the model through chain derivatives. The gradient descent optimization algorithm is used to calculate the gradient layer by layer. The parameters of all encoding branches and transformer backbone networks of the model are corrected according to the preset learning rate. Each traversal of the entire historical dataset is recorded as a training epoch. After each training epoch, the validation set data is called to calculate the total loss of the validation set. The process of predicting features in S102 and constructing the total loss and updating parameters in reverse in S103 is continuously executed in a loop, and the changes in the validation set loss are monitored. When the decrease in the total loss of the validation set is less than the preset threshold within a set number of consecutive epochs, the model is determined to have reached the convergence condition, all training iterations are terminated, and an action prediction world model that can be deployed online is obtained.

[0057] The method for training a motion prediction world model for robot operation provided in this application uses historical datasets, including color visual sequences, piezoresistive electronic skin tactile pressure map sequences, robot motion sequences, and real samples, as input to the model. Based on the historical datasets, the method drives the motion prediction world model to predict future multimodal feature information, including future visual features, future tactile features, and future motion features. Then, using the future multimodal feature information and real samples, a total loss function is constructed, consisting of a multimodal flow matching loss and a tactile structured constraint loss determined based on real samples and future tactile features. The network parameters of the motion prediction world model are iteratively updated according to the total loss function until convergence. During the data input stage, piezoresistive electronic skin tactile pressure map sequences are incorporated to characterize the pressure distribution and physical interaction state of the contact interface between the robot and the object. This corrects the deviation between the temporal environmental change patterns derived solely from color visual sequences and the actual contact mechanics evolution process. The trained motion prediction world model can output motion-related content that conforms to real mechanical laws, improving the accuracy of the robot arm's motion trajectory.

[0058] Combination Figure 2 As shown, in one possible implementation, the action prediction world model includes a visual encoding branch, a tactile encoding branch, and an action encoding branch, all of which share the same transformer backbone network. The step S102 above, which uses the action prediction world model to predict future multimodal feature information based on historical datasets, may include the following steps: Step S2021: Based on the color visual sequence, use the visual coding branch to determine the visual lexical units corresponding to the color visual sequence.

[0059] The visual coding branch can indicate an independent neural network branch of the model used for compression, feature extraction, and serialization encoding of two-dimensional image temporal data.

[0060] Visual lexical units can indicate the fixed length of a color visual sequence after it has undergone visual coding branch convolution, mapping, and dimension normalization.

[0061] Specifically, after determining the color visual sequence, the color visual sequence can be input into the visual encoding branch to output the visual lexical units corresponding to the color visual sequence.

[0062] Step S2022: Based on the piezoresistive electronic skin tactile pressure map sequence, the tactile lexical units corresponding to the piezoresistive electronic skin tactile pressure map sequence are determined using tactile coding branches.

[0063] Piezoresistive electronic skin tactile pressure map sequence can indicate a time sequence composed of frame-by-frame two-dimensional pressure distribution maps periodically collected by the mechanical fingertip piezoresistive sensor array, which can be used to determine the pressure magnitude at various points in the contact area between the robotic arm and the object.

[0064] The tactile coding branch can be understood as a feature extraction network branch designed for two-dimensional pressure distribution maps.

[0065] Tactile lexical units can be understood as serialized embedded vector units generated after features are extracted from the tactile pressure map through the tactile coding branch and the dimensions are normalized.

[0066] Specifically, after determining the piezoresistive electronic skin tactile pressure map sequence, the piezoresistive electronic skin tactile pressure map sequence can be input into the tactile coding branch to output tactile lexical units.

[0067] In one specific implementation, this application uses two chips The tactile data acquired by the piezoresistive electronic skin array, and the raw tactile pressure map tensor obtained from a single time step acquisition, are as follows: ; This is the original piezoresistive electronic skin tactile pressure map tensor; To eliminate amplitude differences between different sensing units and unify the numerical range, each tactile sensing unit (taxel) is normalized point-by-point using the quantiles of the entire dataset. The normalization calculation formula is as follows: ;in, These refer to the height and width positions of the tactile channel and the electronic skin array, respectively. To normalize the tactile value, Specify the 1st percentile for the taxel position in historical data. ε is the 99th percentile of the specified taxel position in the historical data, where ε is a very small constant.

[0068] Based on the cross-modal temporal alignment rules mentioned above, the complete temporally normalized haptic tensor is defined as follows: ;in, For normalized tactile tensor, , These are the grid sizes of the tactile word space. This represents the lexical feature dimension.

[0069] To adapt to the sequence input format of the transformer backbone network, a haptic spatial tokenizer is set up. The mapping relationship is as follows: ; It is a tactile word sequence.

[0070] In a preferred embodiment, the tactile space grid division parameters are set to... , A single tactile time step is output by the word segmenter. Group tactile lexical units; perform word segmentation mapping on the complete temporal tactile tensor step by step, and concatenate to obtain the global tactile lexical sequence: ; Should This refers to the input haptic coding branch, which is used to determine the feature sequence of haptic terms. The specific mapping relationship is as follows: ;in, It is a tactile encoder.

[0071] The tactile spatial segmenter and tactile decoder work together in pairs. On the one hand, they compress the two-dimensional spatial tactile pressure map into one-dimensional serialized tactile words, adapting to the multi-head attention mechanism inside the transformer backbone network to achieve the fusion of visual, tactile, and action multimodal features. On the other hand, they can reconstruct the complete two-dimensional pressure distribution map from the tactile words predicted by the model, preserving the physical contact force meaning of the tactile data and ensuring that the model prediction results can be used to calculate the tactile structured constraint loss.

[0072] Step S2023: Based on the robot action sequence, use the action coding branch to determine the action lexical corresponding to the robot action sequence.

[0073] The action coding branch can be understood as a network branch that performs standardization, mapping, and encoding of one-dimensional motion time sequence values.

[0074] Action lexes can indicate the temporal embedding vector units generated after the robot action sequence has been mapped, normalized, and embedded.

[0075] Specifically, after determining the robot action sequence, the robot action sequence can be input into the action encoding branch to output the action lexes corresponding to the robot action sequence.

[0076] Specifically, step S2023 above may include the following steps: Step a1: Using the motion encoding branch, the robot motion sequence is mapped to a motion canvas of a preset dimension to obtain motion vectors.

[0077] The motion canvas can indicate a predefined structured vector space of fixed dimensions, used to organize disordered and scattered motion values.

[0078] Specifically, after determining the robot's action sequence, the action encoding branch can be further used to map the robot's action sequence to an action canvas of a preset dimension to obtain action vectors.

[0079] As an example, a motion canvas with a fixed size and a one-dimensional linear arrangement is predefined. The vector dimension of this motion canvas is fixed once set. Timing angle data for each of the six rotational joints of the six-axis robotic arm, as well as the opening and closing data of the end effector, are collected. The motion parameters are sequentially and continuously filled into the motion canvas space according to their chronological order, without dividing the various motion parameters into independent blocks or setting hierarchical structures. When the total length of the robot motion sequence to be filled is less than the predetermined dimension of the motion canvas, zero values ​​are uniformly filled into the empty positions at the end of the motion sequence to make it complete. When the timing length of the robot motion sequence exceeds the canvas's capacity, the excess portion is extended forward and truncated to a segment that fits the canvas length. After filling, padding, or truncation, the motion vector is output.

[0080] Step a2 involves performing channel-by-channel quantile normalization on the action vector, constraining the action values ​​corresponding to the action vector to a preset range, and obtaining the target action vector code.

[0081] The preset interval can be a pre-defined interval.

[0082] Target motion vector encoding can indicate that after the motion vector is normalized and scaled, all channel values ​​are limited to a preset range and the numerical distribution is balanced and regular.

[0083] Specifically, after determining the action vector, the action vector can be further processed by channel-by-channel quantile normalization to constrain the action value corresponding to the action vector to a preset range, thereby obtaining the target action vector code.

[0084] As an example, the lower quantile is selected as the lower limit boundary and the higher quantile as the upper limit boundary. Extreme and abnormal motion values ​​that deviate from the normal motion range are filtered out based on this set of boundary thresholds. Each motion parameter channel within the motion vector is used as a calculation unit, and a linear scaling transformation is applied to all values ​​within each channel, uniformly constraining all motion values ​​in each channel to a pre-defined interval. Each motion channel statistically analyzes its own numerical distribution characteristics and performs scaling mapping operations independently; the operations between channels do not interfere with each other, thus generating a standardized target motion vector code.

[0085] Step a3: Determine action lexical units based on the target action vector encoding.

[0086] Specifically, after determining the target action vector encoding, action lexical units can be further determined.

[0087] As an example, a single fully connected linear layer is selected as the dimension mapping module. The target action vector encoding obtained after normalization is used as the input data. The dimension mapping is performed through linear transformation operation to convert the original dimension of the target action vector encoding into a pre-set fixed standard embedding dimension. According to the temporal sequence, the action lexical corresponding to each sampling time point is generated sequentially.

[0088] In a specific implementation example, to ensure compatibility with the diverse native motion vectors acquired and output by robots of different structural configurations, this application uniformly maps various types of original robot motion vectors to a fixed 30-dimensional preset motion canvas. The mapped standard motion vector is denoted as... .

[0089] In a preferred embodiment, the fixed partitions of each channel of the 30-dimensional motion canvas are defined as follows: The channel interval [0:7] is used to store the pose data of the left arm end effector; The channel interval [7:14] is used to store the pose data of the right arm end effector; The channel interval [14:21] is used to store left arm joint motion data; The channel interval [21:28] is used to store right arm joint motion data; Channel 28 corresponds separately to the opening and closing amount of the left gripper; Channel 29 corresponds specifically to the opening / closing amount of the right gripper. Let the robot's native motion vector be... Construct a binary placement matrix After completing the heterogeneous motion dimension mapping, the formula for calculating the mapped 3D standard motion vector is as follows: ; For motion channels that do not exist in the current robot configuration, zero-value filling is uniformly adopted, which satisfies... hour, ,in, This is the set of motion canvas channel indices currently used by the robot. In a specific embodiment of a single-arm robot, the robot's native motion vector is 8-dimensional: ;in, The three-dimensional position coordinates of the robotic arm's end effector. Here, g is the end-effector quaternion, and g is the single gripper opening / closing control variable; the actual set of motion canvas channel indices used in this configuration is: .

[0090] The mapping rule is as follows: fill the end pose data of the original motion vector into the [0:7] channel of the motion canvas, fill the gripper opening / closing amount into the 28th channel, and fill all remaining dimensions of the motion canvas with zero values, that is: ; ; Based on the cross-modal temporal alignment rules mentioned above, the single-channel values ​​within the normalized complete robot motion sequence tensor satisfy: ; in, For action temporal tensor elements, it can represent the normalized action value of the j-th channel, the f-th latent frame, and the p-th action sampling point of the action canvas; Frame index representing visual latent variables. The number of action sampling steps corresponding to a single visual latent variable frame is represented. After completing the heterogeneous action dimension mapping, the standard action vector is standardized according to the channel-wise quantile normalization rule. The normalization calculation formula is: ; in, and ε and 99 represent the 1% quantile and 99% quantile of the j-th action channel in the entire historical dataset, respectively; ε is a minimal constant used to prevent calculation errors due to a zero denominator. These are the normalized action channel values; obtained after normalization processing. That is, the action vectors sent into the action encoding branch and mapped to the action canvas directly support the generation of action vectors in step a1, and then obtain action words through the subsequent encoding process.

[0091] Step S2024: Based on visual lexical units, tactile lexical units, and action lexical units, the future visual features, future tactile features, and future action sequences are predicted using the transformer backbone network.

[0092] Specifically, after determining visual, tactile, and action terms, these terms can be input into the transformer backbone network to predict future visual features, future tactile features, and future action sequences.

[0093] As an example, temporally aligned visual, tactile, and action lexical units are concatenated and fused into a multimodal contextual temporal sequence, which is then fed into the transformer backbone network. The transformer backbone network learns the linkage and evolution relationship between scene visual changes, tactile force fluctuations, and robotic arm movements within the historical temporal sequence based on a unidirectional causal multi-head attention mechanism. Following a sequential temporal logic of visual priority, tactile second, and action last, it deduces step by step and outputs future visual features, future tactile features, and future action sequences in sequence.

[0094] The motion prediction world model training method for robot operation provided in this application embodiment independently encodes color visual sequences, piezoresistive electronic skin tactile pressure map sequences, and robot motion sequences by utilizing visual encoding branches, tactile encoding branches, and motion encoding branches respectively. This method can determine visual, tactile, and motion words with unified dimensions and complete modal features, retaining the scene space features of the color visual sequence, the contact mechanical features of the piezoresistive electronic skin tactile pressure map sequence, and the motion temporal features of the robot motion sequence. This avoids feature loss and coupling interference caused by the mixing of multimodal information. The visual, tactile, and motion words are then input into the transformer backbone network for temporal extrapolation. Based on the contextual information of multimodal word fusion, future visual features, future tactile features, and future motion sequences are predicted. This solves the problem of extrapolating information about environmental and motion changes based solely on the single feature of the color visual sequence, thereby improving the accuracy of robot motion execution.

[0095] like Figure 3 As shown, in one possible implementation, the real samples include real future visual features, real future tactile features, and real future action sequences; the construction of the total loss function based on future multimodal feature information and real samples in step S103 above may include the following steps: Step S3031: Gaussian noise is added to the real future visual features, real future tactile features, and real future action sequences to obtain noisy visual samples, noisy tactile samples, and noisy action samples.

[0096] Noisy visual samples can be understood as noisy visual benchmark samples generated by superimposing Gaussian noise on real future visual features.

[0097] Noisy tactile samples can be understood as noisy tactile reference samples obtained by superimposing Gaussian noise on real future tactile features.

[0098] Noisy motion samples can be understood as noisy motion benchmark samples generated by superimposing Gaussian noise on real future motion sequences.

[0099] Specifically, after determining the real future visual features, real future tactile features, and real future action sequences, noisy visual samples, noisy tactile samples, and noisy action samples can be further determined.

[0100] As an example, Gaussian noise with a fixed standard deviation is configured for each of the three modalities. Specifically, real future visual features are superimposed with Gaussian noise of 0.02 to generate noisy visual samples; real future tactile features are superimposed with Gaussian noise of 0.03 to generate noisy tactile samples; and real future action sequences are superimposed with Gaussian noise of 0.01 to generate noisy action samples.

[0101] As an example, the noise amplitude is dynamically adjusted according to the training rounds: in the early stages of training, the model's fitting ability is weak, so a small noise level is set to avoid difficulty in training convergence; as the training rounds increase, the standard deviation of the Gaussian noise is gradually increased to continuously approach the maximum sensing noise under the robot's extreme working conditions; at the same time, the tactile noise level is finely adjusted according to the material of the interactive object. When grasping hard objects, the electronic skin noise is relatively small, while when grasping soft objects, the sensing disturbance caused by the deformation of the contact surface is greater, so the intensity of the tactile noise is increased accordingly.

[0102] Step S3032: Construct a multimodal flow matching loss based on noisy visual samples, noisy tactile samples, noisy action samples, and future multimodal feature information.

[0103] Specifically, after identifying noisy visual samples, noisy tactile samples, and noisy action samples, we can further combine future multimodal feature information to construct a multimodal flow matching loss.

[0104] As an example, we calculate the matching loss for three sets of modal flows. The visual loss weights are as follows: tactile loss weights Action loss weights ;in, , as well as The sum of the three losses is 1. The three losses are weighted and added together according to their corresponding weights to obtain the multimodal flow matching loss.

[0105] As an example, in constructing the multimodal flow matching loss, the visual fitting error between future visual features and noisy visual samples, the tactile fitting error between future tactile features and noisy tactile samples, and the action fitting error between future action features and noisy action samples are collected in real time. The fitting error values ​​of each of the three modalities are compared and calculated in real time. When the fitting error value of any modality is higher than that of the other two modalities, and the prediction deviation of that modality is large, the weight of the single-modal flow matching loss corresponding to that modality is increased, while the loss of the other two modalities with better fitting effects is reduced appropriately. The weights are used to amplify the proportion of errors in the overall multimodal flow matching loss of modes with larger deviations. When the model updates the network parameters based on backpropagation of the overall loss, it will prioritize reducing the prediction error of the mode, so that the model can focus on optimizing the temporal prediction effect of the mode with excessive deviation. As iterative training continues, the three modes are continuously optimized and adjusted. The fitting error difference between the visual mode, tactile mode and action mode continues to shrink, the fitting accuracy of the three modes gradually tends to be balanced, and the flow matching loss weights corresponding to each mode no longer change significantly. The weight values ​​of each type gradually converge to the pre-set fixed weight ratio.

[0106] Specifically, step S3032 above may include the following steps: Step b1: Determine the visual flow matching loss based on noisy visual samples and future visual features.

[0107] Specifically, the degree of deviation between the overall temporal probability distribution of noisy visual samples and the overall temporal probability distribution of future visual features can be determined, and the visual flow matching loss can be determined by the degree of deviation.

[0108] Step b2: Determine the tactile flow matching loss based on noisy tactile samples and future tactile features.

[0109] The tactile pressure map of each frame within the complete time series is traversed. The deviation of the pressure values ​​of each pixel point between the noisy tactile sample and the future tactile features is statistically analyzed using the pixel mean square error calculation method. During the calculation, the statistical range is defined, and error calculation is performed only on pixels in the effective force area where the robot arm and the object make contact. Pixels in the blank background area of ​​the map without pressure values ​​or contact interaction are excluded and not included in the loss value statistics. The error results corresponding to all effective pixels within the time series are summarized and the arithmetic mean is calculated. This average error value is determined as the tactile flow matching loss.

[0110] Step b3: Determine the action flow matching loss based on noisy action samples and future action features.

[0111] Specifically, for each joint temporal dimension, the Euclidean distance between the noisy motion sample and the future motion features is calculated to characterize the temporal motion deviation of a single joint; where the error assignments for all joints are exactly the same weight; the temporal error results of all joints are accumulated sequentially and the mean is calculated, and the final mean result is used as the motion flow matching loss.

[0112] Step b4 involves weighted fusion of the visual flow matching loss, tactile flow matching loss, and motion flow matching loss to obtain the multimodal flow matching loss.

[0113] Specifically, fixed weight ratios are pre-set for visual flow matching loss, tactile flow matching loss, and motion flow matching loss. Throughout the entire training cycle from the initial iteration to training convergence, the weight values ​​corresponding to the three types of modal losses remain constant and do not change with the training process or the interaction scenario. The three modal loss values ​​are multiplied by their respective preset weights and then summed to generate the multimodal flow matching loss.

[0114] Step S3033: Determine the clean tactile pressure map estimator based on noisy tactile samples and future tactile features.

[0115] The clean tactile pressure map estimator can indicate the noise-free tactile pressure distribution map restored after the offset is corrected for noisy tactile samples, removing sensor random noise.

[0116] Specifically, after determining the noisy tactile samples and future tactile features, a clean tactile pressure map estimator can be further determined.

[0117] As an example, after obtaining frame-by-frame time-series strictly aligned noisy tactile samples and future tactile features, the noisy tactile sample data and future tactile feature data of all time-series frames are used as joint input. A least-squares fitting algorithm is used to fit the overall error of the full-time-series tactile data, solving for the global tactile temporal evolution offset that characterizes the tactile modal noise perturbation pattern throughout the entire time series. After obtaining the global tactile temporal evolution offset, this offset is applied frame-by-frame to the noisy tactile samples at each time step, removing noise components from the noisy tactile samples frame by frame, and uniformly denoising and correcting all tactile pressure maps throughout the entire time series. Through this uniform denoising process across the entire time series, a clean tactile pressure map estimate is obtained.

[0118] In one possible implementation, the real sample (clean modality sample) corresponding to any modality of vision, touch, or action is denoted as . Perform the Gaussian noise sampling and noisy sample construction process: Sampling standard Gaussian noise distribution: ; Sample stream matching noise level parameters , used to control the intensity of noise superposition; Noisy samples are constructed based on clean modal samples, Gaussian noise, and noise level: ;in, For noisy modal samples, The noise mixing coefficient, It is random Gaussian noise; Define the true flow velocity corresponding to modal evolution as the difference between noise and clean samples: ;in, This represents the actual flow rate; Input the noisy samples, noise level parameters, language conditional vectors, and the transformer backbone network context cache K into the model to predict the model flow rate: Where K represents the context features or KV buffer of the transformer backbone network, and e is the language condition vector output by the text encoder. To predict flow velocity for the model, The sampling parameters for flow matching are defined. Based on the difference between the predicted flow velocity and the actual flow velocity, a standard flow matching loss is constructed using a modal effective mask. The loss expression is as follows: ;in, The weighting function corresponds to the noise level. For the corresponding modality, a binary effective mask is used (no mask for visual modality, tactile effective mask for tactile modality, and action effective mask for action modality). This is an element-wise multiplication operation. For flow matching loss, a mask can mask padding values, invalid backgrounds, and missing channels, preventing invalid data from participating in the loss error statistics. The above flow matching calculation logic is uniformly applicable to the three major branches of vision, tactile sensing, and robot motion, respectively solving for three types of single-modal flow matching losses: visual flow matching loss. tactile flow matching loss Action flow matching loss .

[0119] Among them, the visual branch uses real future visual features as clean samples, and the flow matching target is the evolution speed of future visual features; the tactile branch uses real future tactile features as clean samples, and the flow matching target is the evolution speed of future tactile features or tactile latent variables; the action branch uses real future action sequences as clean samples, and the flow matching target is the action evolution speed corresponding to the action canvas of the preset dimension.

[0120] Specifically, step S3033 above may include the following steps: Step c1: Based on noisy tactile samples and future tactile features, fit and solve the temporal evolution offset of tactile modalities.

[0121] The temporal evolution offset of tactile modal can indicate the deviation compensation parameter obtained by fitting two sets of tactile data. It can comprehensively characterize the disturbance deviation caused by Gaussian noise on the amplitude and spatial distribution of tactile pressure throughout the entire time series.

[0122] Specifically, after determining the noisy tactile samples and future tactile features, the temporal evolution offset of the tactile modality can be further determined.

[0123] As an example, the least squares global fitting algorithm is used to summarize the numerical deviation information of all pixels in the tactile pressure map of each frame in the time series, and the overall fitting solution is completed by combining all pixel deviations to calculate the temporal evolution offset of the tactile modality.

[0124] As an example, the single-frame tactile pressure map is divided into three major blocks: the central contact area, the edge transition area, and the blank background area. For each block, the corresponding noisy tactile sample pixel data and future tactile feature pixel data are retrieved, and the temporal evolution offset of the region is fitted independently for each block.

[0125] Step c2: Noise reduction and correction are performed on the noisy tactile samples based on the tactile temporal evolution offset to obtain a clean tactile pressure map estimate.

[0126] Specifically, after determining the tactile temporal evolution offset, the noisy tactile samples can be further denoised to obtain a clean tactile pressure map estimate.

[0127] As an example, the computation process consistently uses the globally unified solution to obtain the tactile temporal evolution offset. Noisy tactile maps are retrieved frame by frame in chronological order, and a uniform subtraction correction operation is performed on all pixels within the map to eliminate the global noise components superimposed on the tactile data in one go. During the noise reduction stage, the same correction magnitude is applied to blank background areas and the effective force contact areas of the robotic arm, ensuring consistent noise reduction standards throughout the process. After all temporal frames have undergone numerical correction and noise removal, all corrected temporal tactile maps are integrated to output a clean tactile pressure map estimate for the complete temporal sequence.

[0128] Step S3034: Determine the tactile structured constraint loss based on the clean tactile pressure map estimator.

[0129] Specifically, after determining the clean tactile stress map estimator, the tactile structured constraint loss can be further determined.

[0130] As an example, spatial distribution features such as pressure magnitude and contact area contour at various locations within the clean tactile pressure map estimator are extracted to calculate the tactile contact structure loss, constraining the spatial morphology of each frame of the two-dimensional pressure map to conform to the real physical appearance of the robot arm contacting the object; all time indices of the continuous time series are traversed, and the temporal difference features of adjacent frames of the clean tactile pressure map estimator and the temporal difference features of adjacent frames of the real future tactile features are obtained respectively, and the tactile temporal smoothing loss is calculated by comparing the two sets of difference features; the tactile contact structure loss and the tactile temporal smoothing loss are weighted and fused to generate the tactile structured constraint loss.

[0131] In one possible implementation, for the flow matching prediction task of tactile modalities, the tactile branch predicted flow velocity directly output by the model is only an abstract evolution vector, which itself does not have physical meaning such as pressure distribution and contact contour, and cannot be directly used to constrain the spatial morphology of the contact between the robot and the object. To address this, this application adds a tactile atlas reverse reconstruction step. Based on noisy tactile samples and the model-predicted tactile flow velocity, a clean tactile pressure map estimate with complete physical pressure meaning is obtained by reverse calculation. The reconstruction calculation formula is as follows: ;in, This is a noisy tactile sample. For flow matching noise level parameters, Predict flow velocity for the tactile branch of the model output. This is the estimate of the clean tactile pressure map obtained from the reconstruction. Mathematical verification can be completed by combining the previously discussed flow-matching sample construction and the definition of true flow velocity: when the model perfectly fits, the predicted flow velocity is equal to the true flow velocity, i.e. Substituting this equation into the reconstructed formula, we can derive: ;in, For a truly clean and tactile pressure feel.

[0132] The derivation results demonstrate that when the model's tactile flow prediction is completely accurate, the clean tactile pressure map estimate output by the reconstruction is completely consistent with the actual future tactile characteristics. Based on the above mechanism, this application limits the application of tactile contact structure loss to the clean tactile pressure map estimate obtained from the reconstruction, rather than directly applying structural constraints to the tactile branch prediction flow velocity, which has no physical meaning. This design is a key technical feature for ensuring that the tactile structured constraints have real contact mechanics physical meaning, solving the problem that simply supervising the flow vector cannot constrain the tactile two-dimensional spatial pressure distribution and contact contour deformation, and providing an interpretable and quantifiable physical tactile benchmark map for subsequent tactile modal denoising and tactile contact structure loss calculation.

[0133] Specifically, step S3034 above may include the following steps: Step d1: Calculate the tactile contact structure loss based on the spatial distribution characteristics of the clean tactile pressure map estimator.

[0134] Spatial distribution characteristics can indicate the overall spatial attributes of each pixel within a single frame of tactile pressure map, including pressure amplitude, contact area contour range, pressure gradient changes, and contact surface deformation.

[0135] The tactile contact structure loss can indicate the loss term of the quantification model predicting the spatial structure deviation of the tactile map, with the spatial pressure morphology of the clean tactile pressure map estimate as a reference. This is used to constrain the tactile pressure distribution of each frame to fit the actual contact appearance between the robot and the object.

[0136] Specifically, after determining the spatial distribution characteristics of the clean tactile pressure map estimator, the tactile contact structure loss can be further calculated.

[0137] As an example, the algorithm iterates through all pixel regions of the clean tactile pressure map estimate frame by frame, calculates the mean square error between the future tactile features and the pixel-wise pressure value of the clean tactile pressure map estimate, and only counts the error of the effective contact pixels of the robot hand. Blank, non-contact background areas are not included in the loss calculation. The average value of all effective pixel errors is used to obtain the tactile contact structure loss.

[0138] As an example, information on pressure gradient changes within the clean tactile pressure map estimator is extracted. Higher error weights are assigned to the contact center region with sharp pressure gradients, while lower weights are assigned to the contact edge region with gentler pressure. The tactile contact structure loss is obtained by weighting the statistical pixel errors.

[0139] Step d2: Calculate the tactile temporal smoothing loss based on the clean tactile stress map estimate and the true future tactile features under continuous time series.

[0140] The tactile temporal smoothing loss can indicate the loss term obtained by solving the difference in tactile pressure changes between adjacent time frames, and is used to constrain the gradual change of tactile force in continuous time moments.

[0141] Specifically, after determining the clean tactile stress map estimator and the true future tactile features under continuous time series, the tactile temporal smoothing loss can be further calculated.

[0142] As an example, the overall pressure difference between two adjacent frames of the clean tactile pressure map estimate and the pressure difference between two adjacent frames of the true future tactile features are calculated sequentially, and the mean square error between the two sets of differences is obtained; the average value of the errors of all adjacent frames in the time series is accumulated to obtain the tactile temporal smoothing loss.

[0143] In one possible implementation, this application applies dynamic constraints to the continuous temporal tactile map through tactile temporal smoothing loss (also known as tactile temporal consistency loss), ensuring that the pressure changes at adjacent time points conform to the real physical interaction laws of the robotic arm grasping, sliding, and releasing, and suppressing the occurrence of high-frequency pressure flickering without physical basis in the model prediction.

[0144] Let the index of the complete tactile time sequence be... This refers to the unified tactile sequence length after cross-modal temporal alignment mentioned earlier.

[0145] Temporal differences between adjacent frames are calculated based on the reconstructed clean tactile pressure map estimate and the true future tactile features, respectively: Predicted side-haptic inter-frame difference (clean haptic pressure map estimator inter-frame difference): ;in, To predict the inter-frame difference of side haptic feedback; Truth-side temporal difference (inter-frame difference of real future tactile features): ; This represents the inter-frame difference of the haptic feedback on the truth side.

[0146] To ensure that the pixels participating in the timing error statistics are valid interactive data in both the current and next frames, a joint valid mask for adjacent frames is constructed: ;in, For the joint effective mask of adjacent frames, The valid mask for the current frame. Used as the effective mask for the next frame.

[0147] A pixel pair is included in the temporal loss calculation only when both the current and next haptic effective masks are 1. If any frame contains filled values, blank non-contact areas, or missing sensor channels, the pixel pair is directly masked and does not participate in error accumulation. Based on two sets of temporal differences and the joint mask of adjacent frames, a normalized haptic temporal smoothing loss is constructed: ;in, This is a loss due to tactile temporal smoothing.

[0148] The tactile temporal smoothing loss is used to prevent division by zero anomalies caused by summing all adjacent masks in the denominator. The core function of this tactile temporal smoothing loss is to constrain the model to learn the dynamic evolution of tactile sensation in real robots, eliminate abrupt and high-frequency pressure jumps and flickering in the predicted sequence, and enable the model output to continuously convey various realistic interactive dynamic processes such as the establishment of contact, slow pressure increase, object surface slippage, gripping and locking, and release.

[0149] Specifically, step d2 above may include the following steps: Step d21: For each time index in the continuous time series, perform difference calculation on the clean tactile pressure map estimate at the current time and the clean tactile pressure map estimate at the next time to obtain the predicted tactile temporal difference features.

[0150] Predicted tactile temporal difference features can indicate the complete temporal change feature sequence formed by summarizing the pressure difference values ​​of the predicted tactile maps of adjacent frames at each time step, which can be used to characterize the dynamic fluctuation law of fingertip force predicted by the model.

[0151] Specifically, after determining the clean tactile pressure map estimator, for each time index in the continuous time series, the difference operation can be performed on the clean tactile pressure map estimator at the current time and the clean tactile pressure map estimator at the next time to obtain the predicted tactile temporal difference features.

[0152] As an example, the tactile maps of the current time step and the next time step, which are adjacent in time sequence, are selected. The corresponding pixels in the two maps are then subtracted sequentially to obtain the pixel-by-pixel difference, which reflects the change in fingertip pressure between adjacent sampling times. After traversing all time steps, the inter-frame difference data from all time steps are aggregated and arranged in an ordered manner to generate the predicted tactile temporal difference features.

[0153] Step d22: For each time index in the continuous time series, perform difference operations on the real future tactile features at the current time and the real future tactile features at the next time to obtain the real tactile temporal difference features.

[0154] The real tactile temporal difference feature can indicate the temporal change sequence obtained by calculating the difference between adjacent frames of the true tactile map, and can be used to characterize the gradual evolution of fingertip force in the physical world.

[0155] Specifically, two temporally adjacent frames are selected, and the first-order temporal difference is solved by pixel-by-pixel subtraction to obtain the real tactile temporal difference features.

[0156] For example, a certain tactile time series includes three real future tactile features corresponding to three time points t0, t1, and t2. Each real future tactile feature is simplified to three pixel pressure values: the pixel values ​​for t0 are 15, 26, and 8, the pixel values ​​for t1 are 13, 22, and 9, and the pixel values ​​for t2 are 10, 18, and 11. Pairwise subtraction operations are performed on each pair of temporally adjacent maps. First, the corresponding pixels of the t0 map are subtracted from each pixel of the t1 map to obtain differences of -2, -4, and 1. Then, the corresponding pixels of the t1 map are subtracted from each pixel of the t2 map to obtain differences of -3, -4, and 2. The two differences are then integrated according to the temporal arrangement to generate real tactile time series difference features.

[0157] Step d23: Based on the predicted tactile temporal difference features and the actual tactile temporal difference features, determine the tactile temporal smoothing loss.

[0158] Specifically, the predicted tactile temporal difference features with perfectly aligned temporal length, map size, and pixel arrangement are compared and calculated item by item with the actual tactile temporal difference features. The overall deviation values ​​of the two sets of temporal difference features are statistically analyzed and integrated to obtain the tactile temporal smoothing loss.

[0159] Step d3 involves weighted fusion of the tactile contact structure loss and the tactile temporal smoothing loss to obtain the tactile structured constraint loss.

[0160] Specifically, after determining the tactile contact structure loss and the tactile temporal smoothing loss, the tactile structured constraint loss can be further determined.

[0161] As an example, the tactile contact structure loss and tactile time smoothing loss are assigned equal weights, and the weight values ​​of the two losses are kept consistent. The tactile structure constraint loss is obtained by weighted summation of the two losses.

[0162] In a specific implementation example, this application performs spatial dimension constraints based on the clean tactile pressure map estimate obtained from the reconstruction, and accurately predicts the contact area contour and pressure amplitude distribution between the robot and the object through a tactile contact structure loss supervision model.

[0163] First, based on the contact pressure threshold γ, a binary mask of the real contact area is constructed from the actual future tactile features. The binary determination formula for a single-point sensing unit is as follows: In this context, the indicator function 1[.] outputs 1 to indicate that the sensing unit has effective contact, and outputs 0 to indicate a non-contact blank area; t is the timing index, c is the tactile channel index, and h and w are the height and width dimension indices of the tactile array, respectively. Binary labels for actual contact areas; The reconstructed clean tactile pressure map estimate is scaled and mapped to obtain the contact prediction logit feature: ; Where α is the logit scaling factor, used to adjust the gradient scale of the classification loss. A tactile contact structure loss is constructed by combining a tactile effective mask. A binary cross-entropy loss with logits (BCEWithLogits) is used to statistically analyze the prediction bias of the contact area pixel by pixel. Simultaneously, fill values, invalid backgrounds, and missing sensor channels are masked. The loss calculation formula is: ;in, For tactile contact structure loss, The cross-entropy of a binary classification system with logits; In an optional optimization implementation, an additional spatial gradient structure loss is introduced to constrain the pressure gradient change trend of the predicted tactile map to match the actual future tactile features, thereby enhancing the ability to restore details of contact surface deformation and pressure transition edges. The expression for the spatial gradient structure loss is: ;in, This is a two-dimensional spatial gradient operator for the height and width dimensions of the tactile array. It also filters out invalid pixels using a tactile effective mask, calculating gradient deviations only in the actual contact areas. During training, the tactile contact structure loss and the optional spatial gradient structure loss can be weighted and summed to form a complete tactile spatial distribution constraint term.

[0164] Step S3035: Perform weighted fusion processing on the multimodal flow matching loss and the tactile structured constraint loss to obtain the total loss function.

[0165] Specifically, after determining the multimodal flow matching loss and the haptic structured constraint loss, the total loss function can be further determined.

[0166] As an example, fixed weight coefficients are pre-set for the multimodal flow matching loss and the tactile structured constraint loss. The calculation results of the multimodal flow matching loss and the tactile structured constraint loss are multiplied by their respective weight coefficients in sequence and then summed to generate the total loss function.

[0167] In one possible implementation example, the total loss function is expressed as: ; in, Matching loss weights to visual flow Matching loss weights to the action flow Matching loss weights to haptic flow Weights for tactile contact structure loss, The haptic temporal smoothing loss weights are pre-defined non-negative constants used to balance the contribution ratio of multimodal flow matching constraints and haptic structured constraints in the overall training optimization.

[0168] By adding a spatial gradient structure loss term to the base total loss, the complete total loss function is: ;in, This is an optional spatial gradient structure loss.

[0169] The method for training a world model for robot operation motion prediction provided in this application adds Gaussian noise to real future visual features, real future tactile features, and real future action sequences to obtain noisy visual samples, noisy tactile samples, and noisy action samples. This simulates real sensor acquisition interference, improves the generalization ability of the model for multimodal temporal prediction, and simulates the interference from real sensor acquisition. The multimodal flow matching loss constructed based on noisy visual samples, noisy tactile samples, noisy action samples, and future multimodal feature information can fit the overall temporal evolution law of visual, tactile, and action multimodal features, ensuring the overall consistency of multimodal prediction. Simultaneously, based on noisy tactile features... The clean tactile stress map estimator is determined based on tactile samples and future tactile features, which can effectively eliminate the interference of Gaussian noise on tactile features and obtain tactile representations that fit the real physical interaction state. Then, the tactile structured constraint loss is determined based on the clean tactile stress map estimator, which can specifically constrain the spatial structure rationality and temporal smoothness of tactile prediction. Finally, the total loss function is obtained by weighted fusion of multimodal flow matching loss and tactile structured constraint loss, which can optimize the prediction details of tactile modality while taking into account the global temporal fitting accuracy of multimodal data, and solve the problem that a single loss constraint cannot take into account both global multimodal matching and accurate tactile structured prediction.

[0170] This embodiment provides a motion prediction method for robot operations, which can be used in computer devices such as computers and servers. Figure 4 This is a flowchart of a motion prediction method for robot operations according to an embodiment of this application, such as... Figure 4 As shown, the process includes the following steps: Step S401: Obtain the context sequence data of the robot in the current environment; wherein, the context sequence data includes: color visual sequence, piezoresistive electronic skin tactile pressure map sequence, and robot action sequence.

[0171] Context sequence data can instruct the robot to continuously perform interactive operations within the working environment. Onboard sensing devices synchronously and continuously collect multi-dimensional time-series sensing data, which are then spliced ​​together after aligning the timestamps to form a continuous historical time sequence segment.

[0172] Specifically, after the robot starts its operation, it synchronously collects color visual sequences, piezoresistive electronic skin tactile pressure map sequences, and robot motion sequences, and uniformly labels global timestamps to achieve temporal alignment. It then sequentially completes sparse sampling of visual frames, normalization of tactile pressure maps per tactile sensor unit, mapping of robot motion vectors to a motion canvas of a preset dimension, and unifying the temporal length by padding with zeros or truncating. Simultaneously, it generates effective motion masks and effective tactile masks, and outputs temporal segments adapted to the input specifications of the converter backbone network as context sequence data.

[0173] Step S402: Based on the context sequence data, using the action prediction world model, perform the determination operation for the future action sequence; wherein, the determination operation includes: Predicting future visual features based on color visual sequences; Predict future tactile features based on future visual features and contextual sequence data; Based on preset language conditions, context sequence data, future visual features, and future tactile features, the future action sequence is determined.

[0174] Preset language conditions can indicate the fixed-dimensional feature vector obtained by encoding the natural language instructions of the task by the TextEncoder.

[0175] Specifically, following a hierarchical, progressive multimodal deduction logic that first processes vision, then touch, and finally generates motion, the regularized context sequence data output from the preceding steps is processed sequentially: First, the color visual sequence is encoded using a visual encoding branch to obtain visual lexical units and stored in a cache. Then, the future visual features are obtained through visual modality diffusion denoising iterative operation and the cache content is updated. Next, the generated future visual features and the original context sequence are fused and fed into the tactile diffusion network. After iterative denoising, the complete future tactile features are obtained and encoded as tactile lexical units to supplement the cache. Finally, the preset language conditions, the original context lexical units, the predicted future visual features and the future tactile features are combined and fed into the action diffusion network. After outputting the action canvas, the robot's original temporal instructions, i.e., the future action sequence, are restored through reverse mapping.

[0176] Step S403: Control the robot to execute the future action sequence.

[0177] Specifically, the control commands corresponding to the future action sequence are transmitted to the robot's underlying motion driver. The driver sequentially parses the end-effector pose, joint motion, and gripper opening and closing parameters according to the time sequence and outputs control signals to drive the robotic arm to run.

[0178] As an example, let's take a single-arm collaborative robot grasping a water cup: the output sequence of future actions includes multiple sets of control parameters. In the first stage, the robot arm is driven to lift and translate by reading the angle parameters of each joint, and the end effector is moved to a preset position on the side of the water cup. In the second stage, the end effector posture parameters are analyzed to adjust the orientation of the end effector, so that the fingertip piezoresistive electronic skin fits against the outer wall of the water cup. In the third stage, the gripper opening and closing parameters are read to control the gripper to slowly close and contact the water cup to form a grip.

[0179] In a possible example, a temporal context cache is constructed based on multimodal tokens and language conditions, future visual features, future piezoresistive electronic skin tactile pressure maps, and future robot action sequences are deduced round by round, and rolling update of the cache is realized in combination with real machine observation. Let the transformer key-value cache corresponding to the r-th round of rolling inference be recorded as , the cache uniformly stores historical visual tokens, historical tactile tokens, historical action tokens and the fixed language condition vector e internally, and the cache set is defined as: ; where, is the transformer key-value cache of the r-th round, ≤r indicates the visual / tactile tokens up to the current round, , , are all historical multimodal tokens in the cache, and <r indicates the action tokens before the current round.

[0180] When each round of rolling inference is started, the currently real-time collected visual observation, tactile observation, and action performed at the previous moment are encoded by the corresponding tokenizer and then updated to the cache, and the cache update operation is expressed as: ; where, , , are the visual / tactile / action encoding mappings respectively; Initialize the visual noise tensor at the last step of diffusion: ; where, is the visual noise tensor at the last step; Iterate in reverse order from the maximum denoising step to to complete stepwise denoising: ; is the intermediate quantity of iterative denoising, indicates the visual flow prediction branch; ; where, is the noise position of the q-th denoising step, is the future visual latent variable of the q-th step, is the future visual latent variable of the q-1-th step; When iterating to the denoising step , the complete predicted future visual features are output: ; is the future visual result obtained when q=0; Write the encoded tokens of the predicted future visual features into the current round cache: ; The future piezoresistive electronic skin tactile pressure mapping prediction tactile modality diffusion inference process is symmetrical with the visual modality architecture: Initialize the final step of diffusion: tactile noise tensor ; Reverse the entire diffusion denoising step to complete the flow velocity prediction and denoising update: ; For flow velocity, ; For the (q-1)th step, the future haptic candidate tensor, For the q-th step future haptic candidate tensor After iteration, the complete sequence of predicted future tactile features is output: ; For future tactile feature sequences; The haptic segmenter is invoked to encode the predicted haptic graph into haptic terms and write them into the current round's context cache. ; The future robot motion prediction process includes motion modality diffusion inference, canvas inverse mapping, and motion output. Initialize the final step of the diffusion noise tensor: ;in, This is the noise tensor for the final action step; Reverse the order of action diffusion denoising iterations: ; ;in, This is the candidate tensor for the q-th step future action; Output a standardized 30-dimensional preset action canvas tensor: ; This represents the future action result obtained when q=0.

[0181] Operator selection based on action inverse The 30-dimensional standard motion canvas is reversed to restore the robot's native motion vectors: ; For the robot's native motion vectors, Place a matrix for the action The corresponding reverse index mapping operator; Output a continuous native motion fragment of length H: Among them, continuous native motion segments are sent to the robotic arm for execution; where H is a configurable parameter, which can be either the full prediction block length or a shorter scroll prediction field of view execution length.

[0182] After the robotic arm completes a motion segment of length H, the sensors acquire a new visual observation for the next moment. tactile observation Perform a cache rollover operation (Roll[.]) to generate a new context cache for the next round of inference: ; After the cache rolling update is completed, the next round of block rolling prediction loop begins. The core innovative advantage of this rolling cache inference mechanism is that the planned actions output by the robot are not only generated based on the current single-frame observation. The action generation process explicitly relies on two types of predicted consequences: future visual evolution and future tactile force changes, which are derived from the model. This aligns with the physical dynamics of real robotic arm grasping and sliding interactions, improving the stability and accuracy of long-term robot operation trajectories.

[0183] In one possible implementation, in a real-world robot deployment scenario, the color vision sequence acquisition and transmission process, the piezoresistive electronic skin tactile pressure map acquisition and transmission process, and the robot motion execution process will each generate inherent transmission delays. The three types of modal data cannot achieve ideal temporal synchronization, and the visual observations, tactile observations, and historical motion commands received by the controller at the same inference timing node correspond to different physical moments. To address this, this application adds a forward dynamics alignment module. Based on the multimodal observation data with temporal discrepancies, it calculates the current temporally unified hidden state and then completes the first round of cache content updates based on this unified hidden state. The forward dynamics alignment module takes into account the color vision observations with delay discrepancies, the piezoresistive electronic skin tactile pressure map observations, and the robot's historical motion sequences over continuous time periods. Through dynamics deduction and calculation, it obtains the current temporally aligned unified hidden state. At the start of each round of rolling inference, instead of directly writing the original asynchronous observation data into the cache, the calculated temporally unified hidden state is updated into the cache.

[0184] In each round of rolling inference, the original multimodal observations with temporal misalignment are no longer directly written to the cache; instead, a cache update operation is performed.

[0185] This implementation can compensate for the inherent time difference between visual modality, tactile modality, and action execution links. Even if there is a continuous asynchronous delay in the three types of perception and execution links during actual device operation, it can still provide unified context information for the timing reference of the converter block rolling prediction, stably maintain the timing logic continuity of the entire rolling prediction mechanism, and avoid the problem of prediction deviation of future visual features, future tactile features, and future action sequences caused by multimodal timing misalignment.

[0186] This embodiment also provides a motion prediction world model training device for robot operations, which is used to implement the above embodiments and preferred embodiments, and will not be repeated as already described. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0187] This embodiment provides a training device for a motion prediction world model for robot operations, comprising: an acquisition module for acquiring historical datasets, wherein the historical datasets include color visual sequences, piezoresistive electronic skin tactile pressure map sequences, robot action sequences, and real samples; a prediction module for predicting future multimodal feature information based on the historical datasets and using the motion prediction world model to be processed, wherein the future multimodal feature information includes future visual features, future tactile features, and future action features; a construction module for constructing a total loss function based on the future multimodal feature information and real samples, wherein the total loss function includes multimodal flow matching loss and tactile structured constraint loss, the tactile structured constraint loss being determined by real samples and future tactile features; and an update module for updating the network parameters of the motion prediction world model based on the total loss function until convergence, thereby obtaining the motion prediction world model.

[0188] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0189] In this embodiment, the motion prediction world model training device for robot operation is presented in the form of functional units. Here, a unit refers to an ASIC (Application Specific Integrated Circuit), a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0190] This application also provides a computer device having the above-described motion prediction world model training apparatus for robot operations.

[0191] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of this application, such as... Figure 5As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 5 Take a processor 10 as an example.

[0192] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0193] The memory 20 stores instructions executable by at least one processor 10 to cause at least one processor 10 to perform the method shown in the above embodiments.

[0194] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0195] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0196] The computer device also includes an input device 30 and an output device 40. The processor 10, memory 20, input device 30, and output device 40 can be connected via a bus or other means. Figure 5 Taking the example of a connection between China and Israel via a bus.

[0197] Input device 30 can receive input numerical or character information, and generate key signal inputs related to user settings and function control of the computer device, such as a touchscreen, keypad, mouse, trackpad, touchpad, joystick, one or more mouse buttons, trackball, joystick, etc. Output device 40 may include display devices, auxiliary lighting devices (e.g., LEDs), and haptic feedback devices (e.g., vibration motors). The aforementioned display devices include, but are not limited to, liquid crystal displays, light-emitting diodes, displays, and plasma displays. In some alternative embodiments, the display device may be a touchscreen.

[0198] The computer device also includes a communication interface for communicating with other devices or communication networks.

[0199] This application also provides a computer-readable storage medium. The methods described in this application can be implemented in hardware or firmware, or implemented as recordable on a storage medium, or implemented as computer code downloaded over a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and subsequently stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium may also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the methods shown in the above embodiments are implemented.

[0200] A portion of this application can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to this application through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0201] Although embodiments of this application have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of this application, and all such modifications and variations fall within the scope defined by the appended claims.

Claims

1. A method for training a world model for action prediction in robot operations, characterized in that, The method is applied to an action prediction world model; wherein the method includes: Acquire historical datasets; wherein, the historical datasets include color visual sequences, piezoresistive electronic skin tactile pressure map sequences, robot action sequences, and real samples; Based on the historical dataset, a world prediction model for the action to be processed is used to predict future multimodal feature information; wherein, the future multimodal feature information includes future visual features, future tactile features, and future action features; Based on the future multimodal feature information and the real samples, a total loss function is constructed; wherein, the total loss function includes multimodal flow matching loss and tactile structured constraint loss, and the tactile structured constraint loss is determined by the real samples and future tactile features; Based on the total loss function, the network parameters of the action prediction world model are updated until convergence, thus obtaining the action prediction world model.

2. The method for training a motion prediction world model for robot operations according to claim 1, characterized in that, The action prediction world model includes a visual encoding branch, a tactile encoding branch, and an action encoding branch, all of which share the same transformer backbone network. The step of predicting future multimodal feature information based on the historical dataset using the action prediction world model to be processed includes: Based on the color visual sequence, the visual coding branch is used to determine the visual lexical units corresponding to the color visual sequence; Based on the piezoresistive electronic skin tactile pressure map sequence, the tactile lexical units corresponding to the piezoresistive electronic skin tactile pressure map sequence are determined using the tactile coding branch; Based on the robot action sequence, the action encoding branch is used to determine the action lexical units corresponding to the robot action sequence; Based on the visual lexical units, the tactile lexical units, and the action lexical units, the future visual features, future tactile features, and future action sequences are predicted using the transformer backbone network.

3. The method for training a motion prediction world model for robot operations according to claim 1, characterized in that, The real samples include real future visual features, real future tactile features, and real future action sequences; the construction of a total loss function based on the future multimodal feature information and the real samples includes: Gaussian noise is added to the real future visual features, real future tactile features, and real future action sequences respectively to obtain noisy visual samples, noisy tactile samples, and noisy action samples. Based on the noisy visual samples, the noisy tactile samples, the noisy action samples, and the future multimodal feature information, a multimodal flow matching loss is constructed; Based on the noisy tactile samples and the future tactile features, determine the clean tactile pressure map estimator; Based on the clean tactile pressure map estimate, the tactile structured constraint loss is determined; The multimodal flow matching loss and the tactile structured constraint loss are weighted and fused to obtain the total loss function.

4. The method for training a motion prediction world model for robot operations according to claim 3, characterized in that, The determination of the tactile structured constraint loss based on the clean tactile pressure map estimate includes: Based on the spatial distribution characteristics of the clean tactile pressure map estimate, the tactile contact structure loss is calculated; Based on the clean tactile pressure map estimate under continuous time series and the real future tactile features, calculate the tactile time smoothing loss; The tactile contact structure loss and the tactile temporal smoothing loss are weighted and fused to obtain the tactile structured constraint loss.

5. The method for training a motion prediction world model for robot operations according to claim 3, characterized in that, Based on noisy visual samples, noisy tactile samples, noisy action samples, and future multimodal feature information, a multimodal flow matching loss is constructed, including: Based on the noisy visual samples and the future visual features, determine the visual flow matching loss; Based on the noisy tactile samples and the future tactile features, determine the tactile flow matching loss; Based on the noisy action samples and the future action features, determine the action flow matching loss; The visual flow matching loss, tactile flow matching loss, and motion flow matching loss are weighted and fused to obtain the multimodal flow matching loss.

6. The method for training a motion prediction world model for robot operations according to claim 2, characterized in that, The step of determining the action lexical corresponding to the robot action sequence using the action encoding branch, based on the robot action sequence, includes: Using the motion encoding branch, the robot motion sequence is mapped to a motion canvas of a preset dimension to obtain motion vectors; The action vector is subjected to channel-by-channel quantile normalization, and the action value corresponding to the action vector is constrained to a preset range to obtain the target action vector code. The action lexical is determined based on the target action vector encoding.

7. The method for training a motion prediction world model for robot operations according to claim 3, characterized in that, The determination of the clean tactile pressure map estimator based on noisy tactile samples and future tactile features includes: Based on the noisy tactile samples and the future tactile features, the temporal evolution offset of the tactile modality is fitted and solved. The noisy tactile samples are denoised and corrected based on the tactile temporal evolution offset to obtain the clean tactile pressure map estimate.

8. The method for training a motion prediction world model for robot operations according to claim 4, characterized in that, The calculation of tactile temporal smoothing loss based on the clean tactile pressure map estimate under continuous time series and the true future tactile features includes: For each time index within a continuous time series, the difference operation is performed on the clean tactile pressure map estimate at the current time and the clean tactile pressure map estimate at the next time to obtain the predicted tactile temporal difference features. For each time index within a continuous time series, the difference operation is performed on the real future tactile features at the current time and the real future tactile features at the next time to obtain the real tactile temporal difference features; Based on the predicted tactile temporal difference features and the actual tactile temporal difference features, the tactile temporal smoothing loss is determined.

9. A motion prediction method for robot operations, characterized in that, The method includes: Acquire context sequence data of the robot in the current environment; wherein, the context sequence data includes: color visual sequence, piezoresistive electronic skin tactile pressure map sequence, and robot action sequence; Based on the context sequence data, a future action sequence determination operation is performed using a pre-trained action prediction world model; wherein the pre-trained action prediction world model is obtained using the action prediction world model training method for robot operations as described in any one of claims 1-8, and the determination operation includes: Based on the color visual sequence, predict future visual features; Based on the future visual features and contextual sequence data, predict future tactile features; The future action sequence is determined based on preset language conditions, context sequence data, future visual features, and future tactile features; Control the robot to execute the future action sequence.

10. A computer device, characterized in that, include: A memory and a processor are communicatively connected, the memory stores computer instructions, and the processor executes the computer instructions to perform the motion prediction world model training method for robot operations as described in any one of claims 1 to 8 and / or the motion prediction method for robot operations as described in claim 9.

Images