Robot tactile representation training method based on mechanical reconstruction constraint and related product

Abstract

The application is suitable for the technical field of robot training, and provides a robot haptic representation training method based on mechanical reconstruction constraints and related products.The method comprises the following steps: collecting a first haptic image, a first dense local force field and a first grid deformation flow, performing parameter optimization and parameter freezing on a first haptic encoder and a first haptic reconstructor to obtain a second haptic encoder and a second haptic reconstructor; calculating a visual prediction loss value, a haptic latent vector prediction loss value and an action block prediction loss value and jointly training a first multi-modal generation backbone network to obtain a second multi-modal generation backbone network; inputting a second haptic image, a second dense local force field and a second grid deformation flow collected in real time into the second multi-modal generation backbone network, and performing cyclic inference until a task is completed.The application enables the robot to have reliable haptic guidance capability in operation tasks, such as grasping delicate fragile objects and continuous contact force control, which are highly dependent on mechanical perception.

Description

Technical Field

[0001] This invention relates to the field of robot training technology, and in particular to a robot tactile representation training method and related products based on mechanical reconstruction constraints. Background Technology

[0002] In contact-intensive robotic tasks, tactile perception is a crucial information source that cannot be replaced by visual information. Existing technologies directly use the raw pixel output of tactile sensors or simple binary contact states as model inputs, without effectively modeling the intrinsic physical relationship between the local force field distribution, surface deformation dynamics, and overall contact mechanics contained in the tactile signals. This results in the learned tactile representations lacking realistic mechanical semantics.

[0003] Existing technologies use appearance reconstruction loss or contrastive learning loss as the sole supervision signal when training tactile latent states. The resulting latent states only reflect the texture and appearance information of the sensing surface, lacking both the dense force distribution of the contact surface and macroscopic information about the overall force on the gripper. This fails to provide an effective feature base for subsequent contact mechanics prediction. Furthermore, existing tactile encoders often employ sequence structures containing historical states. This results in the tactile latent states obtained by the encoder processing the observation sequence and the predicted tactile latent states output by the prediction module at each time step existing in different representation spaces. These two cannot be directly compared or supervised, thus hindering the effective learning of future tactile prediction capabilities. Summary of the Invention

[0004] This invention provides a robot tactile representation training method and related products based on mechanical reconstruction constraints, which solves the technical problems of simplified processing of tactile signals and loss of local contact physical information in the prior art.

[0005] In a first aspect, embodiments of this application provide a robot tactile representation training method based on mechanical reconstruction constraints, including: Based on the first tactile image, the first dense local force field, and the first mesh deformation flow collected by the tactile sensors on both sides of the robot gripper, the parameters of the first tactile encoder and the first tactile reconstructor are optimized and frozen to obtain the second tactile encoder and the second tactile reconstructor. The teaching data is encoded and decoded by the second tactile encoder and the second tactile reconstructor, the visual prediction loss value, the tactile latent vector prediction loss value and the action block prediction loss value are calculated, and the first multimodal generation backbone network is jointly trained to obtain the second multimodal generation backbone network. The real-time acquired second tactile image, second dense local force field, and second grid deformation flow, together with the current visual observation, ontological perception state, and task language conditions, are input into the second multimodal generation backbone network. The action block sequence is decoded and iterated until the task is completed.

[0006] Optionally, in a first implementation of the first aspect of the present invention, the step of optimizing and freezing the parameters of the first tactile encoder and the first tactile reconstructor based on the first tactile image, the first dense local force field, and the first mesh deformation flow acquired by the tactile sensors on both sides of the robot gripper to obtain the second tactile encoder and the second tactile reconstructor includes: The robot gripper acquires a first tactile image, a first dense local force field, and a first grid deformation flow based on the tactile sensors on both sides of its gripper. The first tactile image, the first dense local force field, and the first mesh deformation flow are input into the first tactile encoder for feature extraction and cross-modal cross-attention calculation, respectively, to obtain the single-sided modal interaction feature maps on both sides of the gripper. The channel splicing and feature compression of the single-sided modal interaction feature maps obtained on both sides of the gripper are used to obtain the historical tactile latent vector; The historical tactile latent vectors are input into the first tactile reconstructor, the reconstruction loss is calculated, and the parameters of the first tactile encoder and the first tactile reconstructor are jointly optimized and frozen to obtain the second tactile encoder and the second tactile reconstructor.

[0007] Optionally, in a second implementation of the first aspect of the present invention, the step of inputting the historical tactile latent vector into the first tactile reconstructor, calculating the reconstruction loss, and performing joint parameter optimization and parameter freezing on the first tactile encoder and the first tactile reconstructor to obtain the second tactile encoder and the second tactile reconstructor includes: The historical tactile latent vectors are input into the first tactile reconstructor for parallel decoding to obtain the reconstructed dense local force field, the reconstructed grid deformation flow, and the reconstructed bilateral contact resultant force. The force field reconstruction error is calculated based on the reconstructed dense local force field and the actual dense local force field; the deformation flow reconstruction error is calculated based on the reconstructed mesh deformation flow and the actual mesh deformation flow; the contact resultant force reconstruction error is calculated based on the reconstructed bilateral contact resultant force and the actual bilateral contact resultant force; and the force field reconstruction error, the deformation flow reconstruction error, and the contact resultant force reconstruction error are weighted and summed to obtain the reconstruction loss. Based on the reconstruction loss, the first tactile encoder and the first tactile reconstructor are jointly backpropagated and optimized until convergence. After convergence, the corresponding parameters are frozen to obtain the second tactile encoder and the second tactile reconstructor.

[0008] Optionally, in a third implementation of the first aspect of the present invention, the step of encoding and decoding the teaching data through the second tactile encoder and the second tactile reconstructor, calculating the visual prediction loss value, the tactile latent vector prediction loss value, and the action block prediction loss value, and jointly training the first multimodal generative backbone network to obtain the second multimodal generative backbone network includes: The teaching data is input frame by frame into the second tactile encoder for encoding to obtain a real future tactile latent vector sequence corresponding to each teaching frame, and the real future tactile latent vector sequence is used as a tactile supervision label. The predicted future tactile latent vector sequence predicted by the tactile expert in the first multimodal generation backbone network is input into the second tactile reconstructor for decoding to obtain the predicted dense local force field sequence, the predicted grid deformation flow sequence, and the predicted bilateral contact resultant force sequence. Semantic consistency regularization terms are calculated with the real dense local force field, real grid deformation flow, and real bilateral contact resultant force of the corresponding frame in the teaching data, respectively. The semantic consistency regularization term is weighted and summed with the tactile latent vector prediction loss, visual prediction loss, and action block prediction loss to obtain the total training loss. Based on the total training loss, the first multimodal generative backbone network is optimized by backpropagation. The gradient is transmitted through the second tactile reconstructor to the future tactile latent vector sequence but does not update the parameters of the second tactile reconstructor until convergence, thus obtaining the second multimodal generative backbone network.

[0009] Optionally, in a fourth implementation of the first aspect of the present invention, the step of weighted summing of the semantic consistency regularization term with the tactile latent vector prediction loss value, the visual prediction loss value, and the action block prediction loss value to obtain the total training loss, and using the total training loss to perform backpropagation optimization on the first multimodal generation backbone network, with the gradient penetrating through the second tactile reconstructor to the future tactile latent vector sequence without updating the parameters of the second tactile reconstructor, until convergence, to obtain the second multimodal generation backbone network, includes: The total training loss is obtained by weighting and summing the semantic consistency regularization term, the tactile latent vector prediction loss, the visual prediction loss, and the action block prediction loss. Backpropagation is performed on the first multimodal generation backbone network based on the total training loss. The gradient of the semantic consistency regularization term is passed forward through the computation graph of the second tactile reconstructor to the future tactile latent vector sequence predicted by the tactile experts. The parameters of the second tactile reconstructor do not receive any gradient updates during this process, and the parameter gradients of each expert network are obtained. Visual, tactile, and action terms are input into a masked hybrid self-attention layer for forward computation. A mask matrix constrains the visibility range between different types of terms, enabling visual and tactile terms to interact based on current observation information and mutually visible information. Simultaneously, action terms are prohibited from reading future visual and tactile prediction terms. Since the output of the masked hybrid self-attention layer depends on visual, tactile, and action terms within their visible ranges, the visual prediction loss, the tactile latent vector prediction loss, the action block prediction loss, and the... The gradients generated by each of the semantic consistency regularization terms are propagated through the shared computation graph of the masked hybrid self-attention layer to the input projection parameters and attention parameters corresponding to other visible modalities, forming cross-modal gradient collaboration. After performing gradient norm clipping on the trainable parameters of the first multimodal generative backbone network, the optimizer updates the parameters of the visual expert, the tactile expert, the action expert, and the masked hybrid self-attention layer, and repeats the forward computation, total training loss calculation, backpropagation, and parameter update until the total training loss converges, thus obtaining the second multimodal generative backbone network.

[0010] Optionally, in a fifth implementation of the first aspect of the present invention, the step of inputting the real-time acquired second tactile image, the second dense local force field, and the second mesh deformation flow, along with the current visual observation, the ontological perception state, and the task language conditions, into the second multimodal generation backbone network, decoding to obtain the action block sequence, and iteratively reasoning until the task is completed includes: The real-time acquired second tactile image, second dense local force field, and second grid deformation flow are input into the second tactile encoder for single-frame forward inference to obtain real-time tactile latent vectors. The real-time tactile latent vectors, together with the current visual observation, ontological perception state, and task language conditions, are input into the second multimodal generation backbone network for decoding to obtain an action block sequence of length H frames. Each action word in the action block sequence is used as the target joint position state and is sent to the joint controller for execution in frame order to complete the H-step action. After the H-step action is completed, the third tactile image, the third dense local force field, and the third grid deformation flow of the new frame are acquired and input into the second tactile encoder for single-frame forward inference to obtain the tactile latent vector of the new frame. The real-time tactile latent vector is then updated with the tactile latent vector of the new frame to trigger the next round of decoding. This process is repeated until the task is completed.

[0011] Secondly, embodiments of this application provide a robot tactile representation training device based on mechanical reconstruction constraints, comprising: The parameter optimization unit is used to optimize and freeze the parameters of the first tactile encoder and the first tactile reconstructor based on the first tactile image, the first dense local force field and the first mesh deformation flow collected by the tactile sensors on both sides of the robot gripper, so as to obtain the second tactile encoder and the second tactile reconstructor. The joint training unit is used to encode and decode the teaching data through the second tactile encoder and the second tactile reconstructor, calculate the visual prediction loss value, the tactile latent vector prediction loss value and the action block prediction loss value, and jointly train the first multimodal generation backbone network to obtain the second multimodal generation backbone network. The task execution unit is used to input the real-time acquired second tactile image, second dense local force field, and second grid deformation flow together with the current visual observation, ontological perception state, and task language conditions into the second multimodal generation backbone network, decode it to obtain the action block sequence, and iterate and reason until the task is completed.

[0012] Thirdly, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the above-described robot tactile representation training method based on mechanical reconstruction constraints.

[0013] Fourthly, embodiments of this application provide a readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described robot tactile representation training method based on mechanical reconstruction constraints.

[0014] Fifthly, embodiments of this application provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the above-described robot tactile representation training method based on mechanical reconstruction constraints.

[0015] In one of the solutions provided by the above-mentioned robot tactile representation training method and related products based on mechanical reconstruction constraints, pixel-level spatial registration is performed on the three signals of the first tactile image, the first dense local force field, and the first mesh deformation flow in the contact surface coordinate system. After extracting feature maps with consistent spatial resolution through appearance feature extraction network, force feature extraction network, and deformation flow feature extraction network, respectively, the historical tactile latent vector is obtained by cross-modal cross-attention fusion and bilateral fusion pooling of modal interaction block. This solves the technical problem of simplifying tactile signal processing and losing local contact physical information in the prior art. This invention feeds historical tactile latent vectors into a first tactile reconstructor, simultaneously reconstructing a first dense local force field, a first grid deformation flow, and the resultant force of bilateral contact. The weighted sum of the three reconstruction errors is used as the reconstruction loss to jointly optimize the first tactile encoder and the first tactile reconstructor. The resultant force of bilateral contact is only used as a global mechanical supervision term in the reconstruction loss calculation and does not enter the encoder input channel. Thus, without increasing any encoder input dimension, the macroscopic contact mechanical information of the gripper's overall force is pressed into the tactile latent vector through reconstruction constraints. This allows the obtained historical tactile latent vector to simultaneously carry mechanical information at both the microscopic local contact pattern and the macroscopic overall force levels, solving the technical problem of existing technologies lacking global mechanical constraints and true mechanical semantics in tactile latent states. The first tactile encoder of this invention adopts a single-frame stateless structure, ensuring that the tactile latent vectors obtained by the encoder from the observed frame and the predicted tactile latent vectors output by the prediction module at each time step naturally fall into the same latent space. Therefore, the second tactile encoder directly obtains the future tactile latent vector sequence as a tactile supervision label by encoding the teaching data frame by frame, without requiring any additional alignment modules. This invention uses a second tactile reconstructor with frozen parameters but a differentiable computation graph as a tool for applying semantic consistency regularization. It performs forward decoding on the future tactile latent vector sequence predicted by the first multimodal generation backbone network. The reconstruction error between the decoded mechanical quantities and the actual mechanical quantities constitutes the semantic consistency regularization term. The gradient of this regularization term is propagated back to the predicted future tactile latent vector sequence through the computation graph of the second tactile reconstructor. Throughout the second-stage joint training process, the mechanical semantics of the predicted latent states are continuously constrained, ensuring that the predicted tactile latent vector sequence is consistent with the actual contact mechanical evolution in a mechanical and physical sense, rather than merely fitting the supervision label in the sense of Euclidean distance. This fundamentally solves the problems of existing technologies where tactile future prediction lacks physical constraints and the prediction results lack mechanical interpretability.During the inference phase, real-time tactile latent vectors, together with visual observation, ontological perception state, and task language conditions, drive the second multimodal generation backbone network to decode action block sequences. Tactile information participates in action generation throughout the process in the form of compact latent vectors that carry real contact mechanics semantics, enabling the robot to have reliable tactile guidance capabilities in operation tasks that are highly dependent on mechanical perception, such as grasping delicate and fragile objects and continuous contact force control. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of a robot tactile representation training system based on mechanical reconstruction constraints in one embodiment of the present invention; Figure 2 This is a flowchart illustrating a robot tactile representation training method based on mechanical reconstruction constraints in one embodiment of the present invention. Figure 3 yes Figure 2 A schematic diagram of the implementation process of step S100; Figure 4 yes Figure 2 A schematic diagram of the implementation process of step S200; Figure 5 yes Figure 2 A schematic diagram of the implementation process of step S300; Figure 6 This is an overall architecture diagram of a robot operation model in one embodiment of the present invention; Figure 7 This is an internal structural diagram of a spatial registration fusion tactile encoder according to an embodiment of the present invention; Figure 8 This is a schematic diagram of a robot tactile representation training device based on mechanical reconstruction constraints in one embodiment of the present invention; Figure 9 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof. It should also be understood that, as used in this specification and the appended claims, the term "and / or" refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0020] Furthermore, in the description of this invention and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0021] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of the invention include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0022] It should be understood that the sequence number of each step in the following embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0023] To illustrate the technical solution of the present invention, specific embodiments are described below.

[0024] To address the problems mentioned above in the background art, this application proposes a robot tactile representation training method and related products based on mechanical reconstruction constraints. The robot tactile representation training method based on mechanical reconstruction constraints provided by this invention can be applied to, for example... Figure 1The robot tactile representation training system based on mechanical reconstruction constraints shown includes a tactile representation pre-training module and a multimodal joint training module. The tactile representation pre-training module performs spatial registration, bilateral fusion encoding, and mechanical reconstruction constraint training of three tactile signals, and freezes the tactile encoder and tactile reconstructor after pre-training. The multimodal joint training module receives the frozen model and future tactile latent state supervision targets, jointly trains visual future prediction, tactile future prediction, and action block generation, and inputs the predicted tactile latent states into the frozen tactile reconstructor to form decoding semantic consistency constraints.

[0025] In one embodiment, such as Figure 2 As shown, a robot tactile representation training method based on mechanical reconstruction constraints is provided, which is then applied to... Figure 1 The following is an example of a robot tactile representation training system based on mechanical reconstruction constraints, which includes the following steps: S100: Based on the first tactile image, the first dense local force field, and the first mesh deformation flow collected by the tactile sensors on both sides of the robot gripper, the parameters of the first tactile encoder and the first tactile reconstructor are optimized and frozen to obtain the second tactile encoder and the second tactile reconstructor. S200: The teaching data is encoded and decoded by the second tactile encoder and the second tactile reconstructor. The visual prediction loss value, the tactile latent vector prediction loss value and the action block prediction loss value are calculated. The first multimodal generation backbone network is jointly trained to obtain the second multimodal generation backbone network. S300: The real-time acquired second tactile image, second dense local force field, and second grid deformation flow, together with the current visual observation, ontological perception state, and task language conditions, are input into the second multimodal generation backbone network, decoded to obtain the action block sequence, and iteratively reasoned until the task was completed.

[0026] In this embodiment, the first tactile image, the first dense local force field, and the first grid deformation flow signal are spatially registered at the pixel level in the contact surface coordinate system. After feature maps with consistent spatial resolution are extracted by the appearance feature extraction network, the force feature extraction network, and the deformation flow feature extraction network, respectively, the historical tactile latent vector containing three types of physical information—local contact appearance, contact pressure distribution, and tangential deformation dynamics—is obtained through cross-modal cross-attention fusion and bilateral fusion pooling of the modal interaction block. This solves the technical problem of simplifying tactile signal processing and losing local contact physical information in the prior art. This invention feeds historical tactile latent vectors into a first tactile reconstructor, simultaneously reconstructing a first dense local force field, a first grid deformation flow, and the resultant force of bilateral contact. The weighted sum of the three reconstruction errors is used as the reconstruction loss to jointly optimize the first tactile encoder and the first tactile reconstructor. The resultant force of bilateral contact is only used as a global mechanical supervision term in the reconstruction loss calculation and does not enter the encoder input channel. Thus, without increasing any encoder input dimension, the macroscopic contact mechanical information of the gripper's overall force is pressed into the tactile latent vector through reconstruction constraints. This allows the obtained historical tactile latent vector to simultaneously carry mechanical information at both the microscopic local contact pattern and the macroscopic overall force levels, solving the technical problem of existing technologies lacking global mechanical constraints and true mechanical semantics in tactile latent states. The first tactile encoder of this invention adopts a single-frame stateless structure, ensuring that the tactile latent vectors obtained by the encoder from the observed frame and the predicted tactile latent vectors output by the prediction module at each time step naturally fall into the same latent space. Therefore, the second tactile encoder directly obtains the future tactile latent vector sequence as a tactile supervision label by encoding the teaching data frame by frame, without requiring any additional alignment modules. This invention uses a second tactile reconstructor with frozen parameters but a differentiable computation graph as a tool for applying semantic consistency regularization. It performs forward decoding on the future tactile latent vector sequence predicted by the first multimodal generation backbone network. The reconstruction error between the decoded mechanical quantities and the actual mechanical quantities constitutes the semantic consistency regularization term. The gradient of this regularization term is propagated back to the predicted future tactile latent vector sequence through the computation graph of the second tactile reconstructor. Throughout the second-stage joint training process, the mechanical semantics of the predicted latent states are continuously constrained, ensuring that the predicted tactile latent vector sequence is consistent with the actual contact mechanical evolution in a mechanical and physical sense, rather than merely fitting the supervision label in the sense of Euclidean distance. This fundamentally solves the problems of existing technologies where tactile future prediction lacks physical constraints and the prediction results lack mechanical interpretability. During the inference phase, real-time tactile latent vectors, together with visual observation, ontological perception state, and task language conditions, drive the second multimodal generation backbone network to decode action block sequences. Tactile information participates in action generation throughout the process in the form of compact latent vectors that carry real contact mechanics semantics, enabling the robot to have reliable tactile guidance capabilities in operation tasks that are highly dependent on mechanical perception, such as grasping delicate and fragile objects and continuous contact force control.

[0027] In one embodiment, such as Figure 3 As shown, step S100 specifically includes the following steps: S101: Acquire a first tactile image, a first dense local force field, and a first grid deformation flow based on the tactile sensors on both sides of the robot gripper; S102: Input the first tactile image, the first dense local force field and the first mesh deformation flow into the first tactile encoder for feature extraction and cross-modal cross-attention calculation, and obtain the single-sided modal interaction feature maps of both sides of the gripper respectively; S103: Perform channel splicing and feature compression on the single-sided modal interaction feature maps obtained on both sides of the gripper to obtain the historical tactile latent vector; S104: Input the historical tactile latent vector into the first tactile reconstructor, calculate the reconstruction loss, and perform joint parameter optimization and parameter freezing on the first tactile encoder and the first tactile reconstructor to obtain the second tactile encoder and the second tactile reconstructor.

[0028] In this embodiment, a visual-tactile sensor based on a deformable contact film can be installed on each of the two gripping fingers of the robot gripper. A unified clock source sends a synchronization trigger signal to the two tactile sensors, the robot joint controller, and other acquisition devices, enabling frame-level synchronous acquisition of tactile data from both sides of the gripper at a sampling frequency of 30Hz. At each sampling moment, the original image of the contact film captured by the built-in camera of the tactile sensor is read. Then, based on pre-calibrated camera intrinsic parameters, distortion coefficients, and a mask of the effective area of ​​the contact surface, the original image is processed for distortion correction, cropping, and brightness normalization to obtain the first tactile image. Simultaneously, based on the calibration relationship between contact film deformation and contact force, The three-dimensional deformation information of the contact surface is converted into local force information at the corresponding location, resulting in a first dense local force field. Based on the grid node displacements between the current tactile image and the tactile image at the previous sampling time, a first grid deformation flow is obtained to characterize tangential deformation and shear motion. The first tactile image, the first dense local force field, and the first grid deformation flow are all uniformly mapped to the coordinate system of the tactile sensor contact film surface, ensuring that the same spatial location corresponds to the contact appearance, local force, and deformation displacement, respectively. The same timestamp is added to the data on the left and right sides of the gripper to avoid erroneous bilateral mechanical relationships caused by asynchronous sampling during gripper closure, slippage, or object rotation. For example, the three data streams can be uniformly resampled to a 64×64 spatial grid. The first tactile image retains three color channels, the first dense local force field retains force-related channels at each grid location, and the first grid deformation flow retains two displacement channels in the horizontal and vertical directions. This spatial dimension, as a specific implementation parameter, can be adjusted according to the original resolution of the tactile sensor.

[0029] In this embodiment, the three data streams from each side of the gripper are input into the first tactile encoder. The first tactile encoder adopts a single-frame stateless structure, that is, it only processes the data at the current sampling moment each time, without introducing a loop state or historical frame buffer, so that each tactile latent vector is uniquely determined by the current tactile observation. The first tactile image is input into the appearance feature extraction network to extract the texture changes, contact boundaries and local concavity features of the contact area. The first dense local force field is input into the force feature extraction network to extract the normal force concentration area, force gradient and force distribution features. The first mesh deformation flow is input into the deformation flow feature extraction network to extract the tangential displacement direction, displacement amplitude and local shear change features. The feature maps output by the three feature extraction networks are adjusted to the same spatial resolution. For example, three networks can each output a feature map with a spatial size of 16×16 and 128 channels. Then, cross-modal cross-attention computation is performed at the same spatial location. First, appearance features are used as query information, and force features are used as keys and values, allowing the network to select relevant force-bearing areas based on changes in the contact appearance. Then, force features are used as query information, and deformation flow features are used as keys and values, allowing the network to identify deformation information related to slippage, shear, or contact membrane rebound by combining the force distribution. The cross-attention output is then merged with the corresponding input features through residual connections. The three-way interaction results from each side are concatenated along the channel direction and mapped using 1×1 convolutions to obtain a single-sided modal interaction feature map with a spatial size of 16×16 and 256 channels, thus obtaining single-sided modal interaction feature maps for the left and right sides of the gripper. By encoding independently on each side, premature mixing of left and right gripper information is avoided before the mechanical correlation modeling of the three local features is completed, allowing each single-sided modal interaction feature map to first form a complete description of the corresponding contact surface.

[0030] The unilateral modal interaction feature map on the left side of the gripper is concatenated with the unilateral modal interaction feature map on the right side of the gripper along the channel direction to preserve the relative force, contact position, and deformation differences between the left and right contact surfaces. Then, channel compression and bilateral information mixing are performed through 1×1 convolution. For example, the concatenated 512 channels can be compressed into 256 channels. The spatial dimension is eliminated by global average pooling to obtain a historical tactile latent vector with a fixed dimension of 256. The historical tactile latent vector is generated from a single frame of tactile observation, but its name is used to indicate that the latent vector is used as a component of the observed tactile state and the future tactile supervision sequence in the subsequent multimodal training process, rather than indicating that the first tactile encoder contains historical recursive states. The historical tactile latent vectors are input into the first tactile reconstructor. Dense local force fields, grid deformation flow, and bilateral contact resultant forces are reconstructed by independent decoding paths. The bilateral contact resultant forces include the three-dimensional forces and three-dimensional moments on both sides of the gripper, forming a total of 12-dimensional supervision quantities. The reconstruction loss is formed based on the error between each reconstruction result and the corresponding real data. The reconstruction loss is then used to jointly optimize the first tactile encoder and the first tactile reconstructor through backpropagation. For example, a training setting with an initial learning rate of 0.0001 and a batch size of 64 can be used. The first stage of training is stopped when the reconstruction loss on the validation set does not decrease further for 10 consecutive training cycles. Then, all parameters of the first tactile encoder and the first tactile reconstructor are frozen to obtain the second tactile encoder and the second tactile reconstructor, respectively, so as to fix the latent space and its mechanical decoding relationship used for future tactile supervision.

[0031] In one embodiment, step S104 specifically includes the following steps: The historical tactile latent vectors are input into the first tactile reconstructor for parallel decoding to obtain the reconstructed dense local force field, the reconstructed mesh deformation flow, and the reconstructed resultant force of bilateral contact. The force field reconstruction error is calculated based on the reconstructed dense local force field and the actual dense local force field. The deformation flow reconstruction error is calculated based on the reconstructed mesh deformation flow and the actual mesh deformation flow. The contact resultant force reconstruction error is calculated based on the reconstructed bilateral contact resultant force and the actual bilateral contact resultant force. The force field reconstruction error, deformation flow reconstruction error and contact resultant force reconstruction error are weighted and summed to obtain the reconstruction loss. Based on the reconstruction loss, the first tactile encoder and the first tactile reconstructor are jointly backpropagated and optimized until convergence. After convergence, the corresponding parameters are frozen to obtain the second tactile encoder and the second tactile reconstructor.

[0032] In this embodiment, the historical tactile latent vector is input into the first tactile reconstructor. The fixed-dimensional historical tactile latent vector is expanded into intermediate decoding features with spatial structure through a shared feature mapping layer. The intermediate decoding features are then fed into independent force field decoding heads, deformation flow decoding heads, and contact resultant force decoding heads for parallel processing. The force field decoding head and deformation flow decoding head can use transposed convolution and upsampling layers to restore spatial resolution step by step, so that the output grid maintains a positional correspondence with the real dense local force field and the real grid deformation flow. The contact resultant force decoding head can use a multilayer perceptron to map the intermediate decoding features into a 12-dimensional vector. The 12-dimensional vector corresponds to the three-dimensional contact resultant force and the three-dimensional contact resultant torque on the left and right sides of the gripper, respectively. This results in the reconstruction of the dense local force field, the reconstruction of the grid deformation flow, and the reconstruction of the contact resultant force on both sides. That is, the local force distribution, tangential deformation dynamics, and the overall force state of the gripper are simultaneously restored from the same historical tactile latent vector.

[0033] In this embodiment, to avoid any one of the errors—dense force field, grid deformation flow, contact force, and contact torque—becoming absolutely dominant during training due to differences in units, numerical ranges, and the number of output elements, the various types of real data and corresponding reconstructed data are standardized based on the statistical results of the training set. Then, the three mean square errors are calculated, where the force field reconstruction error is... The deformation flow reconstruction error is The contact resultant force reconstruction error is ,in, , and These represent the effective number of elements in dense local force fields, grid deformation flow, and resultant force from bilateral contact, respectively. , and These represent the actual dense local force field, the actual mesh deformation flow, and the actual resultant force at the current moment, respectively, marked with the upper symbol "". The corresponding variable for "" represents the reconstruction result output by the first tactile reconstructor. Then, according to... Calculating the reconstruction loss, for example, when all three types of supervised data have been standardized to zero mean and unit variance, can be done by... , and All values ​​are set to 1.0. When the calibration noise of the dense local force field is significantly higher than the measurement noise of the contact resultant force, the three values ​​can be set to 0.5, 1.0, and 1.0 respectively, so that the contact resultant force undertakes a stronger global mechanical supervision role. The contact resultant force is only used for loss calculation and is not returned to the first tactile encoder as a fourth input. Therefore, it will not increase the input dimension of the tactile encoder, but it can force the historical tactile latent vectors to retain information sufficient to recover the overall force on both sides through the error gradient.

[0034] In this embodiment, a continuous backpropagation computation graph is established for the first tactile encoder and the first tactile reconstructor. The gradients generated by the three reconstruction errors are first propagated back through their respective decoders to the shared decoding layer and historical tactile latent vectors. Then, the historical tactile latent vectors continue to propagate back to the bilateral fusion layer, cross-modal attention layer, and feature extraction networks of each modality in the first tactile encoder, thereby synchronously updating the parameters of the encoder and reconstructor. For example, training can be performed with a batch size of 64 and an initial learning rate of 0.0001. Model convergence is determined when the reconstruction loss on the validation set does not decrease for 10 consecutive training epochs. After convergence, gradient calculation and optimizer updates are stopped. The parameters of the first tactile encoder are fixed to those of the second tactile encoder, and the parameters of the first tactile reconstructor are fixed to those of the second tactile reconstructor. The second tactile encoder is used to generate stable tactile latent vector supervision labels, while the second tactile reconstructor keeps its parameters frozen but retains a differentiable forward computation path.

[0035] In one embodiment, such as Figure 4 As shown, step S200 specifically includes the following steps: S201: Input the teaching data frame by frame into the second tactile encoder for encoding to obtain a real future tactile latent vector sequence that corresponds one-to-one with the teaching frame, and use the real future tactile latent vector sequence as a tactile supervision label; S202: Input the predicted future tactile latent vector sequence predicted by the tactile expert in the first multimodal generation backbone network into the second tactile reconstructor for decoding to obtain the predicted dense local force field sequence, the predicted mesh deformation flow sequence, and the predicted bilateral contact resultant force sequence. Calculate semantic consistency regularization terms with the real dense local force field, real mesh deformation flow, and real bilateral contact resultant force of the corresponding frame in the teaching data, respectively. S203: The semantic consistency regularization term is weighted and summed with the tactile latent vector prediction loss, visual prediction loss, and action block prediction loss to obtain the total training loss. Based on the total training loss, the first multimodal generation backbone network is optimized by backpropagation. The gradient is passed through the second tactile reconstructor to the future tactile latent vector sequence but the parameters of the second tactile reconstructor are not updated until convergence, thus obtaining the second multimodal generation backbone network.

[0036] In this embodiment, the visual observations, robot joint position states, motion data, tactile images of both sides of the gripper, dense local force fields, and mesh deformation flow are aligned according to the frame-level timestamps of the teaching trajectory, and the starting time of each motion block is used as the basis for alignment. Based on this, extract the subsequent continuous segments from the teaching trajectory. A frame can be used as a future supervision interval, for example, the length of an action block. Set to 16 frames, corresponding to a future evolution range of approximately 0.53 seconds at a sampling frequency of 30Hz; the bilateral tactile images, dense local force fields, and grid deformation flow of each frame within the supervised interval are input into the second tactile encoder with frozen parameters. Since the second tactile encoder adopts a single-frame stateless structure, each teaching frame can be independently mapped to the same fixed tactile latent space, thereby sequentially obtaining... to The future tactile latent vector sequence is constructed and saved as tactile supervision labels in the original chronological order. The second tactile encoder only performs forward inference in this process and does not participate in parameter updates. Its output can also be calculated offline and cached before formal training to avoid repeatedly processing the same teaching tactile data in each training iteration, while ensuring that the tactile supervision labels do not drift with the updates of the first multimodal generation backbone network throughout the second stage of training.

[0037] In this embodiment, the current visual observation at the start time of the action block is used. Current haptic latent vector Current ontology perception state and task language conditions Input the first multimodal generative backbone network, where tactile experts generate a predicted sequence of future tactile latent vectors, with the following relationship: The predicted future tactile latent vector sequence is then compared with the tactile supervision labels according to the same frame index, and the tactile latent vector prediction loss value is calculated. The future tactile latent vector sequence fed into the second tactile reconstructor in S202 is the prediction output of the tactile expert during specific training. Instead of directly feeding the real tactile supervision labels into the second tactile reconstructor to calculate the regularization term, the semantic consistency regularization term can only reflect the original reconstruction capabilities of the frozen encoder and frozen reconstructor, and cannot pass the gradient to the first multimodal generation backbone network. Frame-based haptic latent vector execution This process yields the predicted dense local force field, predicted two-sided contact resultant force, and predicted mesh deformation flow for the corresponding frames. These are then paired one-to-one with the actual dense local force field, actual two-sided contact resultant force, and actual mesh deformation flow in the teaching data, according to their timestamps. After completing the same data standardization process as in the first stage, semantic consistency regularization terms are calculated based on the number of valid elements in each data type and the number of frames in the future supervision interval. Therefore, even if the predicted tactile latent vector is close to the tactile supervision label in numerical distance, as long as it cannot recover the local force field, deformation flow or bilateral overall force state consistent with the real data after being decoded by the second tactile reconstructor, it will still generate a large semantic consistency regularization term, thereby avoiding tactile experts from only learning the surface numerical fitting in the latent space.

[0038] In this embodiment, a future visual state sequence is output simultaneously by a visual expert, a future tactile latent vector sequence is output by a tactile expert, and a sequence of action blocks represented by the target joint position state is output by a motion expert. The visual prediction loss, tactile latent vector prediction loss, and action block prediction loss are calculated respectively, and then weighted and summed with a semantic consistency regularization term to form the total training loss. Backpropagation is performed on the first multimodal generative backbone network, where the visual prediction loss, tactile latent vector prediction loss, and action block prediction loss are applied to the corresponding experts and the shared hybrid self-attention layer, respectively. The gradient of the semantic consistency regularization term passes through the second tactile reconstructor, whose parameters are frozen but whose computation graph remains differentiable, and continues to propagate back to the future tactile latent vector sequence predicted by the tactile expert. The second tactile reconstructor itself does not receive updates from the optimizer. Thus, while keeping the existing mechanical decoding space fixed, the tactile future predicted by the first multimodal generative backbone network is made to continuously satisfy the real contact mechanical constraints. Joint training is completed through information exchange between the visual, tactile, and motion modal flows until the total training loss meets the preset convergence condition, resulting in the second multimodal generative backbone network.

[0039] In one embodiment, step S203 specifically includes the following steps: The total training loss is obtained by weighting and summing the semantic consistency regularization term, the haptic latent vector prediction loss, the visual prediction loss, and the action block prediction loss. Backpropagation is performed on the first multimodal generative backbone network based on the total training loss. The gradient of the semantic consistency regularization term is passed forward through the computation graph of the second tactile reconstructor to the future tactile latent vector sequence predicted by the tactile experts. The parameters of the second tactile reconstructor do not receive any gradient updates in this process, and the parameter gradients of each expert network are obtained. Visual, tactile, and action terms are input into a masked hybrid self-attention layer for forward computation. A mask matrix constrains the visibility range between different types of terms, enabling visual and tactile terms to interact based on current observation information and mutually visible information. Simultaneously, action terms are prohibited from reading future visual and tactile prediction terms. Since the output of the masked hybrid self-attention layer depends on visual, tactile, and action terms within their visible ranges, the visual prediction loss, tactile latent vector prediction loss, action block prediction loss, and semantic consistency are all considered. The gradients generated by each regularization term are propagated through the shared computation graph of the masked hybrid self-attention layer to the input projection parameters and attention parameters corresponding to other visible modalities, forming cross-modal gradient collaboration. After performing gradient norm clipping on the trainable parameters of the first multimodal generative backbone network, the optimizer updates the parameters of the visual expert, the tactile expert, the action expert, and the masked hybrid self-attention layer, and repeats the forward computation, total training loss calculation, backpropagation, and parameter update until the total training loss converges, thus obtaining the second multimodal generative backbone network.

[0040] In this embodiment, during the second-stage joint training, the tactile latent vector prediction loss value output by the tactile expert, the visual prediction loss value output by the visual expert, the action block prediction loss value output by the action expert, and the semantic consistency regularization term formed after the predicted tactile latent vector is decoded by the second tactile reconstructor are obtained respectively, and the four losses are calculated uniformly within the same training batch. Since the dimensions and numerical ranges of the visual prediction results, tactile latent vectors, action words, and dense mechanical quantities are different, the average loss can be calculated according to the number of effective elements of each, avoiding the large number of elements in the dense local force field from affecting the overall training. The training loss does not occupy an unreasonable dominant position. Then, the four losses are weighted and summed. For example, the weights of the visual prediction loss, the tactile latent vector prediction loss, and the action block prediction loss can all be set to 1.0, and the weight of the semantic consistency regularization term can be set to 0.1. This keeps the three-way generation task in the main optimization position, while continuously using dense local force fields, grid deformation flow, and bilateral contact resultant force constraints to predict the mechanical meaning of the tactile latent vector. If the three types of decoding errors are at a similar order of magnitude to other losses after standardization, the weight of the semantic consistency regularization term can also be set to 1.0.

[0041] After obtaining the total training loss in this embodiment, the parameter gradients saved in the previous training batch are cleared, and a joint backpropagation is performed starting from the total training loss. The visual prediction loss value is mainly backpropagated along the visual output layer, visual expert, and hybrid self-attention layer. The tactile latent vector prediction loss value is backpropagated along the tactile output layer, tactile expert, and hybrid self-attention layer. The action block prediction loss value is backpropagated along the action output layer, action expert, and hybrid self-attention layer. The semantic consistency regularization term first penetrates back along the differentiable calculation path of the second tactile reconstructor from the reconstructed dense local force field sequence, the reconstructed grid deformation flow sequence, and the reconstructed bilateral contact resultant force sequence to the future tactile latent vector sequence predicted by the tactile expert, and then continues to be passed to the tactile expert and its upstream hybrid self-attention layer. To achieve the goal of the second haptic reconstructor "participating in gradient propagation but not updating parameters", all parameters of the second haptic reconstructor are marked as untrainable and excluded from the set of parameters to be updated by the optimizer. However, the computation graph between the predicted haptic latent vectors and the output of the second haptic reconstructor cannot be severed, nor can the inference mode that prohibits gradient computation be used during the forward decoding of the second haptic reconstructor, otherwise the semantic consistency regularization term will not be able to constrain the haptic expert. Thus, the gradient of the semantic consistency regularization term with respect to the future sequence of haptic latent vectors can still be computed during backpropagation, but the second haptic reconstructor itself will not generate parameter gradients for the optimizer to update.

[0042] In this embodiment, visual, tactile, and action terms are input into a masked hybrid self-attention layer, enabling information exchange among the three types of terms within the same attention computation space. The mask matrix defines the visibility relationships between these terms, allowing visual and tactile flows to learn scene changes and contact mechanics evolution by combining current observations and mutually visible information. Simultaneously, action flow terms are prohibited from reading future visual and tactile terms, preventing the use of future information unavailable during deployment during the training phase. Task language conditions and the current ontology perception state are injected through cross-attention pathways within each expert, rather than participating in unconstrained information exchange as future modal terms. Since the output of the hybrid self-attention layer depends on visual, tactile, and action terms within the visible range, the gradient generated by any modality loss not only updates the corresponding modality expert but also influences the input projection and attention parameters of other visible modalities through shared attention computation. This forms a mask-constrained cross-modal gradient collaboration, rather than simply adding the gradients of the three experts directly. After gradient calculation is completed, gradient norm pruning can be performed on the trainable parameters of the first multimodal generative backbone network. For example, the maximum gradient norm can be set to 1.0. Then, the optimizer updates the parameters of the visual expert, tactile expert, motion expert, masked hybrid self-attention layer, and cross-attention layers within each stream. Batch forward calculation, total training loss calculation, joint backpropagation, and parameter update are repeated. For example, when the total training loss of the validation set does not decrease for 10 consecutive training epochs, or the relative decrease of adjacent training epochs is less than 0.1% for 5 consecutive times, the training can be considered to have converged. The parameters of the first multimodal generative backbone network at this time are retained as the second multimodal generative backbone network, while the second tactile encoder and the second tactile reconstructor remain frozen throughout the second stage to ensure that the future tactile supervision space and mechanical decoding relationship do not drift with joint training.

[0043] In one embodiment, such as Figure 5 As shown, step S300 specifically includes the following steps: S301: Input the real-time acquired second tactile image, second dense local force field and second grid deformation flow into the second tactile encoder for single-frame forward inference to obtain the real-time tactile latent vector, and input the real-time tactile latent vector together with the current visual observation, ontological perception state and task language conditions into the second multimodal generation backbone network for decoding to obtain an action block sequence of length H frames; S302: Take each action word in the action block sequence as the target joint position state, and send it down to the joint controller for execution in frame order to complete the H-step action; S303: After the H-step action is completed, the third tactile image, the third dense local force field, and the third grid deformation flow of the new frame are acquired and input into the second tactile encoder for single-frame forward inference to obtain the tactile latent vector of the new frame. The real-time tactile latent vector is updated with the tactile latent vector of the new frame to trigger the next round of decoding. The process is repeated until the task is completed.

[0044] In this embodiment, at the start time of the current action block The tactile sensors on both sides of the gripper synchronously acquire tactile data of the current frame at a frequency of 30Hz, consistent with the robot control system. They then sequentially perform tactile image distortion correction, dense local force field generation, displacement extraction of grid nodes in adjacent tactile frames, and spatial registration of the three data streams in the coordinate system of the tactile sensor contact surface, resulting in a second tactile image, a second dense local force field, and a second grid deformation flow. The three tactile data streams from both sides of the gripper are input to a second tactile encoder with frozen parameters. The second tactile encoder extracts appearance features, force field features, and deformation flow features, respectively. After single-sided cross-modal interaction, double-sided feature fusion, and feature compression, the real-time tactile latent vector of the current frame is obtained. The processing relationship can be expressed as follows: ,in, , and These represent the corrected tactile image, dense local force field, and mesh deformation flow of the current frame, respectively. This indicates the second tactile encoder. This represents the real-time tactile latent vector. Since the second tactile encoder uses a single-frame stateless structure, this inference only relies on the tactile observation of the current frame and does not require reading the loop state saved in the previous action block, thus avoiding the accumulation of errors from historical states and keeping the computational latency of each round of tactile encoding stable. Simultaneously, current visual observations are acquired through the top and wrist cameras, and the current joint positions output by the encoders of each robot joint are read as the proprioceptive state. Pre-inputted task instructions are encoded into task language conditions. Then, the real-time tactile latent vector, current visual observation, proprioceptive state, and task language conditions are converted into tactile lexical units, visual lexical units, proprioceptive lexical units, and task lexical units, respectively, and input into the second multimodal generation backbone network. A masked hybrid self-attention layer exchanges visual information, contact mechanics information, and action context within the allowed visible range. Finally, a motion expert decodes and outputs a single sequence of lengths... Frame action block sequence In the same training and inference configuration, the action block length Maintain consistency. For example, during the training phase, keep the action block length consistent. When set to 16 frames, the inference phase synchronization will With 16 frames per second and a 30Hz control frequency, one action block covers approximately 0.53 seconds. If H is set to 8 frames per second for tasks such as grasping fragile objects or initial contact localization, then the future supervision interval, haptic expert prediction sequence, and action block supervision label captured during the training phase are all set to 8 frames per second, corresponding to approximately 0.27 seconds at a 30Hz control frequency.

[0045] In this embodiment, each action term in the action block sequence is interpreted as the target joint position state for the next control step, rather than directly as motor current, joint torque, or underlying instantaneous control commands. These action terms are then sent to the joint controller frame by frame according to their temporal order within the action block. The joint controller receives the first target joint position state in the first control cycle and drives the robotic arm to move. In subsequent control cycles, it sequentially receives the remaining target joint position states until completion. Step-by-step action. Before issuing action terms, validity checks are performed based on the allowable position range, maximum speed, and changes in adjacent target positions for each robot joint. If the target position exceeds the allowable range, amplitude limiting is applied to prevent the generated result from exceeding the robot's execution capabilities, while maintaining the semantics of the joint positions represented by the action terms. After each action term is executed, the joint controller records the actual joint position and execution timestamp for use as a new ontology perception state during the next action block inference. The future tactile latent vector sequence synchronously predicted by the tactile expert can be decoded by the second tactile reconstructor to analyze the changing trends of the future force field, deformation flow, and bilateral contact resultant force. However, this prediction result does not directly modify the currently generated action block; the current action block is still executed according to the predetermined frame order.

[0046] When the After each action token is executed, the start time of the current action block will be changed from... Updated to The tactile sensors on both sides of the gripper re-acquire a new frame of tactile data at the same moment. After the same correction, force field generation, mesh deformation flow calculation, and spatial registration processing as S301, a third tactile image, a third dense local force field, and a third mesh deformation flow are obtained. The three data streams are then input into the second tactile encoder for single-frame forward inference to obtain the new frame of tactile latent vectors. The new frame's tactile latent vector replaces the real-time tactile latent vector used in the previous action block, enabling the new round of decoding to reflect the actual contact position, local force distribution, tangential deformation, and potential slip state formed after the previous action block's execution. Simultaneously, the current visual observations provided by the top and wrist cameras, as well as the proprioceptive state provided by the joint encoder, are updated. This is because the next round of decoding by the second multimodal generation backbone network still requires complete visual, tactile, proprioceptive, and task language conditions, and cannot simply update the tactile input. The updated real-time tactile latent vector, visual observations, and proprioceptive state then drive the second multimodal generation backbone network to generate the next frame of length... The robot executes action blocks within a frame, repeating the sequential issuance of action lexies, acquisition of new observations, and updating of tactile latent vectors. This forms a closed-loop, rolling temporal execution that runs according to the rhythm of the action blocks. When the task status determination result indicates that the target object has reached the designated position, the wiping trajectory has been completed, the grasping and placement operation has ended, or a preset task completion flag has been received, the next round of action block decoding and joint control command issuance stops, thus completing the entire robot operation task.

[0047] like Figure 6 As shown, observations from the top and wrist cameras, action blocks, and tactile information are processed by a visual encoder, a motion encoder, and a spatial registration fusion tactile encoder to form corresponding lexical units; task instructions and ontology-aware information form task / ontology-aware lexical units. These lexical units are input into a hybrid expert joint attention layer and processed by visual, motion, and tactile experts respectively to form visual future predictions, action block outputs, and future tactile state predictions. The three outputs are used to calculate the joint training loss. In an optional implementation, tactile information is further processed by a tactile history encoder and historical states to form contact states, serving as additional context for the joint attention calculation.

[0048] like Figure 7 As shown, the corrected tactile image, dense local force field, and mesh deformation flow are respectively processed through tactile appearance feature extraction, force feature convolutional network, and deformation flow feature convolutional network to obtain corresponding spatial features. The three spatial features are spatially registered according to the positional relationship of the tactile sensor surfaces. The appearance, local force, and deformation information are fused by the modal interaction module, and then the features of the tactile sensors on both sides of the gripper are integrated through dual-sided fusion and pooling to output the step-level tactile latent state. The encoder adopts a single-frame structure, so that the observed tactile state and the time-by-time predicted tactile state are located in the same latent space.

[0049] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0050] In one embodiment, a robot tactile representation training device 800 based on mechanical reconstruction constraints is provided. This robot tactile representation training device 800 based on mechanical reconstruction constraints corresponds one-to-one with the robot tactile representation training method based on mechanical reconstruction constraints described in the above embodiments. For example... Figure 8 As shown, the robot tactile representation training device 800 based on mechanical reconstruction constraints includes: The parameter optimization unit 801 is used to optimize and freeze the parameters of the first tactile encoder and the first tactile reconstructor based on the first tactile image, the first dense local force field and the first mesh deformation flow collected by the tactile sensors on both sides of the robot gripper, so as to obtain the second tactile encoder and the second tactile reconstructor. The joint training unit 802 is used to encode and decode the teaching data through the second tactile encoder and the second tactile reconstructor, calculate the visual prediction loss value, the tactile latent vector prediction loss value and the action block prediction loss value, and jointly train the first multimodal generation backbone network to obtain the second multimodal generation backbone network. The task execution unit 803 is used to input the real-time acquired second tactile image, second dense local force field and second grid deformation flow together with the current visual observation, ontological perception state and task language conditions into the second multimodal generation backbone network, decode to obtain action block sequence, and cyclically reason until the task is completed.

[0051] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0052] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0053] This application also provides a computer device, such as... Figure 9As shown, the computer device includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor. When the processor executes the computer program, it implements the steps in any of the above method embodiments, or when the processor executes the computer program, it implements the functions of each module / unit in the above device embodiments.

[0054] For example, the computer program may be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete this application. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the computer device.

[0055] Those skilled in the art will understand that Figure 9 The computer device described is merely an example and does not constitute a limitation on the computer device. It may include more or fewer components than shown, or combine certain components, or different components. For example, the computer device may also include input / output devices, network access devices, buses, etc.

[0056] The aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), or field-programmable gate arrays (FPGAs). Programmable Gate Array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0057] The memory can be an internal storage unit of the computer device, such as a hard drive or RAM. The memory can also be an external storage device of the computer device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory can include both internal and external storage units of the computer device.

[0058] This application also provides a readable storage medium storing a computer program, which, when executed by a processor, can implement the steps in the above-described method embodiments.

[0059] This application provides a computer program product that, when run on an electronic device, enables the electronic device to implement the steps described in the various method embodiments above.

[0060] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0061] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0062] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0063] In the embodiments provided in this application, it should be understood that the disclosed apparatus / devices and methods can be implemented in other ways. For example, the apparatus / device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0064] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0065] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A robot tactile representation training method based on mechanical reconstruction constraints, characterized in that, include: Based on the first tactile image, the first dense local force field, and the first mesh deformation flow collected by the tactile sensors on both sides of the robot gripper, the parameters of the first tactile encoder and the first tactile reconstructor are optimized and frozen to obtain the second tactile encoder and the second tactile reconstructor. The teaching data is encoded and decoded by the second tactile encoder and the second tactile reconstructor, the visual prediction loss value, the tactile latent vector prediction loss value and the action block prediction loss value are calculated, and the first multimodal generation backbone network is jointly trained to obtain the second multimodal generation backbone network. The real-time acquired second tactile image, second dense local force field, and second grid deformation flow, together with the current visual observation, ontological perception state, and task language conditions, are input into the second multimodal generation backbone network. The action block sequence is decoded and iterated until the task is completed.

2. The robot tactile representation training method based on mechanical reconstruction constraints as described in claim 1, characterized in that, The process of optimizing and freezing the parameters of the first tactile encoder and the first tactile reconstructor based on the first tactile image, the first dense local force field, and the first mesh deformation flow acquired by the tactile sensors on both sides of the robot gripper, to obtain the second tactile encoder and the second tactile reconstructor, includes: The robot gripper acquires a first tactile image, a first dense local force field, and a first grid deformation flow based on the tactile sensors on both sides of its gripper. The first tactile image, the first dense local force field, and the first mesh deformation flow are input into the first tactile encoder for feature extraction and cross-modal cross-attention calculation, respectively, to obtain the single-sided modal interaction feature maps on both sides of the gripper. The channel splicing and feature compression of the single-sided modal interaction feature maps obtained on both sides of the gripper are used to obtain the historical tactile latent vector; The historical tactile latent vectors are input into the first tactile reconstructor, the reconstruction loss is calculated, and the parameters of the first tactile encoder and the first tactile reconstructor are jointly optimized and frozen to obtain the second tactile encoder and the second tactile reconstructor.

3. The robot tactile representation training method based on mechanical reconstruction constraints as described in claim 2, characterized in that, The process of inputting the historical tactile latent vector into the first tactile reconstructor, calculating the reconstruction loss, and performing joint parameter optimization and parameter freezing on the first tactile encoder and the first tactile reconstructor to obtain the second tactile encoder and the second tactile reconstructor includes: The historical tactile latent vectors are input into the first tactile reconstructor for parallel decoding to obtain the reconstructed dense local force field, the reconstructed grid deformation flow, and the reconstructed bilateral contact resultant force. The force field reconstruction error is calculated based on the reconstructed dense local force field and the actual dense local force field; the deformation flow reconstruction error is calculated based on the reconstructed mesh deformation flow and the actual mesh deformation flow; the contact resultant force reconstruction error is calculated based on the reconstructed bilateral contact resultant force and the actual bilateral contact resultant force; and the force field reconstruction error, the deformation flow reconstruction error, and the contact resultant force reconstruction error are weighted and summed to obtain the reconstruction loss. Based on the reconstruction loss, the first tactile encoder and the first tactile reconstructor are jointly backpropagated and optimized until convergence. After convergence, the corresponding parameters are frozen to obtain the second tactile encoder and the second tactile reconstructor.

4. The robot tactile representation training method based on mechanical reconstruction constraints as described in claim 3, characterized in that, The process involves encoding and decoding the teaching data using the second tactile encoder and the second tactile reconstructor, calculating the visual prediction loss, tactile latent vector prediction loss, and action block prediction loss, and jointly training the first multimodal generative backbone network to obtain the second multimodal generative backbone network, including: The teaching data is input frame by frame into the second tactile encoder for encoding to obtain a real future tactile latent vector sequence corresponding to each teaching frame, and the real future tactile latent vector sequence is used as a tactile supervision label. The predicted future tactile latent vector sequence predicted by the tactile expert in the first multimodal generation backbone network is input into the second tactile reconstructor for decoding to obtain the predicted dense local force field sequence, the predicted grid deformation flow sequence, and the predicted bilateral contact resultant force sequence. Semantic consistency regularization terms are calculated with the real dense local force field, real grid deformation flow, and real bilateral contact resultant force of the corresponding frame in the teaching data, respectively. The semantic consistency regularization term is weighted and summed with the tactile latent vector prediction loss, visual prediction loss, and action block prediction loss to obtain the total training loss. Based on the total training loss, the first multimodal generative backbone network is optimized by backpropagation. The gradient is transmitted through the second tactile reconstructor to the future tactile latent vector sequence but does not update the parameters of the second tactile reconstructor until convergence, thus obtaining the second multimodal generative backbone network.

5. The robot tactile representation training method based on mechanical reconstruction constraints as described in claim 4, characterized in that, The step involves weighting and summing the semantic consistency regularization term with the tactile latent vector prediction loss, visual prediction loss, and action block prediction loss to obtain the total training loss. This total training loss is then used to perform backpropagation optimization on the first multimodal generative backbone network. The gradient is transmitted through the second tactile reconstructor to the future tactile latent vector sequence without updating the parameters of the second tactile reconstructor, until convergence, resulting in the second multimodal generative backbone network. This includes: The total training loss is obtained by weighting and summing the semantic consistency regularization term, the tactile latent vector prediction loss, the visual prediction loss, and the action block prediction loss. Backpropagation is performed on the first multimodal generation backbone network based on the total training loss. The gradient of the semantic consistency regularization term is passed forward through the computation graph of the second tactile reconstructor to the future tactile latent vector sequence predicted by the tactile experts. The parameters of the second tactile reconstructor do not receive any gradient updates during this process, and the parameter gradients of each expert network are obtained. Visual, tactile, and action terms are input into a masked hybrid self-attention layer for forward computation. A mask matrix constrains the visibility range between different types of terms, enabling visual and tactile terms to interact based on current observation information and mutually visible information. Simultaneously, action terms are prohibited from reading future visual and tactile prediction terms. Since the output of the masked hybrid self-attention layer depends on visual, tactile, and action terms within their visible ranges, the visual prediction loss, the tactile latent vector prediction loss, the action block prediction loss, and the... The gradients generated by each of the semantic consistency regularization terms are propagated through the shared computation graph of the masked hybrid self-attention layer to the input projection parameters and attention parameters corresponding to other visible modalities, forming cross-modal gradient collaboration. After performing gradient norm clipping on the trainable parameters of the first multimodal generative backbone network, the optimizer updates the parameters of the visual expert, the tactile expert, the action expert, and the masked hybrid self-attention layer, and repeats the forward computation, total training loss calculation, backpropagation, and parameter update until the total training loss converges, thus obtaining the second multimodal generative backbone network.

6. The robot tactile representation training method based on mechanical reconstruction constraints as described in claim 5, characterized in that, The process of inputting the real-time acquired second tactile image, second dense local force field, and second mesh deformation flow, along with the current visual observation, ontological perception state, and task language conditions, into the second multimodal generation backbone network, decoding to obtain an action block sequence, and iteratively reasoning until the task is completed includes: The real-time acquired second tactile image, second dense local force field, and second grid deformation flow are input into the second tactile encoder for single-frame forward inference to obtain real-time tactile latent vectors. The real-time tactile latent vectors, together with the current visual observation, ontological perception state, and task language conditions, are input into the second multimodal generation backbone network for decoding to obtain an action block sequence of length H frames. Each action word in the action block sequence is used as the target joint position state and is sent to the joint controller for execution in frame order to complete the H-step action. After the H-step action is completed, the third tactile image, the third dense local force field, and the third grid deformation flow of the new frame are acquired and input into the second tactile encoder for single-frame forward inference to obtain the tactile latent vector of the new frame. The real-time tactile latent vector is then updated with the tactile latent vector of the new frame to trigger the next round of decoding. This process is repeated until the task is completed.

7. A robot tactile representation training device based on mechanical reconstruction constraints, characterized in that, The steps for implementing the robot tactile representation training method based on mechanical reconstruction constraints as described in any one of claims 1 to 6 include: The parameter optimization unit is used to optimize and freeze the parameters of the first tactile encoder and the first tactile reconstructor based on the first tactile image, the first dense local force field and the first mesh deformation flow collected by the tactile sensors on both sides of the robot gripper, so as to obtain the second tactile encoder and the second tactile reconstructor. The joint training unit is used to encode and decode the teaching data through the second tactile encoder and the second tactile reconstructor, calculate the visual prediction loss value, the tactile latent vector prediction loss value and the action block prediction loss value, and jointly train the first multimodal generation backbone network to obtain the second multimodal generation backbone network. The task execution unit is used to input the real-time acquired second tactile image, second dense local force field, and second grid deformation flow together with the current visual observation, ontological perception state, and task language conditions into the second multimodal generation backbone network, decode it to obtain the action block sequence, and iterate and reason until the task is completed.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the robot tactile representation training method based on mechanical reconstruction constraints as described in any one of claims 1 to 6.

9. A readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the robot tactile representation training method based on mechanical reconstruction constraints as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the robot tactile representation training method based on mechanical reconstruction constraints as described in any one of claims 1 to 6.

Images