A method, system and robot for contact-based operation

Abstract

The application discloses a control method, system and robot for contact operation. The method acquires visual information and first tactile information of a robot performing contact operation, and predicts second tactile information based on the visual information. Then, a residual tactile representation is determined according to the first tactile information and the second tactile information, and the residual tactile representation is used to represent physical contact changes not expressed in the second tactile information. Then, an upcoming contact operation is generated based on the residual tactile representation, and the upcoming contact operation is performed by the robot. The application can accurately perceive key physical contact changes that cannot be captured by vision through the residual tactile representation, and dynamically optimize the operation strategy according to the residual tactile representation, so as to improve the accuracy and stability of the contact operation.

Description

Technical Field

[0001] This application relates to the field of robotics technology, and in particular to a control method, system and robot for contact operation. Background Technology

[0002] With the rapid development of embodied intelligence and general-purpose robotics, contact manipulation has become a core task type in scenarios such as industrial assembly, home services, and precision operations. Tasks such as twisting, plugging, hole fitting, wiping, and fine assembly all rely on the robot's accurate perception and stable control of the physical contact process. The Vision-Language-Action (VLA) model, as an important technical path for robots, achieves task-level action decisions by integrating visual observation, language commands, and the robot's own state, and is a crucial technical support for current robot operation.

[0003] In related technologies, contact-based operation control technology is based on a vision-driven VLA model, which integrates visual observation information, language commands, and robot body state to output low-level action sequences to complete the operation task. However, this type of contact-based operation control relies solely on visual information and cannot perceive the actual physical interaction state in the contact operation. For example, it is difficult to characterize interaction details such as contact shape, friction changes, meshing relationships, minute posture deviations, and occlusion areas, which makes the robot prone to decision-making biases during the contact phase. Summary of the Invention

[0004] This application provides a control method, system, and robot for contact operation to solve the problems of low robot operation accuracy and stability.

[0005] In a first aspect, this application provides a control method for contact-based operation, comprising: Acquire visual and tactile information of the robot performing its first contact operation; Predict the second tactile information based on the visual information; Based on the first tactile information and the second tactile information, a residual tactile representation is determined; the residual tactile representation is used to represent physical contact changes not expressed in the second tactile information. A second contact operation is generated based on the residual tactile representation, and the robot executes the second contact operation; the second contact operation is the contact operation that the robot is about to execute.

[0006] The above technical solution has the following beneficial effects or advantages: Using residual tactile representation as one of the control bases for contact operations breaks through the limitations of relying solely on visual information to control contact operations. Residual tactile representation can filter out conventional contact information that can be predicted by visual information, and only retain physical contact changes that cannot be captured by vision. This enables robots to accurately perceive the real interaction state in visually limited scenarios such as occlusion, fine-grained coordination, and minor deviations, thereby improving the reliability and execution accuracy of contact operations.

[0007] In some embodiments of this application, the second tactile information includes a predicted tactile latent variable; the step of determining the residual tactile representation based on the first tactile information and the second tactile information specifically includes: encoding the first tactile information through a tactile encoder to obtain a real tactile latent variable; and subtracting the real tactile latent variable from the predicted tactile latent variable to obtain the residual tactile representation.

[0008] The above technical solution has the following beneficial effects or advantages: By using a tactile encoder, high-dimensional tactile information is reduced and purified, eliminating redundant noise interference while retaining core tactile features. By encoding the latent variable difference to calculate residual features, the difference between visually predicted tactile information and real tactile information can be accurately extracted, enhancing the expression of high-value contact features.

[0009] In some embodiments of this application, the method further includes: acquiring task instructions and robot body state data, wherein the task instructions are used to instruct the robot to perform a contact operation, and the robot body state data is used to characterize the robot's motion state; the step of generating a second contact operation based on the residual tactile representation specifically includes: generating a potential contact primitive based on the residual tactile representation, wherein the potential contact primitive is used to compress high-dimensional continuous physical contact changes into discrete contact event representations; encoding the visual information, the task instructions, and the robot body state data to obtain context embedding features; and performing action policy reasoning on the potential contact primitives based on the context embedding features to obtain the second contact operation.

[0010] The above technical solution has the following beneficial effects or advantages: By constructing multimodal context embeddings using visual information, task instructions, and ontology states, the limitations of single-modal decision-making are avoided. By transforming high-dimensional continuous residual tactile representations into structured discrete events through latent contact primitives, the modeling difficulty of high-dimensional tactile features can be reduced, and the model training instability can be alleviated. By using multimodal context embeddings as conditions to drive contact primitive reasoning, dynamic adaptation of operation strategies can be achieved, thereby improving the success rate of contact operations and the scene generalization ability.

[0011] In some embodiments of this application, before performing action strategy reasoning on the potential contact primitives, the method further includes: generating a gating signal based on the prediction uncertainty of the second tactile information, the gating signal being used to increase or decrease the weight of the potential contact primitives; and performing weight modulation on the potential contact primitives through the gating signal.

[0012] The above technical solution has the following beneficial effects or advantages: By dynamically adjusting the weights of tactile information based on predictive uncertainty, the operation stages can be accurately distinguished, balancing information utilization efficiency and strategy stability, thereby improving the robot's anti-interference ability and execution robustness in contact operations.

[0013] In some embodiments of this application, the step of generating potential contact primitives based on the residual tactile representation specifically includes: performing feature aggregation on the residual tactile representation through local convolutional residual blocks and global max pooling to obtain a global event vector; obtaining a learnable codebook containing a preset number of codewords, wherein the codewords are used to represent discrete feature vectors; calculating the similarity between the global event vector and the codewords in the learnable codebook; and determining the codeword with the highest similarity as the potential contact primitive.

[0014] The above technical solution has the following beneficial effects or advantages: Convolutional residual blocks and global pooling can efficiently aggregate residual tactile features and generate discrete contact primitives through learnable codebook matching, making contact event expressions more stable and semantic, and enhancing the robot's feature adaptation ability to different contact scenarios when performing contact operations.

[0015] In this embodiment of the application, the calculation of the similarity between the global event vector and the codewords in the learnable codebook further includes: updating the learnable codebook by exponential moving average; and resetting the inactive codewords of the learnable codebook according to a preset period, wherein the inactive codewords are codewords that have not been identified as potential contact primitives within a preset time period.

[0016] The above technical solution has the following beneficial effects or advantages: The exponential moving average update of the codebook enables codewords to adapt to the feature distribution changes of different contact scenarios in real time, ensuring the timeliness and accuracy of potential contact primitive generation; the periodic reset of inactive codewords can effectively avoid the feature collapse problem caused by an excessively high proportion of invalid codewords in the codebook, ensuring that each codeword in the codebook has effective contact semantic expression capabilities.

[0017] In some embodiments of this application, the method further includes: fusing the latent contact primitives and the predicted tactile latent variables through a feature linear modulation layer to obtain reconstructed tactile latent variables; decoding the reconstructed tactile latent variables into reconstructed tactile representations; and determining the reconstructed tactile representations as target constraints, wherein the target constraints are used to constrain the process of generating latent contact primitives.

[0018] The above technical solution has the following beneficial effects or advantages: By using feature linear modulation, the latent contact primitives and predicted tactile latent variables are efficiently fused, ensuring the feature integrity and computational efficiency of the fusion process. By using the reconstruction of tactile representation as a constraint, the generation process of latent contact primitives can be constrained in reverse, ensuring that the generated contact primitives always conform to the real physical contact laws and avoiding invalid or distorted contact features.

[0019] In some embodiments of this application, generating a gating signal based on the prediction uncertainty specifically includes: performing feature transformation on the prediction uncertainty through a multilayer perceptron to obtain prediction features; and performing normalization processing on the prediction features to obtain the gating signal.

[0020] The above technical solution has the following beneficial effects or advantages: Multilayer perceptrons can deeply mine the visual prediction reliability information behind prediction uncertainty, adapt to the differences in uncertainty distribution under different contact scenarios, and achieve accurate generation of gating signals; normalization processing transforms features into controllable values ​​in the 0-1 range, ensuring the smoothness and stability of gating signal adjustment.

[0021] In some embodiments of this application, the weighting modulation of the potential contact primitives by the gating signal specifically includes: mapping the potential contact primitives into tactile projection vectors via linear projection; setting a learnable non-contact default embedding, which is used to mark the scenario of non-contact operation; and performing weighted fusion of the tactile projection vectors and the non-contact default embedding by the gating signal.

[0022] The above technical solution has the following beneficial effects or advantages: Linear projection ensures feature dimension matching, non-contact default embedding accurately distinguishes operation scenarios, and weighted fusion enables adaptive injection of tactile information, which can effectively suppress noise in the non-contact stage and enhance the perception effect in the contact stage.

[0023] Secondly, some embodiments of this application provide a contact-operated control system, including: The vision acquisition module is configured to acquire visual information of the robot performing its first contact operation; The tactile sensing module is configured to acquire first tactile information of the robot performing the first contact operation. The processing module is configured as follows: Predict the second tactile information based on the visual information; Based on the first tactile information and the second tactile information, a residual tactile representation is determined; the residual tactile representation is used to represent physical contact changes not expressed in the second tactile information. A second contact operation is generated based on the residual tactile representation, and the robot executes the second contact operation; the second contact operation is the contact operation that the robot is about to execute.

[0024] The above technical solution has the following beneficial effects or advantages: It can simultaneously collect visual and tactile information during robot contact interaction, predict tactile information through visual information, and then determine residual tactile representation based on the predicted tactile information and the actual collected tactile information. The residual tactile representation can extract physical contact changes that cannot be predicted by vision and eliminate redundant information interference. Subsequent contact operations are generated based on the residual tactile features, which can fit the real physical interaction state and improve the execution accuracy and stability of contact operations.

[0025] Thirdly, this application also provides a robot including a processor and a memory; the memory stores a computer program that, when executed by the processor, implements the method as described in any of the first aspects.

[0026] The above technical solution has the following beneficial effects or advantages: Based on the hardware architecture of processor and memory, it can stably run the core program of contact operation control, accurately perceive microscopic physical contact changes that cannot be captured by visual modalities, optimize contact operation logic, and improve the operation accuracy of robots in fine contact operations such as twisting, insertion, and assembly. Attached Figure Description

[0027] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 A schematic diagram of the architecture of a contact-type operation control system provided for some embodiments of this application; Figure 2 A flowchart illustrating a control method for contact operation provided in some embodiments of this application; Figure 3 A flowchart illustrating the residual tactile representation calculation method provided in some embodiments of this application; Figure 4A schematic diagram of the process for generating a second contact operation based on residual tactile representation provided for some embodiments of this application; Figure 5 A flowchart illustrating the dynamic modulation weighting provided in some embodiments of this application; Figure 6 A schematic diagram of a process for weighting potential contact primitives is provided for some embodiments of this application; Figure 7 A flowchart illustrating the principle of a contact-based operation control method provided in some embodiments of this application; Figure 8 This is a schematic diagram illustrating a scenario of a control method for a robot performing contact operations, provided in some embodiments of this application. Detailed Implementation

[0029] The embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described below do not represent all embodiments consistent with this application. They are merely examples of systems and methods consistent with some aspects of this application as detailed in the claims.

[0030] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.

[0031] In this application, "robot" includes, but is not limited to, industrial robotic arms, collaborative robots, assembly robots, and service robots—devices capable of performing contact-based tasks. These contact-based tasks include, but are not limited to, operations requiring physical contact with objects, such as screwing, plugging, mating, wiping, and grasping. Each contact-based task comprises multiple consecutive contact operations. For example, a light bulb screwing task includes multiple contact operations such as grasping and screwing.

[0032] In some embodiments, the robot may be configured with a contact-operation control system for controlling the robot to perform contact tasks.

[0033] Figure 1 This is a schematic diagram of the architecture of a contact-type operation control system provided for some embodiments of this application. For example... Figure 1 As shown, the system includes a visual acquisition module 110, a tactile acquisition module 120, and a processing module 130.

[0034] In some embodiments, the vision acquisition module 110 is configured to acquire visual information of the robot performing contact operations. The vision acquisition module 110 includes one or more image acquisition devices, such as an RGB camera, a depth camera, a stereo vision camera, etc. In some embodiments, the vision acquisition module 110 uses an RGB-D camera, capable of simultaneously acquiring color images and depth information. The vision acquisition module 110 is installed at different positions on the robot and can acquire visual information from the robot's base view, side view, and wrist view.

[0035] The visual information acquired by the visual acquisition module 110 includes, but is not limited to, the appearance features of the target object such as position, posture, shape, color, and texture, as well as the robot's pose information. The visual information is transmitted to the processing module 130 in the form of an image sequence.

[0036] In some embodiments, the tactile acquisition module 120 is configured to acquire tactile information when the robot performs contact operations. The tactile acquisition module 120 may include tactile sensors mounted on the robot's end effector (such as a robotic gripper), such as dual-finger tactile sensors, multi-finger tactile sensors, or array-type tactile sensors. The tactile sensors can convert physical interaction information such as pressure distribution, local contact morphology, and sliding state of the contact area into a two-dimensional tactile image. After filtering, amplifying, performing analog-to-digital conversion, and image normalization on the acquired raw tactile image, the tactile acquisition module 120 transmits it to the processing module 130 in the form of a digital tactile image.

[0037] The processing module 130 is electrically or communicatively connected to the vision acquisition module 110 and the tactile acquisition module 120, respectively, and is configured to receive visual information transmitted by the vision acquisition module 110 and tactile information transmitted by the tactile acquisition module 120, and control the robot's contact operation based on the received visual and tactile information.

[0038] In some embodiments, the processing module 130 may be a central processing unit (CPU), microcontroller unit (MCU), digital signal processor (DSP), field programmable gate array (FPGA), application specific integrated circuit (ASIC) with data processing capabilities, or a computing device containing the above-mentioned processors.

[0039] In some embodiments, the processing module 130 may pre-store algorithms for contact operation control, such as grasping posture planning algorithms, force-position hybrid control algorithms, compliant control algorithms, etc. The processing module 130 can respond to the task instructions of each contact operation and adjust the pose and applied force of the robot end effector through the above algorithms to perform the corresponding contact operation, such as grasping, assembly, grinding, polishing, etc.

[0040] In related technologies, when robots perform contact tasks, they rely on a vision-driven Visual Alignment (VLA) model, which integrates visual observation information, verbal commands, and the robot's own state to output a low-level action sequence corresponding to the contact operation, thereby completing the contact task. However, the vision-driven VLA model depends solely on visual information and cannot perceive the actual physical interaction state in the contact operation. For example, it is difficult to perceive interaction details such as contact shape, friction changes, meshing relationships, minute posture deviations, and occlusion areas, which makes the robot prone to decision-making biases during the contact phase.

[0041] To address the aforementioned issues, this application provides a control method for contact-based operations, applicable to the robot described in the above embodiments. This method filters out predictable conventional contact information through residual tactile representation, retaining only physical contact changes that cannot be captured visually, such as abnormal collisions, contact establishment, friction changes, thread engagement, and localized compression. This enables the robot to accurately perceive the actual interaction state. Furthermore, the method optimizes the contact-based operation strategy to be executed based on the residual tactile representation, thereby improving the reliability and execution accuracy of the contact-based operation.

[0042] Figure 2 This application provides a flowchart illustrating a control method for contact-based operation in some embodiments, as shown below. Figure 2 As shown, the method includes the following steps: S201. Obtain visual information and tactile information of the robot performing the first contact operation.

[0043] The visual and tactile information corresponding to the robot performing contact operations are acquired simultaneously. The first contact operation is any contact operation performed by the robot in a contact task. For ease of distinction, the tactile information corresponding to the first contact operation is represented as the first tactile information.

[0044] In some embodiments, the visual information includes image data corresponding to the robot performing the first contact operation. The image data may include RGB images from the base view, side view, and wrist view. The first tactile information is the tactile image of the robot performing the first contact operation. For example, if the robot performs the first contact operation at time t, the visual information includes V at time t. t base (RGB image from the base perspective), Vt side (RGB image from a side view) and V t wrist (RGB image from wrist view), tactile information includes I at time t. t tac Tactile images.

[0045] S202, Predicting second tactile information based on visual information.

[0046] The robot's tactile information is predicted based on the acquired visual information to obtain second tactile information. This second tactile information is a predicted value obtained from the visual information regarding the tactile feedback the robot might experience during the first contact operation. For example, based on this visual information, a pre-defined prediction model is used to estimate the tactile feedback the robot might feel during the first contact operation, thereby generating the second tactile information.

[0047] For ease of distinction, in this embodiment of the application, the actual tactile information collected when the robot performs a contact operation is represented as the first tactile information, and the tactile information predicted based on visual information is represented as the second tactile information.

[0048] S203. Determine the residual tactile representation based on the first tactile information and the second tactile information.

[0049] Based on the predicted second tactile information and the actual first tactile information obtained by the robot performing the first tactile operation, the actual physical contact changes not expressed in the predicted second tactile information are identified as residual tactile features. That is, residual tactile features are used to represent physical contact changes not expressed in the second tactile information, such as tactile feedback changes caused by subtle differences in the texture of the object surface, local abrupt changes in contact pressure, and frictional changes caused by slight adjustments in the contact angle.

[0050] S204. Generate a second contact operation based on the residual tactile representation, and have the robot perform the second contact operation.

[0051] After obtaining the residual tactile representation, the contact operation to be executed can be generated based on the residual tactile representation. For ease of explanation, in the embodiments of this application, the contact operation to be executed after the first contact operation is represented as the second contact operation. That is, the second contact operation is the contact operation that the robot is about to execute.

[0052] In other words, the physical contact changes reflected by the residual tactile representation, such as subtle differences in the surface texture of an object, local abrupt changes in contact pressure, or changes in friction caused by minute adjustments in the contact angle, are used to guide parameter adjustments and path planning for the second contact operation. For example, if the residual tactile representation shows a local abrupt change in the current contact pressure, the system can generate control commands to reduce the contact pressure or adjust the contact posture to optimize the execution effect of the second contact operation.

[0053] In this embodiment, a residual tactile representation is obtained by comparing the first tactile information actually collected with the second tactile information predicted based on visual information and subtracting the two. This residual tactile representation can filter out conventional contact information that can be predicted by visual information, retaining only physical contact changes that cannot be captured by vision. This reduces the introduction of redundant noise, enabling the robot to accurately perceive the interaction state of real physical contact in visually limited scenarios such as occlusion, fine-grained coordination, and minute deviations, thereby improving the reliability and execution accuracy of contact operations.

[0054] In some embodiments, task instructions and robot body state data are also acquired. The task instructions are used to instruct the robot to perform contact operations, such as a first contact operation and subsequent contact operations; the robot body state data is used to characterize the robot's motion state.

[0055] For example, when the robot performs its first contact operation at time t, the task instruction is the task instruction L of the task to which the first contact operation belongs, and the robot's body state data is s. t s t Belongs to R 7 Or a higher-dimensional state space, which may include joint angles, joint velocities, end-effector poses, gripper opening and closing amounts, etc.

[0056] For step S202, in some embodiments, visual features of the visual information are extracted, and these visual features are mapped to the variable space of tactile features to obtain the predicted tactile latent variables and their prediction uncertainties. The predicted tactile latent variables and their prediction uncertainties are then used as the second tactile information. Here, the prediction uncertainty is the confidence level regarding the predicted tactile latent variables.

[0057] In some implementations, the process of extracting visual features from visual information and mapping these visual features to a variable space of tactile features is performed by a visual encoder. For example, the visual encoder is a learnable 18-layer residual network (ResNet-18) backbone followed by a multilayer perceptron head. Visual information is input to the visual encoder, which processes the information and outputs the corresponding mean and standard deviation. The mean is used as a prediction of the tactile latent variable, and the variance is used as the prediction uncertainty corresponding to the prediction of the tactile latent variable. In this way, the visual encoder can learn the mapping relationship from visual input to tactile feature distribution parameters end-to-end.

[0058] To improve the prediction speed based on visual information, in some embodiments, the RGB image of the wrist view in the visual information is input into the visual encoder to predict the second tactile information, without the need to introduce the hand view or other multi-view images, thereby effectively reducing the amount of image data processing.

[0059] Because RGB images from the wrist perspective can capture key visual cues related to contact during operation, such as the surface texture of the target object, the deformation trend of the contact area, and the relative positional relationship between the tool and the object, the visual encoder can directly extract and map features from images focused solely on this perspective. This avoids the computational resource consumption and time delays caused by multi-view image fusion, improves the prediction efficiency of second tactile information, and meets the real-time requirements of contact-based operations.

[0060] For example, the RGB image from the wrist view V t wrist Input visual encoder E vis E vis This is a multilayer perceptron head connected to the ResNet-18 backbone network. vis For V t wrist Process the data and output the mean μ. t and standard deviation σ t , where μ t Belongs to R (C'×H'×W') , σ t Belongs to R; C' is relative to V t wrist After feature extraction, the channel dimension, H' versus V t wrist The height dimension after feature extraction, W' is the height dimension of V. t wrist After feature extraction, the width dimension is determined. This leads to the corresponding predicted distribution: ~N(μ t ,σ t 2 I).

[0061] In step S203, in some embodiments, the first tactile information and the second tactile information are subtracted to obtain the corresponding residual tactile features. In this way, the residual tactile features can effectively capture subtle changes caused by real physical contact that are missed in the second tactile information predicted solely by visual information, such as small fluctuations in contact force and differences in friction caused by the texture of the object's surface. This provides supplementary key information for more precise control of the robot to perform contact operations.

[0062] Figure 3 This is a flowchart illustrating a method for calculating residual tactile representations provided in some embodiments of this application. In some embodiments, visual information includes predicted tactile latent variables, as described in the above embodiments. Figure 3 As shown, determining the residual tactile representation specifically includes the following steps: S301. The first tactile information is encoded by a tactile encoder to obtain the real tactile latent variable.

[0063] The first tactile information, acquired in real-world data, is input into a tactile encoder to encode the information, obtaining corresponding variables that serve as latent tactile variables. In some implementations, the tactile encoder is at least one of a Unified Tactile Representation (UniT) type self-supervised tactile representation network, a convolutional autoencoder, a variational autoencoder, or a Transformer-type tactile encoder. For example, the real tactile image I acquired by the tactile sensor... tta c Input haptic encoder E tac The real tactile latent variable z is obtained. t z t =E tac (I t tac ), z t Belongs to R (C'×H'×W') C' is for I t tac After feature extraction, the channel dimension H' is related to I. t tac The height dimension after feature extraction, W' is the height dimension of I. t tac The width dimension after feature extraction.

[0064] In some implementations, the haptic encoder employs a convolutional autoencoder structure. The encoder portion includes three convolutional layers with kernel sizes of 3×3, 3×3, and 2×2, and channel numbers of 32, 64, and 128, respectively. Each layer is followed by a batch normalization layer and a rectified linear unit (ReLU) activation function. The haptic encoder receives a haptic image (16×4×1) and outputs a 256-dimensional true haptic latent variable.

[0065] S302. Subtract the actual tactile latent variables from the predicted tactile latent variables to obtain the residual tactile representation.

[0066] The difference between the actual tactile latent variables and the predicted tactile latent variables is taken, and the resulting variable is used as the residual tactile representation. For example, as shown in the example above, the residual tactile representation r t =z t - .

[0067] Figure 4 This is a schematic diagram illustrating the process of generating a second contact operation based on residual tactile representation, provided for some embodiments of this application. For example... Figure 4 As shown, in some embodiments, generating a second tactile operation based on residual tactile representation specifically includes the following steps: S401. Generate potential contact primitives based on residual tactile representations.

[0068] Among them, latent contact primitives are used to compress high-dimensional continuous physical contact changes into discrete contact event representations. For example, when the residual tactile representation shows an abnormal pressure surge in the contact area, the latent contact primitive may be identified as a "collision establishment" event; if the residual tactile representation shows periodic frictional fluctuations, it may correspond to a "thread engagement" event.

[0069] In some embodiments, the residual tactile representation can be aggregated using local convolutional residual blocks and global max pooling to obtain a global event vector, which represents the intensity and category of the contact event at the current moment. A learnable codebook containing a predetermined number of codewords is then obtained; these codewords represent discrete feature vectors. For example, the learnable codebook may contain 64 codewords, each with a dimension of 128. Then, the similarity between the global event vector and the codewords in the learnable codebook is calculated, and the codeword with the highest similarity is identified as the potential contact primitive.

[0070] In this embodiment, the local convolutional residual block can deeply mine and extract subtle features of local regions in the residual tactile representation. Through multi-layer convolutional operations and skip connections, it preserves local texture information and gradient changes during the contact process, such as pressure distribution differences caused by minute bumps and depressions at the contact interface. Global max pooling further aggregates these local features, filters out the most representative key feature values, and forms a global event vector that reflects the overall contact situation.

[0071] For example, the residual tactile representation r t The event encoder f inputs convolutional residual blocks plus a global max pooling structure φ f φ For r t After processing, the global event vector h is obtained. t Among them, h t =f φ (r t ), h t Belongs to R D .

[0072] In some embodiments, the learnable codebook is trained using a large amount of historical contact information, with each codeword corresponding to a typical contact event.

[0073] In some implementations, when calculating the similarity between the global event vector and each codeword in the codebook, at least one of cosine similarity and Euclidean distance is used. By comparing the spatial distance or directional consistency between vectors based on similarity, the codeword that best matches the current global event vector is determined. The contact event pattern represented by this codeword is the generated potential contact primitive, thereby achieving an effective conversion from continuous high-dimensional physical contact changes to stable discrete event representations.

[0074] For example, constructing a learnable codebook C containing K codewords. The global event vector h t The similarity score is calculated for each codeword, and the cosine similarity is used to select the best matching codeword: q t =argmax(c k ∈C)cos(h t ,c k ).

[0075] To mitigate the risk of codebook collapse, in some embodiments, when calculating the similarity between the global event vector and codewords in the learnable codebook, the learnable codebook is updated using an exponential moving average, enabling codewords to adapt to the feature distribution of different contact scenarios. Additionally, inactive codewords in the learnable codebook are reset at a preset period, where inactive codewords are those not identified as potential contact primitives within a preset time period, such as the last 1000 inferences.

[0076] In some implementations, when updating the learnable codebook using exponential moving average, if a codeword is selected as a potential contact primitive, this codeword is updated according to the following rules: c i =c i +α×(h t -c i ), where α is the learning rate, set to 0.01, h t This is the global event vector.

[0077] In some embodiments, latent contact primitives and predicted tactile latent variables are fused through a feature linear modulation layer to obtain reconstructed tactile latent variables. These reconstructed tactile latent variables are then decoded into reconstructed tactile representations, which are then used as target constraints to govern the generation of latent contact primitives.

[0078] In some implementations, the feature-wise linear modulation layer employs a feature-wise linear modulation (FiLM) mechanism.

[0079] For example, as shown in the example above, q t and The reconstructed tactile latent variables are obtained by fusing through a feature modulation layer. . =FiLM( q t Then, through the tactile decoder D... tac Will Decoding to reconstruct tactile representation (image) , t =D tac( ) Thus, q is constrained through the reconstruction process. t It can express key contact events without deviating from the semantics of real touch.

[0080] In this embodiment of the disclosure, the latent contact primitives and predicted tactile latent variables are input into the feature linear modulation layer, and the FiLM layer will adjust the input based on the latent contact primitives q. tModulation parameters are generated, and the features of each channel of the predicted tactile latent variable are dynamically adjusted. In this way, the latent contact primitives can guide the reconstruction process of the predicted tactile latent variable, so that the reconstructed tactile latent variable not only includes the basic tactile features predicted by vision, but also incorporates the key contact events revealed by the residual information. Subsequently, the reconstructed tactile latent variable is input into the tactile decoder to obtain the reconstructed tactile representation. The reconstructed tactile representation is compared with the real first tactile information, the loss between the two is calculated, and this loss is used as a target constraint for backpropagation to ensure that the generated latent contact primitives can more accurately reflect the real physical contact state.

[0081] S402. Encode the visual information, task instructions, and robot body state data to obtain context embedding features.

[0082] The acquired visual information, task instructions, and robot body state data are encoded to obtain context embedding features. In some implementations, the visual information, task instructions, and robot body state data are input into a pre-trained visual language model to obtain context embedding features. The pre-trained visual language model can simultaneously process information from both visual and linguistic modalities and fuse multimodal data into a unified feature representation.

[0083] For example, the pre-trained visual language model uses a Transformer-based architecture. This model includes an image encoder and a text encoder. The image encoder processes visual information (such as RGB and depth images) to extract visual features like object contours, spatial locations, and color textures. The text encoder transforms task instructions (such as "tighten bolts" or "grab blocks") into word embedding vectors and uses a self-attention mechanism to capture semantic relationships and instruction intent between words. Robot body state data (such as joint angles, end effector pose, velocity, and acceleration) is mapped to a vector space with dimensions consistent with the visual language features through a fully connected network. This vector space is then added or concatenated element-wise with the visual and text features to ultimately generate contextual embedding features that include environmental visual information, task target information, and the robot's own state information.

[0084] S403. Using contextual embedding features as conditions, perform action policy reasoning on potential contact primitives to obtain the second contact operation.

[0085] After obtaining the context embedding features, action policy reasoning is performed on the potential contact primitives based on the context embedding features, thereby obtaining the second contact operation to be executed.

[0086] Because robots may perform contact-based operations in scenarios with varying degrees of contact (stronger or weaker), and the reliability of the predicted second tactile information also differs, some embodiments employ different methods to more accurately adapt to different scenarios, such as... Figure 5 As shown, before performing action policy reasoning on potential contact primitives, the following steps are also included: S501. Generate a gating signal based on the predictive uncertainty of the second tactile information.

[0087] The gating signal is used to increase or decrease the weight of potential contact primitives. In some implementations, a gating network is used to process prediction uncertainty to output a corresponding gating signal. The gating network can be a multilayer perceptron followed by a sigmoid activation function.

[0088] In some embodiments, the prediction uncertainty is feature-transformed using a multilayer perceptron to obtain prediction features. These prediction features are then normalized to obtain a gating signal. For example, as shown in the example above, the prediction uncertainty σ is... t The input is a gating network G, which uses a multilayer perceptron followed by a sigmoid activation function to obtain a gating signal g between 0 and 1. t Specifically: g t =σ(MLP(σ t )), g t It belongs to the interval (0, 1). Thus, when σ... t When it is small, g t A value close to 0 indicates that the currently predicted second tactile information is reliable; conversely, when σ approaches 0, it indicates that the second tactile information is reliable. t When it is small, g t If the value approaches 1, it indicates that the second tactile information predicted is unreliable.

[0089] S502, weighted modulation of potential contact primitives using gating signals.

[0090] After obtaining the gating signal, the weights of the potential contact primitives are dynamically modulated using the gating signal, enabling the generation of more accurate contact operations during the policy reasoning stage. In other words, this embodiment uses the gated potential contact primitives as input data for the policy reasoning stage. Through this dynamic weight modulation mechanism based on prediction uncertainty, the reliability of the potential contact primitives can be improved, making the second contact operation more accurately match the actual operation scenario.

[0091] Figure 6 This is a schematic diagram illustrating a process for weighting potential contact primitives, provided for some embodiments of this application. For example... Figure 6 As shown, in some embodiments, weight modulation of potential contact primitives specifically includes the following steps: S601. Map the potential contact primitives to tactile projection vectors via linear projection.

[0092] Discrete latent contact primitives are mapped into tactile projection vectors via linear projection. For example, a learnable linear transformation matrix can be used to project high-dimensional latent contact primitive vectors into a space that matches the dimension of tactile information, resulting in tactile projection vectors. In this way, abstract latent contact primitives can be transformed into vector forms that are easier to fuse with tactile information and modulate weights.

[0093] In some embodiments, the mapped tactile information dimension is the token dimension of the expert network. For example, as shown in the example above, the discrete contact primitive q is mapped... t The haptic projection vector p is obtained by mapping the haptic projection vector to the token dimension of the motion expert network through a linear projection layer. t : p t =W q ×q t +b q ,

[0094] Among them W q b is a learnable weight matrix q is the learnable bias vector for the linear projection layer.

[0095] S602, Set a learnable contactless default embedding.

[0096] The contactless default embedding, used to label the scenario of contactless operation, is a fixed-dimensional learnable parameter vector with the same dimension as the tactile projection vector obtained in step S601. Thus, by making the contactless default embedding learnable, it can more accurately represent the feature information in the contactless state, thereby enabling effective comparison and fusion with the tactile projection vector in the subsequent weight modulation process.

[0097] For example, in the initial stage, the contactless default embedding is defined as a random initial value. Then, each time the parameters are updated, the gradient descent is used to adjust it based on the loss function calculation results in the contactless operation scenario, so that the contactless default embedding can gradually learn the most representative features in the contactless state.

[0098] S603, weighted fusion of tactile projection vector and non-contact default embedding is performed by gating signal.

[0099] The tactile projection vector and the non-contact default embedding are weighted and fused using a gating signal, thereby achieving weight modulation of the potential contact primitives through the gating signal. For example, as shown in the example above, a learnable non-contact default embedding e0 is set, and the tactile projection vector and the non-contact default embedding are weighted and fused using the following formula: et =g t ×p t +(1-g t )×e0, e t As a potential contact primitive after weight modulation.

[0100] Therefore, when in the free space motion stage (before the robot has made contact), the weighted potential contact primitives are closer to the non-contact default embedding, thus suppressing invalid tactile noise; when in the contact establishment or contact uncertainty stage, the weighted potential contact primitives are closer to the tactile projection vector, thus enhancing the influence of tactile feedback on action decision.

[0101] For generating the second contact operation, in some embodiments, the weighted latent contact primitive is concatenated with a noise token to obtain the corresponding concatenation result. This concatenation result is then input into a motion expert network (MAN). The MAN processes the concatenation result using conditional flow matching or diffusion-based motion generation to output the second contact operation to be executed. The second contact operation consists of a corresponding action sequence, which may include at least one of the following: end-effector pose increment, gripper opening / closing command, and joint control parameters. For example, the action sequence may include angle increments of multiple joints of the end effector.

[0102] In some embodiments, during the phase where the robot performs contact operations, steps S201-S204 are executed cyclically until the robot completes all contact operations. For example, as shown in the example above, after completing steps S201-S204 at time t, the control method continues to execute steps S201-S204 at time t+1 until the robot completes all contact operations. For example, as shown in the example above, the motion expert network outputs a future H-step motion sequence A. t ={a t ,a (t+1) ,...,a (t+H-1)}, that is, the strategy satisfies π(A) t |O t ,L). Among them, O t The visual information, first tactile information, task instructions, and robot body state data acquired at time t are given.

[0103] In some embodiments, steps S202-S203 are performed by a cross-modal predictor, and step S204 is performed by an action expert network. The training of the cross-modal predictor and the action expert network is completed through a two-stage training method. This achieves decoupling optimization of tactile representation learning and action policy learning, improves representation generalization, accelerates policy convergence, and reduces training costs.

[0104] For the first stage, as shown in the example above, the cross-modal predictor is pre-trained, specifically including: acquiring a multi-task interaction dataset and jointly training the visual encoder, haptic encoder, event encoder, codebook, and haptic decoder. The cross-modal prediction loss from joint training is defined as the weighted negative log-likelihood loss. L pred =λ σ log(σ t 2 )+||z t -μ t / σ t 2 .

[0105] Where, λ σ This is the variance penalty coefficient, used to prevent σ. t Unconstrained increases lead to uncertainty collapse. To ensure that discrete contact primitives can still reconstruct realistic tactile sensation, the tactile reconstruction loss is defined as: L rec =|| -I t tac The vector quantization part preferably uses standard vector quantization (VQ) with a committed loss of L. vq =||sg[h t ]-q t +β||h t -sg[q t ] , where sg[ ] represents the stopping gradient operator, and β is the commitment loss weight. The overall training objective of the cross-modal predictor can be written as: L CMP =L rec +λ p ×L pred +L vq , where λ p The weights are used to predict the loss. This function, corresponding to the overall training objective, simultaneously ensures the predictability of visual prior information for tactile latent variables, the semantic stability of discrete contact primitives, and the consistency of tactile reconstruction.

[0106] For the second stage, as shown in the example above, based on freezing or partially freezing the cross-modal predictor parameters, the action expert network is trained. Specifically, this includes generating weighted latent contact primitives, denoted as e. t Then e tThe motion expert network (MAN) is fed into the context embedding. The MAN's action generator is trained using conditional flow matching or diffusion-denoised targets, enabling the action generator to output the corresponding action sequence. During the inference phase, the codebook and cross-modal predictor parameters are not updated. Instead, μ is calculated in real-time based on the acquired visual information, first tactile information, task instructions, and robot body state data. t σ t r t q t and e t It generates action sequences in a scrolling manner. This ensures that even under unknown contact disturbances, abnormal physical contact can still be quickly converted into discrete tactile events that are effective for control.

[0107] Figure 7 This is a flowchart illustrating the principle of a contact-based operation control method provided in some embodiments of this application. Figure 7 As shown, the first stage is residual tactile representation learning. Visual images (visual information) captured by a wrist camera and tactile images (first tactile information) captured by a tactile sensor are used as inputs. Features are extracted by the visual encoder and tactile encoder respectively to obtain the predicted tactile latent variables for visual prediction. The distribution of the predicted tactile latent variables is (μ... t ,σ t ) and the real tactile latent variable z t By calculating the difference between the two, a residual tactile representation is obtained to extract tactile information that cannot be covered by vision. After the event encoder extracts the contact event features corresponding to the residual tactile representation, the quantization layer maps the continuous features to discrete latent contact primitives q through vector quantization. t The tactile image is then reconstructed through a feature linear modulation layer and a tactile decoder. The mean squared error loss is calculated based on the reconstructed tactile image and the tactile image captured by the tactile camera. The event encoder and quantization layer are then optimized in reverse based on the calculated mean squared error loss to indirectly optimize the effectiveness of the residual tactile representation, ensuring that the final learned latent contact primitives (the output of the quantization layer) can truly reflect the semantics of physical contact.

[0108] The second stage inputs multi-view visual images, robot body state data, and task instructions into a large visual language model, generating a multimodal semantic context. Simultaneously, the latent contact primitives output from the quantization layer in the first stage are mapped to tactile projection vectors adapted to the motion expert network via a projection layer. Through dynamic fusion of the tactile projection vectors with embeddable, learnable non-contact default embeddings using gating signals, adaptive control is achieved to suppress ineffective tactile noise during free-space motion and enhance tactile guidance during contact establishment or uncertainty stages, completing weight modulation. Finally, the multimodal semantic context and the weighted tactile projection vectors are input into the motion expert network, denoised, and used to generate robot actions, completing policy learning. This two-stage architecture decouples and optimizes tactile representation learning from action policy learning, filling in visual perception blind spots and improving the robustness and generalization ability of the robot's fine manipulation.

[0109] The following example illustrates a scenario where a robot performs a contact operation.

[0110] Figure 8 This is a schematic diagram illustrating a scenario of a control method for a robot performing contact operations, provided in some embodiments of this application. For example... Figure 8 As shown, the robot operation visual information acquired by the visual acquisition module (scene or wrist camera), tactile images acquired by the tactile acquisition module (tactile sensor), task instructions, and robot body state data are used. The tool center point (TCP pose) of the robot end effector is used as the multimodal input. The residual tactile encoder (the tactile encoder and the module corresponding to visual prediction) extracts the residual tactile representation of predicted tactile information (second tactile information) and real tactile information (first tactile information). The physical contact changes that cannot be covered by vision are encoded into discrete latent contact primitives. The uncertainty of visual prediction of tactile information is used to determine the gating signal, and the weight is dynamically determined by the gating signal. The latent contact primitives are fused with learnable non-contact default embeddings to generate gated latent contact primitives. Finally, the multimodal context embedding features and the gated latent contact primitives are input into the residual tactile vision-language-action (Residual Tactile) module. The Vision-Language-Action (ResTacVLA) core network decouples and optimizes tactile representation learning and action strategy learning, providing a foundation for robot contact tasks such as twisting and plugging.

[0111] Figure 8The following example uses a contact-based task of screwing or light bulb turning. The contact operation is divided into five consecutive stages: ① Approach, ② Pre-contact, ③ Alignment, ④ Turning, and ⑤ Completion. These correspond to the temporal changes of the gating signal: In the approach stage, the robot does not have physical contact with the object, and the gating signal is extremely low, suppressing ineffective tactile noise to ensure stable idle travel; in the pre-contact and alignment stages, the robot approaches the target and establishes contact, and the gating signal gradually rises, gradually activating tactile features to guide precise alignment; in the turning stage, continuous contact operation keeps the gating signal high, and tactile information dominates force control and motion adjustment throughout; in the completion stage, contact disappears, the gating signal drops sharply, and the gating signal switches back to a non-contact mode, filling in visual perception blind spots and helping the robot identify whether the thread is aligned, slipped, and has reached an effective screw-in state.

[0112] Similarly, in some implementations, for contact-based tasks involving plug insertion, pin insertion, and hole assembly, due to obstructions and sub-millimeter-level positional errors in the insertion area, it is difficult to determine whether misalignment, jamming, or localized collisions have occurred using only visual perception. This application utilizes residual tactile representation to quickly identify contact deviations that cannot be explained by vision, thereby correcting the end effector pose and improving the insertion success rate. For contact-based wiping tasks, due to variations in the height of the working plane and visual depth errors, robots are prone to wiping while suspended or exhibiting weak contact. This application uses residual tactile representation to reflect the actual contact state between the end effector and the surface in real time, dynamically adjusting the end effector pressure and trajectory to ensure continuous and effective contact.

[0113] Based on the above-described control method for contact operation, the control system for contact operation provided in this application embodiment is as follows: Figure 2 As shown, in some embodiments, the vision acquisition module 110 is configured to acquire visual information of the robot performing a first contact operation, such as... Figure 7 The wrist camera shown; the tactile acquisition module 120 is configured to acquire first tactile information of the robot performing a first contact operation, such as... Figure 7 The tactile sensor shown. That is, the visual acquisition module 110 and the tactile acquisition module 120 jointly execute S201. The processing module 130 is configured to execute the following program steps: S202, Predicting second tactile information based on visual information.

[0114] S203. Based on the first tactile information and the second tactile information, determine the residual tactile representation. The residual tactile representation is used to represent physical contact changes not expressed in the second tactile information. S204. A second contact operation is generated based on the residual tactile representation, and the robot executes the second contact operation. The second contact operation is the contact operation that the robot is about to perform.

[0115] In some embodiments, the processing module 130 is further configured to: acquire task instructions and robot body state data, wherein the task instructions are used to instruct the robot to perform contact operations, and the robot body state data are used to characterize the motion state of the robot.

[0116] In some embodiments, the processing module 130 performs a second contact operation based on residual tactile representation, specifically configured to: generate a potential contact primitive based on the residual tactile representation, the potential contact primitive being used to compress high-dimensional continuous physical contact changes into discrete contact event representations; encode the visual information, the task instructions, and the robot body state data to obtain context embedding features; and perform action policy reasoning on the potential contact primitive based on the context embedding features to obtain the second contact operation.

[0117] In some embodiments, before the processing module 130 performs action policy reasoning on the potential contact primitives, it is further configured to: generate a gating signal based on the prediction uncertainty of the second tactile information, the gating signal being used to increase or decrease the weight of the potential contact primitives; and perform weight modulation on the potential contact primitives through the gating signal.

[0118] In some embodiments, the processing module 130 performs the generation of potential contact primitives based on the residual tactile representation, specifically configured to: perform feature aggregation on the residual tactile representation through local convolutional residual blocks and global max pooling to obtain a global event vector; obtain a learnable codebook containing a preset number of codewords, the codewords being used to represent discrete feature vectors; calculate the similarity between the global event vector and the codewords in the learnable codebook; and determine the codeword with the highest similarity as the potential contact primitive.

[0119] In some embodiments, when the processing module 130 performs the calculation of the similarity between the global event vector and the codewords in the learnable codebook, it is further configured to: update the learnable codebook by exponential moving average; and reset the inactive codewords of the learnable codebook according to a preset period, wherein the inactive codewords are codewords that have not been identified as the potential contact primitives within a preset time period.

[0120] In some embodiments, the processing module 130 is further configured to: fuse the latent contact primitives and the predicted tactile latent variables through a feature linear modulation layer to obtain reconstructed tactile latent variables; decode the reconstructed tactile latent variables into reconstructed tactile representations; and determine the reconstructed tactile representations as target constraints, the target constraints being used to constrain the process of generating latent contact primitives.

[0121] In some embodiments, the processing module 130 performs the generation of a gating signal based on the prediction uncertainty, specifically configured to: perform feature transformation on the prediction uncertainty through a multilayer perceptron to obtain prediction features; and perform normalization processing on the prediction features to obtain the gating signal.

[0122] In some embodiments, the processing module 130 performs weighted modulation of the potential contact primitives through the gating signal, specifically configured to: map the potential contact primitives to tactile projection vectors via linear projection; set a learnable non-contact default embedding, which is used to mark the scenario of non-contact operation; and perform weighted fusion of the tactile projection vectors and the non-contact default embedding through the gating signal.

[0123] Based on the above-described control method for contact operation, this application also provides a robot, which includes a processor and a memory. The memory stores a computer program that can be executed by at least one processor to cause the at least one processor to perform the steps of the control method for contact operation as described above.

[0124] In some embodiments, the memory may be volatile memory, non-volatile memory, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer. The processor may be a central processing unit, graphics processor, application-specific integrated circuit, field-programmable gate array or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, etc., capable of executing computer programs to achieve precise control of the robot's contact operations, thereby improving the robot's operational adaptability and intelligence level in complex contact environments.

[0125] This application also provides a computer program product, which causes the above-described contact operation control method to be executed when a computer reads and executes the computer program product.

[0126] The computer program product can be a software package or SDK (Software Development Kit), or it can be a computer program stored on a computer-readable storage medium (such as a USB flash drive, solid-state drive, optical disc, or cloud storage space).

[0127] This application also provides a computer-readable storage medium storing a computer program, which, when part or all of the computer program is executed by a computer, causes the above-described control method for contact operation to be executed.

[0128] Similar parts between the embodiments provided in this application can be referred to mutually. The specific implementation methods provided above are only a few examples under the overall concept of this application and do not constitute a limitation on the scope of protection of this application. For those skilled in the art, any other implementation methods extended from the solution of this application without creative effort shall fall within the scope of protection of this application.

Claims

1. A control method for contact-based operation, characterized in that, include: Acquire visual and tactile information of the robot performing its first contact operation; Predict the second tactile information based on the visual information; Based on the first tactile information and the second tactile information, the residual tactile representation is determined; The residual tactile representation is used to represent physical contact changes that are not expressed in the second tactile information; A second contact operation is generated based on the residual tactile representation, and the robot executes the second contact operation; the second contact operation is the contact operation that the robot is about to execute.

2. The method according to claim 1, characterized in that, The second tactile information includes predicted tactile latent variables; the step of determining the residual tactile representation based on the first tactile information and the second tactile information specifically includes: The first tactile information is encoded by a tactile encoder to obtain the true tactile latent variables; The residual tactile representation is obtained by subtracting the actual tactile latent variable from the predicted tactile latent variable.

3. The method according to claim 1, characterized in that, Also includes: Acquire task instructions and robot body state data, wherein the task instructions are used to instruct the robot to perform contact operations, and the robot body state data are used to characterize the robot's motion state; The generation of the second contact operation based on the residual tactile representation specifically includes: Based on the residual tactile representation, latent contact primitives are generated, which are used to compress high-dimensional continuous physical contact changes into discrete contact event representations; The visual information, the task instructions, and the robot body state data are encoded to obtain context embedding features; Using the context embedding features as conditions, action policy reasoning is performed on the potential contact primitives to obtain the second contact operation.

4. The method according to claim 3, characterized in that, Before performing action policy reasoning on the potential contact primitives, the method further includes: A gating signal is generated based on the prediction uncertainty of the second tactile information, and the gating signal is used to increase or decrease the weight of the potential contact primitive; The potential contact primitives are weighted by the gating signal.

5. The method according to claim 3, characterized in that, The generation of potential contact primitives based on the residual tactile representation specifically includes: The residual tactile representation is aggregated using local convolutional residual blocks and global max pooling to obtain a global event vector; Obtain a learnable codebook containing a preset number of codewords, wherein the codewords are used to represent discrete feature vectors; Calculate the similarity between the global event vector and the codewords in the learnable codebook; The codeword with the highest similarity is identified as the potential contact primitive.

6. The method according to claim 5, characterized in that, The calculation of the similarity between the global event vector and the codewords in the learnable codebook further includes: The learnable codebook is updated by an exponential moving average. In addition, the inactive codewords of the learnable codebook are reset according to a preset period, wherein the inactive codewords are codewords that have not been identified as potential contact primitives within a preset time period.

7. The method according to any one of claims 3-6, characterized in that, Also includes: The latent contact primitives and the predicted tactile latent variables are fused through a feature linear modulation layer to obtain the reconstructed tactile latent variables; The reconstructed tactile latent variables are decoded into reconstructed tactile representations; The reconstructed tactile representation is defined as the target constraint, which is used to constrain the process of generating potential contact primitives.

8. The method according to claim 4, characterized in that, The step of generating a gating signal based on the predicted uncertainty specifically includes: The prediction uncertainty is transformed using a multilayer perceptron to obtain the prediction features; The predicted features are normalized to obtain the gated signal.

9. The method according to claim 4, characterized in that, The weighting modulation of the potential contact primitives using the gating signal specifically includes: The potential contact primitives are linearly projected and mapped into tactile projection vectors; Set a learnable contactless default embedding, which is used to mark scenarios of contactless operations; The tactile projection vector and the non-contact default embedding are weighted and fused using the gating signal.

10. A contact-operated control system, characterized in that, include: The vision acquisition module is configured to acquire visual information of the robot performing its first contact operation; The tactile sensing module is configured to acquire first tactile information of the robot performing the first contact operation. The processing module is configured as follows: Predict the second tactile information based on the visual information; Based on the first tactile information and the second tactile information, a residual tactile representation is determined; the residual tactile representation is used to represent physical contact changes not expressed in the second tactile information. A second contact operation is generated based on the residual tactile representation, and the robot executes the second contact operation; the second contact operation is the contact operation that the robot is about to execute.

11. A robot, characterized in that, It includes a processor and a memory; the memory stores a computer program that, when executed by the processor, implements the method as described in any one of claims 1 to 9.

12. A computer program product, characterized in that, When a computer reads and executes a computer program product, the method as described in any one of claims 1 to 9 is performed.

Images