An end-to-end cross-entity generalized visual language action processing method and device based on VLA fused haptic feedback
By unifying tactile data into standard contact tokens and normalizing their physical dimensions, combined with residual motion correction, the problem of migrating and real-time control of visual language motion models across different robot platforms was solved, enabling flexible cross-platform application and real-time adjustment.
Patent Information
- Application Number
- CN202611119035.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-27
- Publication Date
- 2026-11-10
AI Technical Summary
Existing visual language action models cannot achieve cross-entity generalization and real-time closed-loop correction. This leads to differences in tactile sensor types, sampling frequencies, and dimensions between different robot platforms, causing the model to be bound to specific sensors and platforms. This prevents the effective integration of heterogeneous tactile feedback and limits the application of visual language action models in the field of robot operation.
By unifying the physical dimensions and correcting the closed-loop residuals, we acquire language commands, visual observation data, robot body state data, and tactile data, convert them into standard contact tokens, generate a unified state representation using a visual language action model, and achieve cross-platform migration and real-time adaptive control through residual action correction.
It enables the migration, deployment, and real-time adaptive adjustment of visual language action models across different robot platforms, solving the requirements for cross-entity generalization and real-time haptic feedback, and improving the flexibility and efficiency of robot operation.
Smart Images

Figure CN123060643A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of robot control and embodied intelligence technology, and in particular to an end-to-end cross-body generalized visual language action processing method and device based on VLA fused haptic feedback. Background Technology
[0002] Visual language action models unify inputs of images, natural language commands, and robot ontology states into a single model, directly outputting robot actions or action tokens. Leveraging large-scale visual language pre-training knowledge, they understand tasks such as "what to grasp, where to place, and according to what semantic relationships to operate," and demonstrate transferable task knowledge across different robot platforms through cross-ontology datasets and general strategies. In contact-intensive operation scenarios such as assembly, insertion, twisting, and flexible object grasping, existing technologies attempt to introduce tactile or force signals as additional inputs into the visual language action model. Multimodal fusion enhances the model's ability to perceive physical contact states; for example, tactile images or force / torque signals are concatenated with visual features and input into a Transformer architecture, enabling the model to learn the correlation between visual and tactile features during training.
[0003] However, existing technologies suffer from two fundamental drawbacks: First, there is a lack of a unified physical representation benchmark between visual language action models and tactile feedback. Different robots have varying types of tactile sensors (array pressure, visual-tactile, six-dimensional force / torque, joint torque), sampling frequencies, dimensions, and spatial resolutions. Directly stitching raw tactile data into the model binds it to a specific sensor type. When changing robot platforms or tactile sensors, data must be collected and retrained, making cross-entity generalization impossible. Second, the inference frequency of visual language action models is much lower than the tactile sampling frequency. Existing solutions typically read tactile values only once during visual inference or downsample high-frequency tactile samples before inputting them into the model. Short-term slip, impact, and contact establishment events are averaged or lost, preventing the model from instantly correcting generated action blocks based on real-time tactile changes within the control cycle. These shortcomings make it difficult to migrate and deploy the same visual language action model across different robot platforms. In contact-intensive tasks, it cannot simultaneously meet the requirements of cross-entity generalization and real-time closed-loop correction, limiting the practical application of visual language action models in robot manipulation.
[0004] Therefore, this invention proposes an end-to-end cross-entity generalized visual language action processing method and device based on VLA fused haptic feedback. Summary of the Invention
[0005] This invention provides an end-to-end cross-entity generalized visual language action processing method and device based on VLA fusion haptic feedback. By unifying the physical dimensions and correcting the closed-loop residual, it solves the technical problem of adapting heterogeneous haptic data to cross-entity actions, and realizes cross-platform migration and real-time adaptive control of visual language action models in contact-intensive tasks.
[0006] This invention provides an end-to-end cross-ontology generalized visual language action processing method based on VLA-fused haptic feedback, comprising:
[0007] Acquire language commands, visual observation data, robot body state data, tactile data, and target robot body descriptor;
[0008] Multi-rate time synchronization is performed on visual observation data, robot body state data, tactile data and language commands to obtain time-synchronized multimodal data;
[0009] The time-synchronized tactile data is converted into standard contact tokens in the tool coordinate system. The standard contact tokens include contact position, contact normal, normal force, tangential force, contact torque, slip probability, contact stiffness, vibration characteristics, temperature, confidence level, sensor type encoding, and effective field mask. Different robots only share the standard contact tokens and do not share the image size, array resolution, or units of the raw tactile data.
[0010] The visual language action model is used to encode language commands, visual observation data, robot body state data and standard contact tokens. The encoding results are grouped into displacement group, force group and tactile group according to physical dimensions. Within each group, the physical units are normalized to make the same characteristics of different robot bodies comparable under the same dimensions. Cross-group information exchange is carried out between groups to generate a unified state representation.
[0011] Generate a general action block and predict the contact trajectory based on the unified state representation. The general action block includes the end pose change, gripping coordination, target contact force, desired stiffness, desired damping, and contact mode.
[0012] The general motion block is converted into control instructions for the target robot based on the target robot body descriptor and then executed. The body descriptor includes joint topology, degrees of freedom, joint limits, kinematic parameters, end effector type, control mode, maximum speed, maximum torque, control cycle, and fault status.
[0013] Obtain the actual contact token after executing the control command, generate residual actions based on the contact error between the actual contact token and the predicted contact token at the corresponding moment in the predicted contact trajectory, and use the residual actions to correct the general action blocks that have not yet been executed.
[0014] After applying safety constraints to the modified general action block, continue controlling the target robot until the task is completed or a safety stop is triggered.
[0015] Furthermore, the haptic data is converted into standard contact tokens in the tool coordinate system, including:
[0016] When the tactile data is array pressure sensor data, the contact position is calculated using the array unit coordinates and the pressure weighting center, and the contact normal is estimated based on the pressure gradient and the sensor surface normal.
[0017] When the tactile data is from a visual tactile sensor, the contact shape and shear force are estimated by the displacement field of the marked point or optical flow.
[0018] When the tactile data is from a six-dimensional force or torque sensor, the resultant force and resultant torque in the tool coordinate system are obtained through coordinate transformation.
[0019] When the tactile data is joint torque or motor current data, the external force is estimated by subtracting the gravity, inertia and friction terms from the dynamic model and then adding the corresponding confidence value.
[0020] When a robot has multiple tactile sensors, multiple contact tokens are projected onto a unified coordinate system using sensor extrinsic parameters, and a set of contact tokens is generated by confidence weighting or non-maximum suppression.
[0021] Furthermore, multi-rate time synchronization includes:
[0022] Establish a unified time reference and record the acquisition timestamp and arrival timestamp for visual frames, robot states, tactile sampling, and control commands respectively; when devices do not share a hardware clock, estimate clock offset and drift through periodic synchronization messages, round-trip delay estimation, or correlation analysis of common motion events.
[0023] Construct a sliding time window centered on the target control time;
[0024] Time alignment is performed on visual observation data using nearest neighbor or feature interpolation methods.
[0025] Time alignment is performed on the robot's body state data using linear or spline interpolation.
[0026] The maximum normal force, tangential force integral, impact peak, slip spectrum, contact establishment time, contact release time and contact duration are extracted from the tactile data within the time window. The variable number of tactile samples are compressed into a fixed number of tactile event tokens. The tactile event tokens are then processed by the tactile encoder and converted into standard contact tokens.
[0027] When an impact, slippage, or force overshoot trend is detected, fast residual inference is triggered without waiting for the next visual frame.
[0028] Furthermore, the visual language action model is used to encode language commands, visual observation data, robot body state data, and standard contact tokens. The encoding results are grouped according to physical dimensions into displacement, force, and tactile groups, including:
[0029] Output task tokens and constraint tokens via the language encoder;
[0030] The visual encoder outputs scene features, target object features, tool features, and spatial relationship features.
[0031] The body state encoder outputs joint position, joint velocity, end effector pose, and current control mode characteristics.
[0032] The standard contact token and its timing characteristics are output through a tactile encoder.
[0033] The spatial relationship features output by the visual encoder and the end pose features output by the body state encoder are grouped into the displacement group.
[0034] The joint torque characteristics output by the body state encoder and the normal force, tangential force, and contact torque characteristics output by the tactile encoder are grouped into the force group;
[0035] The contact position, slip probability, contact stiffness, vibration characteristics, temperature, confidence level, sensor type encoding, and effective field mask features output by the tactile encoder are categorized into the tactile group;
[0036] Inter-group and cross-group information interaction is achieved through a dimensional perception attention mechanism. The dimensional perception attention mechanism applies cross-group constraints based on the physical dimensional relationship: the force group and the displacement group are coupled through stiffness matrix constraints, the time difference of the displacement group and the force group are coupled through dynamic relationship constraints, and the tactile group and the force group are coupled through contact mechanics relationship constraints, thereby obtaining cross-group interaction characteristics.
[0037] The tactile injection gating coefficients are calculated based on cross-group interaction features, linguistic features, visual features, ontological features, and tactile confidence.
[0038] The gating coefficient is used to control the cross-modal attention fusion intensity of tactile features with displacement, force and language features in cross-group interaction features;
[0039] In the approaching target stage, the gating coefficient is reduced to rely mainly on visual features. In the clamping of fragile objects stage, the gating weight corresponding to the normal force feature is increased. In the screwing bottle cap stage, the gating weight corresponding to the tangential force feature, sliding feature, and contact torque feature is increased. After gating fusion, a unified state representation is generated.
[0040] Furthermore, a general action block and a predicted contact trajectory are generated, and a residual action correction general action block is generated based on the contact error between the actual contact token and the predicted contact trajectory, including:
[0041] The motion generation head outputs a nominal motion block with a length of H control steps at a time. The nominal motion block is a predicted instance of the general motion block at the current control moment, and also outputs the predicted contact trajectory corresponding to the nominal motion block.
[0042] The general action block is organized into a pose layer, force control layer and contact layer according to physical dimensions. Each layer is independently normalized and the layers are coupled through physical relationships.
[0043] The pose layer includes end-effector translation increments, rotation increments, or six-dimensional velocity spinors in the tool coordinate system;
[0044] The force control layer includes the target normal force, the target tangential force range, the target torque, the desired end stiffness, the desired damping, and the compliance direction;
[0045] The contact layer includes contact patterns, normalized grasping coordination, action duration, action confidence, and predicted contact states;
[0046] After executing the first segment of the nominal action block, the actual contact token is obtained. The contact error between the actual contact token and the predicted contact token at the corresponding moment in the predicted contact trajectory is calculated. The contact error includes normal force error, tangential force error, slip probability error, contact position error, and contact torque error.
[0047] The residual action head generates residual actions based on contact error, current semantic features, action stage, and ontology descriptor;
[0048] The residual actions are used to correct the general action blocks that have not yet been executed. The correction method is to superimpose the residual actions with the unexecuted parts of the nominal action blocks and adjust the injection intensity of the residual actions through an error gating function.
[0049] In response to contact errors caused by indicated slippage, residual actions are generated to increase the amount of gripping coordination, change the contact direction, or reduce the tangential velocity.
[0050] In response to contact errors indicating excessive force, residual actions are generated to reduce normal displacement, reduce stiffness, or perform retraction.
[0051] In response to a contact error indicating a jammed connection, a residual action is generated to perform a search motion;
[0052] In response to a contact error indicating a loss of contact, a residual action is generated to re-approach or reposition the control end.
[0053] In response to contact errors caused by an over-limit of the indicated torque, a residual action is generated to reduce the rotation increment or change the rotation direction.
[0054] Furthermore, the ontology descriptor contains a morphological graph structure, which uses joints, links, end effectors, and contacts as nodes, and kinematic connections, mechanical couplings, and symmetry relationships as edges.
[0055] Based on the target robot body descriptor, convert the general action block into control instructions for the target robot, including:
[0056] Input the ontology descriptor into the morphological graph encoder to obtain the ontology vector;
[0057] Based on the general action block and the ontology vector, the action adapter outputs a learnable residual for the analytical kinematic mapping;
[0058] The end effector is mapped to joint space action using the generalized inverse or damped generalized inverse of the Jacobian matrix of the target robot.
[0059] Project the joint space motion onto the target robot's set of joint limit constraints, velocity constraints, torque constraints, and collision constraints;
[0060] When the target robot only supports the position control interface, the target contact force, desired stiffness, and desired damping in the general action block are converted into a small position offset; when the target robot supports torque control or impedance control interface, the torque control parameters or impedance control parameters are directly output.
[0061] Furthermore, it also includes a cross-ontology joint training step:
[0062] Acquire training trajectories from multiple robot platforms. Each training trajectory includes task instructions, visual sequences, ontology state sequences, tactile sequences, robot action sequences, success or failure markers, and corresponding ontology descriptors.
[0063] Convert each robot's actions into a general motion space representation;
[0064] Convert each tactile sequence into a standard contact token sequence;
[0065] The training trajectory is divided into stages: free space stage, pre-contact stage, contact establishment stage, stable operation stage, abnormal recovery stage, and release stage.
[0066] Align different robot features according to physical dimensions: the displacement group features of different robots are aligned with each other, the force group features are aligned with each other, and the tactile group features are aligned with each other. The alignment loss is calculated for features within the same group under a unified dimension.
[0067] The model is trained using action imitation loss, contact token prediction loss, physical dimension grouping and alignment loss, cycle consistency loss (after converting a generic action block into a target robot joint action by a ontology adapter and then reverting it back to a generic action block and comparing it with the original generic action block), safety constraint penalty loss, and random modal occlusion loss as joint training objectives. The random modal occlusion loss is the cross-entropy loss that allows the model to still predict the correct action after randomly occluding at least one modality of visual features, linguistic features, or tactile features with a preset probability during training.
[0068] Furthermore, it also includes new robot adaptation steps:
[0069] Freeze the parameters of the visual encoder, language encoder, and haptic normalization module of the visual-language-action model;
[0070] Obtain the calibration trajectory of the new robot, which includes the free space motion trajectory, the basic grasping trajectory, and the contact calibration trajectory;
[0071] The parameters of the body descriptor encoder and motion adapter of the new robot can be trained using the calibration trajectory, or the parameters can be finely tuned efficiently through a low-rank adaptation layer.
[0072] Furthermore, safety constraints include force constraints, velocity constraints, acceleration constraints, joint limit constraints, collision distance constraints, and confidence constraints;
[0073] Safety constraints are implemented through safety filters, which are constructed using one or more of the following methods: rule constraints, control barrier functions, model predictive control, collision detection, or independent safety controllers.
[0074] When the tactile confidence level falls below the threshold or the tactile sensor communication is interrupted, the confidence level is set to zero and a fault token is generated. The residual motion weight is reduced, the end effector speed and maximum gripping force are limited, and the vision and proprioception state are switched to a conservative mode. Before entering the contact phase, effective tactile feedback must be regained; otherwise, the task is stopped.
[0075] When the force exceeds the limit, the collision risk is high, or the model confidence level is below the threshold, the safety filter will stop, unload, or retract.
[0076] This invention provides an end-to-end cross-proprio generalized visual language action processing device based on VLA fused haptic feedback, comprising:
[0077] The multimodal acquisition module acquires language commands, visual observation data, robot body state data, tactile data, and target robot body descriptors.
[0078] The time synchronization module performs multi-rate time synchronization of visual observation data, robot body state data, tactile data and language commands to obtain time-synchronized multimodal data.
[0079] The haptic standardization module converts the time-synchronized haptic data into standard contact tokens in the tool coordinate system. The standard contact tokens include contact position, contact normal, normal force, tangential force, contact torque, slip probability, contact stiffness, vibration characteristics, temperature, confidence level, sensor type encoding, and valid field mask.
[0080] The multimodal fusion module uses a visual language action model to encode language commands, visual observation data, robot body state data, and standard contact tokens. The encoding results are grouped into displacement, force, and tactile groups according to physical dimensions. Within each group, the data is normalized according to physical units to make similar features of different robot bodies comparable under a unified dimension. Cross-group information exchange is performed between groups to generate a unified state representation.
[0081] The general motion generation module is used to generate general motion blocks and predict contact trajectories based on a unified state representation. The general motion block includes end pose change, grasping coordination, target contact force, desired stiffness, desired damping, and contact mode.
[0082] The body adaptation module converts general motion blocks into control instructions for the target robot based on the target robot's body descriptor and executes them. The body descriptor includes joint topology, degrees of freedom, joint limits, kinematic parameters, end effector type, control mode, maximum speed, maximum torque, control cycle, and fault status.
[0083] The tactile residual correction module obtains the actual contact token after executing the control command, generates residual actions based on the contact error between the actual contact token and the predicted contact token at the corresponding moment in the predicted contact trajectory, and uses the residual actions to correct the general action blocks that have not yet been executed.
[0084] The safety control module applies safety constraints to the modified general action block and outputs them to the target robot until the task is completed or a safety stop is triggered.
[0085] The beneficial effects of this invention compared to existing technologies are as follows: Existing visual-language-action models cannot effectively integrate heterogeneous tactile feedback in contact-intensive tasks, resulting in a disconnect between tactile perception and semantic decision-making. Different robots have varying tactile sensor types, sampling frequencies, dimensions, and control interfaces. Directly splicing raw data leads to models being tied to specific sensors and robot platforms, making data merging difficult and requiring extensive retraining for cross-entity transfer. This invention, by uniformly converting heterogeneous tactile signals into standard contact tokens and sharing only the tokens (not the raw data), normalizing them by physical dimensions to make cross-entity features comparable, and using the residual between predicted contact trajectories and actual tactile sensations for closed-loop correction, enables the migration and deployment of the same model across different robot platforms. In contact-intensive tasks, it adaptively adjusts operating parameters based on real-time tactile feedback.
[0086] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in this application.
[0087] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description
[0088] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:
[0089] Figure 1 This is a flowchart of an end-to-end cross-ontology generalized visual language action processing method based on VLA fused haptic feedback in an embodiment of the present invention.
[0090] Figure 2 This is a flowchart of the cross-ontology joint training process in an embodiment of the present invention. Detailed Implementation
[0091] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.
[0092] refer to Figure 1 and Figure 2 This invention provides an embodiment of an end-to-end cross-ontology generalized visual language action processing method based on VLA-fused haptic feedback, comprising:
[0093] Acquire language commands, visual observation data, robot body state data, tactile data, and target robot body descriptor;
[0094] Multi-rate time synchronization is performed on visual observation data, robot body state data, tactile data and language commands to obtain time-synchronized multimodal data;
[0095] The time-synchronized tactile data is converted into standard contact tokens in the tool coordinate system. The standard contact tokens include contact position, contact normal, normal force, tangential force, contact torque, slip probability, contact stiffness, vibration characteristics, temperature, confidence level, sensor type encoding, and effective field mask. Different robots only share the standard contact tokens and do not share the image size, array resolution, or units of the raw tactile data.
[0096] The visual language action model is used to encode language commands, visual observation data, robot body state data and standard contact tokens. The encoding results are grouped into displacement group, force group and tactile group according to physical dimensions. Within each group, the physical units are normalized to make the same characteristics of different robot bodies comparable under the same dimensions. Cross-group information exchange is carried out between groups to generate a unified state representation.
[0097] Generate a general action block and predict the contact trajectory based on the unified state representation. The general action block includes the end pose change, gripping coordination, target contact force, desired stiffness, desired damping, and contact mode.
[0098] The general motion block is converted into control instructions for the target robot based on the target robot body descriptor and then executed. The body descriptor includes joint topology, degrees of freedom, joint limits, kinematic parameters, end effector type, control mode, maximum speed, maximum torque, control cycle, and fault status.
[0099] Obtain the actual contact token after executing the control command, generate residual actions based on the contact error between the actual contact token and the predicted contact token at the corresponding moment in the predicted contact trajectory, and use the residual actions to correct the general action blocks that have not yet been executed.
[0100] After applying safety constraints to the modified general action block, continue controlling the target robot until the task is completed or a safety stop is triggered.
[0101] In this embodiment, the language instructions are task requirements described in natural language, such as "grab the connector and insert it into the socket, stop applying force after inserting it all the way in." The visual observation data are RGB images or RGBD image sequences captured by a wrist camera or an external camera. The robot body state data includes joint position, joint velocity, end effector pose, and current control mode. The tactile data are raw outputs from one or more of the following: array pressure sensor, visual-tactile sensor, six-dimensional force or torque sensor, joint torque, or motor current. The target robot body descriptor is a set of data describing the target robot's hardware structure and control capabilities. Multi-rate time synchronization is used to solve the frequency inconsistency problem caused by the update frequency of visual observation data (10 to 30 Hz), robot body state data (50 to 200 Hz), and tactile data (200 to 1000 Hz). A sliding time window is constructed centered on the target control moment to align multimodal data of different frequencies to the same time reference, obtaining time-synchronized multimodal data.
[0102] In this embodiment, the standard contact token is a structured vector that uniformly converts heterogeneous tactile data into a tool coordinate system, and it is represented by the following formula:
[0103]
[0104] In the formula, For standard contact tokens, there is a structured vector containing components with different physical dimensions, each component being measured independently according to its own physical dimension; Point of contact, in meters; The contact normal is a unit vector and is dimensionless. Normal force, measured in Newtons; This is the tangential force, measured in Newtons. The contact torque is expressed in Newton-meters. The slip probability has a value ranging from 0 to 1 and is dimensionless. Contact stiffness, measured in Newtons per meter; Vibration characteristics, measured in meters per second; Temperature, in degrees Celsius; The confidence level is a dimensionless value ranging from 0 to 1. The sensor type is encoded; it is an integer and dimensionless. The effective field mask is a binary vector, dimensionless. The tool coordinate system is established with the center point of the robot's end effector tool as the origin. After the tactile data of different robots are projected onto the tool coordinate system, their dimensions and coordinate references are consistent, making the standard contact token comparable and shareable among different robots. Different robots only share the standard contact token, not the image size, array resolution, or units of the raw tactile data. This allows the visual language action model to be independent of specific tactile sensor types, achieving sensor-independent cross-body generalization.
[0105] In this embodiment, the visual language action model is a multimodal deep neural network based on the Transformer architecture. It is constructed as follows: a pre-trained visual Transformer is used as the visual encoder, a pre-trained language Transformer is used as the language encoder, a multilayer perceptron is used as the ontology state encoder, and a one-dimensional temporal convolutional network and a temporal attention network are connected in series as the tactile encoder. The outputs of the four encoders are input to the action generation head after cross-modal interaction through a shared multimodal Transformer backbone. The inputs of the visual language action model are the text token sequence of language instructions, the image token sequence of visual observation data, the numerical vector of robot ontology state data, and the standard contact token sequence. The outputs are a unified state representation, a general action block, and a predicted contact trajectory. The encoded results are grouped into displacement, force, and tactile groups according to physical dimensions. The displacement group includes features in length dimensions such as spatial relationship features and end-effector pose features. The force group includes features in force or torque dimensions such as joint torque features, normal force, tangential force, and contact torque. The tactile group includes features in tactile-specific dimensions such as contact position, slip probability, contact stiffness, vibration features, temperature, confidence level, sensor type encoding, and effective field mask features. Within each group, features are normalized by physical units to make similar features of different robot bodies comparable under a unified dimension. For example, displacement group features are uniformly normalized to meters, and force group features are uniformly normalized to Newtons. Cross-group information exchange generates a unified state representation, which is a latent vector sequence that integrates visual, linguistic, body state, and tactile multimodal information.
[0106] In this embodiment, the general action block is a latent action variable decoupled from the specific robot structure. It includes end-effector pose change, grasping coordination, target contact force, desired stiffness, desired damping, and contact mode. The end-effector pose change is measured in meters and radians; the grasping coordination is a dimensionless scalar ranging from 0 to 1; the target contact force is measured in Newtons; the desired stiffness is measured in Newtons per meter; the desired damping is measured in Newton-seconds per meter; and the contact mode is a discrete enumeration, including free space, pre-contact, stable contact, surface sliding, insertion, twisting, release, and abnormal recovery. The predicted contact trajectory is the future predicted by the visual-language action model. The timing sequence of the standard contact token within each control step. The predicted time domain length is a positive integer. The ontology descriptor is a data set describing the target robot's hardware structure and control capabilities. It includes joint topology, degrees of freedom, joint limits, kinematic parameters, end effector type, control mode, maximum speed, maximum torque, control cycle, and fault status. The joint topology is represented by a graph structure of joints and links. Degrees of freedom are integers, joint limits are in radians, kinematic parameters are link lengths and joint axis directions in meters and radians, end effector types are enumerated as two-finger grippers, dexterous hands, or suction cups, control modes are enumerated as position control, torque control, or impedance control, maximum speed is in radians per second, maximum torque is in Newton-meters, control cycle is in seconds, and fault status is a binary flag. Based on the target robot ontology descriptor, a general motion block is converted into control instructions for the target robot and executed, allowing the same general motion block to be adapted to robots with different joint degrees of freedom and end effector structures.
[0107] In this embodiment, the actual contact token after executing the control command is obtained. A residual action is generated based on the contact error between the actual contact token and the predicted contact token at the corresponding moment in the predicted contact trajectory. The contact error includes normal force error, tangential force error, slip probability error, and contact position error. The residual action is a correction amount for the unexecuted general action block. After correcting the unexecuted general action block using the residual action, safety constraints are applied to the corrected general action block. Safety constraints include force constraints, velocity constraints, acceleration constraints, joint limit constraints, collision distance constraints, and confidence constraints. The target robot continues to be controlled until the task is completed or a safety stop is triggered. This embodiment achieves a unified representation of heterogeneous tactile feedback through standard contact tokens, achieves cross-body feature comparability through physical dimension grouping and normalization, and achieves real-time closed-loop correction through residual actions. This allows the same visual language action model to be migrated and deployed across different robot platforms and adaptively adjusts operating parameters based on real-time tactile feedback in contact-intensive tasks.
[0108] In this embodiment, the task instruction is "grab the connector and insert it into the socket, stop applying force after inserting it all the way in". The system first identifies the position and orientation of the connector and socket through a wrist camera, converts the array pressure sensor data into the contact position, normal force and shear slip probability of the left and right fingers, and converts the six-dimensional force / torque sensor data into the insertion force and eccentric torque in the tool coordinate system. The general motion block generated by the model includes the connector end pose increment, gripper cooperative closure amount, target gripping force, insertion direction stiffness and contact mode. The motion adapter maps the end pose increment to the joint velocity of the seven-degree-of-freedom robotic arm. When the connector and socket make initial contact, the haptic event module generates a contact establishment token, and the model switches the contact mode from free space to insertion mode and reduces the lateral stiffness. If the actual lateral torque is higher than the predicted value, the residual motion head generates a small spiral or cross search motion. If the insertion force continues to increase and the insertion displacement does not increase, it is determined to be stuck and retracted by 1 to 3 mm to realign. When the insertion force decreases and reaches the target depth, it stops advancing and releases the force.
[0109] In another embodiment of the invention, converting haptic data into a standard contact token in the tool coordinate system includes:
[0110] When the tactile data is array pressure sensor data, the contact position is calculated using the array unit coordinates and the pressure weighting center, and the contact normal is estimated based on the pressure gradient and the sensor surface normal.
[0111] When the tactile data is from a visual tactile sensor, the contact shape and shear force are estimated by the displacement field of the marked point or optical flow.
[0112] When the tactile data is from a six-dimensional force or torque sensor, the resultant force and resultant torque in the tool coordinate system are obtained through coordinate transformation.
[0113] When the tactile data is joint torque or motor current data, the external force is estimated by subtracting the gravity, inertia and friction terms from the dynamic model and then adding the corresponding confidence value.
[0114] When a robot has multiple tactile sensors, multiple contact tokens are projected onto a unified coordinate system using sensor extrinsic parameters, and a set of contact tokens is generated by confidence weighting or non-maximum suppression.
[0115] In this embodiment, the array pressure sensor data consists of the outputs of multiple pressure units distributed in a two-dimensional array. The coordinates of each pressure unit are known, and the pressure unit is Pascal. The contact position is calculated by weighting the coordinates of each pressure unit according to the pressure value to find the centroid. The resulting pressure-weighted center is the contact position. The contact normal is estimated by calculating the pressure gradient direction and correcting it using the sensor surface normal as a reference; the resulting unit vector is the contact normal. The visual-tactile sensor data is recorded as an image of the displacement field of marker points on the elastic surface. The contact shape and shear force are estimated by inverting the deformation distribution of the contact area through dense optical flow or the displacement field of the marker points, and the tangential force is calculated from the tangential component of the deformation distribution. Calculating normal force from normal components The data from the six-dimensional force or torque sensor consists of three-axis forces and three-axis torques in the sensor's original coordinate system. These are then rotated to the tool coordinate system through coordinate transformation to obtain the resultant force in the tool coordinate system. sum moment The resultant force is projected along the contact normal to obtain the normal force. Tangential force is obtained by projecting along the tangential direction. .
[0116] In this embodiment, the joint torque or motor current data are the torque or current measurements at the robot joint level, in Newton-meters and amperes, respectively. The external force is estimated by calculating the torque contributions of gravity, inertia, and friction terms through the robot dynamics model and subtracting them from the joint torque. The remaining part is the joint torque generated by the external force. Then, the external force and external torque are obtained by transposing the Jacobian matrix to the end-effector coordinate system. Since there is an accumulation of modeling errors after the joint torque or motor current data is subtracted from the dynamics model, its confidence level is... The confidence level is lower than that of directly measured six-dimensional force or torque sensor data, therefore a correspondingly lower confidence value is added. For example, the confidence level of array pressure sensors and six-dimensional force or torque sensors is 0.9, and the confidence level of joint torque or motor current data is 0.6. When multiple tactile sensors exist on the same robot, multiple contact tokens are projected onto a unified coordinate system using sensor extrinsic parameters. The sensor extrinsic parameters are the homogeneous transformation matrix from the sensor coordinate system to the tool coordinate system. After projection, the contact positions and contact normals of multiple contact tokens are all in the tool coordinate system. A set of contact tokens is generated by confidence weighting or non-maximum suppression. Confidence weighting is a weighted average of each field with the confidence level of each contact token as the weight. Non-maximum suppression is to retain the contact token with the highest confidence level among multiple contact tokens that are spatially close and suppress the others. The resulting set of contact tokens is a set of several non-overlapping standard contact tokens in the tool coordinate system at the same time.
[0117] In another embodiment of the present invention, multi-rate time synchronization includes:
[0118] Establish a unified time reference and record the acquisition timestamp and arrival timestamp for visual frames, robot states, tactile sampling, and control commands respectively; when devices do not share a hardware clock, estimate clock offset and drift through periodic synchronization messages, round-trip delay estimation, or correlation analysis of common motion events.
[0119] Construct a sliding time window centered on the target control time;
[0120] Time alignment of visual observation data is performed using nearest neighbor or feature interpolation methods.
[0121] Time alignment is performed on the robot's body state data using linear or spline interpolation.
[0122] The maximum normal force, tangential force integral, impact peak, slip spectrum, contact establishment time, contact release time and contact duration are extracted from the tactile data within the time window. The variable number of tactile samples are compressed into a fixed number of tactile event tokens. The tactile event tokens are then processed by the tactile encoder and converted into standard contact tokens.
[0123] When an impact, slippage, or force overshoot trend is detected, fast residual inference is triggered without waiting for the next visual frame.
[0124] In this embodiment, the unified time reference is a global time axis, measured in seconds, that is commonly referenced by all sensors and control commands. The acquisition timestamp is the local clock reading at the time the sensor samples, and the arrival timestamp is the local clock reading at the time the data arrives at the processing unit. The difference between the two can be used to estimate transmission delay. When the vision sensor, robot controller, tactile sensor, and control command processing unit do not share a hardware clock, there is clock skew and clock drift between the local clocks of each device. Clock skew is the difference in readings between the clocks of two devices at the same moment, measured in seconds, while clock drift is the rate at which the difference in clock readings between the two devices changes over time, measured in seconds per second. Periodic synchronization messages are messages with transmission timestamps that are periodically exchanged between devices. Clock skew and round-trip delay are estimated by comparing the transmission timestamp and the reception timestamp. Round-trip delay is estimated by measuring half of the round-trip time of the message as an approximation of the one-way transmission delay. Correlation analysis of common motion events involves recording signal sequences of the same physical motion event on different devices. The time difference between the clocks of the two devices is estimated by calculating the peak position of the cross-correlation function of the two signal sequences, thereby estimating the clock skew.
[0125] In this embodiment, the sliding time window is based on the target control time. Centered on, with a width of 2 Time interval [ - , ], The window width is half-width, and the unit is seconds, for example, 0.05 seconds. For visual observation data, time alignment is performed using nearest neighbor or feature interpolation. Nearest neighbor is selected from the visual frames whose timestamps are closest to the target control time. Feature interpolation is the time-weighted average of depth features extracted from adjacent visual frames. For robot body state data, time alignment is performed using linear or spline interpolation. Linear interpolation is linear interpolation between two adjacent robot state sampling points according to the time ratio. Spline interpolation is the value obtained at the target control time after fitting the robot state sequence with a cubic spline function. Within a time window, the following parameters are extracted from tactile data: maximum normal force, tangential force integral, impact peak, slip spectrum, contact establishment time, contact release time, and contact duration. The maximum normal force is the maximum value of the normal force within the window, measured in Newtons (N). The tangential force integral is the integral of the tangential force with respect to time within the window, measured in Newton-seconds (N / s). The impact peak is the maximum value of the time derivative of the normal force within the window, measured in Newton-seconds (N / s). The slip spectrum is the spectral characteristics of the slip probability sequence within the window after Fast Fourier Transform (FFT). The contact establishment time is the moment when the normal force within the window first exceeds the contact threshold, measured in seconds. The contact release time is the moment when the normal force within the window first drops below the contact threshold, measured in seconds. The contact duration is the difference between the contact release time and the contact establishment time, measured in seconds.
[0126] In this embodiment, a variable number of tactile samples are compressed into a fixed number of tactile event tokens, which are expressed by the following formula:
[0127]
[0128] In the formula, For the first The haptic event tokens at each control moment are dimensionless structured vectors; This is a tactile event encoding function, constructed by concatenating a one-dimensional temporal convolutional network and a temporal attention network. The input consists of the raw tactile data sequence within the time window, the sensor type, and the effective field mask. and signal quality The output is a fixed-dimensional haptic event token; For a moment The raw tactile data, the units of which vary depending on the type of sensor; For time variables, the unit is seconds; For sliding time windows, The target control time is in seconds. The width is half the window width, and the unit is seconds. The sensor type and valid field mask is a binary vector, dimensionless. For signal quality, the value ranges from 0 to 1 and is dimensionless. The tactile event encoding function compresses a variable number of raw tactile data samples within a time window into a single fixed-dimensional tactile event token. That is, it outputs one tactile event token at each control moment, ensuring that tactile sensors with different sampling rates produce the same number of tactile event tokens under a unified time reference, thereby eliminating the impact of tactile sampling rate differences on subsequent model inference. Contact events have higher priority than ordinary non-contact sampling. When an impact, slippage, or force exceeding the limit is detected, fast residual inference is triggered without waiting for the next visual frame. Fast residual inference only calls the residual action head to correct the current general action block, without re-executing the complete visual language action model inference, so that high-frequency tactile events can trigger action correction immediately outside the low-frequency inference cycle of the visual language action model.
[0129] In another embodiment of the present invention, a visual language action model is used to encode language commands, visual observation data, robot body state data, and standard contact tokens. The encoding results are grouped into displacement, force, and tactile groups according to physical dimensions, including:
[0130] Output task tokens and constraint tokens via the language encoder;
[0131] The visual encoder outputs scene features, target object features, tool features, and spatial relationship features.
[0132] The body state encoder outputs joint position, joint velocity, end effector pose, and current control mode characteristics.
[0133] The standard contact token and its timing characteristics are output through a tactile encoder.
[0134] The spatial relationship features output by the visual encoder and the end pose features output by the body state encoder are grouped into the displacement group.
[0135] The joint torque characteristics output by the body state encoder and the normal force, tangential force, and contact torque characteristics output by the tactile encoder are grouped into the force group;
[0136] The contact position, slip probability, contact stiffness, vibration characteristics, temperature, confidence level, sensor type encoding, and effective field mask features output by the tactile encoder are categorized into the tactile group;
[0137] Inter-group and cross-group information interaction is achieved through a dimensional perception attention mechanism. The dimensional perception attention mechanism applies cross-group constraints based on the physical dimensional relationship: the force group and the displacement group are coupled through stiffness matrix constraints, the time difference of the displacement group and the force group are coupled through dynamic relationship constraints, and the tactile group and the force group are coupled through contact mechanics relationship constraints, thereby obtaining cross-group interaction characteristics.
[0138] The tactile injection gating coefficients are calculated based on cross-group interaction features, linguistic features, visual features, ontological features, and tactile confidence.
[0139] The gating coefficient is used to control the cross-modal attention fusion intensity of tactile features with displacement, force and language features in cross-group interaction features;
[0140] In the approaching target stage, the gating coefficient is reduced to rely mainly on visual features. In the clamping of fragile objects stage, the gating weight corresponding to the normal force feature is increased. In the screwing bottle cap stage, the gating weight corresponding to the tangential force feature, sliding feature, and contact torque feature is increased. After gating fusion, a unified state representation is generated.
[0141] In this embodiment, the language encoder is a text encoding network built with a pre-trained language Transformer. Its input is a sequence of text tokens representing language instructions, and its output is a task token and a constraint token. The task token is a latent vector encoding task objectives such as "what to grasp" and "where to place," while the constraint token is a latent vector encoding operational constraints such as "handle gently" and "maintain tension." Both are of dimension 1. floating-point vectors is the dimension of the hidden layer of the model, and is a positive integer. The visual encoder is an image encoding network built with a pre-trained visual Transformer. Its input is a sequence of image tokens from visual observation data, and its output is scene features, object features, tool features, and spatial relationship features. Scene features are latent vectors encoding the global scene layout; object features are latent vectors encoding the pose and shape of the object; tool features are latent vectors encoding the pose of the end effector; and spatial relationship features are latent vectors encoding the relative poses of the object and the tool. All four types of features have a dimension of . The body state encoder is a numerical encoding network constructed using a multilayer perceptron. Its input is a numerical vector concatenated from joint position, joint velocity, end-effector pose, and current control mode. The output is a floating-point vector of dimension [missing information]. The ontological state feature vector. The haptic encoder is a sequence encoding network constructed by concatenating a one-dimensional temporal convolutional network and a temporal attention network. Its input is a sequence of standard contact tokens, and its output is the standard contact tokens and their temporal features. The temporal features are latent vectors that encode the temporal relationship of contact events.
[0142] In this embodiment, the spatial relationship features output by the visual encoder and the end-effector pose features output by the body state encoder are grouped into the displacement group. The features in the displacement group are all in the dimension of length or angle, and are normalized to meters or radians according to physical units. The joint torque features output by the body state encoder and the normal force, tangential force, and contact torque features output by the tactile encoder are grouped into the force group. The features in the force group are all in the dimension of force or torque, and are normalized to Newtons or Newton-meters according to physical units. The contact position, slip probability, contact stiffness, vibration features, temperature, confidence level, sensor type encoding, and effective field mask features output by the tactile encoder are grouped into the tactile group. The features in the tactile group are in tactile-specific dimensions and are normalized to their respective physical units. Within each group, normalization by physical units makes similar characteristics of different robot bodies comparable under a unified dimension. For example, after the end-effector pose characteristics of different robots are normalized to meters and radians, their displacement group characteristics can be directly compared. After the normal force characteristics of different robots are normalized to Newtons, their force group characteristics can be directly compared, thereby eliminating the influence of differences in sensor range and units on feature comparison.
[0143] In this embodiment, the dimensional awareness attention mechanism is a multi-head attention network based on a physical dimensional mask. Its construction method is as follows: a physical dimensional mask is superimposed on the attention weight matrix of a standard multi-head attention network. The physical dimensional mask is a binary or soft-weight matrix constructed based on the physical dimensional relationships between displacement, force, and tactile groups. Its construction rules are as follows: for pairs with physical coupling relationships (displacement group and force group, force group and tactile group, displacement group and tactile group), the corresponding positions of the mask are assigned a soft weight of 1 or a positive value to allow information interaction; for pairs without physical coupling relationships, the corresponding positions of the mask are assigned a soft weight of 0 or close to 0 to suppress information interaction without physical meaning. The mask is applied to the attention weight matrix of the standard multi-head attention network using addition or multiplication; that is, the attention weights are added to the mask matrix or multiplied element-wise with the mask matrix before softmax normalization, thereby constraining the range of values for the attention weights between groups. The input of the dimensional awareness attention mechanism is the normalized feature vectors of the displacement, force, and tactile groups, and the output is the cross-group interaction features. Dimensional awareness attention mechanism applies cross-group constraints based on physical dimensional relationships: force groups and displacement groups are coupled and constrained through stiffness matrices. The unit is Newtons per meter (N / m), and its physical meaning is the proportional relationship between force and displacement. The constraint expression is that the force group characteristics are approximately equal to the stiffness matrix. Multiply by the displacement group characteristic, i.e., the force group characteristic equals Multiplying by the characteristics of the displacement group minimizes the coupling constraint loss; the time difference of the displacement group and the force group are coupled and constrained through a dynamic relationship, which is governed by Newton's second law. equal Multiply ,in For mass, the unit is kilogram. The acceleration is the second derivative of displacement with respect to time, measured in meters per second squared. The constraint expression is that the characteristic force of the force group is approximately equal to the mass. Multiplying by the second time derivative of the displacement group, i.e., the characteristic of the force group equals The coupling constraint loss is minimized when multiplying the displacement group features by the second-order difference over time. The tactile group and the force group are coupled and constrained through contact mechanics relationships, which are the relationship between normal force, contact stiffness, and contact deformation. The constraint expression is that the normal force is approximately equal to the contact stiffness multiplied by the contact deformation, where the contact stiffness is in Newtons per meter and the contact deformation is in meters. That is, the coupling constraint loss is minimized when the normal force feature in the tactile group is equal to the contact stiffness feature multiplied by the contact deformation feature. These three types of cross-group constraints are added to the training objective of the dimensional perception attention network as auxiliary loss terms. The auxiliary loss term is the sum of squared residuals of the physical relationships between the features of each group, enabling the dimensional perception attention mechanism to learn inter-group information interaction patterns that conform to physical laws during training, thus acquiring cross-group interaction features.
[0144] In this embodiment, the tactile injection gating coefficient is calculated. The gating coefficient is determined based on cross-group interaction features, language features, visual features, ontological features, and tactile confidence, and is calculated using the following formula:
[0145]
[0146] In the formula, For the first The tactile injection gating coefficient at each control moment is a dimensionless scalar ranging from 0 to 1; It is a sigmoid activation function, which is dimensionless; The gating weight matrix has the following dimensions: Multiply by 5 , The hidden layer dimension is dimensionless; the vector within square brackets is a concatenated vector of cross-group interaction features, linguistic features, visual features, ontological features, and tactile confidence, where the cross-group interaction features are the tactile features processed by the dimension-aware attention mechanism. Because the dimensional perception attention mechanism has already carried out cross-group information exchange between the tactile group, displacement group, and force group, therefore Simultaneously, it carries cross-group interaction information and serves as the input gating coefficient for cross-group interaction feature calculation; For the first The visual features at each control moment are of dimension [missing information]. A floating-point vector; For language features, for dimensions A floating-point vector; For the first The ontological features at each control moment are of dimension [dimensional value]. A floating-point vector; For the first The tactile features at each control moment are of dimension [dimensional value missing]. A floating-point vector; For the first The tactile confidence level at each control moment, with a value ranging from 0 to 1 and dimensionless; Let be the gated bias vector, with dimension . Dimensionless; the semicolon indicates that vectors are concatenated along the feature dimension. Gating coefficients It is used to adaptively control the intensity of haptic injection based on the contact phase and task semantics.
[0147] In this embodiment, a gating coefficient is used to control the cross-modal attention fusion intensity of tactile features, displacement features, force features, and language features in the cross-group interaction features. After gating fusion, a unified state representation is generated, which is expressed by the following formula:
[0148]
[0149] In the formula, For the first The unified state representation after gating and fusion at each control moment is a dimensionless representation. A floating-point vector; For the first The joint features of visual, linguistic, and ontological states at each control moment are represented by a dimension of [missing information]. The floating-point vector is obtained by converting visual features... linguistic features and ontological features After being concatenated into a feature sequence, it is input into a cross-modal self-attention layer sharing a multimodal Transformer backbone. After fusion by a multi-head self-attention network, the first token of the output sequence is taken as the joint feature. ; For the first The tactile injection gating coefficient at each control moment, with a value range of 0 to 1 and dimensionless; It is an element-wise multiplication operator, dimensionless; This is a cross-modal attention function, constructed using a multi-head cross-attention network, with the query vector as input. and key value vector The output is an attention-weighted fusion result of tactile features and visual-language ontology joint features, with dimensions of [missing information]. ; For the first The tactile features at each control moment are of dimension [dimensional value missing]. A floating-point vector. The gating coefficient is reduced during the approach phase to the target. Primarily dependent on visual features, such as Set it to 0.2; increase the gating weight corresponding to the normal force characteristic during the stage of clamping fragile objects, for example... Set it to 0.8; increase the gating weights corresponding to tangential force characteristics, slip characteristics, and contact torque characteristics during the bottle cap twisting stage, for example... Set the value to 0.7. After gating and fusion, a unified state representation is generated. The unified state representation is a sequence of latent vectors that fuses visual, linguistic, ontological, and tactile multimodal information, which is used by the subsequent action generation head to generate general action blocks and predict contact trajectories.
[0150] In another embodiment of the present invention, a general action block and a predicted contact trajectory are generated, and a residual action correction general action block is generated based on the contact error between the actual contact token and the predicted contact trajectory, including:
[0151] The motion generation head outputs a nominal motion block with a length of H control steps at a time. The nominal motion block is a predicted instance of the general motion block at the current control moment, and also outputs the predicted contact trajectory corresponding to the nominal motion block.
[0152] The general action block is organized into a pose layer, force control layer and contact layer according to physical dimensions. Each layer is independently normalized and the layers are coupled through physical relationships.
[0153] The pose layer includes end-effector translation increments, rotation increments, or six-dimensional velocity spinors in the tool coordinate system;
[0154] The force control layer includes the target normal force, the target tangential force range, the target torque, the desired end stiffness, the desired damping, and the compliance direction;
[0155] The contact layer includes contact patterns, normalized grasping coordination, action duration, action confidence, and predicted contact states;
[0156] After executing the first segment of the nominal action block, the actual contact token is obtained. The contact error between the actual contact token and the predicted contact token at the corresponding moment in the predicted contact trajectory is calculated. The contact error includes normal force error, tangential force error, slip probability error, contact position error, and contact torque error.
[0157] The residual action head generates residual actions based on contact error, current semantic features, action stage, and ontology descriptor;
[0158] The residual actions are used to correct the general action blocks that have not yet been executed. The correction method is to superimpose the residual actions with the unexecuted parts of the nominal action blocks and adjust the injection intensity of the residual actions through an error gating function.
[0159] In response to contact errors caused by indicated slippage, residual actions are generated to increase the amount of gripping coordination, change the contact direction, or reduce the tangential velocity.
[0160] In response to contact errors indicating excessive force, residual actions are generated to reduce normal displacement, reduce stiffness, or perform retraction.
[0161] In response to a contact error indicating a jammed connection, a residual action is generated to perform a search motion;
[0162] In response to a contact error indicating a loss of contact, a residual action is generated to re-approach or reposition the control end.
[0163] In response to contact errors caused by an over-limit of the indicated torque, a residual action is generated to reduce the rotation increment or change the rotation direction.
[0164] In this embodiment, the action generation head is an action generation network constructed using a diffusion model. Its construction method is as follows: using a unified state representation as conditional input, it generates action sequences through a denoising diffusion process. The denoising diffusion process includes two stages: forward denoising and backward denoising. In the forward denoising stage, Gaussian noise is gradually added to the real action sequence according to a preset noise scheduling table. In the backward denoising stage, using the unified state representation as conditional input, a noise prediction network is used to gradually predict and subtract noise. For example, the number of denoising steps is 50, and the noise scheduling table uses a cosine scheduling method. The input to the action generation head is the unified state representation, the ontology descriptor vector, and the task instruction token; the output is a sequence of length [missing information]. The nominal action block for each control step and the predicted contact trajectory corresponding to the nominal action block. For the predicted time domain length, is a positive integer, for example Let's take 16. The nominal action block is the action block predicted by the motion generator at the current control moment, before residual correction. The general action block is a representation of the latent motion variables, decoupled from the specific robot structure and hierarchically organized by physical dimensions into a pose layer, force control layer, and contact layer. The nominal action block is the predicted instance of the general action block at the current control moment. Both exist in the same motion space and have the same hierarchical structure of physical dimensions. The nominal action block becomes the final action block after residual correction. The general action block is hierarchically organized by physical dimensions into a pose layer, force control layer, and contact layer. The pose layer includes the end-effector translation increment, rotation increment, or six-dimensional velocity screw in the tool coordinate system. The unit of the end-effector translation increment is meters, the unit of the rotation increment is radians, and the unit of the six-dimensional velocity screw is meters per second and radians per second. The force control layer includes the target normal force, target tangential force range, target torque, desired end-effector stiffness, damping, and compliance direction. The unit of the target normal force and target tangential force range is Newtons, the unit of the target torque is Newton-meters, and the unit of the desired end-effector stiffness is... The unit for end effector stiffness is Newtons per meter (N / m), and the unit for damping is Newton-seconds per meter (N / m). The compliance direction is the direction in which the end effector is allowed to produce displacement and is a unit vector. The contact layer includes contact mode, normalized grasping coordination quantity, motion duration, motion confidence, and predicted contact state. The contact mode is a discrete enumeration quantity. The normalized grasping coordination quantity ranges from 0 to 1 and is dimensionless. The unit for motion duration is seconds. The motion confidence quantity ranges from 0 to 1 and is dimensionless. The predicted contact state is a predicted standard contact token for future contact. Each layer is independently normalized, and the layers are coupled through physical relationships. For example, the target normal force of the force control layer is coupled to the end effector translation increment of the pose layer through the expected stiffness, and the normalized grasping coordination quantity of the contact layer is coupled to the target normal force of the force control layer through the grasping force mapping relationship.
[0165] In this embodiment, after executing the first segment of the nominal action block, the actual contact token is obtained, and the contact error between the actual contact token and the predicted contact token at the corresponding moment in the predicted contact trajectory is calculated, which is expressed by the following formula:
[0166]
[0167] In the formula, For the first The contact error at each control moment is a floating-point vector with the same dimension as the standard contact token, and the units of each component are consistent with the corresponding fields of the standard contact token. For the first The actual contact token at each control moment is the standard contact token obtained by converting tactile data after the first segment of the nominal action block is actually executed. For the first The predicted contact token at each control moment is the predicted value of the standard contact token at the corresponding moment in the predicted contact trajectory. The symbol ^ at the top indicates the prediction estimate. The contact error includes normal force error, tangential force error, slip probability error, and contact position error. The normal force error is the difference between the actual normal force and the predicted normal force, in Newtons. The tangential force error is the difference between the actual tangential force and the predicted tangential force, in Newtons. The slip probability error is the difference between the actual slip probability and the predicted slip probability, dimensionless. The contact position error is the difference between the actual contact position and the predicted contact position, in meters.
[0168] In this embodiment, the residual action head is a residual generation network built with a lightweight multilayer perceptron. Its inputs are contact error, current semantic features, action stage, and ontology descriptor, and its output is a residual action, which is a correction vector of the same dimension as the general action block. The residual action is used to correct general action blocks that have not yet been executed, expressed by the following formula:
[0169]
[0170] In the formula, For from the first Each control moment starts and lasts for a period of time. The action sequence of the control step, i.e., covering the first control step. to The final action block of each control step is output after residual correction and safety constraints, and the unit of each component is consistent with the corresponding field of the general action block; As a safety filter, it is constructed using a control barrier function or model predictive control. The input is a modified action block, and the output is an action block that satisfies the safety constraints. For the nominal action block to For the portion of each control step that has not yet been executed, the unit of each component is consistent with the corresponding field of the general action block; This is the error gating function, constructed using a sigmoid gating network, with the contact error as the input. and tactile confidence The output is a gated scalar with a value range of 0 to 1, and is dimensionless. This is the contact error vector, and the units of each component are consistent with the corresponding fields of the standard contact token; The tactile confidence level ranges from 0 to 1 and is dimensionless. It is an element-wise multiplication operator, dimensionless; The subscript is , The residual action generation function is constructed using a multilayer perceptron and has a learnable parameter set of... The input is the contact error. Ontology descriptor and unified state representation The output is the residual action; For ontology descriptor, To unify the state representation, both are floating-point vectors. When the contact error indicates slippage, the residual action increases the grasping coordination, changes the contact direction, or reduces the tangential velocity; when the contact error indicates overforce, the residual action reduces the normal displacement, reduces stiffness, or performs a pullback; when the contact error indicates insertion jamming, the residual action performs a search motion, which is a probing motion of the end effector in a spiral or cross trajectory within a small range; when the contact error indicates loss of contact, the residual action controls the end effector to re-approach or reposition itself.
[0171] When the contact error indicating torque exceeds the limit, the residual action reduces the rotation increment or changes the rotation direction.
[0172] As a further optimized implementation of the aforementioned embodiments, the core of this embodiment lies in the fact that existing technologies typically rely on a single physical quantity (such as force error or force change rate) to detect contact anomalies, while this embodiment extracts early warning signals from three independent physical dimensions of contact stability, namely spatial stability, temporal response stability, and mechanical property stability, and constructs a multi-level early warning and graded correction mechanism.
[0173] Step S01: Extract real-time values of three independent physical quantities in parallel:
[0174] The system executes the following three sub-steps in parallel at each control moment:
[0175] First sub-step: Calculate the instantaneous position of the center of contact pressure:
[0176] For robots equipped with array-type tactile sensors (such as array pressure sensors or visual-tactile sensors), the system calculates the weighted center of pressure at all contact points from the contact surface pressure distribution output by the sensors as the contact pressure center. The instantaneous position of this pressure center reflects the centroid of the contact point distribution on the contact surface.
[0177] The location of the contact pressure center is calculated as follows: the position coordinates of all contact points are weighted and summed according to the pressure value at each point, and then divided by the sum of the pressure values of all contact points. For visual-tactile sensors, the contact pressure center is calculated using the contact force distribution obtained from the displacement field of the marked points or optical flow inversion.
[0178] Second sub-step: Calculate the phase lag between force and displacement:
[0179] The system records the moment of change in the end effector displacement and the moment of response of the contact force at each control step, and calculates the time difference between the two as the phase lag. Specifically, when a step change in the displacement command or a sudden change in the velocity command is detected, the moment of displacement change is recorded; the change in contact force is continuously monitored, and when the contact force starts to respond from the current value and exceeds the preset response threshold, the moment of force response is recorded; the difference between the two is the phase lag.
[0180] The phase lag is calculated as follows: the moment of change of the end effector displacement is recorded as the displacement change moment; the moment of response of the contact force is recorded as the contact force response moment; the phase lag is equal to the contact force response moment minus the displacement change moment. For the tangential direction, the difference between the tangential displacement change moment and the tangential force response moment is used.
[0181] Under stable contact conditions, the force response to displacement is essentially synchronous, and the phase hysteresis remains within a small, normal range. A sudden increase in phase hysteresis indicates that slippage is occurring at the contact surface or that the characteristics of the contact interface are changing.
[0182] Third sub-step: Calculate the contact stiffness drift ratio:
[0183] The system calculates the current contact stiffness value in real time. Contact stiffness is defined as the ratio of the change in normal force to the change in displacement of the end in the normal direction of contact. Specifically, during the stable contact phase, the increment of normal force between two adjacent control moments is selected and divided by the corresponding increment of normal displacement of the end to obtain the instantaneous contact stiffness value at that moment.
[0184] The system performs time difference analysis on the instantaneous contact stiffness value, that is, subtracts the contact stiffness value of the previous moment from the current contact stiffness value and divides it by the sampling period to obtain the contact stiffness drift rate. The stiffness drift rate reflects the rate and direction of change of contact stiffness over time. A positive value indicates that the contact is softening, i.e., the stiffness is decreasing, while a negative value indicates that the contact is hardening, i.e., the stiffness is increasing.
[0185] Under stable contact conditions, the contact stiffness remains relatively stable, and the stiffness drift rate is close to zero. When slippage begins, the contact stiffness decreases, and the drift rate is positive; when jamming or excessive force is about to occur, the contact stiffness increases, and the drift rate is negative.
[0186] Step S02: Parallel monitoring of three signals and multi-level early warning determination:
[0187] The system compares the three physical quantities obtained in step S01 with their respective preset thresholds:
[0188] The first-level warning signal, namely the pressure center drift rate, is calculated by dividing the positional offset of the pressure center between two consecutive control moments using a standardized sampling period. When the drift rate exceeds a preset first threshold, the system determines that the contact point is undergoing significant movement on the contact surface, triggering the first-level warning signal.
[0189] The reason why the pressure center drift rate is the first-level warning signal is that, in the evolutionary chain of loss of contact stability, the drift of the pressure center is the earliest detectable signal. When an object begins to slightly slide between fingers, the position of the pressure center changes immediately, while the force and stiffness signals have not yet changed significantly.
[0190] The second-level warning signal, namely the phase lag: The system compares the currently calculated phase lag between force and displacement with a preset normal range. When the phase lag exceeds the upper limit of the preset normal range, the system determines that the force response to displacement has a significant delay, triggering the second-level warning signal.
[0191] The reason why phase lag is the second-level warning signal is that when slippage develops from micro-slippage to macro-slippage, the frictional characteristics between the contact surfaces change from static friction to kinetic friction, and the force-displacement response relationship undergoes a fundamental change, manifested as a significant increase in phase lag. At this time, the drift rate of the pressure center has exceeded the threshold for some time, but the magnitude of the force may not have decreased significantly yet.
[0192] The third-level warning signal, namely the stiffness drift rate: The system compares the currently calculated contact stiffness drift rate with preset positive and negative thresholds respectively. When the stiffness drift rate exceeds the positive threshold, the system determines that the contact is softening, i.e., the stiffness is decreasing, triggering the softening warning in the third-level warning signal; when the stiffness drift rate is below the negative threshold, the system determines that the contact is hardening, i.e., the stiffness is increasing, triggering the hardening warning in the third-level warning signal.
[0193] The reason why stiffness drift rate is a level three warning signal is that stiffness is a direct representation of the mechanical properties of the contact interface. When slippage has progressed to the point where the force begins to decrease, the contact stiffness will inevitably change. Stiffness change is the ultimate evidence of loss of contact stability and also the most urgent level of correction signal.
[0194] Step S03: Differentiated Correction Strategy Driven by Level 3 Early Warning:
[0195] The various correction actions described in this step are specific manifestations of the graded response behavior learned by the residual action head in the aforementioned embodiments through end-to-end training after receiving the contact error characteristics constituted by the warning signal.
[0196] Based on the level of the warning signal currently detected in step S02, the system selects the corresponding corrective action from the first-level warning, the second-level warning, and the third-level warning:
[0197] When only the first-level warning signal (pressure center drift rate exceeding the threshold) is detected, the system determines that the contact is in a state of impending slippage and executes the first-level correction action: increasing the gripping coordination of the end effector by a small amount to increase the normal pressure between the contact surfaces to suppress the slippage trend. The magnitude of the first-level correction action is minimal, aiming to eliminate the impending slippage with minimal intervention and avoid overcorrection that could lead to contact state oscillations.
[0198] When both the first-level and second-level warning signals (both pressure center drift rate and phase hysteresis exceeding the threshold) are detected simultaneously, the system determines that slippage is in progress and executes a second-level correction action: while increasing the gripping coordination, it reduces the velocity of the end effector in the tangential direction of the contact surface to suppress the acceleration trend of slippage. The magnitude of the second-level correction action is moderate. At this point, slippage has already clearly occurred, requiring intervention from both directions of increasing normal pressure and decreasing tangential force.
[0199] When a softening warning (positive stiffness drift rate exceeding the threshold) is detected in the third-level warning signal, the system determines that a fundamental change in the mechanical properties of the contact state is occurring, i.e., the contact is softening, the contact force is about to become insufficient, and the mission faces the risk of failure. It then executes a third-level corrective action, controlling the end effector to retract to a safe distance and then approaching the contact surface again at a reduced end speed to re-establish a stable contact state. The third-level corrective action has the largest magnitude, involving a complete re-approach process rather than just parameter adjustments.
[0200] When the hardening warning (negative stiffness drift rate exceeding the threshold) in the third-level warning signal is detected, the system determines that the contact state is undergoing the opposite change, i.e., the contact is hardening. This indicates that the end effector is excessively pressing into the contact surface or is jamming, which will generate contact force exceeding the safe range. The fourth-level correction action is then executed, which controls the displacement of the end effector in the contact normal direction to reduce, while simultaneously reducing the desired stiffness value, in order to reduce the contact force and restore the compliance of the contact interface.
[0201] Step S04: Determining the priority of early warning signals and resolving conflicts:
[0202] The system assigns clear priorities to the three warning signals: the third-level warning signal has the highest priority, followed by the second-level, and the first-level has the lowest priority. This is because: stiffness drift rate directly reflects changes in the mechanical properties of the contact interface and is the most direct evidence of lost contact stability; phase hysteresis reflects a break in the coupling relationship between kinematics and dynamics and is a mid-term signal that contact stability is deteriorating; and pressure center drift rate reflects the spatial movement of the contact point and is an early sign of changes in contact stability.
[0203] When the system detects multiple warning signals at the same time, the corresponding corrective action is triggered according to the warning signal with the highest priority. The specific rules are as follows:
[0204] When any of the Level 3 warning signals (softening or hardening) is detected simultaneously with any other warning signal of any level, the system ignores the other warning signals and only executes Level 3 or Level 4 correction actions. When no Level 3 warning signal is detected but both Level 2 and Level 1 warning signals are detected simultaneously, the system executes Level 2 correction actions. When only a Level 1 warning signal is detected and no higher-level warning signal is detected, the system executes Level 1 correction actions.
[0205] The physical basis of this priority rule is that the higher the level of the warning signal, the more severe the loss of contact stability, and the greater the intervention required. If stiffness drift, i.e., a change in contact mechanical properties, has been detected, then the drift of the pressure center or the delay of phase lag is no longer important, and the most thorough correction must be performed.
[0206] Step S05: Closed-loop verification of the graded correction effect:
[0207] After performing any level of correction, the system continuously monitors the three physical quantities in step S01 to determine whether the contact state has returned to normal.
[0208] When all three physical quantities fall back to their respective normal threshold ranges, the system determines that the contact state has stabilized, stops the current correction action, and resumes normal operation mode.
[0209] If any of the three physical quantities still exceeds the threshold range, the system determines that the contact state has not been restored and maintains the current level of correction action or upgrades to a higher level of correction action.
[0210] When the three physical quantities continue to deteriorate (pressure center drift rate continues to increase, phase lag continues to increase, and stiffness drift rate continues to move away from zero), the system determines that the current level of correction action is invalid and upgrades to a higher level of correction action.
[0211] In another embodiment of the present invention, the ontology descriptor includes a morphology graph structure, which has joints, links, end effectors and contacts as nodes, and kinematic connections, mechanical coupling and symmetry relationships as edges.
[0212] Based on the target robot body descriptor, convert the general action block into control instructions for the target robot, including:
[0213] Input the ontology descriptor into the morphological graph encoder to obtain the ontology vector;
[0214] Based on the general action block and the ontology vector, the action adapter outputs a learnable residual for the analytical kinematic mapping;
[0215] The end effector is mapped to joint space action using the generalized inverse or damped generalized inverse of the Jacobian matrix of the target robot.
[0216] Project the joint space motion onto the target robot's set of joint limit constraints, velocity constraints, torque constraints, and collision constraints;
[0217] When the target robot only supports the position control interface, the target contact force, desired stiffness, and desired damping in the general action block are converted into a small position offset; when the target robot supports torque control or impedance control interface, the torque control parameters or impedance control parameters are directly output.
[0218] In this embodiment, the ontology descriptor includes a morphological graph structure, which is a data structure describing the topology of robot hardware in graph data format. It uses joints, links, end effectors, and contacts as nodes, and kinematic connections, mechanical couplings, and symmetry relationships as edges. Kinematic connections represent parent-child connections between joints and links; mechanical coupling represents the linkage between two joints, such as the coordinated flexion and extension of multiple joints in a dexterous hand; and symmetry relationships represent the equivalence relationships between structurally symmetrical joints or links. The ontology descriptor is input to the morphological graph encoder, which is a graph encoding network constructed using a graph attention network. Its construction method is as follows: the node features of the morphological graph structure are used as input, including node type, link length, joint axis direction, joint limits, and current joint state. Information is propagated between nodes through a multi-layer graph attention mechanism, and the output is an ontology vector. The input of the morphological graph encoder is the node feature matrix and adjacency matrix of the morphological graph structure, and the output is the ontology vector. The ontology vector has a dimension of floating-point vectors, is the dimension of the ontology vector, and is a positive integer.
[0219] In this embodiment, based on the general action block and the body vector, a learnable residual of the analytical kinematic mapping is output through an action adapter. The action adapter is a residual prediction network constructed using a conditional multilayer perceptron. It is constructed by concatenating the general action block and the body vector as conditional input, and outputting a residual term through the multilayer perceptron. This residual term is superimposed on the analytical kinematic mapping to correct the error of the analytical model. The end effector is mapped to a joint space action using the generalized inverse or damped generalized inverse of the target robot's Jacobian matrix, expressed by the following formula:
[0220]
[0221] In the formula, For the first The joint space motion of the target robot at each control moment is a floating-point vector with a dimension equal to the degree of freedom of the target robot, and the unit is radians (position control), radians per second (velocity control), or Newton-meters (torque control). To project joint space motion onto the set of allowable joint constraints, velocity constraints, torque constraints, and collision constraints of the target robot using a constraint projection function. For constraint sets, dimensionless; For action adapter functions, the subscripts are the learnable parameter sets. , Constructed using a conditional multilayer perceptron, with a general action block as input. Ontology vector and current joint status The output is the learnable residual of the analytic kinematic mapping, which is the generalized inverse of the Jacobian matrix. With six-dimensional velocity spinor product This represents the analytical mapping from end-effector velocity to joint velocity without considering model errors; the motion adapter function. The output residuals are superimposed on the analytical kinematic mapping to compensate for the error between the analytical model and the real robot dynamics; For the first A general action block for each control moment. For the ontology vector, For the first The joint state vector at each control moment, in radians; The top right corner Let Jacobian matrix be the generalized inverse or damped generalized inverse of the robot's Jacobian matrix. The dimension is 6 times , The robot's degrees of freedom are represented by the partial derivatives of joint velocities to end-effector velocities, with units of 1 per radian. The dimension of the generalized inverse is... Multiply by 6, the formula for calculating the generalized inverse of damping is: The generalized inverse is equal to transpose multiplied by ( Multiply transpose squared by the identity matrix The inverse matrix of ), where The damping coefficient is dimensionless. It is the identity matrix; For the first The six-dimensional velocity spinor in the general action block at each control moment: the first three dimensions are linear velocity in meters per second, and the last three dimensions are angular velocity in radians per second.
[0222] In this embodiment, joint space motion is projected onto the target robot's set of joint limit constraints, velocity constraints, torque constraints, and collision constraints. Joint limit constraints are the upper and lower limits of joint angles in radians; velocity constraints are the upper and lower limits of joint angular velocity in radians per second; torque constraints are the upper and lower limits of joint torque in Newton-meters; and collision constraints are the distance thresholds at which joint movement does not cause link collisions in meters. The projection method involves truncating the joint space motion to the boundary value when it exceeds any constraint boundary. When the target robot only supports a position control interface, the target contact force and desired stiffness in the general motion block are converted into a small position offset, and the desired damping is converted into a velocity damping term and superimposed on the end effector velocity feedback. The conversion method for the small position offset is based on the desired stiffness. The small position offset is calculated based on the contact force with the target. This small position offset is equal to the target contact force divided by the desired stiffness. The unit is meters. A small positional offset is superimposed on the end-effector translation increment to achieve force control. The conversion method of the velocity damping term is to calculate the damping force based on the product of the desired damping and the end-effector velocity, and then convert the damping force into a velocity correction amount and superimpose it into the end-effector velocity command. When the target robot supports torque control or impedance control interface, the torque control parameters or impedance control parameters are directly output. The torque control parameters are the target joint torque in Newton-meters, and the impedance control parameters are the desired stiffness matrix and the desired damping matrix. The unit of the desired stiffness matrix is Newton-meter, and the unit of the desired damping matrix is Newton-second-meter.
[0223] In another embodiment of the invention, a cross-ontology joint training step is also included:
[0224] Acquire training trajectories from multiple robot platforms. Each training trajectory includes task instructions, visual sequences, ontology state sequences, tactile sequences, robot action sequences, success or failure markers, and corresponding ontology descriptors.
[0225] Convert each robot's action into a general motion space representation;
[0226] Convert each tactile sequence into a standard contact token sequence;
[0227] The training trajectory is divided into stages: free space stage, pre-contact stage, contact establishment stage, stable operation stage, abnormal recovery stage, and release stage.
[0228] Align different robot features according to physical dimensions: the displacement group features of different robots are aligned with each other, the force group features are aligned with each other, and the tactile group features are aligned with each other. The alignment loss is calculated for features within the same group under a unified dimension.
[0229] The model is trained using action imitation loss, contact token prediction loss, physical dimension grouping and alignment loss, cycle consistency loss (after converting a generic action block into a target robot joint action by a ontology adapter and then reverting it back to a generic action block and comparing it with the original generic action block), safety constraint penalty loss, and random modal occlusion loss as joint training objectives. The random modal occlusion loss is the cross-entropy loss that allows the model to still predict the correct action after randomly occluding at least one modality of visual features, linguistic features, or tactile features with a preset probability during training.
[0230] In this embodiment, the multiple robot platforms are a collection of robots with different joint degrees of freedom, different end effector types, and different tactile sensor configurations. For example, they include a six-axis robotic arm with a two-finger gripper and an array of pressure sensors, a seven-axis robotic arm with a dexterous hand and visual-tactile sensors, and a mobile dual-arm robot with a suction cup and a six-dimensional force or torque sensor. The training trajectory is a multimodal data sequence recorded by the robot during task execution. Each training trajectory includes task instructions, a visual sequence, a body state sequence, a tactile sequence, a robot motion sequence, success or failure markers, and a corresponding body descriptor. The task instructions are natural language text; the visual sequence is a set of image frames ordered by time; the body state sequence is a set of joint positions, joint velocities, and end effector poses ordered by time; the tactile sequence is a set of tactile-related data samples ordered by time; the robot motion sequence is a set of joint control variables ordered by time; the success or failure markers are binary flags used to filter valid training samples; and the corresponding body descriptor is the body descriptor of the robot platform.
[0231] In this embodiment, each robot's actions are converted into a general motion space representation. This conversion is achieved by inverse kinematics and control interface derivation, mapping the joint control quantities of each robot into general motion blocks. These general motion blocks include end-effector pose changes, grasping coordination quantities, target contact force, desired stiffness, desired damping, and contact patterns, making the action sequences of different robots comparable within a unified motion space. Each tactile sequence is then converted into a standard contact token sequence. This conversion is performed using the standard contact token conversion method described in Embodiment 2, uniformly converting the tactile data of each robot into standard contact tokens in the tool coordinate system, making the tactile sequences of different robots comparable within a unified token space. The training trajectory is divided into four phases: free space phase, pre-contact phase, contact establishment phase, stable operation phase, anomaly recovery phase, and release phase. The free space phase is the motion phase in which the end effector does not make contact with the object. The pre-contact phase is the phase in which the end effector approaches the target object and before contact is established. The contact establishment phase is the phase in which the normal force first exceeds the contact threshold. The stable operation phase is the phase in which the contact force remains stable and the task continues to be executed. The anomaly recovery phase is the phase in which corrective actions are executed after slippage, over-force, or loss of contact is detected. The release phase is the phase in which the end effector disengages from contact after the task is completed. The phase division is based on the temporal variation characteristics of the normal force, slippage probability, and contact mode.
[0232] In this embodiment, different robot features are aligned according to physical dimensions. Displacement features, force features, and tactile features of different robots are aligned with each other. The alignment loss within the same group is calculated under a unified dimension. The alignment loss by physical dimensions is expressed by the following formula:
[0233]
[0234] In the formula, The loss is grouped and aligned according to physical dimensions, and is a dimensionless scalar; the outer summation symbol is applied to the grouped sets of physical dimensions. Each group Perform a traversal. A dimensionless set consisting of displacement, force, and tactile components; the inner summation symbol applies to all robot platforms. Perform a traversal. and For indexing different robot platforms, dimensionless, and These represent the total number of different robot platforms; the double vertical lines represent vectors. 2-norm, dimensionless; For robot platforms In the current mission phase The normalized eigenvectors of the group For robot platforms Same group under the same task phase The normalized feature vectors are both floating-point vectors, and the units of each component vary depending on the group; the superscript 2 indicates the square of the norm, which is dimensionless. The displacement group feature alignment loss is the mean square error between displacement group features when different robots perform the same task stage, in square meters; the force group feature alignment loss is the mean square error between force group features when different robots perform the same task stage, in Newton squares; and the haptic group feature alignment loss is the mean square error between haptic group features when different robots perform the same task stage, in the square of the unit of each component. Calculating the alignment loss of features within the same group under a unified dimension ensures that different robot platforms produce physically consistent hidden layer representations in the same task stage, thereby enabling the visual language action model to learn a general strategy independent of the specific robot platform.
[0235] In this embodiment, the model is trained using action imitation loss, contact token prediction loss, physical dimension grouping alignment loss, cyclic consistency loss between the general action and the original general action after ontology adaptation, safety constraint penalty loss, and random modal occlusion loss as joint training objectives. The joint training objective is expressed by the following formula:
[0236]
[0237] In the formula, Let be the total loss from joint training, and be a dimensionless scalar. The weighting coefficients for the action imitation loss are dimensionless. The action imitation loss is the mean square error between the model's predicted general action block and the actual general action block in the training trajectory. The unit is the square of each action component unit, which is a scalar after dimensionless conversion. The weighting coefficients for predicting the loss of the contact token are dimensionless. The loss is the contact token prediction loss, and the mean square error between the model's predicted contact trajectory and the actual standard contact token sequence is the square of each token component unit, which is a scalar after dimensionless conversion. The weighting coefficients for the alignment loss grouped by physical dimensions are dimensionless. The loss is grouped and aligned according to physical dimensions, and the mean square error between features of the same group of different robots in the same task stage is a scalar after being dimensionless. The weighting coefficients for the cycle consistency loss are dimensionless. The cyclic consistency loss is the mean square error between the general action block and the original general action block after the general action block is converted into the target robot joint action by the body adapter and then restored by inverse mapping. After being dimensionless, it is a scalar. Inverse mapping is a method to restore the target robot joint action to the general action block by inverse solution of forward kinematics and control interface. Forward kinematics is an analytical method to calculate the end pose by link transformation matrix chain based on the joint state. Inverse solution of control interface is a method to obtain the joint state by integrating the joint control quantity and then apply forward kinematics to inversely deduce the end pose increment. The weighting coefficient for the penalty loss due to safety constraints is dimensionless. The penalty loss is calculated to account for safety constraints, and the sum of penalty terms for action blocks that violate safety constraints is calculated as a dimensionless scalar in the model output. , which are the weighting coefficients of the random modal occlusion loss, and are dimensionless; The random modal occlusion loss is the cross-entropy loss, a dimensionless scalar, which is the loss that allows the model to predict the correct action even after randomly occluding visual, verbal, or tactile modalities during training. All loss terms are dimensionless before calculation, and all weight coefficients are positive, for example, 1.0, 0.5, 0.3, 0.2, 0.1, and 0.2 respectively. The model is trained using the Adam optimizer with a learning rate of 1 x 10^-4 (1 second), a batch size of 256 (dimensionless), and 500 training epochs (dimensionless).
[0238] In another embodiment of the invention, a new robot adaptation step is also included:
[0239] Freeze the parameters of the visual encoder, language encoder, and haptic normalization module of the visual-language-action model;
[0240] Obtain the calibration trajectory of the new robot, which includes the free space motion trajectory, the basic grasping trajectory, and the contact calibration trajectory;
[0241] The parameters of the body descriptor encoder and motion adapter of the new robot can be trained using the calibration trajectory, or the parameters can be finely tuned efficiently through a low-rank adaptation layer.
[0242] In this embodiment, the new robot is a robot platform that did not participate in cross-entity joint training, and its joint degrees of freedom, end effector type, or tactile sensor configuration differs from the robot platforms in the training set. The parameters of the visual encoder, language encoder, and tactile normalization module are frozen. Freezing means not updating the neural network weight parameters of these modules during fine-tuning, so that the encoding capabilities of the visual encoder for images, the language encoder for language instructions, and the normalization capabilities of the tactile normalization module for tactile data remain unchanged, thereby preventing overfitting of these modules due to a small amount of calibration data from the new robot. The visual encoder is an image encoding network built with a pre-trained visual Transformer, the language encoder is a text encoding network built with a pre-trained language Transformer, and the tactile normalization module is the module that performs the standard contact token conversion of Embodiment 2.
[0243] In this embodiment, the calibration trajectory is a multimodal data sequence collected by the new robot under limited task scenarios. The calibration trajectory includes free space motion trajectory, basic grasping trajectory, and contact calibration trajectory. The free space motion trajectory is a sequence of body states recorded by the new robot performing preset joint spatial movements under non-contact conditions. It is used to calibrate the kinematic parameters and joint constraint of the new robot. The kinematic parameters are link length and joint axis direction, in meters and radians. The unit of joint constraint is radians. The basic grasping trajectory is a multimodal data sequence recorded by the new robot performing grasping, handling, and placement on a standard object. It is used to calibrate the end effector type and control mode of the new robot. The end effector type is enumerated as a two-finger gripper, dexterous hand, or suction cup. The control mode is enumerated as position control, torque control, or impedance control. The contact calibration trajectory is a tactile sequence and body state sequence recorded by the new robot pressing a calibration object with known stiffness with a preset torque. It is used to calibrate the mapping relationship between the new robot's tactile sensor output and the standard contact token. The known stiffness of the calibration object is in Newtons per meter.
[0244] In this embodiment, the parameters of the new robot's ontology descriptor encoder and motion adapter are trained using a calibrated trajectory. The ontology descriptor encoder is a graph encoding network constructed with a graph attention network. Its input is the morphological graph structure node feature matrix and adjacency matrix of the new robot's ontology descriptor, and its output is the new robot's ontology vector. The motion adapter is a residual prediction network constructed with a conditional multilayer perceptron. Its input is a general action block, ontology vector, and current joint state, and its output is a learnable residual of the analytical kinematic mapping. The training objectives are action imitation loss and contact token prediction loss. During training, only the parameters of the ontology descriptor encoder and motion adapter are updated, while the parameters of the visual encoder, language encoder, and tactile normalization module remain frozen. Alternatively, efficient parameter fine-tuning can be achieved through a low-rank adaptation layer. The low-rank adaptation layer is a low-rank decomposition matrix connected in parallel with the attention weight matrix of the visual-language-action model. Its construction method is to convert the original weight matrix... The update amount is decomposed into two low-rank matrices. and The product of, where For dimension Multiply matrix, For dimension Multiply matrix, It is a low-rank dimension and much smaller , The hidden layer dimension of the model is used; only the low-rank matrix is updated during training. and Parameters, original weight matrix By keeping the parameters frozen, efficient adaptation to new robotic platforms can be achieved with a small number of trainable parameters, for example... Take 8 When taking 512, the trainable parameters of each attention layer are from The square of 262144 is reduced to 2 multiplied by Multiply That is, 8192. The weight update method of the low-rank adaptation layer is expressed by the following formula:
[0245]
[0246] In the formula, This is the equivalent weight matrix updated by the low-rank adaptation layer, and its dimensions are the same as the original weight matrix. Same, dimensionless; The original weight matrix during the pre-training phase remains frozen and dimensionless during fine-tuning. It is one of the low-rank decomposition matrices with dimension . Multiply Updated during fine-tuning; dimensionless. The second low-rank decomposition matrix, with dimension . Multiply Updated during fine-tuning; dimensionless. Multiply The product of two low-rank matrices with dimension . Multiply , and the original weight matrix Same dimension, dimensionless; The hidden output vector of the attention layer, with dimension . Dimensionless; The input vector of the attention layer has a dimension of The training hyperparameters for fine-tuning the low-rank adaptation layer are: learning rate of 1 x 10^-4 (unit: 1 second), batch size of 32 (dimensionless), number of training epochs of 100 (dimensionless), and the optimizer is Adam.
[0247] In another embodiment of the present invention, the safety constraints include force constraints, velocity constraints, acceleration constraints, joint limit constraints, collision distance constraints, and confidence constraints;
[0248] Safety constraints are implemented through safety filters, which are constructed using one or more of the following methods: rule constraints, control barrier functions, model predictive control, collision detection, or independent safety controllers.
[0249] When the tactile confidence level falls below the threshold or the tactile sensor communication is interrupted, the confidence level is set to zero and a fault token is generated. The residual motion weight is reduced, the end effector speed and maximum gripping force are limited, and the vision and proprioception state are switched to a conservative mode. Before entering the contact phase, effective tactile feedback must be regained; otherwise, the task is stopped.
[0250] When the force exceeds the limit, the collision risk is high, or the model confidence level is below the threshold, the safety filter will stop, unload, or retract.
[0251] In this embodiment, safety constraints are a set of constraints applied to the modified general action block, including force constraints, velocity constraints, acceleration constraints, joint limit constraints, collision distance constraints, and confidence constraints. Force constraints are the upper and lower limits of the end-effector contact force and joint torque, with the end-effector contact force measured in Newtons (N) and the joint torque in Newton-meters (Nm). Velocity constraints are the upper and lower limits of the end-effector linear velocity, end-effector angular velocity, and joint angular velocity, with the end-effector linear velocity measured in meters per second (m / s), the end-effector angular velocity in radians per second (radians per second). Acceleration constraints are the upper and lower limits of the end-effector linear acceleration and joint angular acceleration, with the end-effector linear acceleration measured in meters per second squared (m / s²) and the joint angular acceleration in radians per second squared (radians per second squared). Joint limit constraints are the upper and lower limits of the joint angle, measured in radians. Collision distance constraints are the minimum permissible distance between the end-effector and obstacles in the environment, measured in meters. Confidence constraints are the lower limit of tactile confidence, ranging from 0 to 1 and dimensionless.
[0252] In this embodiment, safety constraints are implemented using one or more of the following methods: rule constraints, control barrier functions, model predictive control, collision detection, or independent safety controllers. Rule constraints involve directly applying upper and lower limits to each component of the action block, truncating the action block component to its boundary value when it exceeds the constraint boundary. The control barrier function is a safety constraint method constructed using control barrier functions. Its construction method defines the safety set as a subset of the state space that satisfies the safety conditions, and the control barrier function... For state scalar function, when When the value is greater than 0, the state is within the safe set. The invariance constraint condition of the control barrier function is expressed by the following formula:
[0253]
[0254] In the formula, The control barrier function is denoted by , and the state is denoted by . A scalar function whose state is within the safe set and dimensionless when it is greater than 0; The symbol for the partial derivative is dimensionless. Divide by To control the barrier function For the state vector The partial derivative vector is dimensionless; The robot's state transition function describes the state. In control input The rate of change under action, dimensionless; This is a vector dot product operator, dimensionless; The greater than or equal to sign is dimensionless; The safety margin coefficient is a positive, dimensionless value, for example, 1.0. This constraint ensures that the safety set is invariant, meaning that once the state trajectory enters the safety set, it does not leave it. The robot's state vector is expressed in radians and meters per second. This is the robot's state transition function. The control input is the action block. The input to the control barrier function is the robot state and the action block to be detected, and the output is the modified action block that satisfies the safety set invariance condition. Model predictive control is a safety constraint method built using rolling optimization, which is constructed by solving for future... The action block within each control step is a quadratic programming problem with optimization variables, safety constraints, and dynamic constraints as constraints, and minimizing task tracking error as the objective function. The model predictive control (MMC) takes the current robot state, the corrected action block, and the reference trajectory as input, and outputs the optimal action sequence that satisfies the safety constraints. Collision detection is implemented using a bounding box hierarchy tree or symbolic distance field. Its inputs are the robot joint states and an environmental obstacle model, and its output is the nearest collision distance in meters. The independent safety controller is a hardware safety module physically isolated from the visual language action model. It is implemented using a field-programmable gate array (FPGA) or microcontroller, takes real-time measurements of force and velocity constraints as input, and outputs an emergency stop signal or torque unloading command.
[0255] In this embodiment, when the tactile confidence level is below a threshold or the tactile sensor communication is interrupted, the confidence level is set to zero and a fault token is generated. The fault token is a binary flag indicating a tactile sensor malfunction, with a value of 0 or 1, and is dimensionless. A fault token of 1 indicates that the tactile sensor is normal, and a value of 0 indicates that the tactile sensor is malfunctioning. Reducing the residual action weight involves lowering the residual action injection intensity coefficient from the normal value of 1.0 to 0.1, significantly weakening the correction effect of the residual action on the general action block. Limiting the end effector speed and maximum clamping force involves lowering the upper limit of the end effector linear velocity from 1 m / s to 0.1 m / s and the maximum clamping force from 20 N to 5 N. Switching to a conservative mode for visual and proprioceptive states means that the visual language action model generates actions based solely on visual and proprioceptive features when tactile information is missing, and the gating coefficient is adjusted accordingly. Forced to 0 to block tactile injection. Before entering the contact phase, valid tactile feedback must be regained. Valid tactile feedback is defined as a tactile confidence level above a threshold and a fault token of 1; otherwise, the task is stopped. When force exceeds limits, collision risk occurs, or model confidence falls below a threshold, the safety filter executes stop, unload, or retraction. Stopping sets the velocity of all robot joints to zero (in radians per second); unloading linearly decays the target contact force from its current value to zero (in Newtons); retraction controls the end effector to retreat a safe distance in the opposite direction of the contact normal (in meters).
[0256] This invention provides an embodiment of an end-to-end cross-proprio generalized visual language action processing device based on VLA fused haptic feedback, comprising:
[0257] The multimodal acquisition module acquires language commands, visual observation data, robot body state data, tactile data, and target robot body descriptors.
[0258] The time synchronization module performs multi-rate time synchronization of visual observation data, robot body state data, tactile data and language commands to obtain time-synchronized multimodal data.
[0259] The haptic standardization module converts the time-synchronized haptic data into standard contact tokens in the tool coordinate system. The standard contact tokens include contact position, contact normal, normal force, tangential force, contact torque, slip probability, contact stiffness, vibration characteristics, temperature, confidence level, sensor type encoding, and valid field mask.
[0260] The multimodal fusion module uses a visual language action model to encode language commands, visual observation data, robot body state data, and standard contact tokens. The encoding results are grouped into displacement, force, and tactile groups according to physical dimensions. Within each group, the data is normalized according to physical units to make similar features of different robot bodies comparable under a unified dimension. Cross-group information exchange is performed between groups to generate a unified state representation.
[0261] The general motion generation module is used to generate general motion blocks and predict contact trajectories based on a unified state representation. The general motion block includes end pose change, grasping coordination, target contact force, desired stiffness, desired damping, and contact mode.
[0262] The body adaptation module converts general motion blocks into control instructions for the target robot based on the target robot's body descriptor and executes them. The body descriptor includes joint topology, degrees of freedom, joint limits, kinematic parameters, end effector type, control mode, maximum speed, maximum torque, control cycle, and fault status.
[0263] The tactile residual correction module obtains the actual contact token after executing the control command, generates residual actions based on the contact error between the actual contact token and the predicted contact token at the corresponding moment in the predicted contact trajectory, and uses the residual actions to correct the general action blocks that have not yet been executed.
[0264] The safety control module applies safety constraints to the modified general action block and outputs them to the target robot until the task is completed or a safety stop is triggered.
[0265] In this embodiment, the multimodal acquisition module is a hardware acquisition unit consisting of a camera, joint encoder, torque sensor, tactile sensor, and communication interface. Its inputs include ambient light, robot joint motion, contact force, and tactile physical quantities. Its outputs include spoken command text, visual observation data image sequences, robot body state data numerical vectors, raw tactile data signals, and target robot body descriptor data structures. The time synchronization module is a hardware time synchronization unit built with a field-programmable gate array (FPGA) or real-time microprocessor. Its inputs are the acquisition timestamps and arrival timestamps of each sensor in the multimodal acquisition module, and its output is the time-synchronized multimodal data. The tactile normalization module is a tactile data conversion unit built with a digital signal processor (DSP) or graphics processor (GPU). Its inputs are the raw tactile data signals, and its output is a standard contact token in the tool coordinate system.
[0266] In this embodiment, the multimodal fusion module is a visual language action model inference unit running on a graphics processor. Its inputs include language commands, visual observation data, robot body state data, and standard contact tokens, and its output is a unified state representation. Internally, the multimodal fusion module groups the encoded results into displacement, force, and tactile groups according to physical dimensions. Within each group, normalization by physical units ensures that similar features of different robot bodies are comparable under a unified dimension. Cross-group information interaction occurs between groups through a dimensional awareness attention mechanism. The general action generation module is an action generation head inference unit running on a graphics processor. Its inputs include the unified state representation and body descriptor vectors, and its outputs are general action blocks and predicted contact trajectories. The body adaptation module is an action adaptation unit running on a graphics processor or a real-time microprocessor. Its inputs include general action blocks, body descriptors, and the current joint state, and its output is the target robot's control commands, which are joint positions, joint velocities, or joint torques, in radians, radians per second, or Newton-meters, respectively.
[0267] In this embodiment, the tactile residual correction module is a residual motion head inference unit running on a graphics processor. Its inputs are the actual contact token, the predicted contact trajectory, the current semantic features, the motion stage, and the ontology descriptor. Its outputs are the residual motion and the general motion block after residual correction. The safety control module is a hardware safety unit built with a field-programmable gate array or a standalone safety controller. Its inputs are the corrected general motion block, the robot's current state, and the tactile confidence level. Its outputs are control commands that meet safety constraints or emergency stop signals. In this embodiment, a multimodal acquisition module, a time synchronization module, a tactile standardization module, a multimodal fusion module, a general motion generation module, a body adaptation module, a tactile residual correction module, and a safety control module are sequentially connected to form an end-to-end processing link. Language commands, visual observation data, robot body state data, tactile related data, and the target robot body descriptor are acquired by the multimodal acquisition module and then processed sequentially by the time synchronization module, tactile standardization module, multimodal fusion module, general motion generation module, body adaptation module, tactile residual correction module, and safety control module. Finally, the data is output to the target robot for execution, forming a closed-loop control with fused tactile feedback. Through the collaborative work of these modules, the device enables the migration and deployment of the same visual language action model across different robot platforms and adaptively adjusts operating parameters based on real-time tactile feedback in contact-intensive tasks.
[0268] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of this invention and its equivalents, this invention also intends to include these modifications and variations.
Claims
1. An end-to-end cross-ontology generalized visual language action processing method based on VLA fusion haptic feedback, characterized in that, include: Acquire language commands, visual observation data, robot body state data, tactile data, and target robot body descriptor; Multi-rate time synchronization is performed on visual observation data, robot body state data, tactile data and language commands to obtain time-synchronized multimodal data; The time-synchronized tactile data is converted into standard contact tokens in the tool coordinate system. The standard contact tokens include contact position, contact normal, normal force, tangential force, contact torque, slip probability, contact stiffness, vibration characteristics, temperature, confidence level, sensor type encoding, and effective field mask. Different robots only share the standard contact tokens and do not share the image size, array resolution, or units of the raw tactile data. The visual language action model is used to encode language commands, visual observation data, robot body state data and standard contact tokens. The encoding results are grouped into displacement group, force group and tactile group according to physical dimensions. Within each group, the physical units are normalized to make the same characteristics of different robot bodies comparable under the same dimensions. Cross-group information exchange is carried out between groups to generate a unified state representation. Generate a general action block and predict the contact trajectory based on the unified state representation. The general action block includes the end pose change, gripping coordination, target contact force, desired stiffness, desired damping, and contact mode. The general motion block is converted into control instructions for the target robot based on the target robot body descriptor and then executed. The body descriptor includes joint topology, degrees of freedom, joint limits, kinematic parameters, end effector type, control mode, maximum speed, maximum torque, control cycle, and fault status. Obtain the actual contact token after executing the control command, generate residual actions based on the contact error between the actual contact token and the predicted contact token at the corresponding moment in the predicted contact trajectory, and use the residual actions to correct the general action blocks that have not yet been executed. After applying safety constraints to the modified general action block, continue controlling the target robot until the task is completed or a safety stop is triggered.
2. The end-to-end cross-ontology generalized visual language action processing method based on VLA fusion haptic feedback according to claim 1, characterized in that, Convert haptic data into standard contact tokens in the tool coordinate system, including: When the tactile data is array pressure sensor data, the contact position is calculated using the array unit coordinates and the pressure weighting center, and the contact normal is estimated based on the pressure gradient and the sensor surface normal. When the tactile data is from a visual tactile sensor, the contact shape and shear force are estimated by the displacement field of the marked point or optical flow. When the tactile data is from a six-dimensional force or torque sensor, the resultant force and resultant torque in the tool coordinate system are obtained through coordinate transformation. When the tactile data is joint torque or motor current data, the external force is estimated by subtracting the gravity, inertia and friction terms from the dynamic model and then adding the corresponding confidence value. When a robot has multiple tactile sensors, multiple contact tokens are projected onto a unified coordinate system using sensor extrinsic parameters, and a set of contact tokens is generated by confidence weighting or non-maximum suppression.
3. The end-to-end cross-ontology generalized visual language action processing method based on VLA fusion haptic feedback according to claim 1, characterized in that, Multi-rate time synchronization includes: Establish a unified time reference and record the acquisition timestamp and arrival timestamp for visual frames, robot states, tactile sampling, and control commands respectively; when devices do not share a hardware clock, estimate clock offset and drift through periodic synchronization messages, round-trip delay estimation, or correlation analysis of common motion events. Construct a sliding time window centered on the target control time; Time alignment is performed on visual observation data using nearest neighbor or feature interpolation methods. Time alignment is performed on the robot's body state data using linear or spline interpolation. The maximum normal force, tangential force integral, impact peak, slip spectrum, contact establishment time, contact release time and contact duration are extracted from the tactile data within the time window. The variable number of tactile samples are compressed into a fixed number of tactile event tokens. The tactile event tokens are then processed by the tactile encoder and converted into standard contact tokens. When an impact, slippage, or force overshoot trend is detected, fast residual inference is triggered without waiting for the next visual frame.
4. The end-to-end cross-ontology generalized visual language action processing method based on VLA fusion haptic feedback according to claim 1, characterized in that, The visual language action model is used to encode language commands, visual observation data, robot body state data, and standard contact tokens. The encoding results are grouped according to physical dimensions into displacement, force, and tactile groups, including: Output task tokens and constraint tokens via the language encoder; The visual encoder outputs scene features, target object features, tool features, and spatial relationship features. The body state encoder outputs joint position, joint velocity, end effector pose, and current control mode characteristics. The standard contact token and its timing characteristics are output through a tactile encoder. The spatial relationship features output by the visual encoder and the end pose features output by the body state encoder are grouped into the displacement group. The joint torque characteristics output by the body state encoder and the normal force, tangential force, and contact torque characteristics output by the tactile encoder are grouped into the force group; The contact position, slip probability, contact stiffness, vibration characteristics, temperature, confidence level, sensor type encoding, and effective field mask features output by the tactile encoder are categorized into the tactile group; Inter-group and cross-group information interaction is achieved through a dimensional perception attention mechanism. The dimensional perception attention mechanism applies cross-group constraints based on the physical dimensional relationship: the force group and the displacement group are coupled through stiffness matrix constraints, the time difference of the displacement group and the force group are coupled through dynamic relationship constraints, and the tactile group and the force group are coupled through contact mechanics relationship constraints, thereby obtaining cross-group interaction characteristics. The tactile injection gating coefficients are calculated based on cross-group interaction features, linguistic features, visual features, ontological features, and tactile confidence. The gating coefficient is used to control the cross-modal attention fusion intensity of tactile features with displacement, force and language features in cross-group interaction features; In the approaching target stage, the gating coefficient is reduced to rely mainly on visual features. In the clamping of fragile objects stage, the gating weight corresponding to the normal force feature is increased. In the screwing bottle cap stage, the gating weight corresponding to the tangential force feature, sliding feature, and contact torque feature is increased. After gating fusion, a unified state representation is generated.
5. The end-to-end cross-ontology generalized visual language action processing method based on VLA fusion haptic feedback according to claim 1, characterized in that, Generate a general action block and a predicted contact trajectory, and generate a residual action correction general action block based on the contact error between the actual contact token and the predicted contact trajectory, including: The motion generation head outputs a nominal motion block with a length of H control steps at a time. The nominal motion block is a predicted instance of the general motion block at the current control moment, and also outputs the predicted contact trajectory corresponding to the nominal motion block. The general action block is organized into a pose layer, force control layer and contact layer according to physical dimensions. Each layer is independently normalized and the layers are coupled through physical relationships. The pose layer includes end-effector translation increments, rotation increments, or six-dimensional velocity spinors in the tool coordinate system; The force control layer includes the target normal force, the target tangential force range, the target torque, the desired end stiffness, the desired damping, and the compliance direction; The contact layer includes contact patterns, normalized grasping coordination, action duration, action confidence, and predicted contact states; After executing the first segment of the nominal action block, the actual contact token is obtained. The contact error between the actual contact token and the predicted contact token at the corresponding moment in the predicted contact trajectory is calculated. The contact error includes normal force error, tangential force error, slip probability error, contact position error, and contact torque error. The residual action head generates residual actions based on contact error, current semantic features, action stage, and ontology descriptor; The residual actions are used to correct the general action blocks that have not yet been executed. The correction method is to superimpose the residual actions with the unexecuted parts of the nominal action blocks and adjust the injection intensity of the residual actions through an error gating function. In response to contact errors caused by indicated slippage, residual actions are generated to increase the amount of gripping coordination, change the contact direction, or reduce the tangential velocity. In response to contact errors indicating excessive force, residual actions are generated to reduce normal displacement, reduce stiffness, or perform retraction. In response to a contact error indicating a jammed connection, a residual action is generated to perform a search motion; In response to a contact error indicating a loss of contact, a residual action is generated to re-approach or reposition the control end. In response to contact errors caused by an over-limit of the indicated torque, a residual action is generated to reduce the rotation increment or change the rotation direction.
6. The end-to-end cross-ontology generalized visual language action processing method based on VLA fusion haptic feedback according to claim 1, characterized in that, The ontology descriptor contains a morphology graph structure, which uses joints, links, end effectors and contacts as nodes, and kinematic connections, mechanical coupling and symmetry relationships as edges. Based on the target robot body descriptor, convert the general action block into control instructions for the target robot, including: Input the ontology descriptor into the morphological graph encoder to obtain the ontology vector; Based on the general action block and the ontology vector, the action adapter outputs a learnable residual for the analytical kinematic mapping; The end effector is mapped to joint space action using the generalized inverse or damped generalized inverse of the Jacobian matrix of the target robot. Project the joint space motion onto the target robot's set of joint limit constraints, velocity constraints, torque constraints, and collision constraints; When the target robot only supports the position control interface, the target contact force, desired stiffness, and desired damping in the general action block are converted into a small position offset; when the target robot supports torque control or impedance control interface, the torque control parameters or impedance control parameters are directly output.
7. The end-to-end cross-ontology generalized visual language action processing method based on VLA fusion haptic feedback according to claim 1, characterized in that, It also includes cross-ontology joint training steps: Acquire training trajectories from multiple robot platforms. Each training trajectory includes task instructions, visual sequences, ontology state sequences, tactile sequences, robot action sequences, success or failure markers, and corresponding ontology descriptors. Convert each robot's actions into a general motion space representation; Convert each tactile sequence into a standard contact token sequence; The training trajectory is divided into stages: free space stage, pre-contact stage, contact establishment stage, stable operation stage, abnormal recovery stage, and release stage. Align different robot features according to physical dimensions: the displacement group features of different robots are aligned with each other, the force group features are aligned with each other, and the tactile group features are aligned with each other. The alignment loss is calculated for features within the same group under a unified dimension. The model is trained using action imitation loss, contact token prediction loss, physical dimension grouping and alignment loss, cycle consistency loss (after converting a generic action block into a target robot joint action by a ontology adapter and then reverting it back to a generic action block and comparing it with the original generic action block), safety constraint penalty loss, and random modal occlusion loss as joint training objectives. The random modal occlusion loss is the cross-entropy loss that allows the model to still predict the correct action after randomly occluding at least one modality of visual features, linguistic features, or tactile features with a preset probability during training.
8. The end-to-end cross-ontology generalized visual language action processing method based on VLA fusion haptic feedback according to claim 7, characterized in that, It also includes the new robot adaptation steps: Freeze the parameters of the visual encoder, language encoder, and haptic normalization module of the visual-language-action model; Obtain the calibration trajectory of the new robot, which includes the free space motion trajectory, the basic grasping trajectory, and the contact calibration trajectory; The parameters of the body descriptor encoder and motion adapter of the new robot can be trained using the calibration trajectory, or the parameters can be finely tuned efficiently through a low-rank adaptation layer.
9. The end-to-end cross-ontology generalized visual language action processing method based on VLA fusion haptic feedback according to claim 1, characterized in that, Safety constraints include force constraints, velocity constraints, acceleration constraints, joint limit constraints, collision distance constraints, and confidence constraints. Safety constraints are implemented through safety filters, which are constructed using one or more of the following methods: rule constraints, control barrier functions, model predictive control, collision detection, or independent safety controllers. When the tactile confidence level is below the threshold or the tactile sensor communication is interrupted, the confidence level is set to zero and a fault token is generated, the residual motion weight is reduced, the end effector speed and maximum clamping force are limited, and the vision and proprioception state conservative mode is switched. Before entering the contact phase, a valid sense of touch must be regained; otherwise, the task will be stopped. When the force exceeds the limit, the collision risk is high, or the model confidence level is below the threshold, the safety filter will stop, unload, or retract.
10. An end-to-end cross-proprio generalized visual language action processing device based on VLA fused haptic feedback, characterized in that, include: The multimodal acquisition module acquires language commands, visual observation data, robot body state data, tactile data, and target robot body descriptors. The time synchronization module performs multi-rate time synchronization of visual observation data, robot body state data, tactile data and language commands to obtain time-synchronized multimodal data. The haptic standardization module converts the time-synchronized haptic data into standard contact tokens in the tool coordinate system. The standard contact tokens include contact position, contact normal, normal force, tangential force, contact torque, slip probability, contact stiffness, vibration characteristics, temperature, confidence level, sensor type encoding, and valid field mask. The multimodal fusion module uses a visual language action model to encode language commands, visual observation data, robot body state data, and standard contact tokens. The encoding results are grouped into displacement, force, and tactile groups according to physical dimensions. Within each group, the data is normalized according to physical units to make similar features of different robot bodies comparable under a unified dimension. Cross-group information exchange is performed between groups to generate a unified state representation. The general motion generation module is used to generate general motion blocks and predict contact trajectories based on a unified state representation. The general motion block includes end pose change, grasping coordination, target contact force, desired stiffness, desired damping, and contact mode. The body adaptation module converts general motion blocks into control instructions for the target robot based on the target robot's body descriptor and executes them. The body descriptor includes joint topology, degrees of freedom, joint limits, kinematic parameters, end effector type, control mode, maximum speed, maximum torque, control cycle, and fault status. The tactile residual correction module obtains the actual contact token after executing the control command, generates residual actions based on the contact error between the actual contact token and the predicted contact token at the corresponding moment in the predicted contact trajectory, and uses the residual actions to correct the general action blocks that have not yet been executed. The safety control module applies safety constraints to the modified general action block and outputs them to the target robot until the task is completed or a safety stop is triggered.
