Robot assembly control method and system integrating vision, force, and proprioception

Abstract

This invention discloses a robot assembly control method and system that integrates vision, force, and proprioception, relating to the field of robot intelligent control technology. The method includes the following steps: acquiring visual images, force data, and proprioception data during the robot assembly process and preprocessing them; extracting features from the visual images and force data separately; fusing the extracted visual and force features to obtain visual-force fusion features; analyzing the visual-force fusion features and proprioception data using a robot assembly prediction model based on diffusion learning to generate flexible assembly instructions; wherein, based on a unidirectional cross-attention mechanism, feature extraction of the force data is performed by introducing a causal mask; and the robot performs assembly operations according to the flexible assembly instructions. This invention enables end-to-end assembly control that balances high real-time performance and high safety.

Description

Technical Field

[0001] This invention relates to the field of robot intelligent control technology, and in particular to a robot assembly control method and system that integrates vision, force perception, and proprioception. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Currently, large-scale robot models integrating vision, language, and force sensing (such as the RT series) have a massive number of parameters (typically in the tens of billions), and a single forward inference often requires a delay of hundreds of milliseconds or even several seconds. In precise industrial contact assembly tasks, once an abnormal collision occurs, the system needs a millisecond-level response speed to adjust its actions. Therefore, these massive models are difficult to deploy directly on edge industrial controllers with limited computing power, which can easily lead to severe delays in collision avoidance intervention.

[0004] While existing lightweight end-to-end control models (such as the basic VisuomotorDiffusion Policy) have high inference frequencies, they generally employ a 'pure vision + proprioception' input paradigm, lacking the perception of end-effector contact force information. In high-precision assembly processes such as gear meshing and insertion, vision is often subject to occlusion or depth estimation errors. Once a workpiece experiences a slight misalignment, the pure vision model cannot perceive the continuously increasing abnormal contact force. The robotic arm will still forcefully apply force along the original trajectory, ultimately causing the robot system to enter a deadlock protection shutdown, or even directly damaging the workpiece and end effector. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the purpose of this invention is to provide a robot assembly control method and system that integrates vision, force, and proprioception to achieve end-to-end assembly control that balances high real-time performance and high safety.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solution: The first aspect of this invention provides a robot assembly control method that integrates vision, force, and proprioception, comprising the following steps: Acquire visual images, force data, and proprioceptive data during the robot assembly process, and perform preprocessing. Feature extraction is performed on visual images and force data respectively. The extracted visual features and force features are fused to obtain visual-force fusion features. A robot assembly prediction model based on diffusion learning is used to analyze the visual-force fusion features and proprioceptive data to generate flexible assembly instructions. Among them, based on a one-way cross attention mechanism, causal masking is introduced to extract features from force data. The robot performs assembly operations according to flexible assembly instructions.

[0007] Further, the specific steps of preprocessing include: The visual images are normalized using mean and standard deviation. The force perception data and proprioceptive data are subjected to minimum-maximum mapping normalization.

[0008] Furthermore, a deep residual network is used as the backbone network to extract features from the visual image. The feature map output by the deep residual network is input into the first branch and the second branch in parallel for processing. The first branch is used to perform spatial dimension flattening operation, and the second branch is used to extract the coordinates of two-dimensional key points of salient features in the visual image using pooling operation.

[0009] Furthermore, the specific steps for fusing the extracted visual and force features are as follows: Independent linear layer mappings are performed for visual features and force features respectively; A multi-head cross-attention mechanism is used for cross-modal attention computation and feature weighting aggregation; The aggregated features are then passed through a linear layer to obtain fused features.

[0010] Furthermore, the specific steps for analyzing visual-mechanical fusion features and proprioceptive data using a diffusion-based robot assembly prediction model are as follows: Visual features, visual-mechanical fusion features, and proprioceptive features are spliced ​​together; The spliced ​​features are then subjected to iterative denoising to obtain the motion trajectory. The action trajectory is extracted by action block segmentation, and the resulting action block sequence is used as the action to be executed in the current control cycle. The extracted action block sequence is restored to physical control commands through an inverse normalization operation.

[0011] Furthermore, an end-to-end supervised learning paradigm is adopted to jointly train the robot assembly prediction model and the feature extraction process.

[0012] A second aspect of the present invention provides a robot assembly control system that integrates vision, force, and proprioception, comprising: The data acquisition module is configured to acquire visual images, force data, and proprioceptive data during the robot assembly process and perform preprocessing. The data processing module is configured to extract features from visual images and force data respectively, fuse the extracted visual features and force features to obtain visual-force fusion features, and use a robot assembly prediction model based on diffusion learning to analyze the visual-force fusion features and proprioceptive data to generate flexible assembly instructions. Among them, based on a one-way cross attention mechanism, causal masking is introduced to extract features from force data. The assembly execution module is configured for the robot to perform assembly operations according to flexible assembly instructions.

[0013] A third aspect of the present invention provides a computer-readable storage medium storing a computer program adapted to be loaded by a processor and executed the steps of the robot assembly control method for the fusion of vision, force, and proprioception as described in the first aspect of the present invention.

[0014] A fourth aspect of the present invention provides a computer device comprising: A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program, which, when executed by the processor, implements the robot assembly control method that integrates vision, force, and proprioception as described in the first aspect of the present invention.

[0015] A fifth aspect of the present invention provides a computer program product or computer program comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of the vision, force, and proprioception fusion robot assembly control method as described in the first aspect of the present invention.

[0016] The above one or more technical solutions have the following beneficial effects: This invention discloses a robot assembly control method and system that integrates vision, force, and proprioception. It considers not only the robot's proprioceptive information, such as vision, joint position sequences and velocity sequences of the robotic arm, and the spatial pose of the end effector, but also force information. By fusing vision, force, and proprioception with a diffusion strategy learning model, it generates flexible assembly instructions for the robot, improving assembly accuracy and efficiency. This invention extracts abnormal contact force trends using a lightweight causal temporal network and dynamically guides visual spatial features through a single-layer cross-attention mechanism. Without significantly increasing model inference latency (maintaining the high-frequency operation capability of a small model), it endows the diffusion strategy model with compliant avoidance and automatic correction capabilities for abnormal assembly conditions, thereby achieving end-to-end assembly control that balances high real-time performance and high safety.

[0017] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of the overall principle of the robot assembly control method that integrates vision, force, and proprioception in Embodiment 1 of the present invention. Detailed Implementation

[0020] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0021] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof. The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0022] Example 1: Embodiment 1 of the present invention provides a robot assembly control method that integrates vision, force, and proprioception, such as... Figure 1 As shown, it includes the following steps: S1: Acquire visual images, force data, and proprioceptive data during the robot assembly process, and perform preprocessing.

[0023] S1.1: Acquire visual images, force data, and proprioceptive data during the robot assembly process.

[0024] In one specific implementation, the visual images are RGB image sequences obtained through a camera during the robot assembly process. Force data [F_x, F_y, F_z, M_x, M_y, M_z] are obtained through force sensors (such as the Onrobot sensor), where F_x, F_y, and F_z are the forces along the x, y, and z axes of the sensor coordinate system, respectively, and M_x, M_y, and M_z are the torques about the x, y, and z axes, respectively. In this embodiment, the robot uses the UR5E robotic arm, and the proprioceptive data includes: joint space data (recording the actual angles and angular velocities of the six joints) and Cartesian space / end-effector data (recording the actual position, orientation, and velocity of the robotic arm's end effector).

[0025] S1.2: Preprocess visual images, force data, and proprioceptive data.

[0026] S1.2.1: Perform mean and standard deviation normalization on the visual image.

[0027] In one specific implementation, for visual images, in order to eliminate the overall brightness drift caused by different lighting environments and camera exposure, and to make it easier for the network to extract edges and object shapes, this embodiment uses mean standard deviation normalization based on the preprocessing module.

[0028] Specifically, assume that the system has pre-calculated the mean values ​​of the image channels on the training set. and standard deviation For each pixel value The calculation formula is: .

[0029] in, These are the normalized pixel values.

[0030] S1.2.2: Perform minimum-maximum mapping normalization on force perception data and proprioceptive data.

[0031] In one specific implementation, for force and proprioceptive data, due to the significant differences in physical dimensions and numerical values ​​between proprioceptive state (unit: meters / radians) and force sensor (unit: Newtons), directly splicing them into the network would cause the gradient generated by large quantities (such as a force of 50 N) to completely overwhelm small quantities (such as a displacement of 0.01 m). Therefore, a minimum-maximum (MIN_MAX) mapping is used to achieve normalization. MIN_MAX normalization forces "displacement" and "contact force" to have equal numerical weights, enabling stable cross-modal attention fusion.

[0032] Specifically, based on the physical limit maximum value recorded in historical data. and minimum value , the original physical quantity A linear mapping is performed to a symmetric dimensionless interval, which in this embodiment is [-1, 1]. The calculation formula is: .

[0033] in, This refers to the normalized physical quantity.

[0034] S2: Extract features from visual images and force data respectively, fuse the extracted visual features and force features to obtain visual-force fusion features, and use a robot assembly prediction model based on diffusion learning to analyze the visual-force fusion features and proprioceptive data to generate flexible assembly instructions. Among them, based on a one-way cross attention mechanism, causal masking is introduced to extract features from force data.

[0035] S2.1: Extract features from the visual image.

[0036] In one specific implementation, the visual feature extraction module of this embodiment uses a deep residual network as the backbone network to extract features from the visual image. The feature map output by the deep residual network is input into the first branch and the second branch in parallel for processing. The first branch is used to perform a flattening operation in the spatial dimension, and the second branch is used to extract the coordinates of two-dimensional key points of significant features in the visual image using a pooling operation.

[0037] Specifically, the deep residual network in this embodiment uses a deep residual network with the fully connected layers removed to preserve the spatial topology of the image, such as ResNet-18.

[0038] The feature map output by the backbone network is fed in parallel into two processing branches: the first branch performs a spatial flattening operation on it, outputting a two-dimensional spatial feature map with spatial index. The first branch serves as the key / value pair for subsequent force-visual cross-attention fusion; the second branch employs spatial softmax pooling to extract the coordinates of two-dimensional key points of significant features in the image. In this embodiment, the second branch focuses on the assembly object features of the robotic arm to improve the prediction accuracy of the subsequent robot assembly prediction model.

[0039] Subsequently, dimensionality reduction is performed using a fully connected layer to form a one-dimensional visual feature representing the global scene state. .

[0040] S2.2: Extract features from force perception data.

[0041] In one specific implementation, this embodiment utilizes a force feature extraction module to extract force features. Since force is low-dimensional, high-frequency temporal information, such as a continuous N-step 6-dimensional force sequence, and the system only focuses on forces that have already occurred, the standard self-attention mechanism involves bidirectional information interaction, which can easily lead to the control strategy relying on future frames during inference. Accurately extracting the temporal evolution trend of contact events (such as sudden collisions and sliding friction), while strictly adhering to the causal time order of the physical world during feature extraction (i.e., the network must not use data from "future" moments to generate data traversal when inferring the current force state), is the primary prerequisite for force features to effectively guide subsequent action generation.

[0042] This embodiment constructs an upper triangular occlusion matrix (i.e., a causal mask) to forcibly block attention calculations from the current time step to future time steps (i.e., decisions are made only based on past and current contact events). Therefore, the force feature extraction module is designed to include one linear layer and M layers of causal transformers, where M is 2 in this embodiment.

[0043] The specific steps are as follows: S2.2.1: Construct the mask matrix.

[0044] To prevent data skipping, this embodiment, in the preprocessing stage, not only normalizes the data but also relies on code to rigidly generate a mask matrix M. The horizontal axis of the matrix represents the querier (current time), and the vertical axis represents the target (target time). The construction rule is to set the lower triangle to 0 and the upper triangle to negative infinity.

[0045] S2.2.2: Use a mask matrix to block attention calculations from the current time step to future time steps.

[0046] First, the force perception data sequence is transformed into query Q and key K through a linear layer. These are multiplied to calculate the original relational scoring table. At this point, the network still calculates the relationships between all time steps.

[0047] Then, a mask matrix is ​​added to the original relational scoring table for forced intervention. The masked matrix is ​​then fed into the Softmax activation function (the purpose is to transform the scores into weight probabilities of 0 to 1). According to exponential operations, all weight coefficients pointing to the future are absolutely forced to become 0. When feature aggregation is performed in the final step, features from future time moments are completely discarded because the weight multiplier is 0. The network can only use past and current features to understand the current state, thus achieving the blocking effect.

[0048] It is important to note that, in order to ensure that information is not lost and gradients are stable, the output of the previous layer is first added to the original input, then subjected to Layer Normalization, and finally used as the input of the next layer.

[0049] Existing pure visual diffusion strategy models are prone to assembly deadlock or workpiece damage due to "blindly applying force" when encountering visual occlusion or minor misalignment. Therefore, this embodiment innovatively introduces a temporal force feature extraction module with causal masking and a unidirectional cross-attention mechanism. Theoretically and structurally, when abnormal contact occurs at the assembly end, the causal Transformer can accurately extract high-frequency force abrupt change trends while blocking interference from future frames. This trend, as a query vector, directly guides the visual network to focus across modalities on the key force-bearing area. This structure endows the system with extremely strong physical correction capabilities, enabling the robot to spontaneously generate compliant yielding movements under complex conditions such as gear meshing and tight insertion, fundamentally eliminating the protective shutdown (deadlock) phenomenon caused by abnormal collisions, and improving the first-time assembly success rate and workpiece yield to industrial-grade usable standards.

[0050] Based on this, this embodiment innovatively introduces a temporal force perception feature extraction module with causal masking and a unidirectional cross-attention mechanism. From a theoretical and structural perspective, its specific physical correction and feature guidance mechanism is as follows: Firstly, regarding temporal blocking and trend extraction, when abnormal contact occurs at the assembly end (e.g., severe oscillation of tangential friction in the XY plane caused by jamming of connectors, or a nonlinear surge in Z-axis contact force caused by rigid collision within several consecutive control cycles), the system acquires a high-frequency sequence of multidimensional force / torque within that continuous time window. The causal Transformer module, through a preset lower triangular causal mask matrix (i.e., forcing the attention weights of future time steps corresponding to the upper triangular region of the matrix to zero), completely blocks the dependence of current-moment features on future frame information at the mathematical calculation level. Under this constraint, the network utilizes its internal multi-head self-attention mechanism to extract high-order temporal latent variables characterizing the aforementioned physical contact patterns such as "collision surge" or "jamming oscillation" by calculating the relative change gradient between the force at the current moment and the historical force sequence.

[0051] Secondly, regarding cross-modal data input and visual guidance, after causal extraction, the original force perception numerical sequence is encoded into a continuous feature vector containing the current collision intensity and historical force evolution patterns, i.e., causal force perception features. Subsequently, this causal force perception feature is directly input into the downstream unidirectional cross-attention network as a query vector; simultaneously, the high-dimensional visual feature map with spatial topology output from the visual feature extraction branch is synchronously input as a key / value pair vector. Inside the cross-attention module, the query vector containing the force abrupt change trend is used to perform a global inner product similarity score on the visual key vector. The above operations calculate the attention weight of the corresponding region based on the degree of force anomaly.

[0052] Through the above mechanism, once a sudden change in force occurs, the change trend vector (Query) immediately guides the visual network (Value) to focus the feature activation region across modalities on the key force area (such as the deformed gripper end or the misaligned workpiece edge). This structure endows the system with extremely strong physical correction capabilities, enabling the robot to spontaneously generate compliant retreat and position fine-tuning movements based on the collaborative mechanism of "force alarm-visual positioning" in complex working conditions such as gear meshing and tight insertion. This fundamentally eliminates the protective shutdown (deadlock) phenomenon caused by abnormal collisions, and improves the first-time assembly success rate and workpiece yield to industrial-grade usable standards.

[0053] S2.3: The extracted visual and force features are fused to obtain visual-force fusion features.

[0054] In one specific implementation, the visual-force feature fusion module achieves cross-modal deep fusion and joint understanding of state between vision and force. In contact assembly processes, the actual force points often occupy only a tiny area of ​​the global visual image, and visual images are high-dimensional spatial data, while force perception is low-dimensional temporal data. Existing conventional fusion methods, which simply concatenate these two heterogeneous features into vectors, struggle to establish a physical mapping relationship between force changes and the corresponding image regions. This embodiment utilizes an asymmetric cross-attention mechanism to "actively guide" the visual network to perform spatial scoring using abnormal changes in contact force. This allows the model to adaptively focus its attention on local image spatial regions highly correlated with the current force state (such as the gripper tip or workpiece edge), thereby achieving a complementary advantage of "force perception locking onto the timing and trend of force, and vision locking onto the force space and structure."

[0055] In a conventional symmetric self-attention mechanism, the query vector (Q), key vector (K), and value vector (V) all come from the same data source. However, the cross-attention mechanism in this embodiment is structurally asymmetric: the system mandates that the temporal features of force perception, after linear mapping, are used only as the query vector (Q), while the high-dimensional visual spatial features, after independent linear mapping, are used as the key vector (K) and value vector (V).

[0056] Therefore, under the above structure, the two modalities do not have equal physical status in the fusion process. Force features (Q) act as an active director, with their inherent "abnormal contact force mutation trend" serving as retrieval conditions; while visual features (K, V) act as a passive knowledge base. This mechanism utilizes force to scan and score the various spatial grids of the visual image, rather than having vision and force score each other in parallel.

[0057] This asymmetric mechanism executes a unidirectional spatial gating information flow. The calculation of the attention weight matrix is ​​dominated solely by force features, and the final weighted aggregation output falls on the spatial dimension of visual features (i.e., generating visual features weighted by force). This module does not include a reverse visual computation path to query force. This unidirectional, asymmetric flow design avoids the computational redundancy and extremely high latency caused by co-attention, while also effectively achieving the technical objective of "using the degree of force anomaly to lock in the visual attention area."

[0058] This embodiment designs a visual-mechanical feature fusion module comprising: an input feature projection linear layer, an N-layer multi-head cross-attention network, and an output projection linear layer. The fusion process specifically includes the following steps: S2.3.1: Perform independent linear layer mapping for visual features and force features respectively.

[0059] Specifically, the output of the force features after passing through a linear layer serves as the query vector (Q), while the visual spatial features... The outputs after passing through the linear layer are used as the key vector (K) and value vector (V), respectively.

[0060] S2.3.2: Utilize multi-head cross-attention mechanism for cross-modal attention calculation and feature weighted aggregation.

[0061] Specifically, Q, K, and V are input into an N-layer multi-head cross-attention network for cross-modal attention calculation and feature weighted aggregation. In this embodiment, N=1 is preferred to avoid modal collapse and greatly reduce computational latency, thus meeting the lightweight requirements of low-level high-frequency control.

[0062] S2.3.3: The aggregated features are then processed through a linear layer to obtain fused features.

[0063] Specifically, a final linear layer is added to output the final fusion features of visual and mechanical perception.

[0064] It is important to note that this embodiment differs from existing multimodal self-attention models such as ViT and Perceiver by using forced force perception as the query and vision as the key, achieving active guidance in a physically meaningful sense. Furthermore, spatial attention is introduced at the feature fusion stage, rather than simply merging at the input layer, overcoming the difficulty in establishing a physical mapping relationship between force changes and the corresponding image region. This embodiment also employs an asymmetric cross-attention mechanism, abandoning the reverse query of force perception by vision in exchange for real-time performance and causal consistency.

[0065] S2.4: Analyze the visual-mechanical fusion features and proprioceptive data using a diffusion-based robot assembly prediction model to generate flexible assembly instructions.

[0066] This embodiment uses a diffusion strategy learning module to generate end-to-end robot assembly actions based on multimodal fusion features, specifically including the following steps: S2.4.1: Construct and train a robot assembly prediction model based on diffusion learning.

[0067] In one specific implementation, this embodiment employs an end-to-end supervised learning paradigm to jointly train the robot assembly prediction model and the feature extraction process. During the training phase, the model uses mean squared error (MSE) as the core loss function. The specific training process and loss calculation method are as follows: S2.4.1.1: Forward Noise Addition and Prediction.

[0068] Specifically, the process begins by acquiring a sequence of real robot motion trajectories from an expert demonstration dataset, and then adding random Gaussian noise to these trajectories during the forward diffusion process. The noisy trajectory, along with the global conditional vector for the current period, is then input into the U-Net conditional denoising network. The network outputs either the predicted noise or the predicted original trajectory. The global conditional vector is a feature vector that integrates visual, force, and proprioceptive features.

[0069] S2.4.1.2: Loss Calculation and Mask Processing.

[0070] Specifically, the mean squared error (MSE) is used to calculate the difference between the noise predicted by the network and the actual Gaussian noise added, thus obtaining the base loss value. To adapt to demonstration trajectories of different lengths, the system masks invalid motion segments that are padded due to insufficient length, and only calculates the final mean MSE loss for valid motion segments.

[0071] S2.4.1.3: Joint optimization and gradient backpropagation.

[0072] Specifically, due to the introduction of unique visual-mechanical cross-modal fusion features into the global conditional vector, when calculating gradients through backpropagation using the MSE loss function, the error signal not only updates the weights of the U-Net itself but also propagates forward along the network structure to the visual feature extraction module, the mechanical feature extraction module, and the visual-mechanical feature fusion module. This end-to-end gradient backpropagation mechanism drives the network to spontaneously and implicitly learn the underlying physical laws: that is, under what mechanical aberration patterns should the visual focus be shifted to where, and what safe avoidance actions should be generated to minimize the loss of prediction noise.

[0073] S2.4.2: Combine visual features, visual-mechanical fusion features, and proprioceptive features.

[0074] In one specific implementation, in order to enhance visual features, global visual features are... The fusion features of visual and force perception, as well as the proprioceptive features preprocessed by minimum-maximum mapping, are sequentially spliced ​​horizontally in the dimension to construct a one-dimensional global conditional vector containing complete scene semantics and force state.

[0075] S2.4.3: Perform iterative denoising on the spliced ​​features to obtain the motion trajectory.

[0076] In one specific implementation, a pure random Gaussian noise sequence of the same length as the predicted field of view is first initialized by a noise module; then, it is input into a one-dimensional U-Net conditional denoising network. This network uses the aforementioned global conditional vector as a feature guide, and then... The process involves iterative denoising to gradually eliminate random noise, ultimately generating a smooth and complete motion trajectory within the prediction field of view.

[0077] Specifically, in the T-step iterative denoising process, the network uses the global conditional vector generated in the previous step, which includes the abnormal contact force trend and the visual focus area, as a bias feature and injects it into the network through the FiLM modulation layer. This conditional vector acts as a physical compliance boundary in each step of the denoising process.

[0078] Specifically, if the assembly process is in normal free space, the conditional vector guides the network to denoise along a directly aligned rigid insertion trajectory; if the current global conditional vector encodes abnormal force interaction features of end-effector jamming or nonlinear collision, U-Net will be strongly guided by these fused features during the denoising iteration, actively penalizing (removing) motion components that continue to exert rigid force along the original trajectory in the motion distribution space, and retaining or generating motion components that move towards the direction of less resistance (such as lateral sliding in the XY plane or small attitude deflection of the end effector).

[0079] Ultimately, after iteratively eliminating random noise, the network not only generates a mathematically smooth sequence of actions, but also a safe control trajectory that is highly consistent with the current assembly conditions in terms of physical semantics and has the ability to actively avoid obstacles and adapt to force, thus ensuring the smooth completion of complex and tight tolerance assembly tasks (such as gear meshing and shaft hole insertion).

[0080] S2.4.4: The action trajectory is extracted by action block segmentation to obtain the action block sequence as the action to be executed in the current control cycle.

[0081] In one specific implementation, a shorter preceding sequence (i.e., action block) is extracted from the complete long motion trajectory generated by U-Net as the action to be executed in the current control cycle.

[0082] This embodiment introduces diffusion denoising and action chunking mechanisms, and deeply collaborates with the pre-embedded visual-force feature fusion module. The high-frequency force-sensory cross-attention features of the pre-embedded module endow the system with extremely sensitive collision radar capabilities, while this module, through high-frequency rolling replanning (executing only a small segment of the predicted trajectory), ensures that once the system senses abnormal force during execution, the global conditional vector of the next frame can immediately guide the diffusion model to sculpt a new trajectory with compliant yielding properties, thereby completely solving the pain point of lacking real-time correction capabilities in long sequence predictions.

[0083] S2.4.4.1: Implicit identification of abnormal forces based on gradient activation mechanism.

[0084] Specifically, in this embodiment, the absolute force perception threshold set manually in traditional industrial control is abandoned, and data-driven implicit distribution recognition is adopted. In the end-to-end training stage, the causal Transformer has implicitly learned the Safe Force Envelope under normal assembly conditions from the expert demonstration data. During online inference, the causal Transformer calculates the temporal gradient (i.e., the first derivative of the contact force) and the fluctuation variance of the force perception sequence for the current continuous N time steps in real time. When the robotic arm has a slight misalignment or jamming, the temporal gradient of the force perception signal will mutate, causing the sequence to deviate from the learned safe envelope distribution in the implicit space of the causal Transformer. This deviation will immediately trigger the high-weight activation of the Query vector in the cross-attention network, thereby completing the recognition of abnormal force at the algorithm level and seamlessly encoding this alarm signal into the global conditional vector of the next frame.

[0085] S2.4.4.2: Intercept the action trajectory using the action interception rule based on Receding Horizon Control (RHC).

[0086] After identifying the anomaly and carving out a new action trajectory by the U-Net, the system strictly follows the Receding Horizon Control rule for action interception. Set the total length of the future action trajectory predicted by the diffusion model at one time to be H (Horizon), such as 16 steps, and the length of the action chunk actually intercepted and sent for execution by the system to be A (ActionChunk), such as 8 steps, and the rule of A < H must be satisfied. After the system finishes executing an action chunk of length A, it unconditionally discards the remaining H - A steps of data in the predicted trajectory and forces the system to collect the latest visual and force perception observation data to trigger the next round of H-step prediction. This sliding window interception rule with a long prediction and a short execution ensures that the control loop can refresh the action instructions at an extremely high frequency (such as the interception period is less than 0.1 seconds).

[0087] It should be particularly noted that to ensure the safety of the compliant yielding action and the stability of the robotic arm, the intercepted action chunk data must meet the following two core conditions: Temporal consistency condition: The time span corresponding to the length A of the intercepted action chunk must be less than or equal to the minimum effective response time window of the underlying force perception sensor. This ensures that before the current action chunk is executed, the system will never cause an increase in physical interference due to the execution of overly long stale predictions.

[0088] The first-order kinematic continuity condition states that the position and velocity components of the newly extracted motion block in the starting frame must maintain first-order continuity with the last frame of the motion extracted in the previous cycle. That is, when the U-Net denoises and generates a new trajectory, the body state features in the global conditional vector provide initial position anchor constraints, thereby ensuring that the underlying joint motors do not experience severe jitter or overshoot due to command step changes when the robotic arm performs compliant retreat or trajectory correction.

[0089] S2.4.5: The extracted action block sequence is restored to physical control commands through an inverse normalization operation.

[0090] In one specific implementation, this embodiment uses an inverse normalization operation to restore the extracted action block sequence into absolute physical control commands (such as joint angles or end-effector pose increments) that the robotic arm can recognize, and then sends them to the robot's underlying layer for execution. After the current block is completed, the remaining trajectory is immediately discarded, and multimodal data for the next cycle is re-acquired to enter the next round of inference loop.

[0091] In this embodiment, after dozens of denoising calculations using the U-Net diffusion model, the final action output by the network is a purely abstract numerical value within the range [-1, 1]. Post-processing (MIN_MAX inverse mapping) is then necessary; otherwise, the robot will have no idea how to execute the action.

[0092] Specifically, let's assume the abstract action output by the network is... (Between [-1, 1]) To convert to physical world velocity, position, or torque commands The formula is: .

[0093] Assuming the angle at which the gripper closes is controlled, the extreme value is set as follows: If the network outputs a predicted value of "yielding" one step under abnormal stress conditions... Substituting into the formula, we get... This completes the translation from abstract AI signals to precise physical instructions.

[0094] It is important to note that during assembly, the stress points often occupy only a very small area of ​​the overall visual image. Visual images are high-dimensional, low-frequency spatial data, while force perception is low-dimensional, high-frequency temporal data. Simply concatenating these two heterogeneous features into vectors is insufficient to establish an inherent physical mapping between them. The key challenge in achieving deep cross-modal fusion lies in how to leverage abnormal changes in contact force to "actively guide" the visual network, enabling the model to adaptively focus its attention on local image spatial regions highly correlated with the current stress state (such as the gripper tip or workpiece edge). This achieves a complementary advantage where "force perception locks in the timing and trend of stress, while vision locks in the space and structure of stress."

[0095] To balance high-frequency real-time control with overall assembly efficiency and address the "large model, high latency" problem, existing multimodal robot basic large models (VLAs) suffer from extremely large parameter counts and inference latency typically exceeding hundreds of milliseconds. This forces robots to significantly reduce their operating speed during assembly to wait for model instructions, severely slowing down production cycle time (CT). This embodiment employs a minimalist architecture design, such as a preferred use of a low-layer (e.g., 2-layer) force-cause encoder and a single-layer (N=1) cross-attention fusion network, to achieve extremely high-density alignment of heterogeneous data (image space and force-sensory sequence) while minimizing additional computational overhead. Combined with an action chunking mechanism, the policy network in this embodiment can operate at a high-frequency rolling frequency of 100Hz (e.g., >50Hz). This allows the robot to maintain high-speed assembly movements while still possessing millisecond-level abnormal force response capabilities, significantly improving overall assembly efficiency without sacrificing safety.

[0096] S3: The robot performs assembly operations according to flexible assembly instructions.

[0097] Compared to traditional control methods, the method described in this embodiment reduces the engineering deployment cost of multimodal control systems and improves generalization capabilities. Traditional robotic multimodal assembly requires manually setting extremely complex visual servo rules and force impedance / admittance control parameters (such as manually adjusting stiffness and damping matrices), and recalibration is necessary during production changes, resulting in extremely high engineering deployment costs. This embodiment employs an end-to-end diffusion learning paradigm, using the mean squared error (MSE) loss function to jointly optimize predicted actions and expert demonstration trajectories. Global backpropagation of gradient signals enables the system to implicitly and automatically learn the complex nonlinear mapping between visual pixels, contact torque, and underlying joint movements. When faced with workpieces of different shapes or changes in ambient lighting, only a small amount of expert demonstration data for new scenarios is needed to quickly fine-tune the model, significantly reducing the debugging cycle and cross-task generalization cost of the multimodal assembly system.

[0098] Example 2: Embodiment 2 of the present invention provides a robot assembly control system that integrates vision, force, and proprioception, comprising: The data acquisition module is configured to acquire visual images, force data, and proprioceptive data during the robot assembly process and perform preprocessing. The data processing module is configured to extract features from visual images and force data respectively, fuse the extracted visual features and force features to obtain visual-force fusion features, and use a robot assembly prediction model based on diffusion learning to analyze the visual-force fusion features and proprioceptive data to generate flexible assembly instructions. Among them, based on a one-way cross attention mechanism, causal masking is introduced to extract features from force data. The assembly execution module is configured for the robot to perform assembly operations according to flexible assembly instructions.

[0099] Example 3: Embodiment 3 of the present invention provides a computer-readable storage medium storing a computer program adapted for loading by a processor and executing the steps of the robot assembly control method for the fusion of vision, force, and proprioception as described in Embodiment 1 of the present invention.

[0100] Example 4: Embodiment 4 of the present invention provides a computer device, the device comprising: A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program, which, when executed by the processor, implements the steps in the robot assembly control method for the fusion of vision, force, and proprioception as described in Embodiment 1 of the present invention.

[0101] Example 5: Embodiment 5 of the present invention provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps in the vision, force, and proprioception fusion robot assembly control method as described in Embodiment 1 of the present invention.

[0102] The steps and methods involved in Examples 2, 3, 4 and 5 above correspond to those in Example 1. For specific implementation methods, please refer to the relevant description section of Example 1.

[0103] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0104] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium that a computer can access or a data processing device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, an optical medium, or a semiconductor medium, etc.

[0105] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A robot assembly control method integrating vision, force perception, and proprioception, characterized in that, Includes the following steps: Acquire visual images, force data, and proprioceptive data during the robot assembly process, and perform preprocessing. Feature extraction is performed on visual images and force data respectively. The extracted visual features and force features are fused to obtain visual-force fusion features. A robot assembly prediction model based on diffusion learning is used to analyze the visual-force fusion features and proprioceptive data to generate flexible assembly instructions. Among them, based on a one-way cross attention mechanism, causal masking is introduced to extract features from force data. The robot performs assembly operations according to flexible assembly instructions.

2. The robot assembly control method integrating vision, force, and proprioception as described in claim 1, characterized in that, The specific steps of preprocessing include: The visual images are normalized using mean and standard deviation. The force perception data and proprioceptive data are subjected to minimum-maximum mapping normalization.

3. The robot assembly control method integrating vision, force, and proprioception as described in claim 1, characterized in that, A deep residual network is used as the backbone network to extract features from visual images. The feature maps output by the deep residual network are input into the first branch and the second branch in parallel for processing. The first branch is used to perform spatial dimension flattening operation, and the second branch is used to extract the coordinates of two-dimensional key points of salient features in the visual image using pooling operation.

4. The robot assembly control method integrating vision, force, and proprioception as described in claim 1, characterized in that, The specific steps for fusing the extracted visual and force features are as follows: Independent linear layer mappings are performed for visual features and force features respectively; A multi-head cross-attention mechanism is used for cross-modal attention computation and feature weighting aggregation; The aggregated features are then passed through a linear layer to obtain fused features.

5. The robot assembly control method integrating vision, force, and proprioception as described in claim 1, characterized in that, The specific steps for analyzing visual-mechanical fusion features and proprioceptive data using a robot assembly prediction model based on diffusion learning are as follows: Visual features, visual-mechanical fusion features, and proprioceptive features are spliced ​​together; The spliced ​​features are then subjected to iterative denoising to obtain the motion trajectory. The action trajectory is extracted by action block segmentation, and the resulting action block sequence is used as the action to be executed in the current control cycle. The extracted action block sequence is restored to physical control commands through an inverse normalization operation.

6. The robot assembly control method integrating vision, force, and proprioception as described in claim 1, characterized in that, An end-to-end supervised learning paradigm is used to jointly train the robot assembly prediction model and the feature extraction process.

7. A robot assembly control system integrating vision, force perception, and proprioception, characterized in that, include: The data acquisition module is configured to acquire visual images, force data, and proprioceptive data during the robot assembly process and perform preprocessing. The data processing module is configured to extract features from visual images and force data respectively, fuse the extracted visual features and force features to obtain visual-force fusion features, and use a robot assembly prediction model based on diffusion learning to analyze the visual-force fusion features and proprioceptive data to generate flexible assembly instructions. Among them, based on a one-way cross attention mechanism, causal masking is introduced to extract features from force data. The assembly execution module is configured for the robot to perform assembly operations according to flexible assembly instructions.

8. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the robot assembly control method that integrates vision, force, and proprioception as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded by a processor and executed as described in any one of claims 1-6, which is a robot assembly control method that integrates vision, force, and proprioception.

10. A computer device, characterized in that, include: A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program, which, when executed by the processor, implements the robot assembly control method that integrates vision, force, and proprioception as described in any one of claims 1-6.

Images