Robotic flexible adaptive assembly system based on multi-modal perception and digital twinning

Abstract

The present application relates to the technical field of automation assembly, and particularly relates to a robot flexible self-adaptive assembly system based on multi-modal perception and digital twinning, which comprises: a visual perception unit used to extract shaft hole edge features and feature definition; a force perception unit used to obtain force moment and signal fluctuation component; a twinning evaluation unit used to fuse the shaft hole edge features, the feature definition and the force moment through a digital twinning model, and output centering state data containing translation deviation, angle deviation, contact mode index, visual trust weight and force trust weight; and a control unit used to determine a compliance control parameter set with a non-diagonal stiffness term based on the centering state data, correct the compliance control parameter set according to the signal fluctuation component, and generate a control instruction to indirectly adjust assembly contact force and feed speed. The present application realizes accurate estimation of shaft hole assembly centering state and adaptive adjustment of compliance control parameters.

Description

Technical Field

[0001] This invention relates to the field of automated assembly technology, and in particular to a robotic flexible adaptive assembly system based on multimodal perception and digital twins. Background Technology

[0002] In the field of robotic automated assembly, precision shaft and hole assembly is a typical task in industrial scenarios such as electronics manufacturing and automotive parts assembly. As the manufacturing industry's requirements for production flexibility and product consistency continue to increase, traditional teach-and-playback and offline programming methods, lacking adaptability to environmental changes and workpiece variations, are no longer sufficient to meet the demands for high-precision and high-efficiency assembly. Therefore, guidance technologies based on external sensors have been widely researched and applied. Vision sensing is one of the most important methods, using industrial cameras or depth cameras to acquire workpiece images, extracting features, and estimating pose to guide the robot to complete initial positioning. However, vision systems are sensitive to factors such as lighting conditions, workpiece surface reflection, and partial occlusion, often limiting their measurement accuracy and stability in complex industrial environments. Another commonly used technology is force-guided assembly, which uses force sensors installed at the robot's end effector to detect contact forces in real time and apply control strategies, enabling the robot to adjust its posture after contact through force feedback, achieving passive alignment of the shaft and hole. While this method has some environmental adaptability, it lacks global spatial information, and when initial deviations are large, it is prone to repeated trials or even jamming, leading to extended assembly cycles.

[0003] In recent years, digital twin technology has been introduced into manufacturing systems, enabling the simulation, monitoring, and optimization of assembly processes by constructing high-fidelity models of physical entities in virtual space. However, most existing digital twin applications are limited to offline simulation or condition monitoring. Although assembly schemes combining vision and force perception have been reported, they mostly operate in a sequential switching manner or employ fixed adjustment strategies, failing to fully explore the complementarity and dynamic control potential of multimodal information, and the system's adaptive capabilities still need improvement.

[0004] Chinese Patent Publication No. CN118013838A discloses an intelligent flexible assembly method for 3C products, comprising: constructing an intelligent assembly production line; constructing a multimodal virtual environment based on the intelligent assembly production line; establishing a perception model including vision, touch, and depth in the multimodal virtual environment to collect multimodal teaching data; establishing a skill knowledge base based on the parsing of the multimodal teaching data; generating a primitive strategy sequence for operation tasks based on the skill knowledge base; and using a randomization method to transfer the primitive strategy sequence from the digital twin environment to the real environment for use by the robot in the intelligent assembly production line. Assembly of lines; it can be seen that the intelligent flexible assembly method for 3C products has the following problems: its multimodal perception is mainly used for the collection of teaching data and skill learning in a virtual environment, and the reliability of visual and force data is not evaluated and fused online in the actual assembly process, which leads to a significant decrease in the accuracy of shaft hole alignment state estimation when the perception conditions deteriorate; at the same time, the method relies on offline generated primitive strategy sequences and lacks adaptive adjustment of compliance control parameters based on real-time contact state and force signal fluctuations, making it difficult to cope with contact shape changes and various disturbances in the assembly process, and the system has insufficient assembly success rate and stability under complex working conditions. Summary of the Invention

[0005] To address this, the present invention provides a robot flexible adaptive assembly system based on multimodal perception and digital twins, which overcomes the problems of inaccurate state estimation and poor adaptability in robot assembly in the prior art.

[0006] To achieve the above objectives, the present invention provides a robot flexible adaptive assembly system based on multimodal perception and digital twin, comprising: A visual perception unit is used to acquire a visual image of the assembly area and extract the shaft hole edge features and feature sharpness from the visual image. The assembly execution unit includes a robotic arm module and an end effector module; A force sensing unit is disposed between the robotic arm module and the end effector module to acquire contact torque signals and extract force spin and signal fluctuation components from the torque signals. The twin evaluation unit is connected to the visual perception unit and the force perception unit respectively. It is used to determine the relative pose parameters of the shaft hole based on the edge features of the shaft hole, the feature sharpness, and the force rotation through a preset digital twin model, and output centering state data. The centering state data includes translational deviation, angular deviation, contact pattern index, visual trust weight, and force trust weight. A control unit, connected to the twin evaluation unit, the force sensing unit, and the assembly execution unit, is used to determine a set of compliance control parameters based on the alignment state data, and to correct the set of compliance control parameters according to the signal fluctuation components. Based on the corrected set of compliance control parameters, control commands are generated and output to the assembly execution unit to indirectly adjust the contact force and feed rate during the assembly process. The set of compliance control parameters includes a stiffness coefficient matrix with off-diagonal stiffness terms, damping characteristic parameters in a specified direction, and a reference force offset.

[0007] Furthermore, the visual perception unit acquires a grayscale image and a point cloud image of the shaft hole region, extracts the edge features of the shaft hole based on the grayscale image, extracts the normal vector features of the shaft hole end face based on the point cloud image, and calculates the feature sharpness based on the ratio of the number of effective edge points in the shaft hole edge features to the total number of extracted points.

[0008] Furthermore, the force sensing unit includes a six-dimensional force sensor, and the contact torque signal includes force components and torque components; the force sensing unit calculates the direction and line of action of the force vector based on the force components and torque components, as the force spinor; the force sensing unit performs high-pass filtering on the contact torque signal, and determines the signal fluctuation component based on the amplitude of the filtered contact torque signal.

[0009] Furthermore, the digital twin model in the twin evaluation unit includes a geometric kinematics sub-model and a contact mechanics sub-model; the geometric kinematics sub-model is determined based on the geometric data of the workpiece and the kinematic parameters of the robotic arm of the assembly execution unit; the contact mechanics sub-model is determined based on the elastic modulus and friction coefficient of the workpiece material.

[0010] Furthermore, the twin evaluation unit updates the relative pose parameters of the shaft hole based on the edge features of the shaft hole and the force rotation, and determines the translational deviation and the angular deviation based on the updated relative pose parameters of the shaft hole.

[0011] Furthermore, the twin evaluation unit determines the directions of the translational deviation and the angular deviation based on the comparison between the direction of the force spinor and the deviation direction estimated based on the edge features of the shaft hole; and determines the contact pattern index based on the higher-order statistical moments of the force spinor in the time series and the contact area displayed in the point cloud of the visual image.

[0012] Further, the twin evaluation unit acquires the feature sharpness and the expected force spin; compares the feature sharpness with a preset sharpness threshold; if the feature sharpness is greater than the preset sharpness threshold, then sets the visual trust weight to be greater than the force trust weight; if the feature sharpness is less than or equal to the preset sharpness threshold, then calculates the directional deviation angle and amplitude deviation rate between the force spin and the expected force spin, determines the force trust weight based on the directional deviation angle and the amplitude deviation rate, and reduces the visual trust weight.

[0013] Furthermore, the control unit determines the direction of action of the off-diagonal stiffness terms in the stiffness coefficient matrix based on the direction of the translational deviation, and determines the value of each off-diagonal stiffness term based on the magnitude of the translational deviation and the angular deviation.

[0014] Further, the control unit determines the base gain value of the damping characteristic parameter based on the contact mode index, and performs a weighted correction on the base gain value based on the visual trust weight and the force trust weight to obtain the gain of the damping characteristic parameter; and determines the reference force bias based on the contact mode index, the visual trust weight and the force trust weight.

[0015] Furthermore, the control unit acquires the signal fluctuation component and compares the signal fluctuation component with a preset fluctuation threshold; if the signal fluctuation component is greater than the preset fluctuation threshold, the damping coefficient corresponding to the assembly feed direction in the damping characteristic parameter is increased, and the rate of change of each element in the stiffness coefficient matrix is ​​decreased.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention acquires multimodal data through visual perception unit and force perception unit, uses digital twin model to fuse heterogeneous information online and generate centering state data containing deviation degree and trust weight. The control unit dynamically adjusts the set of compliant control parameters containing off-diagonal stiffness terms accordingly, and indirectly controls the assembly contact force and feed speed. This overcomes the problems of inaccurate centering state estimation and poor environmental adaptability of existing robot assembly, realizes flexible adaptive control of shaft and hole precision assembly, and effectively improves the assembly success rate and process stability.

[0017] Furthermore, this invention extracts feature sharpness and dynamically allocates trust weights for visual and force perception based on its values, automatically enhancing the force perception guidance weights when illumination changes or occlusion occurs, thus ensuring the stability of centering state estimation under different working conditions.

[0018] Furthermore, this invention integrates geometric kinematics and contact mechanics constraints through a digital twin model, unifying visual edge features and force spindles into a common state space, thereby improving the physical consistency and accuracy of the estimation of the relative pose parameters of the shaft and hole.

[0019] Furthermore, this invention determines the contact mode index by jointly using the higher-order statistical moments of the force spinor and the contact area of ​​the point cloud, providing an effective identification of the current contact state for impedance control and avoiding mismatch of control parameters due to contact mode switching.

[0020] Furthermore, this invention directly determines the magnitude and direction of the off-diagonal stiffness term by means of translational deviation and angular deviation, thereby enabling the stiffness matrix to generate a directional translational-rotational coupling correction effect, which accelerates the assembly alignment convergence process.

[0021] Furthermore, by comparing the force signal fluctuation component with a preset threshold, the present invention increases the damping in the feed direction and limits the rate of stiffness change when the force fluctuation exceeds the limit, effectively suppressing contact vibration and parameter mutation during the assembly process.

[0022] Furthermore, this invention uses a contact mode index and a trust weight to weight and correct the damping gain and reference force bias, so that the compliance control parameters are matched with the perceived reliability and physical contact state in real time, thereby improving the system's adaptive capability. Attached Figure Description

[0023] Figure 1 This is a connection block diagram of the robot flexible adaptive assembly system based on multimodal perception and digital twin of the present invention; Figure 2 This is the overall logic block diagram of the robot flexible adaptive assembly system based on multimodal perception and digital twin of the present invention; Figure 3 This is a logic block diagram of the twin evaluation unit of the robot flexible adaptive assembly system based on multimodal perception and digital twin of the present invention; Figure 4 This is a logic block diagram of the control unit of the robot flexible adaptive assembly system based on multimodal perception and digital twin of the present invention. Detailed Implementation

[0024] To make the objectives and advantages of the present invention clearer, the present invention will be further described below with reference to embodiments; it should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0025] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0026] It should be noted that in the description of this invention, the terms "upper", "lower", "left", "right", "inner", "outer", etc., which indicate directions or positional relationships, are based on the directions or positional relationships shown in the accompanying drawings. This is only for the convenience of description and is not intended to indicate or imply that the device or element must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation of this invention.

[0027] Furthermore, it should be noted that, in the description of this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0028] Please see Figure 1 and Figure 2 The diagrams shown are, respectively, a connection block diagram and an overall logic block diagram of the robot flexible adaptive assembly system based on multimodal perception and digital twin of the present invention; the present invention provides a robot flexible adaptive assembly system based on multimodal perception and digital twin, comprising: A visual perception unit is used to acquire a visual image of the assembly area and extract the shaft hole edge features and feature sharpness from the visual image. Specifically, the visual perception unit acquires a grayscale image and a point cloud image of the shaft hole region, extracts the edge features of the shaft hole based on the grayscale image, extracts the normal vector features of the shaft hole end face based on the point cloud image, and calculates the feature sharpness based on the ratio of the number of effective edge points in the shaft hole edge features to the total number of extracted points.

[0029] In one specific embodiment, the visual perception unit includes an RGB-D camera to simultaneously acquire grayscale and depth images of the assembly area. The depth image is converted into 3D point cloud data using camera intrinsic parameters. The visual perception unit converts the grayscale image into a single-channel grayscale image and defines a rectangular region of interest (ROI) within the image to delineate the image range of the shaft hole end face. The Canny edge detection operator is applied to this ROI to extract all edge points, the number of which is recorded as the total number of extracted points. Subsequently, connected component labeling and shape filtering are performed on the extracted edge points: connected components with a length lower than a preset length threshold are deleted, and edge segments with local orientation consistency lower than a preset angle threshold are removed. The edge points retained after the above filtering are considered valid edge points, and their number is recorded as the number of valid edge points. The visual perception unit calculates the ratio of the number of valid edge points to the total number of extracted points, and uses this ratio as the feature sharpness. Simultaneously, in the 3D point cloud, the visual perception unit extracts a local point cloud subset based on the spatial range corresponding to the ROI, and uses a random sampling consistency algorithm to fit the plane containing the shaft hole end face, outputting the unit normal vector of this plane as the normal vector feature of the shaft hole end face.

[0030] In this embodiment, the feature sharpness value is between 0 and 1. The higher the feature sharpness value, the greater the proportion of identifiable true edges of the shaft holes in the image among all extracted edges, and the higher the reliability of the visual perception data; the lower the feature sharpness value, the greater the proportion of noisy edges or broken edges in the extraction result, and the lower the visual extraction quality.

[0031] Feature clarity provides a quantitative basis for dynamically adjusting the trust weights of visual and force perception in subsequent multimodal fusion. Specifically, parameters such as the high and low thresholds, connected component length threshold, and orientation consistency angle threshold of the Canny edge detection operator can be adjusted offline based on the actual camera resolution and workpiece edge imaging features. In alternative implementations, a deep learning-based edge detection network can be used to replace the Canny operator, or an adaptive thresholding method can be used to automatically determine edge detection parameters based on local image contrast.

[0032] Understandably, in shaft-hole assembly scenarios, genuine part edges possess continuous, smooth, and directional structural properties, while pseudo-edges caused by oil stains, reflections, shadows, or imaging noise exhibit chaotic and discontinuous spatial distribution. By employing dual criteria of connected component length filtering and directional consistency filtering, structural edges can be effectively distinguished from interference signals. When the assembly environment has good lighting and the workpiece surface condition is ideal, the vast majority of edge points detected belong to the genuine shaft-hole contour. After filtering, the number of effective edge points is very close to the total number of extracted points, and the calculated feature sharpness approaches 1. If local occlusion or strong reflection occurs on-site, genuine edges are broken or weakened in the image, and some originally continuous edges are fragmented into short segments, which are then eliminated in length filtering. Consequently, the ratio of effective edge points to the total number of extracted points decreases, reducing feature sharpness. Through this change in ratio, the system can perceive the confidence level of the visual channel in real time without external reference or manual intervention, providing feature support for dynamically adjusting the trust weights of visual and force perception in multimodal fusion.

[0033] The assembly execution unit includes a robotic arm module and an end effector module; In one specific embodiment, the assembly execution unit includes a robotic arm module consisting of a six-axis industrial robot, and a pneumatic parallel gripper module mounted on the robot's end flange as an end effector module. The gripper module's gripper has adjustable stroke and is adapted to the geometry of the workpiece to be assembled, providing a stable and reliable gripping force. The motion control of the robotic arm module is completed by the robot's built-in controller. The controller receives the end-effector target pose command from the control unit and drives the joints to move to perform the assembly task through built-in inverse kinematics and joint space trajectory planning algorithms.

[0034] It is understandable that this embodiment uses a conventional industrial robot as the execution platform for assembly actions, aiming to provide the system with a high-precision, repeatable position control foundation. At the same time, its open end-effector interface facilitates the integration of force sensing units and various end-effector tools, enabling subsequent compliant control strategies to indirectly change the actual contact force and feed speed by adjusting the target pose.

[0035] A force sensing unit is disposed between the robotic arm module and the end effector module to acquire contact torque signals and extract force spin and signal fluctuation components from the torque signals. Specifically, the force sensing unit includes a six-dimensional force sensor, and the contact torque signal includes force components and torque components; the force sensing unit calculates the direction and line of action of the force vector based on the force components and torque components, as the force spinor; the force sensing unit performs high-pass filtering on the contact torque signal, and determines the signal fluctuation component based on the amplitude of the filtered contact torque signal.

[0036] In one specific embodiment, the force sensing unit includes a six-dimensional force sensor, which is rigidly mounted between the end flange of the robotic arm module and the end effector module. The six-dimensional force sensor continuously outputs force components in three orthogonal directions and torque components around three orthogonal axes at a preset sampling frequency. The force sensing unit processes the above force and torque components to extract information in two dimensions: one is the force spinor, which describes the spatial distribution characteristics of the contact force; the other is the signal fluctuation component, which describes the degree of change of the contact force over time.

[0037] The force spinor is extracted as follows: the direction of the force vector synthesized from the three force components is taken as the direction of action of the force spinor; the position of the line of action of the force in the sensor coordinate system is determined by the ratio of the torque component to the force component in the principal force direction, which serves as the characterization of the line of action of the force spinor. The calculation of the position of the line of action of the force is based on the simplified relationship of point load in the force spinor theory, which can provide the spatial orientation information of the contact point in the end coordinate system with low computational overhead. The direction of action of the force spinor indicates the main force direction of the assembly contact force, and the position of the line of action of the force reflects the approximate spatial orientation of the actual contact point between the shaft and the hole. Together, they constitute the spatial geometric information describing the current contact state.

[0038] The signal fluctuation component is extracted as follows: The force sensing unit performs high-pass filtering on each axial force component signal to filter out low-frequency trend components caused by factors such as gravity and slow off-center loading, while retaining high-frequency oscillation components caused by local collisions, frictional slippage, or unstable contact. Subsequently, the root mean square value of each axial force signal within a sliding time window after filtering is taken as the fluctuation amplitude of that axis. The combined result of the fluctuation amplitudes of each axis is determined as the signal fluctuation component. The cutoff frequency of the high-pass filter and the length of the sliding time window can be adjusted according to the actual assembly speed and sensor noise characteristics.

[0039] Understandably, the amplitude of the signal fluctuation component indirectly reflects whether there are abnormal dynamics such as contact impact, edge collision, or impending jamming during the assembly process. The larger the amplitude of the fluctuation component, the stronger the high-frequency oscillation of the contact force, and the more unstable the current contact state; the smaller the amplitude of the fluctuation component, the more gradual the change in contact force, and the relatively stable the contact process.

[0040] Understandably, the force sensing unit's dual-dimensional signal processing aims to simultaneously acquire the static spatial characteristics and dynamic stability characteristics of the contact state. Force spinor extraction focuses on the spatial distribution attributes of the contact force vector, while signal fluctuation component extraction focuses on the drastic changes in the contact force over time. These two dimensions of information are independent yet complementary: the force spinor provides directionality for deviation estimation in the subsequent twin evaluation unit, while the signal fluctuation component provides the control unit with a reference for the timing and amplitude of real-time compensation of compliance control parameters. When the signal fluctuation component exceeds a preset fluctuation threshold, it indicates instability in the contact process. Based on this, the control unit corrects the compliance control parameters to suppress fluctuations and restore stable contact.

[0041] The twin evaluation unit is connected to the visual perception unit and the force perception unit respectively. It is used to determine the relative pose parameters of the shaft hole based on the edge features of the shaft hole, the feature sharpness, and the force rotation through a preset digital twin model, and output centering state data. The centering state data includes translational deviation, angular deviation, contact pattern index, visual trust weight, and force trust weight. Please continue reading. Figure 3 As shown, it is a logic block diagram of the twin evaluation unit of the robot flexible adaptive assembly system based on multimodal perception and digital twin of the present invention. Specifically, the digital twin model in the twin evaluation unit includes a geometric kinematics sub-model and a contact mechanics sub-model; the geometric kinematics sub-model is determined based on the workpiece's geometric data and the mechanical arm kinematic parameters of the assembly execution unit; the contact mechanics sub-model is determined based on the workpiece's material elastic modulus and friction coefficient.

[0042] In one specific embodiment, the digital twin model is implemented as a software module within the twin evaluation unit, including a geometric kinematics sub-model and a contact mechanics sub-model. The geometric kinematics sub-model is established using the robot's forward kinematics method, based on the workpiece's geometric data such as the nominal dimensions, tolerance range, and chamfer geometry parameters of the shaft and hole to be assembled, as well as the kinematic parameters of the robotic arm's joints in the assembly execution unit, including the lengths, torsion angles, and offsets. It can calculate the spatial pose of the shaft and hole in the reference coordinate system based on real-time joint angle feedback. The contact mechanics sub-model, based on the workpiece's material elastic modulus and friction coefficient, combined with Hertzian point contact mechanics formulas and the Coulomb friction model, calculates the corresponding desired contact force spindle under given shaft-hole relative pose parameters.

[0043] The geometric data of the workpiece is directly derived from the part drawings or CAD models; the kinematic parameters of the robotic arm are taken from the DH parameter table in the robot product manual; and the elastic modulus and friction coefficient of the material are obtained by looking up tables in the standard mechanics of materials handbook.

[0044] Understandably, the geometric kinematics sub-model provides the geometric reference for the relative pose parameters of the shaft and hole in the twin evaluation unit, while the contact mechanics sub-model provides the mapping relationship from relative pose to force response. Together, they enable the digital twin to simultaneously reflect the motion constraints and mechanical constraints in the assembly process.

[0045] Understandably, the digital twin model employs a two-sub-model approach to differentiate between geometric and physical constraints during assembly. This allows pose estimation to obtain accurate spatial references based on robot kinematics and workpiece geometry data, while also incorporating physical causes of force signals based on material contact characteristics. The geometric kinematics sub-model ensures that the system clearly defines the relative relationship between the shaft and the hole in space, regardless of the robot's movement; the contact mechanics sub-model enables the system to predict the direction and magnitude of the contact force that should occur under given pose deviations. This invention allows subsequent deviation calculations and trust weight adjustments to simultaneously consider geometric consistency and physical rationality, thereby improving the reliability of centering state estimation.

[0046] Specifically, the twin evaluation unit updates the relative pose parameters of the shaft hole based on the edge features of the shaft hole and the force rotation, and determines the translational deviation and the angular deviation based on the updated relative pose parameters of the shaft hole.

[0047] Specifically, the twin evaluation unit determines the directions of the translational deviation and the angular deviation based on the comparison between the direction of the force spinor and the deviation direction estimated based on the edge features of the shaft hole; and determines the contact pattern index based on the higher-order statistical moments of the force spinor in the time series and the contact area displayed in the point cloud of the visual image.

[0048] Specifically, the twin evaluation unit acquires the feature sharpness and the expected force spin; compares the feature sharpness with a preset sharpness threshold; if the feature sharpness is greater than the preset sharpness threshold, then sets the visual trust weight to be greater than the force trust weight; if the feature sharpness is less than or equal to the preset sharpness threshold, then calculates the directional deviation angle and amplitude deviation rate between the force spin and the expected force spin, determines the force trust weight based on the directional deviation angle and the amplitude deviation rate, and reduces the visual trust weight.

[0049] In one specific embodiment, the twin evaluation unit generates visual observations and force observations based on the received shaft hole edge features and force rotation.

[0050] The visual observations are generated as follows: based on the edge features of the shaft hole, the translational deviation and angular deviation of the shaft hole axis are calculated based on visual estimation through edge matching and pose inverse calculation algorithms.

[0051] The force perception observations are generated as follows: based on the direction of the force spinor and the position of the force line of action, the translational and angular deviations based on the force perception estimate are calculated in reverse using the contact mechanics sub-model in the digital twin model. During the reverse calculation process, the contact mechanics sub-model matches the current force spinor with the mapping relationship between the relative pose parameters of the shaft and hole and the contact force in the digital twin model, determining the force perception observations through physical constraints.

[0052] The twin assessment unit performs a weighted fusion of visual and force observations to update translational and angular deviations. The weighted fusion method is as follows: feature sharpness is used as the weight for visual estimation, and the complement of feature sharpness relative to 1 is used as the weight for force estimation. The translational deviations of the visual and force estimates are weighted and summed to obtain the translational deviation, and the angular deviations of the visual and force estimates are weighted and summed to obtain the angular deviation.

[0053] Understandably, the purpose of this weighted fusion method is to ensure that when feature sharpness is high, translational and angular deviations rely more on visual estimation results to fully utilize the accuracy advantage of vision in spatial positioning. When feature sharpness decreases due to factors such as changes in illumination, occlusion, or reflection, translational and angular deviations automatically transition to force estimation results to maintain the continuity of centering state estimation. Feature sharpness is directly used as a fusion weight here, achieving a smooth switch from vision-dominated to force-dominated, avoiding the state estimation jumps that may be caused by threshold-based hard switching.

[0054] The rule for determining the deviation direction is as follows: compare the angle between the direction vector of the force spinor and the deviation direction vector estimated based on the edge features of the shaft hole. If the angle is less than 90°, it indicates that the contact direction perceived by the force is basically consistent with the deviation direction estimated by the vision in space, and the direction of the translational deviation is taken as the direction estimated by the vision; if the angle is greater than or equal to 90°, it indicates that there is a significant contradiction between the direction perceived by the force and the direction estimated by the vision, and the opposite direction of the force spinor is taken as the direction of the translational deviation.

[0055] It is understandable that when visual and force perception make the same judgment about the direction of deviation, the spatial positioning of vision can be trusted. When the two contradict each other, it usually means that the visual estimation is interfered with. For example, ambiguous matching of symmetrical edges leads to misjudgment of direction. In this case, the actual contact direction reflected by the force perception signal is more physically reliable. Therefore, the force perception signal is used to correct the direction judgment.

[0056] The twin evaluation unit also determines the contact mode index based on the higher-order statistical moments of the force spinor over time and the contact area displayed in the point cloud of the visual image. Specifically, the twin evaluation unit extracts the temporal signal of the force spinor within a preset sliding time window and calculates the skewness and kurtosis of the temporal signal. Skewness reflects the degree of asymmetry in the force signal distribution; the larger the absolute value of the skewness, the more uneven the distribution of the force signal in the positive and negative directions, usually corresponding to unilateral contact or asymmetric contact. Kurtosis reflects the sharpness of the force signal distribution; the greater the kurtosis exceeds the reference value corresponding to the normal distribution, the more peak pulse components there are in the force signal, usually corresponding to multi-point contact or intermittent collision. At the same time, the twin evaluation unit uses the point cloud data output by the visual perception unit to analyze the contact area of ​​the shaft-hole end face. It determines the distribution of contact points by the distance between each point in the point cloud and the fitted plane of the shaft-hole end face, and calculates the spatial area covered by the contact points as the contact area. The contact mode index is determined by a weighted average of three factors: the absolute value of skewness, the excess kurtosis, and the ratio of the contact area to the nominal area of ​​the shaft end face. Each weighting coefficient is determined through calibration based on the statistical and spatial geometric characteristics of the force signal under known contact modes.

[0057] It is understandable that there is a correlation between the magnitude of the contact mode index and the contact state as follows: a low contact mode index corresponds to a single-point contact state, where there is only one main contact point between the shaft and the hole; a medium contact mode index corresponds to a multi-point contact state, where there are two or more discrete contact points between the shaft and the hole; and a high contact mode index corresponds to a surface contact state, where a large area of ​​contact has been formed between the mating surfaces of the shaft and the hole. This contact mode index provides an effective identification of the current contact state for the subsequent control unit to select matching compliance control parameters, enabling the system to adjust the control strategy according to changes in the contact pattern and avoid assembly failures caused by mismatch between control parameters and the contact state.

[0058] In an alternative implementation, the calculation of higher-order statistical moments of the force signal can further include cumulants of order four or higher to more finely characterize the tail features of the force signal distribution. In another alternative implementation, the extraction of the contact area can employ an edge detection method based on the depth discontinuity of the point cloud instead of a plane fitting method. In yet another alternative implementation, the contact pattern index can be replaced by a pre-trained classifier that directly outputs the contact pattern category based on skewness, kurtosis, and contact area, instead of a weighted summation-based exponential representation.

[0059] The twin evaluation unit also determines visual trust weights and force-feel trust weights based on feature sharpness and expected force spin. The expected force spin is calculated in the forward direction by the contact mechanics sub-model in the digital twin model based on the currently estimated shaft-hole relative pose parameters, representing the contact force spin predicted by the physical model under the current estimated pose deviation. The logic for determining the trust weights is as follows: Feature sharpness is compared with a preset sharpness threshold. If the feature sharpness is greater than the preset sharpness threshold, it indicates that the quality of the visual perception data meets the reliability requirements. In this case, the visual trust weight is set to a higher value, and the force-feel trust weight is set to a lower value, with the system using the visual channel as the primary driver for centering state estimation. If the feature sharpness is less than or equal to the preset sharpness threshold, it indicates that the quality of the visual perception data has deteriorated due to environmental interference. In this case, the directional deviation angle and amplitude deviation rate between the actual force spin and the expected force spin are calculated. The directional deviation angle is the spatial angle between the direction of action of the actual force spin and the direction of action of the expected force spin, reflecting the degree of conformity between the force-feel signal and the physical model prediction in the directional dimension. The amplitude deviation rate is the ratio of the absolute value of the difference between the actual force spinor amplitude and the expected force spinor amplitude to the expected force spinor amplitude, reflecting the degree of agreement between the force sensor signal and the physical model prediction in the intensity dimension. Based on the orientation deviation angle and the amplitude deviation rate, a nonlinear mapping function is used to determine the force sensor trust weight. This mapping function has the following characteristics: when both the orientation deviation angle and the amplitude deviation rate are small, the force sensor trust weight approaches 1, indicating a high degree of consistency between the force sensor signal and the digital twin physical prediction, and a highly reliable force sensor channel; as the orientation deviation angle or the amplitude deviation rate increases, the force sensor trust weight smoothly decreases, approaching 0, to limit the impact of unreliable force sensor information on subsequent control. The visual trust weight is taken as the complement of the force sensor trust weight relative to 1. The preset sharpness threshold is determined based on the effective edge recognition rate under typical lighting conditions. The steepness and offset of the nonlinear mapping function are calibrated based on the directional sensitivity and amplitude accuracy of the force sensor.

[0060] Understandably, when visual conditions are good, the system prioritizes the spatial localization capabilities of vision to quickly converge to the centering state. When visual conditions deteriorate, the system switches to relying on force signals for centering estimation, but it does not unconditionally trust force. Instead, it verifies the physical plausibility of the force signals by comparing them with the expected force spin predicted by the digital twin model. Force signals are only assigned a high trust weight when the actual force spin matches the physical model prediction. Thus, when vision fails, the system does not blindly rely on force signals that may also be affected by interference. Instead, it uses the digital twin model as a benchmark for physical consistency verification, achieving a closed-loop evaluation of the reliability of the force channel.

[0061] In an alternative implementation, the nonlinear function mapping the orientation deviation angle and amplitude deviation rate to the force perception trust weight can take different monotonically decreasing S-shaped function forms, as long as it satisfies the qualitative trend that the smaller the deviation, the higher the trust. In another alternative implementation, when the feature sharpness is greater than a preset sharpness threshold, the specific values ​​of the visual trust weight and the force perception trust weight can be finely adjusted according to the continuous change of feature sharpness, rather than being fixed to preset values, so that the weights change continuously with visual quality.

[0062] Understandably, by using the geometric kinematics sub-model and the contact mechanics sub-model in the digital twin model, the geometric features extracted from vision and the contact force information perceived by force perception are uniformly mapped onto the common state space of the relative pose parameters of the shaft and hole, thus solving the problem of fusion of heterogeneous sensor data at the physical representation level. Secondly, the pose deviation weighted fusion strategy with feature clarity as the weight is essentially a feedforward trust adjustment, ensuring that the system converges quickly when visual conditions are good and smoothly transitions to force guidance when visual conditions deteriorate. Thirdly, the introduction of the contact mode index enables the system to recognize the contact phase. This index integrates the statistical characteristics of force signals and the geometric characteristics of visual space, and can effectively distinguish different physical processes such as single-point contact, multi-point contact, and surface contact, providing a basis for targeted adjustment of the compliance control strategy. Finally, the calculation of trust weights forms a closed-loop evaluation of the quality of perceived data. By comparing the actual force signals with the expected signals predicted by the digital twin, the credibility of the force channel is quantified in real time. Visual clarity is used as a priori reference, so that the weight allocation of vision and force not only reflects the changes in environmental conditions, but also considers the physical consistency of the model prediction, thereby ensuring the stability and accuracy of the centering deviation estimation under various interference factors.

[0063] A control unit, connected to the twin evaluation unit, the force sensing unit, and the assembly execution unit, is used to determine a set of compliance control parameters based on the alignment state data, and to correct the set of compliance control parameters according to the signal fluctuation components. Based on the corrected set of compliance control parameters, control commands are generated and output to the assembly execution unit to indirectly adjust the contact force and feed rate during the assembly process. The set of compliance control parameters includes a stiffness coefficient matrix with off-diagonal stiffness terms, damping characteristic parameters in a specified direction, and a reference force offset.

[0064] Please continue reading. Figure 4 As shown, it is a logic block diagram of the control unit of the robot flexible adaptive assembly system based on multimodal perception and digital twin of the present invention. Specifically, the control unit determines the direction of action of the off-diagonal stiffness terms in the stiffness coefficient matrix based on the direction of the translational deviation, and determines the value of each off-diagonal stiffness term based on the magnitude of the translational deviation and the angular deviation.

[0065] Specifically, the control unit determines the base gain value of the damping characteristic parameter based on the contact mode index, and performs a weighted correction on the base gain value based on the visual trust weight and the force trust weight to obtain the gain of the damping characteristic parameter; and determines the reference force bias based on the contact mode index, the visual trust weight and the force trust weight.

[0066] Specifically, the control unit acquires the signal fluctuation component and compares the signal fluctuation component with a preset fluctuation threshold; if the signal fluctuation component is greater than the preset fluctuation threshold, the damping coefficient corresponding to the assembly feed direction in the damping characteristic parameter is increased, and the rate of change of each element in the stiffness coefficient matrix is ​​decreased.

[0067] In one specific embodiment, the control unit obtains alignment status data from the twin evaluation unit, including translational deviation, angular deviation, contact pattern index, visual trust weight, and force trust weight, and obtains signal fluctuation components from the force sensing unit. Based on the above data, the control unit determines a compliance control parameter set, which includes a stiffness coefficient matrix with off-diagonal stiffness terms, damping characteristic parameters in a specified direction, and a reference force offset. The control unit generates control commands based on the compliance control parameter set and outputs them to the assembly execution unit to indirectly adjust the contact force and feed rate during the assembly process.

[0068] The process of determining the set of compliant control parameters by the control unit includes three aspects: determining the off-diagonal stiffness terms in the stiffness coefficient matrix, determining the damping characteristic parameters and the reference force bias, and correcting the above parameters according to the signal fluctuation components.

[0069] In determining the off-diagonal stiffness terms, the control unit determines the direction of action of the off-diagonal stiffness terms based on the direction of the translational deviation, and determines the value of each off-diagonal stiffness term based on the magnitude of the translational deviation and the magnitude of the angular deviation.

[0070] Specifically, the control unit maps the direction vector of the translational deviation and the rotational axis of the angular deviation to the cross-coupling position between the translational and rotational degrees of freedom in the stiffness coefficient matrix. The sign of the off-diagonal stiffness term is set opposite to the deviation direction, so that the coupling force or torque generated by the off-diagonal stiffness term points in the direction of reducing deviation. The value of the off-diagonal stiffness term is determined jointly by the translational and angular deviations: the larger the translational deviation, the larger the value of the off-diagonal stiffness term; the larger the angular deviation, the larger the value of the off-diagonal stiffness term. When both the translational and angular deviations are small, the value of the off-diagonal stiffness term is reduced accordingly to weaken the coupling correction effect and avoid over-adjustment.

[0071] It is understandable that during the shaft-hole assembly alignment process, translational deviation and angular deviation are physically coupled: the tilt of the shaft changes the contact point position, and the eccentricity of the contact force generates a torque. Traditional impedance control typically uses a diagonal stiffness matrix, decoupling the degrees of freedom, and cannot actively utilize this physical coupling to accelerate alignment convergence. This embodiment introduces off-diagonal terms into the stiffness coefficient matrix, allowing translational deviation to trigger a corrective torque in the rotational direction, and angular deviation to trigger a corrective force in the translational direction. This synergistic corrective effect of translation and rotation allows the shaft to autonomously adjust its posture according to the hole's constraints during insertion, rather than relying solely on the position correction of a single degree of freedom. The value of the off-diagonal stiffness term dynamically adjusts with the deviation; a larger deviation enhances the corrective coupling effect, while a smaller deviation automatically weakens the coupling effect, preventing oscillations caused by excessive coupling correction near the alignment position.

[0072] In determining the damping characteristic parameters and the reference force bias, the control unit determines the base gain value of the damping characteristic parameters based on the contact mode index, and then weights and corrects the base gain value according to the visual trust weight and the force trust weight to obtain the gain of the damping characteristic parameters. The damping coefficient in the feed direction is taken as this gain. Simultaneously, the control unit determines the amplitude of the reference force bias based on the contact mode index, the visual trust weight, and the force trust weight, with the direction of the reference force bias taken as along the assembly feed direction.

[0073] Specifically, there is a pre-defined correspondence between the contact mode index and the base gain value: when the contact mode index indicates single-point contact, the base gain value is at a relatively low level. At this time, there is only one main contact point between the shaft and the hole, the contact constraint is weak, and a small damping is sufficient to meet the stability requirements; when the contact mode index indicates multi-point contact, the base gain value is at a medium level. At this time, there are multiple discrete contact points, and a larger damping is required to dissipate the contact impact energy; when the contact mode index indicates surface contact, the base gain value is at a relatively high level. At this time, a large area of ​​contact has been formed between the mating surfaces, the risk of friction and jamming increases, and a larger damping is required to ensure the smoothness of the feed.

[0074] The gain of the damping characteristic parameter is obtained after weighted correction of the base gain value. The weighted correction method is to multiply the base gain value by a weighted combination of visual trust weight and force trust weight. The contributions of the visual trust weight and force trust weight in the weighted combination are adjusted by corresponding correction factors: a damping correction factor under visual guidance and a damping correction factor under force guidance. A higher visual trust weight indicates that the system is currently using vision as the primary method for centering estimation. In this case, the damping gain is appropriately increased to enhance motion stability under visual guidance. A higher force trust weight indicates that the system is currently using force as the primary method for centering estimation. In this case, the damping gain is kept at a moderate level to ensure the responsiveness of force feedback. When both the visual trust weight and the force trust weight are low, it indicates that the reliability of both perception channels is insufficient. In this case, the weighted damping gain automatically tends to be conservative, reducing assembly speed in exchange for safety.

[0075] The amplitude of the reference force bias is determined based on the contact mode index and the trust weight. A larger contact mode index results in a larger reference force bias amplitude, providing sufficient auxiliary insertion force to overcome the frictional resistance caused by the increased contact area. A larger proportion of the force-feed trust weight in the sum of the visual and force-feed trust weights indicates a greater reliance on the force-feed channel, and the amplitude of the reference force bias increases accordingly to actively apply an auxiliary force along the feed direction to guide the shaft into the hole. When the force-feed trust weight is relatively small, the amplitude of the reference force bias decreases accordingly to avoid applying inappropriate auxiliary force when force-feed is unreliable.

[0076] Understandably, by using a contact mode index and trust weights to weight and correct the damping gain and reference force bias, the compliance control parameters are matched in real time with perceived reliability and physical contact status. Under single-point contact and visual reliability, the system feeds rapidly with low damping; under surface contact and force-sensory reliability, the system inserts smoothly with higher damping and a larger auxiliary force; under deteriorating perception conditions and low trust weights, the system automatically adopts conservative control parameters to prioritize assembly safety. This parameter matching mechanism allows the system to select appropriate control strategies under different operating conditions, enhancing its adaptability.

[0077] In correcting compliance control parameters based on signal fluctuation components, the control unit compares the signal fluctuation components with a preset fluctuation threshold. If the signal fluctuation component is greater than the preset fluctuation threshold, it indicates that there are unstable factors in the current contact process. The control unit performs a dual correction: increasing the damping coefficient in the damping characteristic parameters corresponding to the assembly feed direction, while decreasing the rate of change of each element in the stiffness coefficient matrix.

[0078] The feed direction damping coefficient is increased by proportionally amplifying it based on the original feed direction damping coefficient, according to the degree to which the signal fluctuation component exceeds a preset fluctuation threshold; the greater the degree of exceedance, the larger the amplification coefficient. The stiffness matrix change rate is decreased by proportionally attenuating it based on the original stiffness change rate, according to the degree to which the signal fluctuation component exceeds a preset fluctuation threshold; the greater the degree of exceedance, the larger the attenuation. The preset fluctuation threshold is set based on the normal fluctuation level of the force signal under stable contact conditions. The proportion of the signal fluctuation component exceeding the preset fluctuation threshold is mapped to the damping amplification and rate attenuation parameters, and the smoothness calibration is adjusted according to the requirements for force fluctuation suppression and the necessary parameters.

[0079] Understandably, when high-frequency fluctuations in the force signal exceed limits during assembly, it usually indicates contact impact, edge collision, or impending jamming. Increasing the feed direction damping in this situation aims to increase energy dissipation and reduce end-effector jitter, using damping force to absorb and suppress vibration energy. Simultaneously, reducing the stiffness matrix change rate is crucial because when high-frequency fluctuations occur in the contact area, the deviation estimation itself may be affected by noise. If the stiffness parameter changes rapidly accordingly, a positive feedback loop can easily form, where fluctuations cause parameter changes, which in turn exacerbate the fluctuations. By limiting the stiffness change rate, the update speed of the stiffness parameter is slowed during fluctuations, preventing noise from propagating and amplifying through the parameter channel, ensuring a smooth transition of control parameters. When the signal fluctuation component falls below the preset fluctuation threshold, the feed direction damping coefficient and stiffness change rate return to their original levels, and the system resumes normal control rhythm.

[0080] In the alternative implementation, the correspondence between the base gain value and the contact mode index can be replaced by a continuous function mapping instead of a piecewise lookup table, so that the damping characteristic parameters change continuously with the contact mode index, avoiding parameter jumps at the contact mode switching boundary.

[0081] In an alternative implementation, when weighting the damping gain, the combination of visual trust weights and force trust weights can employ different weighting functions, as long as the qualitative trend of appropriately increasing damping when the visual trust weight increases and maintaining a moderate damping response when the force trust weight increases is satisfied. In another alternative implementation, when weighting the damping gain, the larger of the visual trust weights and force trust weights can be used as the dominant weight, and its corresponding correction factor can be used for separate correction, replacing the weighted combination of the two.

[0082] In an alternative implementation, the magnitude of the reference force bias can be determined by directly multiplying the contact mode index by the ratio of the force trust weight to the sum of the visual trust weight and the force trust weight, wherein a small constant is added to the denominator to prevent the denominator from being zero.

[0083] In an alternative implementation, the dual correction of the signal fluctuation component can be extended to more control parameters, such as synchronously adjusting the amplitude of the reference force bias according to the signal fluctuation component, and appropriately reducing the auxiliary insertion force when the fluctuation exceeds the limit to reduce additional disturbances in the unstable contact state.

[0084] The control unit integrates the determined stiffness coefficient matrix, damping characteristic parameters, and reference force offset into a complete set of compliant control parameters. Based on this parameter set, it runs an impedance control algorithm to generate correction commands for the target pose of the robotic arm's end effector. After receiving these correction commands, the robotic arm module in the assembly execution unit drives the movement of each joint through built-in inverse kinematics and joint space trajectory planning algorithms, indirectly adjusting the contact force and feed speed between the shaft and hole to complete the flexible adaptive alignment and insertion process of the shaft and hole assembly.

[0085] Understandably, the control unit transforms the alignment status information and dynamic characteristics of the force signal into directional adjustments to the three dimensions of the impedance controller: stiffness, damping, and force bias. Translational and angular deviations reflect the degree of positional deviation between the shaft and the hole and their spatial distribution. By setting off-diagonal terms corresponding to the deviation direction in the stiffness matrix, the system can generate a synergistic corrective effect of translation and rotation when performing assembly actions at the end. The contact pattern index and visual and force-feedback weights provide a basis for setting the damping characteristics and reference force bias from the perspectives of physical contact morphology and sensor reliability, respectively. Signal fluctuation component monitoring provides an indication of contact instability, triggering stability protection corrections of the control parameters. The entire parameter generation process enables the assembly system to smoothly and reliably complete alignment and workpiece insertion without directly closing the loop servo force or feed rate.

[0086] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A flexible adaptive assembly system for robots based on multimodal perception and digital twins, characterized in that, include: A visual perception unit is used to acquire a visual image of the assembly area and extract the shaft hole edge features and feature sharpness from the visual image. The assembly execution unit includes a robotic arm module and an end effector module; A force sensing unit is disposed between the robotic arm module and the end effector module to acquire contact torque signals and extract force spin and signal fluctuation components from the torque signals. The twin evaluation unit is connected to the visual perception unit and the force perception unit respectively. It is used to determine the relative pose parameters of the shaft hole based on the edge features of the shaft hole, the feature sharpness, and the force rotation through a preset digital twin model, and output centering state data. The centering state data includes translational deviation, angular deviation, contact pattern index, visual trust weight, and force trust weight. A control unit, connected to the twin evaluation unit, the force sensing unit, and the assembly execution unit, is used to determine a set of compliance control parameters based on the alignment state data, and to correct the set of compliance control parameters according to the signal fluctuation components. Based on the corrected set of compliance control parameters, control commands are generated and output to the assembly execution unit to indirectly adjust the contact force and feed rate during the assembly process. The set of compliance control parameters includes a stiffness coefficient matrix with off-diagonal stiffness terms, damping characteristic parameters in a specified direction, and a reference force offset.

2. The robot flexible adaptive assembly system based on multimodal perception and digital twin as described in claim 1, characterized in that, The visual perception unit acquires a grayscale image and a point cloud image of the shaft hole region, extracts the edge features of the shaft hole based on the grayscale image, extracts the normal vector features of the shaft hole end face based on the point cloud image, and calculates the feature sharpness based on the ratio of the number of effective edge points in the shaft hole edge features to the total number of extracted points.

3. The robot flexible adaptive assembly system based on multimodal perception and digital twin as described in claim 1, characterized in that, The force sensing unit includes a six-dimensional force sensor, and the contact torque signal includes force components and torque components. The force sensing unit calculates the direction and line of action of the force vector based on the force component and the torque component, and uses it as the force spinor; the force sensing unit performs high-pass filtering on the contact torque signal, and determines the signal fluctuation component based on the amplitude of the filtered contact torque signal.

4. The robot flexible adaptive assembly system based on multimodal perception and digital twin as described in claim 1, characterized in that, The digital twin model in the twin evaluation unit includes a geometric kinematics sub-model and a contact mechanics sub-model; the geometric kinematics sub-model is determined based on the workpiece's geometric data and the mechanical arm kinematic parameters of the assembly execution unit; the contact mechanics sub-model is determined based on the workpiece's material elastic modulus and friction coefficient.

5. The robot flexible adaptive assembly system based on multimodal perception and digital twin according to claim 4, characterized in that, The twin evaluation unit updates the relative pose parameters of the shaft hole based on the edge features of the shaft hole and the force rotation, and determines the translational deviation and the angular deviation based on the updated relative pose parameters of the shaft hole.

6. The robot flexible adaptive assembly system based on multimodal perception and digital twin according to claim 5, characterized in that, The twin evaluation unit determines the directions of the translational deviation and the angular deviation based on the comparison between the direction of the force spinor and the deviation direction estimated based on the edge features of the shaft hole; and determines the contact pattern index based on the higher-order statistical moments of the force spinor in the time series and the contact area displayed in the point cloud of the visual image.

7. The robot flexible adaptive assembly system based on multimodal perception and digital twin according to claim 4, characterized in that, The twin evaluation unit acquires the feature sharpness and expected force spin; compares the feature sharpness with a preset sharpness threshold; if the feature sharpness is greater than the preset sharpness threshold, then sets the visual trust weight to be greater than the force trust weight. If the feature sharpness is less than or equal to the preset sharpness threshold, then the directional deviation angle and amplitude deviation rate between the force spin and the expected force spin are calculated, the force perception trust weight is determined based on the directional deviation angle and the amplitude deviation rate, and the visual trust weight is reduced.

8. The robot flexible adaptive assembly system based on multimodal perception and digital twin according to claim 1, characterized in that, The control unit determines the direction of action of the off-diagonal stiffness terms in the stiffness coefficient matrix based on the direction of the translational deviation, and determines the value of each off-diagonal stiffness term based on the magnitude of the translational deviation and the angular deviation.

9. The robot flexible adaptive assembly system based on multimodal perception and digital twin according to claim 8, characterized in that, The control unit determines the base gain value of the damping characteristic parameter based on the contact mode index, and performs a weighted correction on the base gain value based on the visual trust weight and the force trust weight to obtain the gain of the damping characteristic parameter; and determines the reference force bias based on the contact mode index, the visual trust weight and the force trust weight.

10. The robot flexible adaptive assembly system based on multimodal perception and digital twin according to claim 1, characterized in that, The control unit acquires the signal fluctuation component and compares the signal fluctuation component with a preset fluctuation threshold. If the signal fluctuation component is greater than the preset fluctuation threshold, the damping coefficient corresponding to the assembly feed direction in the damping characteristic parameter is increased, and the rate of change of each element in the stiffness coefficient matrix is ​​decreased.

Images