Hardware workpiece assembly positioning method and system based on multi-sensor fusion

Abstract

This invention discloses a multi-sensor fusion method and system for assembling and positioning hardware workpieces, relating to the field of workpiece assembly and positioning technology. The method includes: controlling a programmable lighting unit to project a preset optical pattern onto the workpiece and acquiring workpiece images using a vision sensor; calculating the initial pose of the workpiece and guiding an actuator to move the workpiece to a preset range of coordinates at the target position; fusing visual pose data with distance data from a proximity sensor to collaboratively evaluate and correct the workpiece pose upon reaching the preset range of coordinates, fixing it to the assembly execution coordinate position; and finally compensating for the workpiece pose and executing the assembly operation. This invention solves the technical problems of insufficient accuracy in hardware workpiece assembly and positioning and unstable assembly processes in existing technologies, achieving precise hardware workpiece assembly and positioning and stable assembly processes, thus improving assembly accuracy and safety.

Description

Technical Field

[0001] This invention relates to the field of workpiece assembly and positioning technology, and specifically to a method and system for assembling and positioning hardware workpieces using multi-sensor fusion. Background Technology

[0002] In the field of hardware assembly, traditional assembly positioning methods often rely on a single sensor to acquire workpiece information, resulting in insufficient positioning accuracy and poor adaptability. For example, relying solely on a vision sensor is susceptible to factors such as workpiece material reflection and complex surface texture, leading to inaccurate workpiece image feature extraction and subsequent deviations in initial pose calculation. Using only a proximity sensor only acquires distance data, failing to comprehensively reflect the workpiece's three-dimensional posture and thus hindering high-precision assembly requirements. Furthermore, in existing technologies using multiple sensors collaboratively, inconsistent sensor coordinate systems and asynchronous data timestamps often result in large data fusion errors, preventing accurate collaborative evaluation and correction of workpiece pose. In addition, the lack of suitable force-guided micro-manipulation strategies for assembly scenarios with different geometric constraints easily leads to problems such as untimely pose compensation and abnormal assembly forces / torques, resulting in low assembly efficiency, high workpiece damage risk, and difficulty meeting the high-precision and high-stability assembly production requirements of modern hardware.

[0003] Existing technologies suffer from a lack of precision in the positioning of hardware components during assembly, resulting in an unstable assembly process. Summary of the Invention

[0004] This application provides a multi-sensor fusion method and system for assembling and positioning hardware workpieces, which is used to address the technical problems of lack of accuracy in the assembly and positioning of hardware workpieces and the instability of the assembly process in the prior art.

[0005] In view of the above problems, this application provides a method and system for assembling and positioning hardware workpieces using multi-sensor fusion.

[0006] The first aspect of this application provides a method for assembling and positioning hardware workpieces using multi-sensor fusion, the method comprising:

[0007] The programmable lighting unit projects a preset optical pattern onto the workpiece and acquires an image of the workpiece using a vision sensor. Based on the workpiece image, the initial pose of the workpiece is calculated, and the actuator is guided to move the workpiece to a preset range of coordinates according to the initial pose. The visual pose data and the distance data from the proximity sensor are fused to perform a collaborative evaluation and correction of the workpiece pose upon reaching the preset range of coordinates, and then fix it to the assembly execution coordinate position. Based on the geometric constraint type of the target assembly, the corresponding force-guided micro-operation strategy is selected and executed. According to the assembly execution coordinate position and the contact information fed back by the torque sensor, the pose of the workpiece is finally compensated, and the assembly operation is executed.

[0008] A second aspect of this application provides a multi-sensor fusion-based hardware workpiece assembly and positioning system, the system comprising:

[0009] The workpiece image acquisition module controls the programmable lighting unit to project a preset optical pattern onto the workpiece and acquires the workpiece image using a vision sensor. The workpiece movement module calculates the initial pose of the workpiece based on the workpiece image and guides the actuator to move the workpiece to a preset range of coordinates according to the initial pose. The assembly execution module fuses visual pose data with distance data from a proximity sensor to collaboratively evaluate and correct the workpiece pose upon reaching the preset range of coordinates, fixing it to the assembly execution coordinate position. The pose compensation module selects and executes the corresponding force-guided micro-operation strategy based on the geometric constraint type of the target assembly, and performs final compensation for the workpiece pose according to the assembly execution coordinate position and the contact information fed back by the torque sensor, executing the assembly operation.

[0010] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0011] The system controls a programmable lighting unit to project a preset optical pattern onto the workpiece and acquires an image of the workpiece using a vision sensor. It calculates the initial pose of the workpiece and guides the actuator to move the workpiece within a preset coordinate range based on this initial pose. By fusing visual pose data with distance data from a proximity sensor, the system collaboratively evaluates and corrects the workpiece pose once it reaches the preset coordinate range, fixing it to the assembly execution coordinate position. Based on the geometric constraints of the target assembly, it selects and executes a corresponding force-guided micro-operation strategy to ultimately compensate for the workpiece pose and perform the assembly operation. This achieves precise positioning and stable assembly of hardware workpieces, improving assembly accuracy and safety. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 A schematic flowchart of a multi-sensor fusion method for assembling and positioning hardware workpieces provided in this application embodiment;

[0014] Figure 2 A schematic diagram of the structure of the multi-sensor fusion hardware workpiece assembly and positioning system provided in the embodiments of this application.

[0015] Explanation of reference numerals in the attached drawings: 10 for workpiece image acquisition module, 20 for workpiece movement module, 30 for assembly execution module, and 40 for pose compensation module. Detailed Implementation

[0016] This application provides a multi-sensor fusion method and system for assembling and positioning hardware workpieces, which addresses the technical problems of insufficient accuracy in the assembly and positioning of hardware workpieces and unstable assembly processes in the prior art.

[0017] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0018] Example 1, as Figure 1 As shown, this application provides a multi-sensor fusion method for assembling and positioning hardware workpieces, the method comprising:

[0019] Step S100: Control the programmable lighting unit to project a preset optical pattern onto the workpiece, and use a vision sensor to acquire an image of the workpiece.

[0020] Specifically, the system first configures a suitable preset optical pattern based on the structural and material characteristics of the target metal workpiece. The structural characteristics include the workpiece's geometry, hole distribution, and edge contours, while the material characteristics include metal reflectivity and surface roughness. The preset optical pattern can be selected from sinusoidal stripes, pseudo-random speckle, grid patterns, or dot matrix patterns. For highly reflective metal workpieces, pseudo-random speckle is preferred to reduce reflective interference, while grid patterns are used to enhance edge recognition for workpieces with complex contours. After the optical pattern configuration is completed, the system controls the programmable illumination unit to accurately project the preset optical pattern onto the workpiece surface, ensuring that the optical pattern fully overlaps with key workpiece features such as assembly holes and splicing edges. At the same time, a vision sensor, such as an industrial CCD camera, is activated to align and acquire images of the workpiece, capturing image information containing clear optical marks and workpiece details, ultimately obtaining a workpiece image that meets the requirements of subsequent pose calculations.

[0021] Step S200: Calculate the initial pose of the workpiece based on the workpiece image, and guide the actuator to move the workpiece to the target position within the preset range of coordinates according to the initial pose.

[0022] Specifically, firstly, a three-dimensional spatial coordinate system is established based on the alignment relationship between the preset optical pattern in the workpiece image and the target metal workpiece. Then, according to the coordinate positioning relationship of the optical pattern in the three-dimensional spatial coordinate system, a point cloud generation algorithm is used to convert the feature points of the optical pattern and the feature points of the workpiece surface into a three-dimensional point cloud. Subsequently, the generated point cloud is registered with a pre-stored workpiece template point cloud. This registration accurately locates the orientation coordinates of the target metal workpiece in the three-dimensional spatial coordinate system, thus obtaining the initial pose of the workpiece. After completing the initial pose calculation, a movement command is issued to the actuator based on this initial pose, guiding the actuator to grasp the workpiece and move it within the preset coordinate range of the target position. This achieves a rough positioning near the assembly target, preparing for subsequent fine adjustments. Simultaneously, during this process, the structural type of the target assembly is identified in advance, determining whether it is external splicing or internal nesting, laying the foundation for subsequent sensor-adaptive control. Furthermore, after subsequent fine adjustments, the workpiece needs to reach the aligned target assembly position, which is usually above the assembly.

[0023] Step S300: Integrate visual pose data and distance data from proximity sensors to collaboratively evaluate and correct the workpiece pose that has reached the preset range of coordinate circles, and fix it to the assembly execution coordinate position.

[0024] Specifically, the timestamps of the vision sensor, proximity sensor, and torque sensor are first unified through hardware trigger signals or a precise clock synchronization protocol to achieve time alignment of data from different sensors, providing a foundation for time consistency in subsequent data fusion. Then, data processing and pose collaborative correction are carried out: First, the transformation relationship between the vision sensor coordinate system, the proximity sensor coordinate system, and the robot base coordinate system is established, and the transformation matrix is ​​obtained through calibration to ensure coordinate transformation accuracy. Next, the workpiece pose data acquired by the vision sensor is transformed to the robot base coordinate system according to the transformation relationship to obtain the visual pose estimate. Simultaneously, the distance data between the workpiece and the assembly target measured by the proximity sensor is combined with the robot's current pose to calculate the distance-constrained pose estimate of the workpiece in the robot base coordinate system. Finally, the two pose estimates are fused. First, the robot's movement posture path is established, and when the visual sensor... When the measurement data arrives, the observation residuals of the visual pose estimate and the predicted pose state corresponding to the pose path are calculated and the state estimate is updated. When the distance observation data arrives, the observation residuals of the distance-constrained pose estimate are calculated and the state estimate is updated. Then, based on the Mahalanobis distance of the observation residuals of each sensor, the confidence weights of the two in the fusion are dynamically adjusted, and the optimal pose estimate of the collaborative evaluation is output. Finally, based on the optimal pose estimate, the pose correction amount of the robot end is analyzed, and the robot is controlled to perform correction motion. The workpiece pose that has reached the preset range coordinate circle is corrected and fixed to the assembly execution coordinate position, providing a precise pose reference for subsequent fine assembly operations.

[0025] Step S400: Based on the geometric constraint type of the target assembly, select and execute the corresponding force-guided micro-operation strategy, and perform final compensation on the workpiece's pose according to the assembly execution coordinate position and the contact information fed back by the torque sensor, and execute the assembly operation action.

[0026] Specifically, the geometric constraint type of the target assembly is clearly defined, which includes at least external splicing constraints and internal nested constraints. Then, the corresponding force-guided micro-operation strategy is selected according to different constraint types: If it is an external splicing constraint, a force-sensing leveling strategy is adopted. The torque signal generated by the contact between the workpiece and the assembly target is collected in real time by a torque sensor. The changing characteristics of the torque signal are analyzed, and the workpiece is controlled to rotate and fine-tune around an axis parallel to the contact plane until the torque approaches zero, so as to achieve full fit between the workpiece and the target plane; If it is an internal nested constraint, a force-searching centering strategy is adopted. The lateral force signal generated by the contact is collected by a torque sensor. The distribution law of the lateral force signal is analyzed, and the workpiece is controlled to translate and search in a direction parallel to the contact plane until the lateral force approaches zero, so as to complete the center alignment between the workpiece and the target hole. During the execution of the micro-operation strategy, based on the previously determined assembly execution coordinates and combined with the contact information fed back by the torque sensor in real time, including the magnitude of the contact force and the direction of the torque, the workpiece's posture is finally compensated at the millimeter or even micrometer level to ensure that the workpiece's posture accurately matches the assembly requirements. Subsequently, the actuator is controlled to perform the assembly operation. Throughout the assembly process, the assembly force and torque are continuously monitored based on the force sensor or torque sensor, and the monitoring data is compared with the preset qualified assembly model. If abnormal situations such as a sudden increase in assembly force or torque exceeding the threshold occur, an adaptive adjustment command is immediately triggered, such as correcting the assembly speed, fine-tuning the workpiece posture, or a safety retraction command, controlling the actuator to drive the workpiece back to the preset safe position, ensuring that the assembly operation is completed stably and reliably.

[0027] In one possible implementation, step S400 further includes:

[0028] Step S410: During the assembly process, the assembly force and torque are monitored based on force sensors or torque sensors and compared with the preset qualified assembly model. If an abnormality occurs, an adaptive adjustment or safety rollback command is triggered.

[0029] Specifically, throughout the entire process from initial contact with the assembly target to final assembly, force or torque sensors continuously capture real-time data on assembly forces and torques, including the magnitude, direction, and trend of forces at different assembly stages, such as the contact alignment stage, insertion and fitting stage, and fastening completion stage. Simultaneously, a pre-stored qualified assembly model is invoked. This model contains threshold ranges for assembly forces and torques at each stage, set for the current hardware workpiece assembly scenario, such as the maximum permissible lateral force during the contact alignment stage and the upper limit of torque fluctuation during the insertion and fitting stage. The real-time data collected by the sensors is then compared moment-by-moment with the standard parameters in the qualified assembly model. If the comparison reveals that the real-time assembly force exceeds the corresponding stage force threshold, the torque fluctuation exceeds the allowable range, or there are abnormal situations such as a sudden increase / decrease in force value, the corresponding instructions will be triggered immediately according to the type and severity of the abnormality: For minor abnormalities, such as the torque being slightly higher than the threshold but still within the controllable range, an adaptive adjustment instruction will be triggered to fine-tune the assembly speed, correct the workpiece posture, or adjust the assembly pressure by controlling the actuator, so that the assembly process returns to a normal state; For severe abnormalities, such as the assembly force being far beyond the safety threshold, or a sudden change in torque that may cause damage to the workpiece or equipment, a safety retraction instruction will be triggered to control the actuator to quickly retract the workpiece to a preset safe position, such as 10~20mm above the assembly target, while pausing the assembly operation and issuing an abnormality alarm to avoid workpiece damage, equipment failure, or assembly accuracy failure, and to ensure the safety and reliability of the entire assembly process.

[0030] In one possible implementation, step S400 further includes:

[0031] Step S420: The geometric constraint types include at least external splicing constraints and internal nested constraints. When it is an external splicing constraint, a force-sensing leveling strategy is adopted. In this strategy, by analyzing the torque signal generated by the contact, the workpiece is controlled to rotate and fine-tune around an axis parallel to the contact plane until the torque approaches zero, so that the workpiece and the target plane are fully aligned.

[0032] Step S430: When there is an internal nested constraint, a force search centering strategy is adopted. In this strategy, by analyzing the lateral force signal generated by the contact, the workpiece is controlled to perform translational search in a direction parallel to the contact plane until the lateral force approaches zero, and the center of the workpiece is aligned with the target hole.

[0033] Specifically, adaptation operations are performed based on the geometric constraint types of the target assembly. These geometric constraint types must at least include external splicing constraints and internal nested constraints. When an external splicing constraint is identified, a force-sensory leveling strategy is used to perform pose correction and fitting operations. The assembly scenario characteristics corresponding to external splicing constraints are that two or more workpieces are in contact and connected externally. Typical examples include chassis cover assembly, corner butt joints of door and window frames, and welding joints of two metal plates. The geometric constraint characteristics are: clearly defined contact surfaces, contact lines, or contact points, such as planar fitting and edge butt joints; constraints are open and easily observable; and the main constraint directions are clear, such as the main constraints of Z-axis translation and X and Y-axis rotation in planar fitting. The overall correction approach uses the "surface, line, and point" characteristics as a guide to achieve pose alignment. In practice, the robot first uses vision sensors to perform a rough approximation: 3D vision is used to locate the edges, corners, and other features of the two workpieces to be joined, calculating the approximate pose the robot needs to move in, and controlling the actuators to move the workpieces to the position where they are about to contact the target workpiece. Then, a force-driven leveling strategy is used for plane or edge alignment: the robot switches to impedance control mode and moves the workpieces towards the target plane in a compliant manner. When some points on the workpiece first contact the target plane, the force sensors detect non-zero Z-axis forces and X and Y-axis torques. By analyzing the torque signals, the robot determines which side of the workpiece is being aligned. If the workpiece is too high, then control the workpiece to produce a slight rotation around the X-axis and / or Y-axis until the Z-axis force is evenly distributed and the torque approaches zero. At this point, the planes of the two workpieces are parallel. Finally, with the help of vision or force, the final positioning is completed: on the basis of the plane being parallel, control the workpiece to move along the plane in the X and Y directions. Through visual positioning pin holes or force sensing of the contact of side blocks and edges, the precise positioning in the X, Y and Yaw directions is completed, and finally the workpiece is fully fitted with the target plane. The whole process is similar to the operation logic of putting a lid on: first roughly placing it, then flattening and adjusting it, and finally precisely sealing it.

[0034] For scenarios involving internal nested constraints in the target assembly, a force-searching centering strategy is employed to achieve precise center alignment between the workpiece and the target orifice. The assembly scenario corresponding to internal nested constraints is characterized by one workpiece (insertion) partially or completely entering the internal space of another workpiece (enclosing part). Typical examples include bearings being installed into bearing housings, pins being inserted into holes, and gears being fitted into shafts. The geometric constraints are characterized by a closed state; after insertion begins, the insert is surrounded by the enclosing part, rendering visual perception completely ineffective. There is a unilateral constraint, meaning the inner wall of the hole only allows for slight radial movement of the shaft. The reaction force after contact is clearly directional and extremely sensitive to angular deviations; even slight tilting can easily cause the insertion to jam. The overall correction approach uses the "orifice" as a guide to conduct self-centering and compliant insertion operations. In practice, the initial positioning is first completed using a vision sensor: the vision system accurately identifies the position of the hole on the housing and guides the robot to move the insert to a preset safe height directly above the center of the hole, usually 5-10mm above the hole opening, to avoid collisions and facilitate subsequent pressing operations; then, the robot switches to a force-driven search strategy for hole opening search and centering: when there is a millimeter-level initial positional deviation between the insert and the hole, the robot first controls the insert to slowly move downwards along the Z-axis. When the bottom of the insert contacts the chamfer or the hole opening plane, a lateral force (Fx, Fy directions) is generated. The torque sensor captures this lateral force signal in real time and feeds it back to the system; after sensing the lateral force, the robot immediately controls the insert to stop pressing down and simultaneously drives the robot to... The insert moves slightly in the opposite direction of the lateral force, with a movement range of tens to hundreds of micrometers, to avoid excessive movement that could damage the workpiece surface. Then, the insert is pressed down along the Z-axis again. This process of "pressing down - sensing lateral force - fine-tuning in the opposite direction - pressing down again" is repeated multiple times until the lateral force signal approaches zero. At this point, it indicates that the insert has slid into the center of the hole, completing the force search or peak-valley alignment. Although the subsequent insertion phase begins, the robot switches to compliant mode, allowing the insert to deviate slightly when subjected to lateral forces to compensate for errors. At the same time, the Z-axis force is monitored to ensure normal operation. However, the core hole alignment is completed, laying a crucial coaxiality foundation for subsequent compliant insertion and effectively solving the problem of accurate positioning after visual failure under internal nested constraints.

[0035] In one possible implementation, step S100 further includes:

[0036] Step S110: Configure optical patterns based on the structural and material characteristics of the target metal workpiece.

[0037] Step S120: Align and acquire the target metal workpiece using the optical pattern to obtain an image of the workpiece.

[0038] Specifically, based on the structural and material characteristics of the target metal workpiece, a suitable optical pattern is configured to lay the foundation for accurate image acquisition of the workpiece. In practice, the structural characteristics of the target metal workpiece are first comprehensively analyzed to clarify its geometry, such as plate-like, shaft-like, block-like, or irregularly shaped structures with complex protrusions / grooves; the distribution of holes, including the number of holes, hole diameter, hole spacing, and hole distribution pattern; and the type of edge contour, such as straight edges, curved edges, and irregular serrated edges. These structural characteristics directly determine the key areas for identification that the optical pattern needs to highlight. For example, for workpieces with multiple holes, the edge markings of the holes need to be strengthened; for workpieces with complex contours, the entire contour area needs to be covered. Simultaneously, the material characteristics of the workpiece are analyzed, focusing on the reflectivity of the metal, such as high-reflectivity materials like stainless steel and low-reflectivity materials like cast iron, and surface roughness, such as smooth polished surfaces, frosted surfaces, and surfaces with processing textures. Different materials have significantly different light reflection and scattering characteristics, requiring adjustments to the optical pattern type to reduce interference. Based on the above feature analysis, suitable optical patterns are selected and configured from a pre-set optical pattern library. Available optical patterns include sinusoidal stripes, pseudo-random speckle, grid patterns, or dot matrix patterns. For highly reflective workpieces, pseudo-random speckle patterns are prioritized, utilizing their irregular texture distribution to reduce image overexposure or feature blurring caused by reflection. For workpieces requiring precise positioning of holes and edges, grid or dot matrix patterns are configured, establishing a clear positional association between the clear lines / dot matrix and workpiece features, facilitating subsequent image feature extraction. For workpieces requiring 3D shape calculation through pattern deformation, sinusoidal stripe patterns are configured, relying on the deformation patterns of the stripes on the workpiece surface to improve the accuracy of 3D information capture. The entire configuration process ensures that the optical patterns effectively highlight the key features of the workpiece, avoid imaging interference caused by material characteristics, and provide optimal optical conditions for image acquisition.

[0039] Using a pre-configured optical pattern as the core, the target metal workpiece is aligned and acquired to obtain high-quality workpiece images. First, the spatial position and operating parameters of the programmable illumination unit and vision sensor, such as an industrial CCD camera, are adjusted: on the one hand, the projection angle and intensity of the illumination unit are calibrated to ensure that the optical pattern completely and uniformly covers the observation area of ​​the target metal workpiece, especially focusing on key features such as holes, edges, and splicing interfaces; on the other hand, the focal length, exposure time, and sensitivity of the vision sensor are adjusted to avoid image blurring or overexposure due to excessively strong or weak light, while ensuring that the sensor's shooting angle can fully capture the superposition effect of the optical pattern and the workpiece surface. Subsequently, the programmable illumination unit is controlled to precisely project the pre-configured optical pattern, such as pseudo-random speckle or grid patterns, onto the surface of the target metal workpiece. Real-time monitoring ensures effective alignment between the optical pattern and the key features of the workpiece, meaning that the lines, dots, or speckle textures of the optical pattern form a clear positional correlation with the edges of holes and contour lines of the workpiece, without significant offset or obstruction. Once the optical pattern projection is stable and aligned, the vision sensor is triggered to acquire image data containing clear optical marks and workpiece details, ultimately generating a workpiece image that meets the requirements of subsequent pose calculations. This image must meet the requirements of no optical pattern distortion, no blurring of workpiece features, and complete information on key parts, providing reliable visual data support for subsequent image-based 3D coordinate establishment, point cloud generation, and initial pose calculation.

[0040] In one possible implementation, step S200 further includes:

[0041] Step S210: Establish three-dimensional spatial coordinates based on the alignment relationship between the optical pattern and the target metal workpiece.

[0042] Step S220: Generate a point cloud according to the coordinate positioning relationship of the optical pattern in the three-dimensional space coordinates.

[0043] Step S230: Register the point cloud with the workpiece template, locate the orientation coordinates of the target hardware workpiece in three-dimensional space, and obtain the initial pose.

[0044] Specifically, from the workpiece images collected in the early stage, the key alignment information between the optical pattern and the workpiece is accurately identified and extracted. That is, the characteristic points of the optical pattern, such as the line intersections of the grid pattern, the center of the dot matrix pattern, and the texture feature points of the pseudo-random speckle, are clearly identified and correlated with the key features of the target hardware workpiece surface, such as the edge points of the holes, the corner points of the workpiece, and the boundary points of the splicing interface. These corresponding points are the basic reference to ensure that the subsequent coordinate system is accurately bound to the actual shape of the workpiece. Next, a three-dimensional spatial coordinate system is constructed based on the preset parameters of the vision sensor: with the optical center of the vision sensor lens as the origin of the coordinate system, the horizontal direction parallel to the workpiece image plane is defined as the X-axis, the vertical direction parallel to the image plane is defined as the Y-axis, and the direction perpendicular to the image plane and pointing towards the workpiece is defined as the Z-axis. At the same time, the intrinsic parameters of the vision sensor, such as the lens focal length and pixel size, and the extrinsic parameters, such as the tilt angle of the sensor during installation and the initial vertical distance from the workpiece placement plane, are called. Through the alignment relationship between the optical pattern feature points and the workpiece feature points, the two-dimensional pixel coordinates of the key feature points on the workpiece surface are converted into preliminary three-dimensional spatial coordinate data by combining the sensor parameters. Finally, a three-dimensional spatial coordinate framework that can accurately map the real spatial position distribution of the workpiece is built, providing a precise coordinate reference for generating point clouds and calculating the initial pose of the workpiece in subsequent steps.

[0045] For configured optical patterns, such as sinusoidal fringes, pseudo-random speckle, grid patterns, or dot matrix patterns, intrinsic parameter data such as camera focal length, pixel size, and principal point coordinates are first obtained using a visual sensor intrinsic parameter calibration tool, such as a checkerboard calibration board. Combined with extrinsic parameters from sensor installation, a conversion model between the pixel coordinates of the optical pattern and the three-dimensional spatial coordinates is established. For grid or dot matrix patterns, a feature point extraction-coordinate mapping approach is used: corner detection functions from the OpenCV algorithm library, such as the cornerHarris algorithm, extract the pixel coordinates of grid intersections or dot matrix centers in the image. These coordinates are then substituted into the conversion model to calculate the X, Y, and Z coordinates of each feature point in three-dimensional space. The X / Y coordinates are converted from pixel coordinates using intrinsic parameters, and the Z coordinate is determined by the distance parameter in the extrinsic parameters. A neighborhood interpolation algorithm is then used to supplement the transition coordinates between feature points, forming a preliminary point cloud. For sinusoidal fringes, a phase calculation-depth mapping approach is used: utilizing Fourier transform... Leaf transform contouring technology extracts the phase of the acquired stripe image and performs phase unwrapping to obtain the phase information of each pixel on the workpiece surface. Combined with the stripe projection frequency and sensor parameters, the depth value (Z coordinate) corresponding to the pixel is calculated, and then converted into three-dimensional spatial coordinates to generate a dense point cloud. For pseudo-random speckle patterns, a stereo matching-3D reconstruction approach is used: speckle images from different viewpoints are acquired through a binocular vision system, and the SIFT algorithm is used to extract speckle feature points and complete cross-viewpoint matching. The three-dimensional spatial coordinates of the matched point pairs are calculated using triangulation principles, and mismatched points are filtered to generate a point cloud that meets the accuracy requirements. Finally, a point cloud deduplication algorithm is used to remove duplicate coordinate points, and a voxel grid downsampling method is used to optimize the point cloud density, retaining high-density point clouds in key feature areas and simplifying point clouds in flat areas. This ensures that the generated point cloud not only fully reflects the workpiece geometry but also accurately corresponds to the positioning relationship of the optical pattern in three-dimensional spatial coordinates, providing a reliable data foundation for subsequent point cloud registration.

[0046] The initial pose positioning of the workpiece is achieved through point cloud registration. The specific execution process is as follows: First, the pre-stored workpiece template point cloud is called. This template point cloud is generated based on the design 3D model or high-precision scanning data of the target hardware workpiece. It contains the complete geometric feature coordinates of the workpiece in the standard posture, such as the standard 3D coordinates of key parts such as holes, corners, and edges. It has been pre-calibrated with the established 3D spatial coordinate system to ensure that the coordinate reference is consistent. Subsequently, point cloud registration is carried out in stages: First, coarse registration is performed using a feature-based sampling consistency initial registration algorithm to extract key feature points from the generated actual workpiece point cloud. For example, feature vectors at hole edges and corners are calculated using the FPFH feature descriptor and matched with corresponding feature points in the template point cloud. Mismatched points are eliminated through random sampling, and the translation and rotation parameters of the actual point cloud relative to the template point cloud are initially calculated. The actual point cloud is adjusted to be roughly aligned with the template point cloud, reducing the initial attitude deviation. Then, fine registration is performed using an iterative nearest-point algorithm. The distance from each point in the actual point cloud to the nearest point in the template point cloud is calculated iteratively using the coarse registration result as the initial value. The translation and rotation matrix is ​​optimized by minimizing the Euclidean distance error function until the error converges to a preset threshold, usually controlled within 0.05mm, to meet the coarse positioning accuracy requirements of hardware workpiece assembly. Finally, based on the optimal transformation matrix obtained through precise registration, which includes the translation amounts in the X, Y, and Z axes and the rotation angles around the X, Y, and Z axes, the specific attitude coordinates of the target hardware workpiece in three-dimensional space are determined. That is, the position coordinates of the workpiece center on the X, Y, and Z axes, and the rotation angles of the workpiece around the X, Y, and Z axes. These parameters are integrated as the initial pose of the workpiece, providing accurate attitude data support for guiding the actuator to move the workpiece to the target position within a preset range of coordinates.

[0047] In one possible implementation, step S110 further includes:

[0048] The optical patterns include sinusoidal fringes, pseudo-random speckle, grid patterns, or dot matrix patterns.

[0049] Specifically, the optical patterns cover four categories: sinusoidal stripes, pseudo-random speckle, grid patterns, and dot matrix patterns. Each type of pattern needs to be selected specifically based on the structural and material characteristics of the target hardware workpiece to adapt to the visual requirements of different assembly and positioning scenarios. Among them, the sinusoidal stripe pattern, due to its periodic texture characteristics, is preferred when the three-dimensional shape of the workpiece surface needs to be accurately obtained. The deformation generated on the workpiece surface after the stripes are projected can be combined with the phase solution algorithm to calculate the depth information of each point on the workpiece surface. It is suitable for hardware workpieces with complex surfaces or those requiring high-precision three-dimensional reconstruction. The pseudo-random speckle pattern, due to its irregular texture and uniform distribution, can effectively reduce the interference of highly reflective metal workpieces, such as stainless steel workpieces, on image acquisition. It is widely used in scenarios where the workpiece material has high reflectivity and smooth surface. The stereo matching of speckle feature points can achieve stable recognition of the workpiece pose. The grid pattern has clear line intersection features and is selected when the workpiece has clear hole positions, edges and other structured features. It is convenient to quickly extract the grid intersections and key features of the workpiece, such as the alignment relationship of hole edges and corners, and improve the feature positioning efficiency. The dot matrix pattern uses discrete point centers as the core recognition marks and is suitable for scenarios where the workpiece surface features are relatively simple, such as flat metal plates and regular block workpieces. The coordinate mapping of the dot matrix center can quickly establish the relationship between the workpiece and the three-dimensional space coordinates, simplifying the subsequent pose calculation process. The selection of four types of optical patterns provides a suitable visual solution for image acquisition of hardware workpieces with different characteristics, ensuring that the subsequent initial pose calculation based on the image can obtain accurate visual data support.

[0050] In one possible implementation, step S300 further includes:

[0051] Step S310: Establish the transformation relationship between the vision sensor coordinate system, the proximity sensor coordinate system, and the robot base coordinate system.

[0052] Step S320: Convert the workpiece pose data acquired by the vision sensor to the robot base coordinate system to obtain the visual pose estimate.

[0053] Step S330: Combine the distance data measured by the proximity sensor with the robot's current pose to calculate the estimated distance-constrained pose of the workpiece in the robot's base coordinate system.

[0054] Step S340: Perform data fusion on the visual pose estimate and the distance-constrained pose estimate to obtain the optimal pose estimate for collaborative evaluation.

[0055] Step S350: Analyze the pose correction amount of the robot end effector based on the optimal pose estimation value, and use the pose correction amount to control the robot to perform the correction motion.

[0056] Specifically, the establishment of multi-coordinate system transformation relationships begins with coordinate system calibration using calibration tools. For the vision sensor coordinate system and the robot base coordinate system, a checkerboard calibration board is used. The calibration board is fixed at a known position in the robot base coordinate system, and the vision sensor is controlled to capture images of the calibration board in different postures. By calculating the coordinate correspondence between the feature points of the calibration board in the two coordinate systems, a transformation matrix from the vision sensor coordinate system to the robot base coordinate system is generated. For the proximity sensor coordinate system and the robot base coordinate system, a proximity sensor, such as a laser displacement sensor, is fixed on the robot's end effector. The robot is controlled to move the sensor to multiple preset calibration positions in the robot base coordinate system. The measurement values ​​of the sensor at each position and the coordinates in the robot base coordinate system are recorded. The transformation matrix from the proximity sensor coordinate system to the robot base coordinate system is obtained by fitting using the least squares method. Finally, a unified transformation relationship is established between the vision sensor coordinate system, the proximity sensor coordinate system, and the robot base coordinate system.

[0057] The coordinate transformation of the visual pose data is performed by first obtaining the original pose data of the workpiece from the vision sensor, including the position coordinates and rotation angle of the workpiece in the vision sensor coordinate system. Then, the transformation matrix from the vision sensor coordinate system to the robot base coordinate system is called, and the original pose data is substituted into the transformation formula to perform coordinate transformation. The posture parameters of the workpiece in the vision sensor coordinate system are converted into the corresponding position and angle information in the robot base coordinate system. Finally, the visual pose estimate is obtained to ensure that the visual data can be consistent with the coordinate reference of the robot motion control.

[0058] To calculate the distance-constrained pose estimation, the distance data between the workpiece and the assembly target, measured in real time by proximity sensors, is first collected. This includes the vertical distance from the workpiece surface to the target plane and the radial distance from the workpiece edge to the target opening. Simultaneously, the current pose information fed back by the robot control system is acquired, including the position coordinates and attitude angle of the robot's end effector. Combining the fixed installation relationship between the proximity sensor and the robot end effector, and the established transformation relationship from the proximity sensor coordinate system to the robot base coordinate system, the distance data and the robot's current pose are fused and calculated. The possible position range of the workpiece in the robot base coordinate system is derived through geometric relationships, and pose options that do not meet the distance constraints are eliminated. Finally, the distance-constrained pose estimation value of the workpiece in the robot base coordinate system is determined.

[0059] First, a preset posture path for the robot to move the workpiece is established. This path includes the predicted posture states of the robot at different time points, serving as a benchmark for subsequent pose estimation comparisons. When visual observation data, i.e., visual pose estimation values, are transmitted to the system, the observation residual between the visual pose estimation value and the predicted posture state at the corresponding time point in the posture path is calculated. The state estimation of the workpiece's current posture is adjusted and updated based on the magnitude of the residual. Similarly, when the distance observation data collected by the proximity sensor, i.e., the distance-constrained pose estimation value, arrives, the above operation is repeated, calculating its observation residual with the corresponding predicted posture state and updating the state estimation again. Subsequently, based on the Mahalanobis distance of the observation residuals from each sensor, the confidence weights of the two types of pose estimation values ​​are dynamically assigned during the fusion process. If the observation residual of a certain type of pose estimation value is small, it indicates that its deviation from the predicted posture state is smaller and its reliability is higher, and the corresponding confidence weight will be increased; conversely, the weight of the pose estimation value with a larger residual will be decreased. Finally, a weighted fusion algorithm is used to integrate the visual pose estimate and the distance-constrained pose estimate, combining the advantages of both, the positional accuracy of the visual data and the constraint stability of the distance data, to output the optimal pose estimate for collaborative evaluation that accurately reflects the actual pose of the workpiece.

[0060] The system acquires the actual pose data of the robot's end effector, including its X, Y, and Z axis position coordinates in the robot's base coordinate system, as well as its rotation angles around these axes. This data is fed back in real-time by the robot's encoder or position sensors. Then, the optimal pose estimate is compared dimension-by-dimensionally with the current actual pose data. Through coordinate difference calculation and attitude angle deviation analysis, the required pose corrections for the robot's end effector are determined. Specifically, this includes position corrections in the X, Y, and Z axes to compensate for positional deviations (e.g., adjusting the X-axis to the right by 0.5mm and the Z-axis downwards by 0.3mm), and attitude corrections around the X, Y, and Z axes to compensate for angular deviations (e.g., rotating -0.2° around the Y-axis to correct tilt). After obtaining the pose corrections, they are converted according to the robot control system's instruction format to generate control instructions containing parameters such as travel distance, rotation angle, speed, and acceleration. This ensures that the instructions conform to the robot's kinematic constraints, such as maximum travel speed and maximum rotational angular velocity. Finally, the converted control commands are sent to the robot driver. The driver drives the robot joints to move according to the command parameters, which drives the end effector and the grasped metal workpiece to accurately complete the pose correction. During the correction process, the workpiece pose changes are monitored in real time by vision sensors or proximity sensors. If a deviation exceeds the threshold during the correction process, the control commands will be fine-tuned in time until the robot end pose is completely matched with the optimal pose estimate, providing accurate posture assurance for fixing the workpiece to the assembly execution coordinate position in the future.

[0061] In one possible implementation, step S340 further includes:

[0062] Step S341: Establish the robot's movement posture path.

[0063] Step S342: Based on the attitude path, when the visual observation data arrives, calculate the observation residual between the visual pose estimate and the predicted attitude state corresponding to the attitude path, and update the state estimate.

[0064] Step S343: Based on the attitude path, when the distance observation data arrives, calculate the observation residual between the distance-constrained pose estimate and the predicted attitude state corresponding to the attitude path, and update the state estimate.

[0065] Step S344: Based on the Mahalanobis distance of the observation residuals of each sensor, dynamically adjust the confidence weights of the corresponding visual pose estimate and the distance-constrained pose estimate in the fusion, and output the optimal pose estimate.

[0066] Specifically, firstly, the motion target of the workpiece is clearly defined, i.e., moving from the currently reached target position within a preset coordinate range to the assembly execution coordinate position that needs to be fixed later, thus determining the starting and ending coordinates of the path. Then, combining the robot's kinematic parameters, such as joint range of motion, maximum speed, acceleration limits, workpiece weight and size, to avoid workpiece collisions or motion instability during path planning, and considering the sensor data acquisition time interval, ensuring that key nodes in the path match the sensor data update rhythm, a path planning algorithm, such as the A* algorithm, is used to generate a continuous and smooth attitude path. This attitude path needs to be time-dimensional, clearly defining the robot's predicted attitude state at different times, including the position coordinates of the robot's end effector, i.e., the X, Y, and Z axis directions and attitude angles, i.e., the rotation angles around the X, Y, and Z axes. Furthermore, the attitude changes between adjacent moments must meet the requirements of robot motion continuity, avoiding sudden stops, sharp turns, or other situations exceeding motion constraints. Meanwhile, path planning needs to reserve key nodes for sensor data fusion to ensure that when subsequent visual observation data and distance observation data arrive, they can accurately match the corresponding predicted posture state in the path, and finally form a complete posture path that can both meet the robot's motion constraints and adapt to the needs of multi-sensor data fusion.

[0067] The system monitors real-time visual observation data transmitted from the vision sensor, specifically the visual pose estimation values ​​converted to the robot's base coordinate system. Upon data arrival, the system accurately matches the predicted pose state at the corresponding moment in the pose path based on the data's timestamp information. This predicted pose state includes the robot's end effector's position coordinates and attitude angles at that moment, and is a baseline pose pre-calculated based on a preset motion trajectory and the robot's kinematic characteristics. Subsequently, the observation residuals are calculated through a dimensional comparison: in the position dimension, the differences between the X, Y, and Z axis coordinates in the visual pose estimation and the corresponding coordinates in the predicted pose state are calculated to obtain the position residuals; in the attitude dimension, the differences between the rotation angles around the X, Y, and Z axes in the visual pose estimation and the corresponding angles in the predicted pose state are calculated to obtain the attitude residuals. The position and attitude residuals are integrated to form a complete set of observation residuals, which directly reflects the degree of deviation between the visually observed actual pose and the predicted pose along the preset path. Finally, a Kalman filter state estimation algorithm is used to dynamically update the workpiece's current pose state estimate based on the calculated observation residuals. The algorithm combines the measurement noise characteristics of the vision sensor, such as pre-calibrated pixel errors and coordinate transformation errors, to give reasonable confidence to the observation residuals. While retaining the effective information in the original state estimation, it incorporates new visual observation data, making the updated state estimation closer to the actual pose of the workpiece.

[0068] The system receives distance observation data transmitted from proximity sensors in real time. This data represents the distance measurements taken by the proximity sensors between the workpiece and the target assembly position. Combined with the robot's current pose and feedback from the robot's encoder, including the position and orientation of the end effector, this data is geometrically derived and converted into a distance-constrained pose estimate of the workpiece in the robot's base coordinate system, ensuring consistency between the data coordinate reference and the orientation path. Once the distance-constrained pose estimate is generated, its timestamp is used to match the predicted orientation state at the corresponding moment from the orientation path. This state includes the robot's end effector's preset position coordinates and orientation angles at that moment, representing ideal pose parameters based on path planning. Subsequently, the observation residual is calculated: in the position dimension, the X, Y, and Z axis coordinates of the distance-constrained pose estimate and the predicted orientation state are compared to obtain the positional deviations along each axis; in the orientation dimension, the focus is on angular deviations related to the distance constraint, such as whether the workpiece's tilt causes distance measurement deviations. The difference between the two values ​​of rotation angles around the X, Y, and Z axes is calculated, and these are integrated to form a complete observation residual. This residual directly reflects the degree of deviation between the actual pose derived from the distance observation and the path-predicted pose. Finally, a state estimation algorithm consistent with the visual observation data processing logic, such as Kalman filtering, is adopted. Combined with the measurement noise characteristics of the proximity sensor, such as the pre-calibrated distance measurement error range, reasonable weights are assigned to the observation residuals. The pose information of the distance dimension is integrated into the current workpiece pose state estimation. The updated state estimation takes into account both the positional accuracy of the visual data and the constraint reliability of the distance data.

[0069] The calculated visual observation residuals and distance observation residuals are extracted, and the Mahalanobis distance of each type of residual is calculated. This Mahalanobis distance is combined with pre-statistically calculated covariance matrices of measurement errors from various sensors, such as the coordinate measurement error matrix of the visual sensor and the distance measurement error matrix of the proximity sensor. This eliminates the differences in data dimensions across different dimensions while accurately quantifying the statistical significance of the residuals deviating from the ideal state. A smaller Mahalanobis distance for a certain type of residual indicates higher consistency with the predicted pose path and stronger data reliability. Subsequently, the confidence weights of the two types of pose estimates are dynamically assigned based on the Mahalanobis distance: a negative correlation is established between Mahalanobis distance and confidence weight, meaning that the smaller the Mahalanobis distance, the higher the confidence weight of the corresponding pose estimate, and vice versa. For example, if the Mahalanobis distance of the visual observation residual is 0.8 (low residual range) and the Mahalanobis distance of the distance observation residual is 2.1 (high residual range), the confidence weight of the visual pose estimate will be higher than that of the distance-constrained pose estimate, ensuring that the more reliable data dominates the fusion process. Finally, the system employs a weighted fusion algorithm to integrate the visual pose estimate and the distance-constrained pose estimate according to dynamically adjusted confidence weights. Using position coordinates and attitude angles as fusion dimensions, the corresponding parameters of the two types of estimates are weighted and summed to obtain a comprehensive pose parameter that balances the positional accuracy of visual data and the stability of distance data constraints. This parameter is then output as the optimal pose estimate for collaborative evaluation.

[0070] In one possible implementation, step S300 further includes:

[0071] By using hardware trigger signals or precise clock synchronization protocols, the timestamps of vision sensors, proximity sensors, and torque sensors are unified to align the time of data from different sensors.

[0072] Specifically, to ensure that the data collected by the vision sensor, proximity sensor, and torque sensor can be accurately and collaboratively used for workpiece pose calculation, time alignment of the data from each sensor needs to be achieved through hardware trigger signals or precise clock synchronization protocols. If a hardware trigger signal is used, a hardware trigger circuit is built to synchronously send a unified trigger pulse signal to the three sensors. When the trigger signal is generated, the vision sensor immediately starts workpiece image acquisition, the proximity sensor synchronously begins distance measurement, and the torque sensor records the torque feedback data at that moment. This ensures that all three types of sensors start data acquisition at the same physical moment, guaranteeing data time consistency from the source. Alternatively, if a precise clock synchronization protocol, such as the PTP precision time protocol, is selected, all sensors are connected to the same clock synchronization network. Using a preset high-precision reference clock, such as the master clock of the robot control system, as a reference, the local clocks of the vision sensor, proximity sensor, and torque sensor are calibrated in real time through the protocol. This unifies the timestamps of each sensor to the same time reference, eliminating time deviations caused by clock drift between different sensors. After time synchronization is completed, timestamp tags under a unified reference are added to the data collected by each sensor. In the subsequent fusion of visual pose data and proximity sensor distance data, and pose compensation by combining torque sensor feedback information, the timestamps can be used to quickly match multi-source sensor data at the same time or with very small time intervals. This avoids fusion errors or pose calculation deviations caused by data time misalignment and provides a reliable time reference guarantee for the effective collaborative processing of multi-sensor data.

[0073] Example 2, based on the same inventive concept as the multi-sensor fusion method for assembling and positioning hardware workpieces in the aforementioned examples, such as... Figure 2 As shown, this application provides a multi-sensor fusion-based hardware workpiece assembly and positioning system. The system and method embodiments in this application are based on the same inventive concept. The system includes:

[0074] The workpiece image acquisition module 10 is used to control the programmable lighting unit to project a preset optical pattern onto the workpiece and to acquire the workpiece image using a vision sensor.

[0075] The workpiece moving module 20 is used to calculate the initial pose of the workpiece based on the workpiece image, and guide the actuator to move the workpiece to the target position within a preset range of coordinates according to the initial pose.

[0076] The assembly execution module 30 is used to fuse visual pose data and distance data from proximity sensors to collaboratively evaluate and correct the workpiece pose that has reached the preset range of coordinate circles, and fix it to the assembly execution coordinate position.

[0077] The pose compensation module 40 is used to select and execute the corresponding force-guided micro-operation strategy according to the geometric constraint type of the target assembly, and perform final pose compensation of the workpiece according to the assembly execution coordinate position and the contact information fed back by the torque sensor, and execute the assembly operation action.

[0078] Furthermore, the system is also used to implement the following functions:

[0079] During the assembly process, the assembly force and torque are monitored based on force or torque sensors and compared with the preset qualified assembly model. If an abnormality occurs, an adaptive adjustment or safety rollback command is triggered.

[0080] Furthermore, the system is also used to implement the following functions:

[0081] The geometric constraint types include at least external splicing constraints and internal nested constraints. When it is an external splicing constraint, a force-sensing leveling strategy is adopted. In this case, by analyzing the torque signal generated by the contact, the workpiece is controlled to rotate and fine-tune around an axis parallel to the contact plane until the torque approaches zero, so that the workpiece and the target plane are fully aligned. When it is an internal nested constraint, a force-searching centering strategy is adopted. In this case, by analyzing the lateral force signal generated by the contact, the workpiece is controlled to translate and search in a direction parallel to the contact plane until the lateral force approaches zero, so that the center of the workpiece and the target hole are aligned.

[0082] Furthermore, the system is also used to implement the following functions:

[0083] Based on the structural and material characteristics of the target hardware workpiece, an optical pattern is configured; the target hardware workpiece is aligned and acquired using the optical pattern to obtain an image of the workpiece.

[0084] Furthermore, the system is also used to implement the following functions:

[0085] Based on the alignment relationship between the optical pattern and the target metal workpiece, a three-dimensional spatial coordinate system is established; according to the coordinate positioning relationship of the optical pattern in the three-dimensional spatial coordinate system, a point cloud is generated; the point cloud is registered with the workpiece template to perform the attitude coordinate positioning of the target metal workpiece in the three-dimensional spatial coordinate system, thereby obtaining the initial pose.

[0086] Furthermore, the system is also used to implement the following functions:

[0087] The optical patterns include sinusoidal fringes, pseudo-random speckle, grid patterns, or dot matrix patterns.

[0088] Furthermore, the system is also used to implement the following functions:

[0089] Establish the transformation relationship between the vision sensor coordinate system, the proximity sensor coordinate system, and the robot base coordinate system; transform the workpiece pose data acquired by the vision sensor to the robot base coordinate system to obtain the visual pose estimate; combine the distance data measured by the proximity sensor with the robot's current pose to calculate the distance-constrained pose estimate of the workpiece in the robot base coordinate system; perform data fusion on the visual pose estimate and the distance-constrained pose estimate to obtain the optimal pose estimate for collaborative evaluation; analyze the pose correction amount of the robot end effector based on the optimal pose estimate, and use the pose correction amount to control the robot to perform corrective motion.

[0090] Furthermore, the system is also used to implement the following functions:

[0091] Establish a pose path for robot movement; based on the pose path, when visual observation data arrives, calculate the observation residual between the visual pose estimate and the predicted pose state corresponding to the pose path and update the state estimate; based on the pose path, when distance observation data arrives, calculate the observation residual between the distance-constrained pose estimate and the predicted pose state corresponding to the pose path and update the state estimate; dynamically adjust the confidence weights of the corresponding visual pose estimate and the distance-constrained pose estimate in the fusion based on the Mahalanobis distance of the observation residuals of each sensor, and output the optimal pose estimate.

[0092] Furthermore, the system is also used to implement the following functions:

[0093] By using hardware trigger signals or precise clock synchronization protocols, the timestamps of vision sensors, proximity sensors, and torque sensors are unified to align the time of data from different sensors.

[0094] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, the above description focuses on specific embodiments of this specification. Additionally, the processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired results. In some implementations, multitasking and parallel processing are possible or may be advantageous.

[0095] The above description is only a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0096] This specification and accompanying drawings are merely illustrative examples of this application and are intended to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from its scope. Therefore, if such modifications and variations fall within the scope of this application and its equivalents, this application intends to include such modifications and variations.

Claims

1. A multi-sensor fusion method for assembling and positioning hardware workpieces, characterized in that, include: The programmable lighting unit is controlled to project a preset optical pattern onto the workpiece, and a vision sensor is used to acquire an image of the workpiece. The initial pose of the workpiece is calculated based on the workpiece image, and the actuator is guided to move the workpiece to the target position within a preset range of coordinates according to the initial pose. By integrating visual pose data and distance data from proximity sensors, the pose of the workpiece that has reached the preset range of coordinate circles is evaluated and corrected collaboratively, and then fixed to the assembly execution coordinate position. Based on the geometric constraint type of the target assembly, the corresponding force-guided micro-operation strategy is selected and executed. According to the assembly execution coordinate position and the contact information fed back by the torque sensor, the workpiece pose is finally compensated and the assembly operation action is executed. Based on the type of geometric constraints of the target assembly, select and execute the corresponding force-guided micromanipulation strategy, including: The geometric constraint types include at least external splicing constraints and internal nested constraints. When it is an external splicing constraint, a force-sensing leveling strategy is adopted. In this strategy, by analyzing the torque signal generated by the contact, the workpiece is controlled to rotate and fine-tune around an axis parallel to the contact plane until the torque approaches zero, so that the workpiece and the target plane are fully aligned. When there is an internal nested constraint, a force search centering strategy is adopted. In this strategy, by analyzing the lateral force signal generated by the contact, the workpiece is controlled to perform translational search in a direction parallel to the contact plane until the lateral force approaches zero, and the center of the workpiece is aligned with the target hole.

2. The multi-sensor fusion method for assembling and positioning hardware workpieces according to claim 1, characterized in that, Performing assembly operations also includes: During the assembly process, the assembly force and torque are monitored based on force or torque sensors and compared with the preset qualified assembly model. If an abnormality occurs, an adaptive adjustment or safety rollback command is triggered.

3. The multi-sensor fusion method for assembling and positioning hardware workpieces according to claim 1, characterized in that, Controlling a programmable lighting unit to project a preset optical pattern onto a workpiece and acquiring an image of the workpiece using a vision sensor includes: Based on the structural and material characteristics of the target hardware workpiece, an optical pattern is configured; The optical pattern is used to align and acquire images of the target metal workpiece.

4. The multi-sensor fusion method for assembling and positioning hardware workpieces according to claim 3, characterized in that, Calculating the initial pose of the workpiece based on the workpiece image includes: Based on the alignment relationship between the optical pattern and the target metal workpiece, a three-dimensional spatial coordinate system is established. A point cloud is generated according to the coordinate positioning relationship of the optical pattern in the three-dimensional space. The point cloud is registered with the workpiece template to determine the orientation coordinates of the target hardware workpiece in three-dimensional space, thereby obtaining the initial pose.

5. The multi-sensor fusion method for assembling and positioning hardware workpieces according to claim 3, characterized in that, The optical patterns include sinusoidal fringes, pseudo-random speckle, grid patterns, or dot matrix patterns.

6. The multi-sensor fusion method for assembling and positioning hardware workpieces according to claim 1, characterized in that, By fusing visual pose data with distance data from proximity sensors, the pose of a workpiece that has reached a preset coordinate range is collaboratively evaluated and corrected, including: Establish the transformation relationship between the vision sensor coordinate system, the proximity sensor coordinate system, and the robot base coordinate system; The workpiece pose data acquired by the vision sensor is converted to the robot's base coordinate system to obtain the visual pose estimate. By combining the distance data measured by the proximity sensor with the robot's current pose, the estimated distance-constrained pose of the workpiece in the robot's base coordinate system is calculated. The visual pose estimate and the distance-constrained pose estimate are fused to obtain the optimal pose estimate for collaborative evaluation. Based on the optimal pose estimation value, the pose correction amount of the robot end effector is analyzed, and the pose correction amount is used to control the robot to perform corrective motion.

7. The multi-sensor fusion method for assembling and positioning hardware workpieces according to claim 6, characterized in that, The visual pose estimate and the distance-constrained pose estimate are fused to obtain the optimal pose estimate for collaborative evaluation, including: Establish the robot's movement posture path; Based on the attitude path, when the visual observation data arrives, the observation residual between the visual pose estimate and the predicted attitude state corresponding to the attitude path is calculated and the state estimate is updated. Based on the attitude path, when the distance observation data arrives, the observation residual between the distance-constrained pose estimate and the predicted attitude state corresponding to the attitude path is calculated and the state estimate is updated. Based on the Mahalanobis distance of the observation residuals of each sensor, the confidence weights of the corresponding visual pose estimate and the distance-constrained pose estimate in the fusion are dynamically adjusted, and the optimal pose estimate is output.

8. The multi-sensor fusion method for assembling and positioning hardware workpieces according to claim 1, characterized in that, The fusion of visual pose data and distance data from proximity sensors also previously included: By using hardware trigger signals or precise clock synchronization protocols, the timestamps of vision sensors, proximity sensors, and torque sensors are unified to align the time of data from different sensors.

9. A multi-sensor fusion-based hardware workpiece assembly and positioning system, characterized in that, The system is used to implement the multi-sensor fusion assembly and positioning method for hardware workpieces according to any one of claims 1-8, and the system includes: The workpiece image acquisition module is used to control the programmable lighting unit to project a preset optical pattern onto the workpiece and to acquire workpiece images using a vision sensor. The workpiece movement module is used to calculate the initial pose of the workpiece based on the workpiece image, and guide the actuator to move the workpiece to a target position within a preset range of coordinates according to the initial pose. The assembly execution module is used to fuse visual pose data and distance data from proximity sensors to collaboratively evaluate and correct the workpiece pose that has reached the preset range of coordinate circles, and fix it to the assembly execution coordinate position. The pose compensation module is used to select and execute the corresponding force-guided micro-operation strategy according to the geometric constraint type of the target assembly. It performs final pose compensation on the workpiece by interpreting the contact information fed back by the torque sensor in combination with the assembly execution coordinate position, and then executes the assembly operation action.

Images