Mechanical arm fine positioning and closed-loop calibration method and system based on heterogeneous visual fusion

Abstract

The invention discloses a mechanical arm fine positioning and closed-loop calibration method and system based on heterogeneous visual fusion, and the method comprises the steps: heterogeneous data synchronous collection and cross-modal joint calibration, semantic segmentation based on prompt learning and high-precision mask generation. Point cloud refining and edge noise elimination in a view cone space; and differential compensation and closed-loop feedback control based on a golden model. A heterogeneous visual fusion framework is constructed, and 3D point cloud noise is constrained by using a sub-pixel edge of a high-resolution 2D image, so that the positioning precision breaks through a sub-millimeter level; a segmentation large model with zero sample generalization ability and a prompt learning mechanism are introduced, so that the system can adapt to various workpieces under a complex background without repeated training; refining point cloud and a standard three-dimensional gold model are registered and fed back to the mechanical arm in a closed loop mode, so that the system can compensate and calibrate drift and accumulative errors in real time, and long-term stable operation is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of hand-eye coordination, in particular to a mechanical arm fine positioning and closed-loop calibration method and system based on heterogeneous vision fusion. BACKGROUND

[0002] With the in-depth development of "Industry 4.0" and intelligent manufacturing technology, the industrial robot operation scene is transforming from traditional rough operations such as carrying and stacking to fine and high-precision operations such as precision assembly, biological medicine, and semiconductor packaging. In the above application scenarios, such as micro test tube grabbing in biological experiments, chip alignment in electronic assembly, and precision part assembly, the end effector of the mechanical arm is often required to achieve sub-millimeter level (such as below 0.2 mm) positioning accuracy.

[0003] To achieve this goal, the machine vision system plays a crucial role in the positioning and guidance of industrial robots. The current mainstream visual positioning technology can be divided into two categories: two-dimensional (2D) vision-based and three-dimensional (3D) vision-based methods. However, in actual precision operation scenarios, both have obvious limitations: Firstly, for 3D vision technology, the depth sensors it relies on, such as structured light, time of flight, and binocular stereo vision, have inherent defects in physical principles. Especially in the edge of the object, high-reflective surface, or texture missing area, due to optical diffraction, multi-path reflection, and signal mixing factors, the three-dimensional point cloud data collected is prone to noise such as flying points and mixed pixels, resulting in point cloud divergence at the object contour and low data reliability. The object pose calculated from this will have a positioning error of usually millimeters at the contour edge, which is difficult to meet the precision requirements of precision assembly.

[0004] Secondly, although high-resolution 2D vision systems can provide sub-pixel level edge and texture information, they lack depth dimension in essence. Traditional 2D vision positioning is usually based on the assumption of a fixed height plane. Once there is a height change, tilt, or stacking of the target object, relying only on 2D images will result in positioning failure of the mechanical arm in the depth direction, and even cause collision.

[0005] In addition, in actual production environments, whether 2D or 3D vision is used, most systems still rely on an open-loop control framework, i.e., through hand-eye calibration to convert the vision coordinate system to the robot base coordinate system and then directly guide the movement. Over a long period of operation, factors such as thermal deformation of mechanical structures, vibration, and wear of parts will cause the calibration parameters to drift, and the system cannot sense and compensate for such errors in real time, resulting in a gradual decline in positioning accuracy over time.

[0006] To address these issues, some attempts have been made in existing technologies. For example, some solutions attempt to fuse 2D and 3D visual data, but these often remain at the level of simple data stitching or sequential processing, failing to fully utilize high-precision 2D edge information to effectively constrain and clean up noise in 3D point clouds. Other solutions introduce artificial intelligence models for target recognition and segmentation, but these are often limited to pure image processing and do not form a closed loop with the robot's physical execution. Still other methods use template matching or iterative nearest-point algorithms for pose estimation, but these are still limited by the influence of noisy point clouds and lack a dynamic compensation mechanism for accumulated system errors.

[0007] Therefore, existing technologies still lack a high-precision, robust robotic arm visual positioning and calibration method that can systematically integrate the detail advantages of high-resolution 2D images with the spatial information of 3D point clouds, achieve noise suppression at the sensor level, and form closed-loop error compensation at the control level. Summary of the Invention

[0008] To address the aforementioned technical problems in related technologies, this invention proposes a method and system for precise positioning and closed-loop calibration of robotic arms based on heterogeneous vision fusion, which can overcome the above-mentioned shortcomings of existing technologies.

[0009] To achieve the above-mentioned technical objectives, the technical solution of the present invention is implemented as follows: A method for precise positioning and closed-loop calibration of a robotic arm based on heterogeneous vision fusion includes the following steps: S1. Heterogeneous data synchronous acquisition and cross-modal joint calibration: A high-resolution 2D industrial camera and an industrial-grade 3D depth sensor are rigidly fixed on the same bracket and clocked by hardware triggering; the 2D camera and 3D sensor are jointly calibrated using a non-coplanar calibration plate to obtain the rigid body transformation matrix from the 3D sensor coordinate system to the 2D camera coordinate system. S2. Semantic segmentation and high-precision mask generation based on cue learning: Target detection is performed on the RGB image acquired in S1 to generate target bounding boxes as cue; The RGB image and the cue are input into a pre-trained high-quality segmentation model to output a sub-pixel precision binarized mask of the target workpiece. S3. Point cloud refinement and edge noise removal in the view frustum space: Using the rigid body transformation matrix and 2D camera intrinsic parameters obtained in S1, the original 3D point cloud acquired in S1 is projected onto the image plane where the binarized mask is located; 3D points whose projection points are located in the foreground area of ​​the mask are retained, and based on the distance field information of the mask, edge 3D points whose distance from the object contour is less than a preset threshold are removed to obtain the core point cloud. S4. Differential compensation and closed-loop feedback control based on the gold model: The core point cloud obtained in S3 is registered with the standard 3D model of the target workpiece, and the rigid body transformation matrix describing the deviation between the current workpiece posture and the standard posture is calculated; the deviation matrix is ​​transformed to the coordinate system of the robot arm base to generate the posture compensation amount, and drive the robot arm to perform closed-loop calibration and positioning.

[0010] Furthermore, in step S1, the optical axis angle between the 2D camera and the 3D sensor is less than 15 degrees, and the field of view overlap rate is greater than 80%; the clock synchronization configuration is master-slave trigger mode to ensure that the time deviation between the exposure center time of the RGB image and the acquisition center time of the 3D point cloud is less than 1ms.

[0011] Furthermore, the cross-modal joint calibration in step S1 specifically includes: using a non-coplanar calibration plate with a high-precision 2D dot array on its surface and the center of the dots having reflectivity differences; controlling the robotic arm to drive the acquisition system to simultaneously acquire 2D images and 3D point clouds in multiple poses within the workspace; extracting the pixel coordinates of the center of the dots in the 2D image and their corresponding three-dimensional spatial coordinates in the 3D point cloud, constructing 3D-2D corresponding point pairs, and using the PnP algorithm combined with nonlinear optimization to solve the rigid body transformation matrix.

[0012] Furthermore, in step S2, the high-quality segmentation model is the High-Quality SegmentAnything Model; the object detection uses a lightweight object detection network, and the detected bounding boxes are scaled out and used as prompt input.

[0013] Further, in step S3, the removal of edge 3D points based on the distance field information of the mask includes: performing Euclidean distance transformation on the binarized mask to obtain a distance field map; setting a safety threshold, and for each 3D point to be screened, querying the value of its projected pixel coordinates in the distance field map, and if the value is less than the safety threshold, then removing it.

[0014] Furthermore, in step S4, the standard three-dimensional model is obtained by scanning a standard workpiece with a high-precision 3D scanner or by sampling from a CAD design model; the registration adopts a point-to-surface iterative nearest-point algorithm, and the initial transformation provided by the center position of the target detection box and the average depth of the 3D sensor is used as the initial value.

[0015] Furthermore, the method also includes a dimensional decoupling step applied to multi-feature target localization: in step S2, multiple prompts are provided to guide the segmentation model to output masks or precise coordinates of different components of the target; wherein, the translation of the target in the XY plane and the rotation around the Z axis are determined by the coordinates of at least two feature points extracted from the 2D image; the height of the target and the tilt angle around the X and Y axes are determined by the planar parameters fitted by the 3D point cloud refined by the corresponding mask.

[0016] Furthermore, the dimensional decoupling step is specifically used in the alignment scenario of CPU chip and socket: the mask of the socket outer frame and the center coordinates of the positioning marker points are obtained by segmentation model; the original point cloud is refined and fitted to a plane using the outer frame mask to obtain the height and tilt attitude; the position and rotation deviation in the plane are calculated using the center coordinates of the marker points and the ideal coordinates.

[0017] A robotic arm precision positioning system based on heterogeneous vision fusion for implementing the method includes: The heterogeneous vision acquisition module includes a rigidly mounted high-resolution 2D industrial camera and an industrial-grade 3D depth sensor, as well as a hardware triggering unit for synchronizing their clocks. The data processing module is used to perform cross-modal joint calibration, semantic segmentation based on cue learning, point cloud refinement in view frustum space, and difference compensation calculation based on the golden model. The closed-loop control module is used to convert the calculated pose compensation amount into control commands for the robotic arm, driving the robotic arm to complete precise positioning and operation.

[0018] The beneficial effects of this invention are as follows: By constructing a heterogeneous visual fusion framework and utilizing the sub-pixel edge constraints of high-resolution 2D images to constrain 3D point cloud noise, the positioning accuracy is reduced to sub-millimeter level (<0.2mm). By introducing a large segmentation model with zero-sample generalization capability and a cue-based learning mechanism, the system can adapt to various workpieces in complex backgrounds without repeated training. By registering the refined point cloud with a standard 3D gold model and feeding it back to the robotic arm in a closed loop, the system can compensate for calibration drift and accumulated errors in real time, achieving long-term stable operation. This method improves accuracy, generalization, and stability while providing reliable technical support for precision assembly and automated operation. Detailed Implementation

[0019] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention are within the scope of protection of the present invention.

[0020] A method for precise positioning and closed-loop calibration of a robotic arm based on heterogeneous vision fusion according to an embodiment of the present invention includes the following steps: S1. Heterogeneous data synchronous acquisition and cross-modal joint calibration: A high-resolution 2D industrial camera and an industrial-grade 3D depth sensor are rigidly fixed on the same bracket and clocked by hardware triggering; the 2D camera and 3D sensor are jointly calibrated using a non-coplanar calibration plate to obtain the rigid body transformation matrix from the 3D sensor coordinate system to the 2D camera coordinate system. S2. Semantic segmentation and high-precision mask generation based on cue learning: Target detection is performed on the RGB image acquired in S1 to generate target bounding boxes as cue; The RGB image and the cue are input into a pre-trained high-quality segmentation model to output a sub-pixel precision binarized mask of the target workpiece. S3. Point cloud refinement and edge noise removal in the view frustum space: Using the rigid body transformation matrix and 2D camera intrinsic parameters obtained in S1, the original 3D point cloud acquired in S1 is projected onto the image plane where the binarized mask is located; 3D points whose projection points are located in the foreground area of ​​the mask are retained, and based on the distance field information of the mask, edge 3D points whose distance from the object contour is less than a preset threshold are removed to obtain the core point cloud. S4. Differential compensation and closed-loop feedback control based on the gold model: The core point cloud obtained in S3 is registered with the standard 3D model of the target workpiece, and the rigid body transformation matrix describing the deviation between the current workpiece posture and the standard posture is calculated; the deviation matrix is ​​transformed to the coordinate system of the robot arm base to generate the posture compensation amount, and drive the robot arm to perform closed-loop calibration and positioning.

[0021] Preferably, in step S1, the optical axis angle between the 2D camera and the 3D sensor is less than 15 degrees, and the field of view overlap rate is greater than 80%; the clock synchronization configuration is master-slave trigger mode to ensure that the time deviation between the exposure center time of the RGB image and the acquisition center time of the 3D point cloud is less than 1ms.

[0022] Preferably, the cross-modal joint calibration in step S1 specifically includes: using a non-coplanar calibration plate with a high-precision 2D dot array on its surface and the center of the dots having reflectivity differences; controlling a robotic arm to drive the acquisition system to simultaneously acquire 2D images and 3D point clouds in multiple poses within the workspace; extracting the pixel coordinates of the center of the dots in the 2D image and their corresponding three-dimensional spatial coordinates in the 3D point cloud, constructing 3D-2D corresponding point pairs, and using the PnP algorithm combined with nonlinear optimization to solve the rigid body transformation matrix.

[0023] Preferably, in step S2, the high-quality segmentation model is High-Quality Segment AnythingModel; the target detection uses a lightweight target detection network, and the detected bounding boxes are scaled out and used as prompt input.

[0024] Preferably, in step S3, the removal of edge 3D points based on the distance field information of the mask includes: performing Euclidean distance transformation on the binarized mask to obtain a distance field map; setting a safety threshold, and for each 3D point to be screened, querying the value of its projected pixel coordinates in the distance field map, and if the value is less than the safety threshold, then removing it.

[0025] Preferably, in step S4, the standard three-dimensional model is obtained by scanning a standard workpiece with a high-precision 3D scanner or by sampling from a CAD design model; the registration adopts a point-to-surface iterative nearest point algorithm, and the initial transformation provided by the center position of the target detection box and the average depth of the 3D sensor is used as the initial value.

[0026] Preferably, the method further includes a dimensional decoupling step for multi-feature target localization: in step S2, multiple prompts are provided to guide the segmentation model to output masks or precise coordinates of different components of the target; wherein, the translation of the target in the XY plane and the rotation around the Z axis are determined by the coordinates of at least two feature points extracted from the 2D image; the height of the target and the tilt angle around the X and Y axes are determined by the planar parameters fitted by the 3D point cloud refined by the corresponding mask.

[0027] Preferably, the dimensional decoupling step is specifically used in the alignment scenario of CPU chip and socket: the mask of the socket outer frame and the center coordinates of the positioning marker points are obtained by segmentation model; the original point cloud is refined and fitted to a plane using the outer frame mask to obtain the height and tilt attitude; the position and rotation deviation in the plane are calculated using the center coordinates of the marker points and the ideal coordinates.

[0028] A robotic arm precision positioning system based on heterogeneous vision fusion for implementing the method includes: The heterogeneous vision acquisition module includes a rigidly mounted high-resolution 2D industrial camera and an industrial-grade 3D depth sensor, as well as a hardware triggering unit for synchronizing their clocks. The data processing module is used to perform cross-modal joint calibration, semantic segmentation based on cue learning, point cloud refinement in view frustum space, and difference compensation calculation based on the golden model. The closed-loop control module is used to convert the calculated pose compensation amount into control commands for the robotic arm, driving the robotic arm to complete precise positioning and operation.

[0029] To facilitate understanding of the above technical solutions of the present invention, the following detailed description of the above technical solutions of the present invention will be provided through specific usage methods.

[0030] In practical application, the method and system for precise positioning and closed-loop calibration of a robotic arm based on heterogeneous vision fusion according to the present invention are as follows.

[0031] Step S1: This step aims to establish a hardware system capable of simultaneously acquiring high-resolution texture information and spatial depth information, and to unify the coordinate systems of both. Specifically, it includes: 1. Construction of a heterogeneous data acquisition system: Hardware selection and installation: Rigidly fix the high-resolution 2D industrial camera (resolution better than 20 megapixels) and the industrial-grade 3D depth sensor (such as structured light or ToF camera) on the same mechanical bracket. The angle between their optical axes should be less than 15 degrees to ensure a field of view overlap of more than 80%, and the installation height should cover the robotic arm's working space (e.g., 300-600mm from the worktable).

[0032] Clock synchronization configuration: Connect the external trigger interfaces of both devices via hardware trigger lines and connect them to the same synchronization controller. Set the master-slave trigger mode to ensure that the time deviation between the exposure center time of the RGB image and the acquisition center time of the 3D point cloud is less than 1ms, thereby achieving spatiotemporal alignment of the data.

[0033] 2. Cross-modal joint calibration: Calibration target fabrication: A custom-designed non-coplanar Halcon dot calibration plate is used. The surface of the calibration plate is printed with a high-precision 2D dot array (for 2D camera calibration), and the center of the dots has specific reflectivity difference characteristics (for 3D camera recognition).

[0034] Data acquisition: Control the robotic arm to drive the heterogeneous camera module to perform multi-angle movements (no less than 15 poses) in the workspace, and simultaneously acquire 2D images and 3D point cloud data at each position.

[0035] Intrinsic parameter calibration: Using Zhang Zhengyou's calibration method and plane fitting algorithm respectively, the intrinsic parameter matrix (focal length, principal point, distortion coefficient) of the 2D camera and the intrinsic parameter model of the 3D sensor are calculated.

[0036] External parameter joint solution: Extract the center pixel coordinates (u,v) of the dots in the 2D image, and simultaneously extract the spatial coordinates (X,Y,Z) of the corresponding center points in the 3D point cloud. Construct a set of corresponding point pairs from 3D to 2D, and use the Perspective-n-Point (PnP) algorithm combined with nonlinear optimization (such as the Levenberg-Marquardt algorithm) to solve for the rigid body transformation matrix (including the rotation matrix R and translation vector t) from the 3D sensor coordinate system to the 2D camera coordinate system. This matrix serves as the benchmark for subsequent cross-modal data fusion.

[0037] Step S2: Semantic segmentation based on cue learning. This step utilizes object detection algorithms to generate cue information, guiding the segmentation model to output a high-precision mask. Specifically, it includes: 1. Target detection generates a prompt box: Dataset preparation: Collect multi-angle RGB images of the target workpiece, annotate the bounding boxes of the target, and construct a training set.

[0038] Model Training and Inference: A lightweight object detection network (such as YOLOv8-Nano or NanoDet) is used for training. The RGB images acquired in real time in step S1 are input into the trained network, which outputs the predicted bounding boxes of the target workpiece. .

[0039] Hint generation: Format the predicted bounding box to the "Box Prompt" input format required by the HQ-SAM model. To improve robustness, the bounding box can be appropriately scaled out (e.g., outsized by 5%) to ensure it completely covers the target object.

[0040] 2. High-precision mask generation: Model loading: Preload pre-trained HQ-SAM (High-Quality Segment Anything Model) model weights.

[0041] Cue-guided segmentation: The original RGB image and the generated Box Prompt are simultaneously input into the HQ-SAM encoder. The model decoder, based on the semantic features within the cue box, focuses on the target region through an attention mechanism, outputting a binary mask M with the same resolution as the original image. In this mask, regions with a pixel value of 1 represent the foreground, and regions with a pixel value of 0 represent the background. Because HQ-SAM is optimized for edge details, this mask achieves sub-pixel-level segmentation accuracy at object edges.

[0042] Step S3: Point cloud refinement in the view frustum space. This step utilizes a 2D high-precision mask to construct a constrained region in 3D space, removing noisy point clouds. Specifically, it includes: Cone construction and projection: Using the joint extrinsic matrix and the 2D camera intrinsic matrix K obtained from step S1 calibration, a projection mapping function from 3D space to 2D image plane is established. Traverse each point in the original sparse point cloud and calculate its projected pixel coordinates on the 2D image; View frustum determination (intersection preservation): Check the corresponding values ​​of the projected coordinates in the mask image M. If M(u,v)=1 (i.e., it falls in the foreground region), then the 3D point is retained; if M(u,v)=0, it is directly discarded. This process is equivalent to constructing a view frustum in 3D space with the camera optical center as the vertex and the 2D mask as the cross section, and only retaining the point cloud inside the view frustum.

[0043] 2. Edge Erosion Denoising: Edge distance field calculation: Perform a Euclidean distance transform on the binarized mask image M to obtain the distance field image D. The value of each pixel in the image represents its Euclidean distance (in pixels) to the nearest background pixel (i.e., the edge of the object).

[0044] Noise point removal: Set a safety threshold t (e.g., t=5 pixels, corresponding to a physical space edge shrinkage of approximately 0.5mm). For each retained point, query its projected coordinates in the distance field map. If the value is negative, it indicates that the 3D point is located in a high-noise region at the edge of the object's outline (prone to generating flying points), and is removed; only points that are not noisy are retained.

[0045] Core point cloud output: The remaining point cloud set after the above two steps of filtering is defined as the "core point cloud". This point cloud only contains high-confidence data of the central region of the object's surface, completely removing edge flying point noise.

[0046] Step S4: Differential Compensation Based on the Golden Model. This step guides the robotic arm calibration by calculating the relative deviation between the measured point cloud and the standard template. Specifically, it includes: 1. Obtaining the Gold Model: A high-density standard point cloud template is obtained by offline scanning of a standard workpiece using a high-precision 3D scanner or by sampling directly from the workpiece's CAD design model. The origin of the workpiece's local coordinate system is defined as the reference benchmark under the ideal posture.

[0047] 2. ICP registration calculation deviation: Initial value assignment: Using the center position of the target detection box in step S2 and the average depth of the 3D sensor, an initial transformation matrix is ​​provided for the ICP algorithm to prevent it from getting trapped in local optima.

[0048] Iterative optimization: The core point cloud (source point cloud) and the golden model (target point cloud) are input into the Point-to-Plane (ICP) algorithm. The algorithm iteratively finds the nearest corresponding points, minimizing the spatial error function between the two sets of point clouds.

[0049] Deviation matrix output: After the algorithm converges, the final rigid body transformation matrix (including rotation and translation) is output. The physical meaning of this matrix is: the amount of spatial transformation required to transform the current actual workpiece posture to the ideal standard posture.

[0050] 3. Closed-loop feedback control: Coordinate system transformation: The deviation matrix based on the camera coordinate system is transformed to the robot arm base coordinate system using the hand-eye calibration matrix to obtain the pose compensation amount of the robot arm end effector.

[0051] Inverse kinematics compensation: The current end-effector pose is superimposed with compensation to calculate the target pose. Inverse kinematics calculation is then performed by the robot controller to drive the robot arm to fine-tune the end-effector position and orientation, ensuring precise alignment with the workpiece and completing closed-loop calibration.

[0052] The present invention will be further described in detail below with reference to specific embodiments.

[0053] Example 1: High-throughput test tube grasping scenario in a smart laboratory This embodiment is applied to an automated biological laboratory, where a robotic arm is required to precisely pick up transparent centrifuge tubes of a specific size from a densely packed test tube rack.

[0054] Step 1: Synchronous Acquisition of Heterogeneous Data A high-precision industrial camera (5472×3648 resolution) is mounted at the end effector of the robotic arm as a high-frequency sensing unit, and a high-precision structured light camera as a low-frequency spatial sensing unit. The extrinsic parameter matrices of both cameras are pre-calibrated. The controller triggers a data acquisition command to obtain a high-resolution RGB image. and original point cloud .at this time, There are obvious streaks and noise at the edge of the test tube.

[0055] Step 2: Semantic-driven high-precision contour extraction Will After downsampling, the image is input into the YOLO network to identify the detection frame of the target test tube. The original high-resolution image is then processed. The detection bounding box is used as input for the segmentation model. The model outputs a high-confidence binarized mask. This mask precisely outlines the circular contour of the test tube cap, with pixel-level precision.

[0056] Step 3: Point cloud cleaning based on mask projection Traversal For each point in the array, calculate its projected coordinates in the 2D plane. Only retain the points whose projected coordinates fall within the specified range. Points within the foreground area.

[0057] Core operation (edge ​​erosion): Performs secondary filtering on the retained points. Calculates the projection points to... Shortest Euclidean distance of the contour boundary Set an erosion threshold. (For example, the corresponding physical space is approximately 0.5mm). If The point was determined to be an "unreliable edge point" and was removed. The final point cloud was obtained. It only contains the purest surface data at the center of the object.

[0058] Step 4: ICP-based deviation calculation and execution Retrieve the standard point cloud template for this model of centrifuge tube from the database. (Representing a perfect positive attitude). The Point-to-Plane ICP algorithm is used to calculate... Compared to rigid body transformation matrix .Will The compensation command is converted into the center point (TCP) of the robotic arm's end effector. After the robotic arm performs the compensation, it grasps the tube, ensuring that the center of the gripper is perfectly aligned with the center of the test tube.

[0059] Example 2: Precision assembly of 3C electronics (CPU chip and socket alignment) This embodiment is applied to an electronic product production line, where CPU chips need to be precisely placed into motherboard sockets. The challenge in this scenario lies in the dense pins and severe metal reflection inside the socket, which causes numerous artifacts and data holes in the 3D point cloud in that area.

[0060] Step 1: Synchronous Acquisition of Heterogeneous Data 1.1 Hardware Configuration: A heterogeneous vision system similar to that in Example 1 is adopted. Due to the tiny features of the slot, a lens with higher magnification (such as a 25mm or 35mm telecentric lens) is selected for the 2D camera, and a laser line scan camera with higher precision is selected for the 3D sensor.

[0061] 1.2 Data Acquisition: Synchronously acquire high-definition images of the CPU socket on the motherboard. Compared with the original point cloud . The image clearly shows the slot's plastic frame, metal pressure bar, and "mark points" used for alignment. There is almost no noise in the slot pin area, but there is relatively reliable depth information on the surface of the plastic frame.

[0062] Step 2: Feature extraction based on multi-objective semantics (dimensional decoupling) 2.1 Multi-target segmentation: Input the segmentation model. Several hints are provided to guide the model in segmenting the two key targets: Target A (for 3D pose): The plastic outer frame of the slot. Output mask. .

[0063] Target B (for 2D positioning): Two circular marker points at the corner of the slot. Output their sub-pixel center coordinates. and .

[0064] 2.2 Dimensional Information Decoupling: This step is the core of this embodiment. We no longer expect the 3D point cloud to provide all 6D pose information, but instead perform task decomposition: XY plane positioning and in-plane rotation (X,Y,Rz): uniquely determined by the center coordinates of two Mark points extracted from the 2D image.

[0065] Height and spatial tilt (Z, Rx, Ry): determined by surface data of the slot frame in the 3D point cloud.

[0066] Step 3: Point cloud refinement and plane fitting based on decoupling information 3.1 Point cloud cleaning of the framework: using masks For the original point cloud The cleaning process (including foreground preservation and edge erosion) yields a clean point cloud containing only the surface of the slot frame. .

[0067] 3.2 Planar attitude fitting: for point clouds The RANSAC (Random Sample Consensus) algorithm is applied to fit an optimal plane. The normal vector of this plane uniquely determines the tilt orientation (Rx, Ry) of the slot in space, while the average height of the plane determines the Z coordinate.

[0068] Step 4: Deviation Calculation and Execution Based on the "Virtual Gold Model" 4.1 Constructing a Virtual Gold Model: The "gold model" here is not a complete 3D point cloud, but a set of ideal parameters: ideal 2D coordinates of Mark points. and Ideal plane normal vector: (i.e., perfectly horizontal); Ideal Z-height: .

[0069] 4.2 Calculate the deviation matrix : XYRz deviation: Comparison with measured Mark point coordinates Using ideal coordinates, the translational deviations in the X and Y directions and the rotational deviations around the Z axis are calculated.

[0070] ZRxRy deviation: By comparing the measured plane normal vector with the ideal normal vector, the tilt angle deviation around the X and Y axes is calculated; by comparing the measured Z height with the ideal Z height, the translation deviation in the Z direction is calculated.

[0071] 4.3 Compensation and Implementation: The deviations from the above six dimensions are combined into the final compensation matrix. The data is sent to the robotic arm. After the robotic arm performs the compensation action, it accurately places the CPU chip into the slot in the correct orientation, avoiding 3D positioning failure caused by pin reflection.

[0072] In summary, by employing the technical solutions described above in this invention, a heterogeneous visual fusion framework is constructed, utilizing the sub-pixel edge constraints of high-resolution 2D images to limit 3D point cloud noise, thereby achieving a positioning accuracy breakthrough to the sub-millimeter level (<0.2mm). By introducing a large segmentation model with zero-sample generalization capability and a cue-based learning mechanism, the system can adapt to various workpieces in complex backgrounds without repeated training. Furthermore, by registering the refined point cloud with a standard 3D gold model and providing closed-loop feedback to the robotic arm, the system can compensate for calibration drift and accumulated errors in real time, achieving long-term stable operation. This method improves accuracy, generalization, and stability while providing reliable technical support for precision assembly and automated operation.

[0073] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for precise positioning and closed-loop calibration of a robotic arm based on heterogeneous vision fusion, characterized in that, Includes the following steps: S1. Heterogeneous data synchronous acquisition and cross-modal joint calibration: A high-resolution 2D industrial camera and an industrial-grade 3D depth sensor are rigidly fixed on the same bracket and clocked by hardware triggering; the 2D camera and 3D sensor are jointly calibrated using a non-coplanar calibration plate to obtain the rigid body transformation matrix from the 3D sensor coordinate system to the 2D camera coordinate system. S2. Semantic segmentation and high-precision mask generation based on cue learning: Target detection is performed on the RGB image acquired in S1, and target bounding boxes are generated as cues; The RGB image and the prompt are input into a pre-trained high-quality segmentation model, which outputs a sub-pixel precision binarized mask of the target workpiece. S3. Point cloud refinement and edge noise removal in the view frustum space: Using the rigid body transformation matrix and 2D camera intrinsic parameters obtained in S1, the original 3D point cloud acquired in S1 is projected onto the image plane where the binarized mask is located; 3D points whose projection points are located in the foreground area of ​​the mask are retained, and based on the distance field information of the mask, edge 3D points whose distance from the object contour is less than a preset threshold are removed to obtain the core point cloud. S4. Differential compensation and closed-loop feedback control based on the gold model: The core point cloud obtained in S3 is registered with the standard 3D model of the target workpiece, and the rigid body transformation matrix describing the deviation between the current workpiece posture and the standard posture is calculated; the deviation matrix is ​​transformed to the coordinate system of the robot arm base to generate the posture compensation amount, and drive the robot arm to perform closed-loop calibration and positioning.

2. The method for precise positioning and closed-loop calibration of a robotic arm based on heterogeneous vision fusion according to claim 1, characterized in that, In step S1, the optical axis angle between the 2D camera and the 3D sensor is less than 15 degrees, and the field of view overlap rate is greater than 80%; the clock synchronization configuration is master-slave trigger mode to ensure that the time deviation between the exposure center time of the RGB image and the acquisition center time of the 3D point cloud is less than 1ms.

3. The method for precise positioning and closed-loop calibration of a robotic arm based on heterogeneous vision fusion according to claim 1 or 2, characterized in that, The cross-modal joint calibration in step S1 specifically includes: using a non-coplanar calibration plate with a high-precision 2D dot array on its surface and the center of the dots having reflectivity differences; controlling the robotic arm to drive the acquisition system to simultaneously acquire 2D images and 3D point clouds in multiple poses within the workspace; extracting the pixel coordinates of the center of the dots in the 2D image and their corresponding three-dimensional spatial coordinates in the 3D point cloud, constructing 3D-2D corresponding point pairs, and using the PnP algorithm combined with nonlinear optimization to solve the rigid body transformation matrix.

4. The method for precise positioning and closed-loop calibration of a robotic arm based on heterogeneous vision fusion according to claim 1, characterized in that, In step S2, the high-quality segmentation model is the High-Quality Segment Anything Model; the object detection uses a lightweight object detection network, and the detected bounding boxes are scaled out and used as prompt input.

5. The method for precise positioning and closed-loop calibration of a robotic arm based on heterogeneous vision fusion according to claim 1, characterized in that, In step S3, the removal of edge 3D points based on the distance field information of the mask includes: performing Euclidean distance transformation on the binarized mask to obtain a distance field map; setting a safety threshold, and for each 3D point to be filtered, querying the value of its projected pixel coordinates in the distance field map. If the value is less than the safety threshold, it is removed.

6. The method for precise positioning and closed-loop calibration of a robotic arm based on heterogeneous vision fusion according to claim 1, characterized in that, In step S4, the standard 3D model is obtained by scanning a standard workpiece with a high-precision 3D scanner or by sampling from a CAD design model; the registration adopts the point-to-surface iterative nearest point algorithm, and the initial transformation provided by the center position of the target detection box and the average depth of the 3D sensor is used as the initial value.

7. The method for precise positioning and closed-loop calibration of a robotic arm based on heterogeneous vision fusion according to claim 1, characterized in that, The method also includes a dimensional decoupling step applied to multi-feature target localization: in step S2, multiple prompts are provided to guide the segmentation model to output masks or precise coordinates of different components of the target; wherein, the translation of the target in the XY plane and the rotation around the Z axis are determined by the coordinates of at least two feature points extracted from the 2D image; the height of the target and the tilt angle around the X and Y axes are determined by the planar parameters fitted by the 3D point cloud refined by the corresponding mask.

8. The method for precise positioning and closed-loop calibration of a robotic arm based on heterogeneous vision fusion according to claim 7, characterized in that, The dimensional decoupling step is specifically used in the alignment scenario of CPU chip and socket: the mask of the outer frame of the socket and the center coordinates of the positioning marker points are obtained by segmentation model; the original point cloud is refined and fitted to a plane using the outer frame mask to obtain the height and tilt attitude; the position and rotation deviation in the plane are calculated using the center coordinates of the marker points and the ideal coordinates.

9. A robotic arm precision positioning system based on heterogeneous vision fusion for implementing the method of any one of claims 1-8, characterized in that, include: The heterogeneous vision acquisition module includes a rigidly mounted high-resolution 2D industrial camera and an industrial-grade 3D depth sensor, as well as a hardware triggering unit for synchronizing their clocks. The data processing module is used to perform cross-modal joint calibration, semantic segmentation based on cue learning, point cloud refinement in view frustum space, and difference compensation calculation based on the golden model. The closed-loop control module is used to convert the calculated pose compensation amount into control commands for the robotic arm, driving the robotic arm to complete precise positioning and operation.