Robot vision perception guidance method and system based on multi-source data fusion

Abstract

The application relates to the technical field of robot vision guidance, in particular to a robot vision perception guidance method and system based on multi-source data fusion, which comprises the following steps: collecting multi-modal images of a power secondary wiring box; extracting visual features and solving a preliminary pose; laser multi-point sampling and normal vector deviation calculation; attitude compensation and depth reference locking; secondary vision collection and accurate positioning: a depth camera is controlled to collect secondary high-resolution images, and sub-pixel level coordinates of screw center points in the wiring box are extracted; three-dimensional space coordinates of the screw center points under a mechanical arm base coordinate system are calculated, and a mechanical arm end tool is guided to perform work. The final positioning accuracy of millimeter-level or even sub-millimeter-level micro targets such as bolts and wiring columns is greatly improved, so that the automatic operation guidance precision of the sub-millimeter level is stably realized.

Description

Technical Field

[0001] This invention relates to the field of robot vision guidance technology, specifically to a robot vision perception guidance method and system based on multi-source data fusion. Background Technology

[0002] In the automated maintenance of smart grids, power robots have been widely used in the inspection and maintenance of secondary junction boxes in substations. The interior space of the secondary junction box is small, with densely distributed air switches, terminals and fastening screws. To automate the disassembly and assembly or perform live maintenance, the robot's end effector must have sub-millimeter positioning accuracy.

[0003] In existing technologies, power robots are typically guided using industrial cameras or depth cameras for target recognition and localization. However, in outdoor power operation environments, issues such as drastic changes in ambient light, strong reflections from metal surfaces, and complex backgrounds with intertwined cables cause significant systematic errors in the depth camera's point cloud data along the Z-axis. Furthermore, when the camera's imaging plane is not parallel to the junction box plane, visual processing results in severe projection distortion, amplifying pixel positioning errors for tiny targets like screws. This leads to a deviation in the normal vector of the tool center point (TCP) at the robotic arm's end effector, preventing the actuator from accurately engaging small targets and easily causing operation failure or equipment damage. Therefore, eliminating environmental interference and achieving high-precision attitude alignment and sub-millimeter-level positioning has become a pressing technical challenge for the industry. Summary of the Invention

[0004] To address the aforementioned problems, this invention provides a robot visual perception guidance method based on multi-source data fusion, comprising: Acquire multimodal images of the power secondary junction box; Visual feature extraction and preliminary pose calculation: The acquired multimodal images are input into a pre-trained multi-task deep learning model, which outputs the semantic segmentation mask of the junction box and the pixel coordinates of at least 3 structural key points in parallel; the preliminary pose transformation matrix of the junction box relative to the coordinate system of the robotic arm end effector is solved by combining the intrinsic parameter matrix of the depth camera and the standard physical model of the junction box. Laser multi-point sampling and normal vector deviation calculation: Based on the initial pose, at least 3 non-collinear sampling points are selected in the junction box plane area for ranging to obtain a set of sampling ranging values; the junction box plane is fitted based on the spatial coordinates of the sampling points, and the angle between the plane normal vector and the axial vector of the robot arm tool center point is calculated to obtain the normal vector deviation value; Attitude compensation and depth reference locking: Construct an attitude correction increment matrix based on the normal vector deviation value, drive the robotic arm to perform attitude rotation compensation until the angle between the normal vectors of the end effector plane and the junction box plane is not greater than a preset parallel threshold; collect laser ranging values ​​under parallel attitude to lock the zero-position depth reference of the robotic arm operation. Secondary visual acquisition and precise positioning: Control the depth camera to perform secondary high-resolution image acquisition, extract the sub-pixel level coordinates of the screw center point inside the junction box; combine the linear projection relationship under the depth reference and parallel plane constraints to calculate the three-dimensional spatial coordinates of the screw center point in the robot arm base coordinate system, and guide the robot arm end tool to perform the operation.

[0005] The acquisition of multimodal images of the power secondary junction box specifically involves using a pre-calibrated depth camera at the end of a robotic arm to simultaneously acquire RGB color images and 3D depth point cloud images of the power secondary junction box in the work scene, thereby constructing a multimodal image.

[0006] The specific operation for solving the preliminary pose transformation matrix of the junction box relative to the coordinate system of the robotic arm end effector is as follows: The pixel coordinates of key feature points output by the multi-task deep learning model are compared with the spatial point coordinates corresponding to the physical model of the junction box to establish a spatial point pair matching relationship. A perspective projection equation is constructed by combining the camera intrinsic parameter matrix of the depth camera; The spatial coordinate set of corresponding feature points is obtained based on the physical model of the junction box. The spatial transformation matrix of the robotic arm end relative to the junction box is obtained by minimizing the reprojection error, which serves as the initial pose transformation matrix.

[0007] The formula for minimizing the reprojection error is as follows: Where K is the intrinsic parameter matrix; R is the rotation matrix; T is the translation vector; P ωi This is the set of spatial coordinates of the feature points corresponding to the physical model of the junction box. p i This is the set of pixel coordinates for key feature points.

[0008] The specific operations for laser multi-point sampling and normal vector deviation calculation are as follows: Guided by the initial pose, the robotic arm moves to the preset sampling pose. Three non-collinear sampling points A, B, and C are selected on the junction box plane. The laser rangefinder mounted on the end of the robotic arm measures the distance to the selected sampling points to obtain depth values. d A , d B , d C ; Based on the depth value, determine the coordinates of three sampling points in the coordinate system of the robotic arm's end effector, denoted as P. A (x A ,y A ,d A ), P B (x B ,y B ,z B ) and P C (x C ,y C ,z C ); Construct two vectors in the plane using three non-collinear points, and then obtain the normal vector of the junction box plane through the cross product of these vectors. ; Based on the axis vector of the TCP at the end of the robotic arm The normal vector of the junction box plane After normalization, calculate the vector relative to the TCP axis. The included angle is the normal vector deviation value, which includes pitch deviation. α Yaw deviation β .

[0009] The attitude compensation process is as follows: based on pitch deviation... α Yaw deviation β Construct the attitude correction increment matrix R adj The robotic arm is controlled to rotate around the X-axis, centered on the current TCP position. α Correct the pitch deviation, then rotate around the Y-axis. β Correct the yaw deviation; repeat the S3 laser multi-point sampling and normal vector deviation calculation until the angle between the normal vectors of the end effector plane and the junction box plane is not greater than the preset parallel threshold.

[0010] The zero-position depth reference for locking the robotic arm operation is specifically as follows: after controlling the robotic arm to rotate and correct to a parallel posture, the laser rangefinder is used to remeasure the non-collinear sampling points A, B, and C to obtain the depth value. d A , d B , d C ;by As a zero-position depth reference for robotic arm operations.

[0011] The specific operation for calculating the three-dimensional spatial coordinates of the screw center point in the robot arm's base coordinate system is as follows: The depth camera is controlled to perform secondary image acquisition on the junction box, and the center point of the target screw is extracted at the sub-pixel level to obtain the sub-pixel coordinates of the screw center point. us , v s ); Calculate the physical offset (ΔX, ΔY) based on the linear projection relationship between the depth datum and the parallel plane constraint: in,( u 0 , v 0 ( ) represents the coordinates of the camera's principal point. f x , f y Let x be the focal length of the camera in the x and y directions. Zero depth reference; Based on the physical offset and the coordinates of the current robotic arm end effector in the base coordinate system, the spatial coordinates of the screw center point in the robotic arm's base coordinate system are calculated as follows: Among them, P tcp The coordinates of the current robotic arm end effector in the base coordinate system.

[0012] The multi-task deep learning model includes a shared backbone network, three parallel task head branches, and a multi-task weighted loss function. The parallel task head branches include a target detection branch for outputting the bounding box and confidence score of the junction box; a semantic segmentation branch for outputting the pixel-level segmentation mask of the junction box; and a keypoint regression branch for outputting the pixel coordinates of multiple structural corner points of the junction box.

[0013] A robot vision perception guidance system that implements the multi-source data fusion-based robot vision perception guidance method described above includes: The image acquisition module is used to acquire multimodal images of the power secondary junction box; The feature extraction and pose calculation module is used to input the acquired multimodal images into a pre-trained multi-task deep learning model and output the semantic segmentation mask of the junction box and the pixel coordinates of at least 3 structural key points in parallel; combined with the intrinsic parameter matrix of the depth camera and the standard physical model of the junction box, the module solves the preliminary pose transformation matrix of the junction box relative to the coordinate system of the robotic arm end effector. The calculation module is used to select at least three non-collinear sampling points in the junction box plane area based on the initial pose to measure the distance and obtain a set of sampled distance values; based on the spatial coordinates of the sampling points, the junction box plane is fitted, and the angle between the plane normal vector and the axial vector of the robot arm tool center point is calculated to obtain the normal vector deviation value. The attitude compensation module is used to construct an attitude correction increment matrix based on the normal vector deviation value, drive the robotic arm to perform attitude rotation compensation until the angle between the normal vectors of the end effector plane and the junction box plane is not greater than a preset parallel threshold; and collect laser ranging values ​​under parallel attitude to lock the zero depth reference of the robotic arm operation. The precision positioning module is used to control the depth camera to perform secondary high-resolution image acquisition and extract the sub-pixel level coordinates of the screw center point inside the junction box. Combining the linear projection relationship under the depth reference and parallel plane constraints, it calculates the three-dimensional spatial coordinates of the screw center point in the robot arm base coordinate system and guides the end effector of the robot arm to perform the operation.

[0014] Beneficial effects: This invention is a robot visual perception guidance method and system based on multi-source data fusion. It enhances the robustness of target recognition in complex backgrounds by simultaneously extracting the semantic and geometric features of the target through a multi-task deep learning model. Combined with the active measurement characteristics of laser ranging, it effectively eliminates depth perception errors caused by changes in outdoor lighting and metal reflections.

[0015] A high-precision laser rangefinder is introduced as an active measurement unit to perform real-time and active geometric compensation on the preliminary pose estimation results based on vision. By fitting the target plane through laser multi-point sampling, the posture deviation of the robotic arm end is calculated and compensated in real time. This can effectively detect and correct the deviation of the normal vector direction between the tool center point TCP plane and the target working surface, fundamentally eliminating the problem of inaccurate final positioning caused by the combined effects of depth camera system errors, lens projection distortion, and environmental interference.

[0016] After establishing the ideal operational geometry of parallel posture through compensation, a secondary visual precision positioning process is performed. This cleverly simplifies the originally complex three-dimensional spatial search and positioning problem into a linear projection calculation problem in a two-dimensional plane. This not only significantly improves computational efficiency, but more importantly, it greatly enhances the final positioning accuracy of millimeter-level or even sub-millimeter-level micro-targets such as bolts and terminals, thereby stably achieving sub-millimeter-level automated operation guidance accuracy.

[0017] In summary, this invention significantly improves the reliability, accuracy, and overall efficiency of power robots in performing automated tasks such as junction box maintenance and screw tightening. Attached Figure Description

[0018] The solutions and advantages of this application will become clear to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of the invention.

[0019] In the attached diagram: Figure 1This is a flowchart illustrating the robot vision perception guidance method of the present invention. Detailed Implementation

[0020] Exemplary embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings.

[0021] Example This embodiment provides a robot vision perception guidance method based on multi-source data fusion, applied to a robotic arm, such as... Figure 1 As shown, the robot has an embedded computing control unit connected to a multimodal perception unit and a robotic arm controller, which is used to build a multi-task deep learning model for data processing and robotic arm control. The robotic arm is equipped with a depth camera and a high-precision laser rangefinder to acquire multimodal images, including RGB, depth, and infrared data, for comprehensive environmental perception; the robotic arm also serves as an execution unit, with an electric screwdriver tool mounted at its end.

[0022] The specific implementation steps of the method are as follows: S1. Acquire multimodal images of the power secondary junction box: Simultaneously acquire RGB color images and 3D depth point cloud images of the power secondary junction box in the operation scene using a pre-calibrated depth camera at the end of the robotic arm to construct a multimodal image.

[0023] The depth camera at the end effector of the robotic arm is pre-calibrated, and pixel coordinates are established after calibration. u , v ) and spatial point P in the camera coordinate system c (x c ,y c ,z c Mapping relationship: in,( u 0 , v 0 ( ) represents the coordinates of the camera's principal point. f x , f y Let x be the focal length of the camera in the x and y directions.

[0024] RGB color images of the work area are acquired using a depth camera mounted at the end of the robotic arm. rgb With depth point cloud image D depth .

[0025] S2. Visual Feature Extraction and Preliminary Pose Calculation: Input the acquired multimodal images into a pre-trained multi-task deep learning model, and output the semantic segmentation mask of the junction box and the pixel coordinates of at least 3 structural key points in parallel; combine the intrinsic parameter matrix of the depth camera and the standard physical model of the junction box to solve the preliminary pose transformation matrix of the junction box relative to the coordinate system of the robotic arm end effector.

[0026] The multi-task deep learning model described in S2 includes a shared backbone network, three parallel task head branches, and a multi-task weighted loss function.

[0027] The shared backbone network uses ResNet50 combined with FPN feature pyramid network to extract multi-scale visual features.

[0028] The three parallel task head branches include: an object detection branch, which outputs the bounding box and confidence score of the junction box; a semantic segmentation branch, which outputs the pixel-level segmentation mask of the junction box; and a keypoint regression branch, which outputs the pixel coordinates of multiple structural corner points of the junction box. Multi-task weighted loss function: Among them, L det For target detection loss, L seg For cross-entropy splitting loss, L kp This represents the loss at key points due to mean square error.

[0029] Using a multi-task deep learning model to analyze RGB color images rgb Extract the edge mask of the junction box and identify m key feature points. The set of pixel coordinates of the key feature points is { p i ( u i , v i )| i =1,2,…,m}.

[0030] The acquired RGB color image I rgb The input is into a pre-trained multi-task deep learning model; this model adopts an encoder-decoder structure, where the encoder is a ResNet50 and the decoder contains three parallel branches: Target detection branch: Outputs the bounding box and confidence score of the junction box to assist in filtering valid regions; Semantic segmentation branch: Outputs pixel-level classification results with the same resolution as the input image, where the pixel value corresponding to the junction box area is 1 and the background is 0, thus obtaining the edge mask of the junction box; Keypoint Regression Branch: This branch outputs the pixel coordinates of multiple structural key points of the junction box by regressing the heatmap or directly regressing the coordinates, denoted as set {p i ( u i , v i )| i =1,2,…,m}; The three parallel task head branches mentioned above share the same deep features extracted by the same backbone network, perform parallel computation during inference, and finally extract the junction box outline from the segmentation mask and obtain at least 3 pixel coordinate points that can be used for pose calculation from the key point regression branch.

[0031] The standard physical model of the junction box, which is integrated with the depth camera, is used to solve for the preliminary pose transformation matrix of the junction box relative to the coordinate system of the robotic arm's end effector. The specific operation is as follows: The pixel coordinates of key feature points output by the multi-task deep learning model are compared with the spatial point coordinates corresponding to the physical model of the junction box to establish a spatial point pair matching relationship. Construct the perspective projection equation by combining the camera intrinsic parameter matrix of the depth camera: Where s is the scale factor; K is the intrinsic parameter matrix; ( u i , v i R represents the pixel coordinates of the key feature points; R is the rotation matrix; T is the translation vector; X i ,Y i ,Z i ) represents the spatial coordinates of the feature points corresponding to the physical model of the junction box.

[0032] Based on the physical model of the junction box, obtain the set of spatial coordinates of the corresponding feature points {P}. ωi (X i ,Y i Z i The spatial transformation matrix of the robotic arm's end effector relative to the junction box is obtained by minimizing the reprojection error, serving as the initial pose transformation matrix; the formula for minimizing the reprojection error is: The rotation matrix R and translation vector T are obtained by minimizing the reprojection error, thus yielding the preliminary pose transformation matrix. .

[0033] S3. Laser Multi-Point Sampling and Normal Vector Deviation Calculation: Based on the initial pose, at least three non-collinear sampling points are selected within the junction box plane area for ranging, obtaining a set of sampling ranging values; the junction box plane is fitted based on the spatial coordinates of the sampling points, and the angle between the plane's normal vector and the axial vector of the robotic arm tool center point is calculated to obtain the normal vector deviation value. The specific operation is as follows: S301. Guide the robotic arm to move to the preset sampling pose based on the initial pose, select three non-collinear sampling points A, B, and C on the junction box plane, and use the laser rangefinder mounted on the end of the robotic arm to measure the distance to the selected sampling points to obtain the depth value. d A , d B , d C .

[0034] S302. Based on the depth value, determine the coordinates of the three sampling points in the coordinate system of the robotic arm's end effector, denoted as P. A (x A ,y A ,d A ), P B (x B ,y B ,z B ) and P C (x C ,y C ,z C ).

[0035] Where, x A y A These are the projected coordinates of sampling point A on the XY plane at the end of the robotic arm; d A Let A be the Z-axis distance of sampling point A relative to the TCP at the end of the robotic arm, as measured by the laser rangefinder. Similarly, (x B ,y B ,z B ), (x C ,y C ,z C ) are the coordinates of sampling points B and C, respectively.

[0036] S303. Construct two vectors in the plane using three non-collinear points, and then obtain the normal vector of the junction box plane through the vector cross product. The calculation method is as follows: Junction box plane normal vector : S304, Based on the axis vector of the TCP at the end of the robotic arm The normal vector of the junction box plane After normalization, calculate the vector relative to the TCP axis. The included angle is the normal vector deviation value, which includes pitch deviation. α Yaw deviation β .

[0037] The normal vector of the junction box plane is normalized to: The unit plane normal vector With TCP axis vector The included angle is: Pitch deviation α : .

[0038] Yaw deviation β : .

[0039] S4. Attitude Compensation and Depth Reference Locking: Construct an attitude correction increment matrix based on the normal vector deviation value, drive the robotic arm to perform attitude rotation compensation until the angle between the normal vectors of the end effector plane and the junction box plane is not greater than the preset parallel threshold; collect laser ranging values ​​under parallel attitude to lock the zero-position depth reference of the robotic arm operation.

[0040] The attitude compensation process described in S4 is as follows: based on pitch deviation... α Yaw deviation β Construct the attitude correction increment matrix R adj The robotic arm is controlled to rotate around the X-axis, centered on the current TCP position. α Correct the pitch deviation, then rotate around the Y-axis. β Correct the yaw deviation; repeat the S3 laser multi-point sampling and normal vector deviation calculation until the angle between the normal vectors of the end effector plane and the junction box plane is not greater than the preset parallel threshold.

[0041] After the robotic arm is rotated and corrected to a parallel posture, a laser rangefinder is used to remeasure the distances at the non-collinear sampling points A, B, and C to obtain depth values. d A , d B , d C ;by As a zero-position depth reference for robotic arm operations.

[0042] S5. Secondary Visual Acquisition and Precise Positioning: Control the depth camera to perform secondary high-resolution image acquisition, extracting the sub-pixel level coordinates of the screw center point inside the junction box; combine the linear projection relationship under the depth reference and parallel plane constraints to calculate the three-dimensional spatial coordinates of the screw center point in the robot arm's base coordinate system, guiding the robot arm's end effector to perform the operation. The specific operation is as follows: S501 controls the depth camera to perform secondary image acquisition on the junction box, and extracts the sub-pixel level of the center point of the target screw to obtain the sub-pixel level coordinates of the screw center point. u s , v s ); S502. Calculate the physical offset (ΔX, ΔY) based on the linear projection relationship between the depth datum and the parallel plane constraint: in,( u 0 , v 0 ( ) represents the coordinates of the camera's principal point. f x , f y Let x be the focal length of the camera in the x and y directions.

[0043] S503. Based on the physical offset and the coordinates of the current robotic arm end effector in the base coordinate system, calculate the spatial coordinates of the screw center point in the robotic arm's base coordinate system as follows: Among them, P tcp The coordinates of the current robotic arm end effector in the base coordinate system.

[0044] S504. Guide the electric screwdriver tool at the end of the robotic arm to move above the screw and perform the operation.

[0045] In addition, a robot vision perception guidance system based on multi-source data fusion is also provided, including: The image acquisition module is used to acquire multimodal images of the power secondary junction box; The feature extraction and pose calculation module is used to input the acquired multimodal images into a pre-trained multi-task deep learning model and output the semantic segmentation mask of the junction box and the pixel coordinates of at least 3 structural key points in parallel; combined with the intrinsic parameter matrix of the depth camera and the standard physical model of the junction box, the module solves the preliminary pose transformation matrix of the junction box relative to the coordinate system of the robotic arm end effector. The calculation module is used to select at least three non-collinear sampling points in the junction box plane area based on the initial pose to measure the distance and obtain a set of sampled distance values; based on the spatial coordinates of the sampling points, the junction box plane is fitted, and the angle between the plane normal vector and the axial vector of the robot arm tool center point is calculated to obtain the normal vector deviation value. The attitude adjustment module is used to construct an attitude correction increment matrix based on the normal vector deviation value, drive the robotic arm to perform attitude rotation compensation until the angle between the normal vectors of the end effector plane and the junction box plane is not greater than a preset parallel threshold; and collect laser ranging values ​​under parallel attitude to lock the zero depth reference of the robotic arm operation. The coordinate positioning module is used to control the depth camera to perform secondary high-resolution image acquisition, extract the sub-pixel level coordinates of the screw center point inside the junction box; combined with the linear projection relationship under the constraints of the depth reference and the parallel plane, it calculates the three-dimensional spatial coordinates of the screw center point in the robot arm base coordinate system, and guides the end effector of the robot arm to perform the operation.

[0046] While exemplary embodiments of the invention have been described herein, many other variations or modifications conforming to the principles of the invention can be directly determined or derived from the disclosure of this invention without departing from its spirit and scope. Therefore, the scope of the invention should be understood and recognized to cover all such other variations or modifications.

Claims

1. A robot visual perception guidance method based on multi-source data fusion, characterized in that, include: Acquire multimodal images of the power secondary junction box; Visual feature extraction and preliminary pose calculation: The acquired multimodal images are input into a pre-trained multi-task deep learning model, which outputs the semantic segmentation mask of the junction box and the pixel coordinates of at least 3 structural key points in parallel; the preliminary pose transformation matrix of the junction box relative to the coordinate system of the robotic arm end effector is solved by combining the intrinsic parameter matrix of the depth camera and the standard physical model of the junction box. Laser multi-point sampling and normal vector deviation calculation: Based on the initial pose, at least 3 non-collinear sampling points are selected in the junction box plane area for ranging to obtain a set of sampling ranging values; the junction box plane is fitted based on the spatial coordinates of the sampling points, and the angle between the plane normal vector and the axial vector of the robot arm tool center point is calculated to obtain the normal vector deviation value; Attitude compensation and depth reference locking: Construct an attitude correction increment matrix based on the normal vector deviation value, drive the robotic arm to perform attitude rotation compensation until the angle between the normal vectors of the end effector plane and the junction box plane is not greater than a preset parallel threshold; collect laser ranging values ​​under parallel attitude to lock the zero-position depth reference of the robotic arm operation. Secondary visual acquisition and precise positioning: Control the depth camera to perform secondary high-resolution image acquisition, extract the sub-pixel level coordinates of the screw center point inside the junction box; combine the linear projection relationship under the depth reference and parallel plane constraints to calculate the three-dimensional spatial coordinates of the screw center point in the robot arm base coordinate system, and guide the robot arm end tool to perform the operation.

2. The robot visual perception guidance method based on multi-source data fusion according to claim 1, characterized in that, The acquisition of multimodal images of the power secondary junction box specifically involves using a pre-calibrated depth camera at the end of a robotic arm to simultaneously acquire RGB color images and 3D depth point cloud images of the power secondary junction box in the work scene, thereby constructing a multimodal image.

3. The robot visual perception guidance method based on multi-source data fusion according to claim 1, characterized in that, The specific operation for solving the preliminary pose transformation matrix of the junction box relative to the coordinate system of the robotic arm end effector is as follows: The pixel coordinates of key feature points output by the multi-task deep learning model are compared with the spatial point coordinates corresponding to the physical model of the junction box to establish a spatial point pair matching relationship. A perspective projection equation is constructed by combining the camera intrinsic parameter matrix of the depth camera; The spatial coordinate set of corresponding feature points is obtained based on the physical model of the junction box. The spatial transformation matrix of the robotic arm end relative to the junction box is obtained by minimizing the reprojection error, which serves as the initial pose transformation matrix.

4. The robot visual perception guidance method based on multi-source data fusion according to claim 3, characterized in that, The formula for minimizing the reprojection error is as follows: Where K is the intrinsic parameter matrix; R is the rotation matrix; T is the translation vector; P ωi This is the set of spatial coordinates of the feature points corresponding to the physical model of the junction box. p i This is the set of pixel coordinates for key feature points.

5. The robot visual perception guidance method based on multi-source data fusion according to claim 1, characterized in that, The specific operations for laser multi-point sampling and normal vector deviation calculation are as follows: Guided by the initial pose, the robotic arm moves to the preset sampling pose. Three non-collinear sampling points A, B, and C are selected on the junction box plane. The laser rangefinder mounted on the end of the robotic arm measures the distance to the selected sampling points to obtain depth values. d A , d B , d C ; Based on the depth value, determine the coordinates of three sampling points in the coordinate system of the robotic arm's end effector, denoted as P. A (x A ,y A ,d A ), P B (x B ,y B ,z B ) and P C (x C ,y C ,z C ); Construct two vectors in the plane using three non-collinear points, and then obtain the normal vector of the junction box plane through the cross product of these vectors. ; Based on the axis vector of the TCP at the end of the robotic arm The normal vector of the junction box plane After normalization, calculate the vector relative to the TCP axis. The included angle is the normal vector deviation value, which includes pitch deviation. α Yaw deviation β .

6. The robot visual perception guidance method based on multi-source data fusion according to claim 5, characterized in that, The attitude compensation process is as follows: based on pitch deviation... α Yaw deviation β Construct the attitude correction increment matrix R adj The robotic arm is controlled to rotate around the X-axis, centered on the current TCP position. α Correct the pitch deviation, then rotate around the Y-axis. β Correct the yaw deviation; repeat the S3 laser multi-point sampling and normal vector deviation calculation until the angle between the normal vectors of the end effector plane and the junction box plane is not greater than the preset parallel threshold.

7. The robot visual perception guidance method based on multi-source data fusion according to claim 1, characterized in that, The zero-position depth reference for locking the robotic arm operation is specifically as follows: after controlling the robotic arm to rotate and correct to a parallel posture, the laser rangefinder is used to remeasure the non-collinear sampling points A, B, and C to obtain the depth value. d A , d B , d C ;by As a zero-position depth reference for robotic arm operations.

8. The robot visual perception guidance method based on multi-source data fusion according to claim 1, characterized in that, The specific operation for calculating the three-dimensional spatial coordinates of the screw center point in the robot arm's base coordinate system is as follows: The depth camera is controlled to perform secondary image acquisition on the junction box, and the center point of the target screw is extracted at the sub-pixel level to obtain the sub-pixel coordinates of the screw center point. u s , v s ); Calculate the physical offset (ΔX, ΔY) based on the linear projection relationship between the depth datum and the parallel plane constraint: in,( u 0 , v 0 ( ) represents the coordinates of the camera's principal point. f x , f y Let x be the focal length of the camera in the x and y directions. Zero depth reference; Based on the physical offset and the coordinates of the current robotic arm end effector in the base coordinate system, the spatial coordinates of the screw center point in the robotic arm's base coordinate system are calculated as follows: Among them, P tcp The coordinates of the current robotic arm end effector in the base coordinate system.

9. The robot visual perception guidance method based on multi-source data fusion according to claim 1, characterized in that, The multi-task deep learning model includes a shared backbone network, three parallel task head branches, and a multi-task weighted loss function. The parallel task head branches include a target detection branch for outputting the bounding box and confidence score of the junction box, a semantic segmentation branch for outputting the pixel-level segmentation mask of the junction box, and a key point regression branch for outputting the pixel coordinates of multiple structural corner points of the junction box.

10. A robot vision perception guidance system that implements the robot vision perception guidance method based on multi-source data fusion as described in claim 1, characterized in that, include: The image acquisition module is used to acquire multimodal images of the power secondary junction box; The feature extraction and pose calculation module is used to input the acquired multimodal images into a pre-trained multi-task deep learning model and output the semantic segmentation mask of the junction box and the pixel coordinates of at least 3 structural key points in parallel; combined with the intrinsic parameter matrix of the depth camera and the standard physical model of the junction box, the module solves the preliminary pose transformation matrix of the junction box relative to the coordinate system of the robotic arm end effector. The calculation module is used to select at least three non-collinear sampling points in the junction box plane area based on the initial pose to measure the distance and obtain a set of sampled distance values; based on the spatial coordinates of the sampling points, the junction box plane is fitted, and the angle between the plane normal vector and the axial vector of the robot arm tool center point is calculated to obtain the normal vector deviation value. The attitude compensation module is used to construct an attitude correction increment matrix based on the normal vector deviation value, drive the robotic arm to perform attitude rotation compensation until the angle between the normal vectors of the end effector plane and the junction box plane is not greater than a preset parallel threshold; and collect laser ranging values ​​under parallel attitude to lock the zero depth reference of the robotic arm operation. The precision positioning module is used to control the depth camera to perform secondary high-resolution image acquisition and extract the sub-pixel level coordinates of the screw center point inside the junction box. Combining the linear projection relationship under the depth reference and parallel plane constraints, it calculates the three-dimensional spatial coordinates of the screw center point in the robot arm base coordinate system and guides the end effector of the robot arm to perform the operation.

Images