A device, system, and method for vision guidance of a barrier-blocking robot.

By combining a multi-camera system with a target pose tracking algorithm based on deep learning, the problems of personnel radiation exposure and insufficient positioning accuracy during the installation process of the blocking robot are solved, realizing the fully automated installation and disassembly of the blocking robot and improving the applicability and accuracy of the system.

CN119839856BActive Publication Date: 2026-11-10RES INST OF NUCLEAR POWER OPERATION +1
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202510179419.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2025-02-19
Publication Date
2026-11-10
Estimated Expiration
2045-02-19

AI Technical Summary

Technical Problem

Existing technologies have problems such as high radiation exposure for personnel during the installation of high-water-level plugs in steam generators, significant impact of human factors on installation quality, insufficient depth positioning accuracy, and cumbersome and dangerous calibration processes, making it impossible to achieve fully automated visual guidance for plugging robots.

Method used

A multi-camera system combined with a deep learning target pose tracking algorithm is used to achieve real-time pose tracking and control of the target at the end of the robotic arm through automated calibration and hand-eye calibration. The PNP algorithm and ICP algorithm are combined to accurately calculate the target pose, and the robotic arm motion control commands are output through the industrial control computer processor to complete the blocking operation process.

Benefits of technology

The system enables fully automated installation and disassembly of the blocking robot in a radiation environment, improving depth positioning accuracy, reducing manual intervention, and enhancing the system's applicability and scalability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119839856B_ABST
    Figure CN119839856B_ABST
Patent Text Reader

Abstract

This invention belongs to the field of nuclear power maintenance technology, specifically relating to a device, system, and method for visual guidance of a blockage robot. It includes a work site camera, a target plate, an industrial control computer processor, and a remote monitoring terminal. The target plate is mounted on the end effector of the robotic arm. The work site camera is connected to the industrial control computer processor, which is connected to the remote monitoring terminal, which has a display screen. The advantages of this invention are: it enables automated hand-eye calibration and multi-camera calibration of the guidance system without manual intervention; it employs a deep learning-based target detection and tracking algorithm, providing highly robust target tracking performance under visible light conditions; and it combines the flexibility and ease of use of a single-camera system with the high accuracy of multi-camera 3D pose estimation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the field of nuclear power maintenance technology, specifically relating to a device, system, and method for visual guidance of a blockage robot. Background Technology

[0002] The high-water-level plug plate (hereinafter referred to as the plug plate) of the steam generator is used to isolate the main pipeline under high-water-level conditions, ensuring that other maintenance work in the steam generator water chamber can be carried out simultaneously during the high-water-level overhaul of the nuclear power plant, thus saving overhaul time. Currently, the installation and removal of the plug plate in the primary side water chamber of the steam generator is generally carried out manually. Because the water chamber space is a high-dose radiation area and the space is narrow, there is a high radiation dose to personnel during the manual installation and removal of the plug plate, and the installation quality is greatly affected by human factors. Therefore, many research institutions at home and abroad have begun to explore the use of automated installation equipment to replace operators in completing the work inside the water chamber. However, due to the high technical difficulty, there are still no mature automated installation engineering products applied to the field. The installation process of the high-water-level plug plate of the steam generator can be mainly divided into three processes: robot body installation, plug plate and sealing component installation, and bolt tightening. Some operations are completed by remote control by personnel outside the steam generator water chamber, while others are completed by the plug plate installation and removal robot.

[0003] The installation of the plug and seals is mainly completed by remote control operation of the plugging robot, and consists of four steps:

[0004] (1) The operator sends the block plate into the block plate inlet, and the internal robotic arm picks up the inserted block plate.

[0005] (2) The robotic arm moves and places the clamped block at the inner ring flange opening;

[0006] (3) Align the center of the robotic arm screwing assembly with the center of the inner flange opening;

[0007] (4) Unfold the screwing assembly, move it around the center to each bolt, and tighten the bolt.

[0008] The existing technical solutions mainly include the following two types:

[0009] The first method is a single camera + target approach:

[0010] A single camera is used to detect the target point pattern. Target detection employs infrared detection, taking advantage of the strong infrared reflectivity of the target points, which appear as bright points in the infrared image. Simple thresholding is used to extract connected regions, yielding the pixel coordinates of the target points in the image. Combined with the world coordinates of the target points on the target board, the PNP (Perspective-n-Point) algorithm is used to calculate the target board's pose in the camera coordinate system. The robot arm's pose at the moment of successful grasping is pre-calibrated, and during actual control, repositioning control is performed according to the calibrated target pose.

[0011] For scenarios involving blocked surfaces, existing visual guidance methods for single-camera targets have the following drawbacks:

[0012] (1) Using infrared detection of target points can only perform target point pose tracking tasks and cannot handle other tasks that require normal visible light image recognition. The blocking operation involves the grabbing of the blocking plate, the positioning of the inner flange ring center, the placement of the blocking plate and the tightening of bolts. Since a target cannot be set at the inner flange ring opening, the center position of the flange hole can only be obtained through image recognition positioning methods. The flange opening cannot be observed by infrared images alone.

[0013] (2) It can only guarantee the accuracy of target pose calculation on the same plane as the target point. When the target point is not on the same plane, there will be a large deviation, especially in the estimation of the depth direction. The blocking operation involves positioning and guidance in the depth direction, which requires high-precision positioning information in the depth direction, which is difficult to meet under single-camera conditions.

[0014] (3) Existing single-camera guidance systems use repositioning control, which requires pre-calibration of the robotic arm's pose when it successfully grasps the object. The calibration process requires manual intervention and is cumbersome. This does not effectively meet the requirements for automated blockage operations.

[0015] The second method is a multi-camera + target approach.

[0016] In existing multi-camera target vision guidance systems, the target detection principle is basically the same as that of single-camera systems. It also relies on the unique material characteristics of the target to extract high-brightness points in infrared images using threshold segmentation, and then calculates the pixel coordinates of the target points in the multi-camera images. Finally, based on pre-defined conversion relationships between cameras, the three-dimensional coordinates of the target points are calculated using the multi-viewpoint projection principle, thereby obtaining the three-dimensional pose of the target plate.

[0017] For scenarios involving blocked panels, existing visual guidance methods for multi-camera targets have the following drawbacks:

[0018] (1) Similar to the single-camera case, infrared detection of target points can only be used for target point pose tracking tasks and cannot handle other tasks that require normal visible light image recognition. The plate blocking operation involves the grabbing of the plate, the positioning of the inner flange ring center, the placement of the plate and the tightening of bolts. Since a target cannot be set at the inner flange ring opening, the center position of the flange hole can only be obtained through image recognition positioning methods. The flange opening cannot be observed by infrared images alone.

[0019] (2) The installation and layout of multiple cameras need to be designed in advance, and the extrinsic parameters (coordinate transformation relationships between cameras) between cameras need to be accurately calibrated in advance. The usual calibration method is to manually move a calibration rod or calibration plate within the calibration area after completing the calibration of the intrinsic parameters and distortion parameters of each camera, so as to cover all areas and calibrate the transformation relationship of each camera, so as to keep the calibration error in the whole scene to a small level. However, the space for installing the plate is relatively small and there is radiation dose, so it is dangerous for personnel to enter for calibration, making the calibration process difficult to implement.

[0020] (3) In multi-camera scenarios, at least two cameras need to simultaneously observe the same target point. However, the blocking robot needs to traverse a large area during the blocking process, which significantly limits the use of multi-camera mode. Based on the above analysis of the blocking robot's operating scenarios and existing methods, this invention proposes a target pose estimation framework combining single and multi-camera methods, along with automated hand-eye calibration and multi-camera calibration methods. This addresses the issues of inaccurate depth estimation with single cameras and limited field of view with multi-camera methods, while also resolving calibration problems and system applicability issues in existing visual guidance methods. Summary of the Invention

[0021] The purpose of this invention is to provide a device, system, and method for visual guidance of a blocking robot. The method uses visual guidance by setting up a camera to photograph the target at the end of the robotic arm, the object to be grasped, or the target at the location of the positioning hole, thereby obtaining the pose of the robotic arm and the final position and posture for performing the operation. The motion control command of the robotic arm is output through a control algorithm to guide and control the blocking robot to complete the above four-step remote control operation process, replacing manual remote control operation and realizing a fully automated guidance and control task.

[0022] The technical solution of the present invention is as follows: A device for visual guidance of a blocking robot includes a work site camera, a target plate, an industrial control computer processor and a remote monitoring terminal, wherein the target plate is installed on the end of the robotic arm, the work site camera is connected to the industrial control computer processor, the industrial control computer processor is connected to the remote monitoring terminal, and the remote monitoring terminal has a display screen.

[0023] The on-site camera system is a multi-camera system, which includes multiple sets of cameras. At least two sets of cameras can cover the observation range to ensure that the three-dimensional pose of the target within the operation range can be acquired.

[0024] The industrial control computer processor controls multiple cameras on site to acquire on-site image data, calculates the end-effector pose of the robotic arm based on the acquired multi-camera image data, identifies the target object and estimates its pose, and outputs control signals for the robotic arm based on the pose offset between the two objects to control the movement of the robotic arm.

[0025] The display on the remote monitoring terminal is used by operators to observe images of the work site, the position of the robotic arm end effector, and the target position.

[0026] A system for visual guidance of a blocking robot includes: an automated calibration algorithm module and a target point pose tracking algorithm module;

[0027] The automated calibration algorithm module calibrates the multi-camera system. The robotic arm moves along a pre-set trajectory, and at each point, the robotic arm pauses for 3 seconds. The cameras in the multi-camera system capture images and record the current TCP pose of the robotic arm's end effector until all trajectory points have been traversed. After acquiring all image data, the extrinsic parameters between the cameras and the coordinate system transformation matrix between the cameras are calculated. After completing the multi-camera calibration, based on the captured images and the TCP pose of the robotic arm's end effector, the transformation matrix between the camera coordinate system and the robotic arm's base coordinate system is calculated to obtain the hand-eye calibration parameters.

[0028] The target point pose tracking algorithm module extracts target points in real time from the on-site operation images obtained by the cameras in the multi-camera system. Based on the position of the target points and the actual local coordinate system coordinates of the target points, it calculates the pose of the target plate. Based on the pose of the target plate, it obtains the pose of the end-effector coordinate system of the robotic arm, realizing real-time end-effector pose tracking. At the same time, it performs pose offset calculation with the target pose obtained by the multi-camera system. Based on the pre-calibrated hand-eye alignment results, it converts the results into motion control quantities for the robotic arm and sends them to the robotic arm actuator to execute the operation.

[0029] A method for vision guidance of a blocking robot includes the following steps:

[0030] Step 1: Perform multi-camera calibration to obtain the internal parameter matrix and distortion coefficient calibration;

[0031] Step 2: Run the preset trajectory points to perform hand-eye calibration;

[0032] Step 3: Run a deep learning-based target pose tracking algorithm to track and obtain the pose of the robotic arm's end effector in real time.

[0033] Step 4: Use template matching to obtain the position of the target in each image of the multi-camera system. Based on the matching points and the multi-camera calibration parameters, calculate the three-dimensional coordinates of the target matching points in the camera coordinate system. Combined with the target size data, use the PNP algorithm to calculate the target pose.

[0034] Step 5: Convert the target pose to the robot arm's base coordinate system to obtain the robot arm's motion control parameters, control the robot arm's movements, and at the same time combine the robot arm's end-effector pose tracking results to determine whether the robot arm has reached the target position. If it has not reached the position, the target pose difference is converted into motion increments to continue generating motion control parameters until the pose difference between the two is less than the set threshold.

[0035] Step 1 includes the following:

[0036] Step 11: Set several trajectory points, control the robotic arm to move along the trajectory points, and obtain the target board images captured by each camera at each trajectory point;

[0037] Step 12: Calculate and output the pose of the target plate in each camera image;

[0038] Step 13: Establish equations based on the three-dimensional coordinates of the target plate in different camera coordinate systems. The coordinates of the target point in camera 1 are X... C1 The coordinates in camera 2 are X C2 Based on formula (1), establish the equation:

[0039] X C2 =T 1t2 *X C1 (1)

[0040] Step 14: The coordinate transformation matrix T from camera 1 to camera 2 can be solved using the RANSAC algorithm. 1t2 .

[0041] Step 2 includes the following:

[0042] Step 21: First, set a section of the robotic arm's running trajectory points, and set the trajectory of several points;

[0043] Step 22: After the robotic arm moves to the designated pose state of the trajectory points in sequence, it pauses for 3 seconds and collects the image data and corresponding robotic arm TCP pose data at the specific points.

[0044] Step 23: The obtained image data is used to extract the target plate and calculate the pose of the target plate using the corresponding pose estimation algorithm;

[0045] Step 24: Obtain the 3D coordinates (X) of the target plate in the camera coordinate system based on the image. CBased on offline TCP calibration, the three-dimensional coordinates (X, Y) of the target plate at each point in the coordinate system of the robotic arm base can be obtained. B Equation (2) is established, and the RANSAC algorithm is used to solve the equation to obtain the transformation matrix T of the camera relative to the coordinate system of the robot arm base. CtB This is recorded as the output result of hand-eye calibration.

[0046] X B =T CtB *X C (2).

[0047] Step 3 includes the following:

[0048] Step 31: Dataset Generation

[0049] The standard target template image is rotated, translated, sheared, perspective-adjusted, and scaled, and certain grayscale changes and noise are added. At the same time, the mask bounding box of the target template image is also subjected to the same geometric processing as the target image to obtain the ground truth bounding box of the target block. The small amount of real dataset obtained on site is labeled to obtain the real dataset.

[0050] Step 32: Model Training

[0051] The detection network uses the YOLOv7 network and is trained using a simulated dataset.

[0052] Step 33: Model Optimization

[0053] We fine-turing and optimizing the initial detection network trained on simulated data using a small amount of real-world scene data obtained on-site.

[0054] Step 34: Target Tracking

[0055] Each frame of image data is processed by a detection network to obtain target detection results, and then the DeepSort tracking algorithm is used to track the target in each frame.

[0056] Step 35: Pose Estimation

[0057] After the target board tracking and recognition is completed, the target points in the target board pattern are detected and extracted to obtain the pixel coordinates of the target points. Then, the target points in the target pattern are matched with the standard target template to obtain the correspondence between the pixels in the image block and the target template points, thereby obtaining the world coordinates of the target points. The world coordinates are a rectangular coordinate system established with the center of the target template as the center point, the length and width of the template as the X and Y axes respectively, and the plane where the target template is located perpendicularly outward as the Z axis.

[0058] Step 36: Based on the obtained target pose and the transformation relationship between the calibrated target coordinate system and the TCP tool coordinate system at the end of the robotic arm, the end pose of the robotic arm can be obtained.

[0059] Step 35 includes the following:

[0060] Step 351: Target point extraction

[0061] Circular targets are used as markers, with the target background set to white and the target markers set to black to clearly distinguish them. Target detection first involves bilateral filtering of the tracked target pattern image to enhance image clarity, followed by Gaussian smoothing to remove some noise. Then, the MSER region extraction algorithm is used to extract color-consistent regions with strong grayscale contrast. A non-maximum region suppression algorithm is then employed to obtain accurate target marker regions. The non-maximum region suppression algorithm fits an ellipse shape to each region based on the MSER region extraction results, determining the ellipse's center, major axis radius, minor axis radius, and rotation angle. The area of ​​the fitted ellipse and its similarity to a circle are used for weighted scoring. During non-maximum region suppression, the algorithm iterates from highest to lowest score, discarding regions with an overlap greater than a certain threshold with the currently selected highest-scoring region. After this iterative process, the regions closest to the target markers are retained.

[0062]

[0063] Where s1 represents the area of ​​the region, s2 is the proportion of the darker areas within the region, s3 is the ratio of the minor axis to the major axis, and r a r is the length of the minor axis b The length of the major axis;

[0064] Step 352: Target pose estimation based on ICP

[0065] After obtaining the pixel coordinates of the target points in the target board, the category information output during target detection and recognition can be directly mapped to the target's standard template. A two-dimensional point sparse matching method based on ICP is used. During ICP matching, the PCA algorithm is first used to fit the target pixels and the main direction of the target board. Then, the target image is rotated to the main direction of the target board. Next, center point normalization is used to translate and align the target image with the target template, obtaining a coarse alignment result after rotation and translation. Further fine alignment is performed using the ICP algorithm, which iteratively selects the nearest neighbor point as the matching point, calculates the transformation matrix, calculates the error after transformation, and then transforms the entire set of points according to the transformation matrix. This nearest neighbor search and matching process is repeated until the final transformation error change is less than a certain threshold, obtaining the final fine alignment result.

[0066] After aligning the target image with the target template, the pixel coordinates of the target points in the target image are obtained according to the pre-designed world coordinates of the target template points. The one-to-one correspondence between the pixel coordinates and world coordinates of the target points in the target image is then substituted into the PNP equation to calculate the corresponding target pose.

[0067]

[0068] Where [u,v] are pixel coordinates, K is the camera intrinsic parameter matrix, [R|t] is the pose matrix to be obtained, and [x,y,z] are world coordinates;

[0069] Step 353: Multi-camera pose fusion

[0070] After the target detection and pose calculation steps described above, the pose of the tracked target within the field of view of a single camera is obtained. When using a multiple camera system, the same target appears in the fields of view of multiple cameras. By simultaneously solving the multi-camera projection equations, the pose of the target is calculated.

[0071] A point (x) in the world coordinate system i y i , z i After projection imaging, the actual projection point in the j-th camera image is... The corresponding points detected at this time are The intrinsic parameter matrix of camera j is K j The transformation matrix (extrinsic parameter matrix) relative to the reference coordinate system is T. j ;

[0072] When multiple cameras are observing simultaneously, the back projection error is used for modeling:

[0073]

[0074] Where j = 1...n represents the observation data from cameras 1 to n, and i = 1...m represents the pixel value or world coordinate system value of the corresponding point within a single field of view.

[0075] The beneficial effects of this invention are as follows: This invention can replace the existing remote-controlled operation method of blocking robots, and is used to automatically guide blocking robots to perform blocking installation and disassembly operations. Furthermore, this invention can achieve automated hand-eye calibration and multi-camera calibration of the guidance system without human intervention. Employing a deep learning target detection and tracking algorithm, it can provide highly robust target tracking performance under visible light shooting conditions. By using single-camera independent pose estimation and multi-single-camera pose fusion, this system combines the flexibility and ease of use of a single-camera system with the accuracy of multi-camera 3D pose estimation, giving the device and system high accuracy, applicability, and scalability. Attached Figure Description

[0076] Figure 1 This is a schematic diagram of a device for visual guidance of a blocking robot provided by the present invention;

[0077] Figure 2 Schematic diagram of the installation site for the blocking operation.

[0078] Figure 3 This is a schematic diagram of a vision-guided system for a blocking robot provided by the present invention;

[0079] Figure 4 Flowchart of an automated multi-camera calibration method;

[0080] Figure 5 Flowchart of an automated hand-eye calibration method;

[0081] Figure 6 This is a flowchart of a deep learning-based target pose tracking algorithm.

[0082] Figure 7 The target plate pattern used in some of the samples;

[0083] Figure 8 This is an example of simulating the generation of training data in this invention;

[0084] Figure 9 Here is a flowchart of the Kalman filtering algorithm;

[0085] Figure 10 Here is a flowchart of the target point extraction algorithm;

[0086] Figure 11 This is a diagram showing the region extraction results before nonmaximum region suppression;

[0087] Figure 12 This is a diagram showing the region extraction results after nonmaximum region suppression;

[0088] Figure 13 This is a schematic diagram of the multi-camera reprojection principle. Detailed Implementation

[0089] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0090] This invention provides a device, system, and method for visual guidance of a blocking robot. Employing visual guidance, a camera is mounted to capture images of the robot arm's end-effector target, the object to be grasped, or the target at the location of a positioning hole. By acquiring the robot arm's pose and the final position and posture for the task, a control algorithm outputs motion control commands for the robot arm. This automatically guides and controls the blocking robot to complete the aforementioned four-step remote-controlled operation, replacing manual remote control and achieving fully automated guidance and control. This invention does not include the original blocking robot components; it only relates to the visual guidance component.

[0091] A device and system for vision guidance of a blocking robot first fixes a target plate printed with a target pattern onto a blocking plate, and measures and calibrates the gripping position of the blocking plate and the position of the bolt holes relative to the center coordinate system of the target plate. The camera used for guidance is calibrated offline to obtain the intrinsic parameters and distortion parameters of each camera. The target plate is fixed to the end effector of the blocking robot, and offline TCP calibration is completed, i.e., calibrating the position of the robotic arm's end effector relative to the origin of the target point.

[0092] Then, the calibrated guide camera and configured light source are installed in the corresponding positions, the power is connected, the camera image signal output terminal is connected to the control computer of the blocking robot, and the software system is installed into the control system.

[0093] Then, the visual guidance software system of this invention is started, and the automated hand-eye calibration and multi-camera calibration tools are run to complete the hand-eye calibration and multi-camera calibration.

[0094] Next, the guided operation mode will be activated, and the automated installation operation of the robotic arm will be executed sequentially according to the previous steps for installing the blocking plate with the remote-controlled robotic arm.

[0095] After completing the task, exit the system and turn off the device power.

[0096] like Figure 1 As shown, a device for visual guidance of a blocking robot includes: a work site camera, a target plate, an industrial control computer processor, and a remote monitoring terminal. The target plate is mounted on the end effector of the robotic arm. The work site camera is connected to the industrial control computer processor, and the industrial control computer processor is connected to the remote monitoring terminal, which has a display screen.

[0097] The on-site cameras employ a multi-camera system to ensure that at least two sets of cameras can cover the observation range, guaranteeing the acquisition of the three-dimensional pose of the target within the operation area. The target plate mounted at the end of the robotic arm uses a dotted target (e.g., ...). Figure 7(As shown) The pattern can be printed and pasted onto a rigid aluminum plate, or directly printed or sprayed onto the aluminum plate, ensuring the allowable dimensional accuracy of the pattern. After printing, it needs to be installed at the end of the robotic arm and fixed to the end tool (airlock). The coordinate system pose of the end tool can be calculated based on the pose of the target plate.

[0098] The industrial control computer processor controls multiple cameras on site to acquire on-site image data, calculates the end-effector pose of the robotic arm based on the acquired multi-camera image data, identifies the target (flange hole, plug plate, bolt, etc.) and estimates its pose, and outputs control signals for the robotic arm based on the pose offset between the two to control the movement of the robotic arm.

[0099] The display on the remote monitoring terminal is used by operators to observe images of the work site, the position of the robotic arm end effector, the position of the target, and other results.

[0100] A system for visual guidance of a blocking robot includes an automated calibration algorithm module and a target point pose tracking algorithm module. After the robotic arm and camera system are installed on site, the system first runs the automated calibration algorithm module.

[0101] The automated calibration algorithm module calibrates the multi-camera system. The robotic arm moves along a pre-set trajectory, stopping for 3 seconds at each point. The cameras in the multi-camera system capture images and record the current TCP pose of the robotic arm's end effector until all trajectory points have been traversed and all image data have been acquired. The extrinsic parameters between the cameras and the coordinate system transformation matrix between the cameras are then calculated. After the multi-camera calibration is completed, the transformation matrix between the camera coordinate system and the robotic arm's base coordinate system is calculated based on the captured images and the TCP pose of the robotic arm's end effector to obtain the hand-eye calibration parameters.

[0102] After completing the hand-eye calibration, the robotic arm operation process can begin.

[0103] The target pose tracking algorithm module is a deep learning-based target pose tracking algorithm module. It extracts target points in real time from the on-site operation images obtained by the cameras in the multi-camera system. Based on the position of the target points and the actual local coordinate system coordinates of the target points, it calculates the pose of the target plate. Based on the pose of the target plate, it obtains the pose of the end-effector coordinate system of the robotic arm, realizing real-time end-effector pose tracking. At the same time, it performs pose offset calculation with the target pose obtained by the multi-camera system. Based on the pre-calibrated hand-eye alignment results, it converts the results into motion control quantities for the robotic arm and sends them to the robotic arm actuator to execute the operation.

[0104] A method for vision guidance of a blocking robot includes the following steps:

[0105] Step 1: After the vision-guided hardware device and robotic arm are installed, the multi-camera calibration method flowchart is as follows: Figure 4 As shown. Multi-camera calibration relies on targets mounted on the end effector of a robotic arm, and each camera has already undergone single-camera calibration, which can be performed using a checkerboard method. Accurate internal parameter matrices and distortion coefficients are obtained. This part of the calibration only needs to be completed beforehand, and can be used in subsequent processes. The specific process is as follows:

[0106] Step 11: Set several trajectory points, control the robotic arm to move along the trajectory points, and obtain the target board images captured by each camera at each trajectory point;

[0107] Step 12: Calculate and output the pose of the target plate in each camera image based on the target pose tracking module (this algorithm can be obtained using a deep learning-based target pose tracking algorithm module);

[0108] Step 13: Establish equations based on the 3D coordinates of the target plate in different camera coordinate systems. For example, the coordinates of the target point in camera 1 are X... C1 The coordinates in camera 2 are X C2 Based on formula (1), establish the equation:

[0109] X C2 =T 1t2 *X C1 (1)

[0110] Step 14: The coordinate transformation matrix T from camera 1 to camera 2 can be solved using the RANSAC algorithm. 1t2 .

[0111] Note: When solving for the extrinsic parameters (camera coordinate system transformation matrix) between multiple cameras, one camera can be used as a fixed reference coordinate system, and the transformation relationships between the other cameras and the reference camera coordinate system can be solved separately.

[0112] Step 2: Run the preset trajectory points to perform automated hand-eye calibration.

[0113] The TCP calibration of a robotic arm mainly involves determining the coordinate transformation relationship between the origin position of the calibration plate fixed to the end effector and the TCP position set at the end effector. Typically, for ease of subsequent calculations, the calibration plate is set as the TCP point. The flowchart is as follows: Figure 5 As shown, the specific process is as follows:

[0114] Step 21: First, set a section of the robotic arm's running trajectory points. Usually, several (e.g., 50) points can be set for the trajectory. The more trajectory points set, the higher the subsequent calibration accuracy and reliability.

[0115] Step 22: After the robotic arm moves to the designated pose state of the trajectory points in sequence, it pauses for 3 seconds and collects the image data and corresponding robotic arm TCP pose data at the specific points.

[0116] Step 23: The obtained image data can be used to calculate the pose of the target board using target board extraction and corresponding pose estimation algorithms (this algorithm can be obtained using a deep learning-based target pose tracking algorithm module).

[0117] Step 24: Obtain the 3D coordinates (X) of the target plate in the camera coordinate system based on the image. C Based on offline TCP calibration, the three-dimensional coordinates (X, Y) of the target plate at each point in the coordinate system of the robotic arm base can be obtained. B Equation (2) is established, and the RANSAC (Random Sample Consensus) algorithm is used to solve the equation to obtain the transformation matrix T of the camera relative to the coordinate system of the robotic arm base. CtB This is recorded as the output result of the hand-eye calibration.

[0118] X B =T CtB *X G (2)

[0119] Step 3: Run the deep learning-based target pose tracking algorithm. The algorithm flowchart is as follows: Figure 6 The robot arm's end-effector pose is tracked and acquired in real time. The specific process is as follows:

[0120] Step 31: Dataset Generation (Offline Stage)

[0121] Using industrial scene images with steel and machinery as backgrounds, closely resembling actual work scenarios, as background images, and comparing them with standard target template images (such as...) Figure 7 As shown, after performing certain geometric transformations such as rotation, translation, shearing, perspective, and scaling, and adding certain grayscale changes and noise, the image is randomly pasted onto the background image. Simultaneously, the background image undergoes random affine transformations, color and brightness changes, random noise additions, and corresponding random motion blur. The mask bounding box of the target template image also undergoes the same geometric processing as the target image to obtain the ground truth bounding box of the target block. An example of generating the dataset is shown below. Figure 8 As shown.

[0122] The small amount of real dataset obtained on-site was labeled to obtain the real dataset.

[0123] Step 32: Model Training (Offline Phase)

[0124] The detection network uses the YOLOV7 network (YOLO stands for You Only Look Once), and is trained using a simulated dataset.

[0125] Step 33: Model Optimization (Offline Phase)

[0126] We fine-turing and optimizing the initial detection network trained on simulated data using a small amount of real-world scene data obtained on-site.

[0127] Step 34: Target Tracking (Online Use)

[0128] Each frame of image data is processed by a detection network to obtain target detection results, and then the DeepSort tracking algorithm is used to track the target in each frame.

[0129] After using the YOLO network to detect the target region in each frame of a video image, the accuracy of the detection results cannot be absolutely guaranteed. Therefore, it is necessary to combine the detection results of previous frames to determine the current detection result. Furthermore, in some cases, changes in lighting and occlusion may cause the target in certain frames to be undetected, so it is also necessary to optimize the detection results from previous moments.

[0130] This invention introduces the DeepSort target tracking framework to further process the detection results. The DeepSort framework mainly includes Kalman prediction, deep feature matching, and the Hungarian algorithm.

[0131] The Kalman prediction algorithm process is as follows: Figure 9 As shown:

[0132] in Let R be the state vector of the target at time k-1. k-1 Let u be the covariance matrix between the state variables at time k-1. k-1 z represents the control quantity of the system at time k. k Represents the measured value at time k. A, B, and H are state transformation matrices, which are adjustment coefficients during the state transformation process (constant here). Matrix B represents the gain of the optional control input u, and matrix H represents the state variable x. k For the measured variable z k The gain is given by Q, where Q is the process excitation noise covariance matrix and R is the observation noise covariance matrix. Q and R are constants.

[0133] The depth features are obtained by using the output vector of the YOLO network BackBone main frame as the depth features of the target pattern. The similarity of the feature vectors is calculated using the cosine function. At the same time, the state information predicted by Kalman is combined with the Hungarian algorithm to match and associate the detected targets in consecutive frames. In particular, for partially occluded low-confidence candidate targets, the correct low-confidence detection targets are output based on the degree of correlation of the similarity between the targets in consecutive frames.

[0134] Step 35: Pose Estimation

[0135] After target tracking and recognition are completed, target points in the target pattern need to be detected and extracted to obtain their pixel coordinates. Then, the target points in the target pattern need to be matched with a standard target template to obtain the correspondence between pixels in the image block and points in the target template, thus obtaining the world coordinates of the target points. World coordinates are a Cartesian coordinate system established with the center of the target template as the center point, the length and width directions of the template as the X and Y axes respectively, and the plane containing the target template pointing perpendicularly outward as the Z axis. The world coordinates of the target points can be obtained in advance based on the actual size of the template and the design of the points.

[0136] After obtaining the correspondence between the world coordinates and pixel coordinates of the target point, we can use the PNP algorithm to calculate the pose of the target center in the camera coordinate system.

[0137] The above describes the target pose obtained from a single camera. When multiple cameras are arranged in the scene, and the same target is simultaneously calculated in the field of view of two cameras, the target pose can be further optimized by using the coordinate system transformation matrix calibrated between the cameras, based on the principle of reprojection and the reprojection equation, to obtain an accurate three-dimensional spatial pose.

[0138] Step 351: Target point extraction

[0139] Because the target plate pattern is designed with a strong grayscale contrast to distinguish the target points, and in order to improve the positioning accuracy and reliability of the target points, regular corner points or geometric patterns such as circles are usually used to facilitate accurate detection and positioning.

[0140] In this invention, we use circles as target points, clearly distinguishing them by setting the target plate background color to white and the target points to black. However, images acquired in real-world shooting often suffer from noise, lighting variations, and ambient light interference, leading to reduced contrast and blurred edges. Furthermore, changes in viewing angle often cause geometric distortion; a circle will project as an ellipse in non-orthographic imaging. Therefore, an ellipse extraction method is required during the detection process. The detailed target point detection and extraction process is as follows... Figure 10 As shown.

[0141] The target detection process begins with bilateral filtering of the tracked target pattern image to enhance image clarity, followed by Gaussian smoothing to remove some noise. Then, the MSER (Maximally Stable Extremal Regions) algorithm is used to extract regions with strong gray-level contrast and consistent color. Since region extraction often involves a region being encompassed by multiple extraction boxes, a non-maximum region suppression algorithm is typically employed to obtain accurate target point regions.

[0142] Non-maximum region suppression (NMR) involves fitting an ellipse to the shape of each region based on the MSER region extraction results. The fitted ellipse has a center, major axis radius, minor axis radius, and rotation angle. The fitted ellipse is weighted and scored based on its area and its similarity to a circle (see Formula 3). During NMR, the region is traversed from highest to lowest score. Each iteration removes regions with an overlap greater than a certain threshold with the currently selected region that has the highest score. After this iterative process, the region that is closest to the target point is retained.

[0143]

[0144] Where s1 represents the area of ​​the region, s2 is the proportion of the darker areas within the region, and s3 is the ratio of the minor axis to the major axis (r). a r is the length of the minor axis b (Length of the major axis).

[0145] After nonmaximum region suppression, relatively accurate target point extraction results can be obtained. However, erroneous points may still appear in some cases, especially when the target is rotated, resulting in background scenes in the identified target area. The background extraction results will interfere with the target point extraction. Therefore, it is necessary to further screen the target points. The usual method is to screen them based on the shape constraints of the extracted target points and the similarity of the target template. The main criteria are the ratio of the major axis to the minor axis and the area of ​​the darker region of the target point. In addition, the target templates detected by YOLO network usually have category labels. The distribution of standard target points corresponding to the category, such as the number of target points, can be compared to judge and erroneous target points can be removed, and the final detection and localization results can be output. Figure 11 The results are before nonmaximum region suppression. Figure 12 The results are after nonmaximum region suppression. It is clear that duplicate and slightly offset detection results can be filtered out to obtain stable and accurate detection boxes.

[0146] Step 352: Target pose estimation based on ICP

[0147] After obtaining the pixel coordinates of the target points in the target template, we can directly match the category information output during target detection and recognition with the standard template of the target. However, since the actual target pattern and the standard target template usually have geometric changes such as rotation, scaling, and misalignment, they cannot be directly matched with the corresponding points in the target template. At this time, it is necessary to further match with the target template to obtain the correspondence between the extracted target pixels and the target template points.

[0148] This invention employs a two-dimensional sparse point matching method based on ICP (Iterative Closest Point). Because the target board is typically designed in a non-fully stacked state to provide directional guidance, it is crucial to ensure its ability to indicate the target's direction. Therefore, during ICP matching, the PCA (principle component analysis) algorithm is first used to fit the target pixels and the target board's main direction. Then, the target image is rotated to the target board's main direction. Next, center point normalization is used to translate and align the target image with the target template, yielding a coarse alignment result after rotation and translation. Further fine alignment is achieved using the ICP algorithm, which iteratively selects the nearest neighbor point as the matching point, calculates the transformation matrix, calculates the transformation error, and then transforms the entire set of points according to the transformation matrix. This nearest neighbor search and matching process is repeated until the final transformation error is less than a certain threshold, resulting in the final fine alignment result.

[0149] After aligning the target image with the target template, the pixel coordinates of the target points in the target image and the world coordinates can be obtained according to the pre-designed world coordinates of the target template points. This correspondence can then be substituted into the PNP equation (as shown in Formula 4) to calculate the corresponding target pose.

[0150]

[0151] Where [u,v] are pixel coordinates, K is the camera intrinsic parameter matrix, [R|t] is the desired pose matrix (extrinsic parameter matrix), and [x,y,z] are world coordinates.

[0152] Step 353: Multi-camera pose fusion

[0153] After the target detection and pose calculation steps described above, the pose of the tracked target within the field of view of a single camera can be obtained. When using a multi-camera system, the same target appears in the fields of view of multiple cameras. In this case, we can simultaneously solve the multi-camera projection equations to calculate a more accurate target pose. A schematic diagram of the multi-camera reprojection principle is shown below. Figure 13 As shown.

[0154] A point (x) in the world coordinate system iy i , z i After projection imaging, the actual projection point in the j-th camera image is... The corresponding points detected at this time are The intrinsic parameter matrix of camera j is K j The transformation matrix (extrinsic parameter matrix) relative to the reference coordinate system is T. j .

[0155] When multiple cameras are observing simultaneously, the back projection error is used for modeling:

[0156]

[0157] Where j = 1...n represents the observation data from cameras 1 to n, and i = 1...m represents the pixel value or world coordinate system value of the corresponding point within a single field of view. This is the multi-view reprojection error equation to be optimized. By minimizing Equation 5, the accurate pose [R|t] is obtained. Thus, the 3D pose result of the target board in multiple cameras can be obtained.

[0158] Step 36: Based on the obtained target pose and the transformation relationship between the calibrated target coordinate system and the TCP tool coordinate system at the end of the robotic arm, the end pose of the robotic arm can be obtained.

[0159] Step 4: Use template matching to obtain the position of the target in each image of the multi-camera system. Based on the matching points and the multi-camera calibration parameters, calculate the three-dimensional coordinates of the target matching points in the camera coordinate system. Combined with the target size data, use the PNP algorithm to calculate the target pose.

[0160] Step 5: Following the hand-eye calibration parameters, the target pose is converted into the robot arm's base coordinate system to obtain the robot arm's motion control parameters. The robot arm's movements are controlled, and the end-effector pose tracking results are combined to determine whether the robot arm has reached the target position. If the position has not been reached, the target pose difference is converted into motion increments, and motion control parameters are generated again until the pose difference is less than a set threshold.

Claims

1. A method for visual guidance of a blocking robot, characterized in that, Includes the following steps: Step 1: Perform multi-camera calibration to obtain the internal parameter matrix and distortion coefficient calibration; Step 2: Run the preset trajectory points to perform hand-eye calibration; Step 3: Run a deep learning-based target pose tracking algorithm to track and obtain the pose of the robotic arm's end effector in real time; Step 3 includes the following: Step 31: Dataset generation; The standard target template image is rotated, translated, sheared, perspective-adjusted, and scaled, and certain grayscale changes and noise are added. At the same time, the mask bounding box of the target template image is also subjected to the same geometric processing as the target image to obtain the ground truth bounding box of the target block. The small amount of real dataset obtained on site is labeled to obtain the real dataset. Step 32: Model training; The detection network uses the YOLOv7 network and is trained using a simulated dataset. Step 33: Model optimization; We fine-turing and optimizing the initial detection network trained on simulated data using a small amount of real-world scene data obtained on-site. Step 34: Target tracking; Each frame of image data is processed by a detection network to obtain target detection results, and then the DeepSort tracking algorithm is used to track the target in each frame. Step 35: Pose estimation; After the target board tracking and recognition is completed, the target points in the target board pattern are detected and extracted to obtain the pixel coordinates of the target points. Then, the target points in the target pattern are matched with the standard target template to obtain the correspondence between the pixels in the image block and the target template points, thereby obtaining the world coordinates of the target points. The world coordinates are a rectangular coordinate system established with the center of the target template as the center point, the length and width of the template as the X and Y axes respectively, and the plane where the target template is located perpendicularly outward as the Z axis. Step 35 includes the following: Step 351: Target point extraction; Circular targets are used as markers, with the target background set to white and the target markers set to black to clearly distinguish them. Target detection first involves bilateral filtering of the tracked target pattern image to enhance image clarity, followed by Gaussian smoothing to remove some noise. Then, the MSER region extraction algorithm is used to extract color-consistent regions with strong grayscale contrast. A non-maximum region suppression algorithm is then employed to obtain accurate target marker regions. The non-maximum region suppression algorithm fits an ellipse shape to each region based on the MSER region extraction results, determining the ellipse's center, major axis radius, minor axis radius, and rotation angle. A weighted score is calculated based on the area of ​​the fitted ellipse and its similarity to a circle. During non-maximum region suppression, the algorithm iterates from highest to lowest score, eliminating regions with an overlap greater than a certain threshold with the currently selected highest-scoring region. After this iterative process, the regions closest to the target markers are retained. (3) , , , , in, Indicates the area of ​​the region. This represents the percentage of darker areas within the region. It is the ratio of the minor axis length to the major axis length. For the minor axis length, The length of the major axis; Step 352: Target pose estimation based on ICP; Step 353: Multi-camera pose fusion; Step 36: Based on the obtained target pose and the transformation relationship between the calibrated target coordinate system and the TCP tool coordinate system of the robotic arm end effector, the pose of the robotic arm end effector can be obtained; Step 4: Use template matching to obtain the position of the target in each image of the multi-camera system. Based on the matching points and the multi-camera calibration parameters, calculate the three-dimensional coordinates of the target matching points in the camera coordinate system. Combined with the target size data, use the PNP algorithm to calculate the target pose. Step 5: Convert the target pose to the robot arm's base coordinate system to obtain the robot arm's motion control parameters, control the robot arm's movements, and at the same time combine the robot arm's end-effector pose tracking results to determine whether the robot arm has reached the target position. If it has not reached the position, the target pose difference is converted into motion increments to continue generating motion control parameters until the pose difference between the two is less than the set threshold.

2. The method for visual guidance of a blocking robot as described in claim 1, characterized in that, Step 1 includes the following: Step 11: Set several trajectory points, control the robotic arm to move along the trajectory points, and obtain the target board images captured by each camera at each trajectory point; Step 12: Calculate and output the pose of the target plate in each camera image; Step 13: Establish equations based on the 3D coordinates of the target plate in different camera coordinate systems. The coordinates of the target point in camera 1 are: The coordinates in camera 2 are Based on formula (1), establish the equation: (1), Step 14: The coordinate transformation matrix from camera 1 to camera 2 can be solved using the RANSAC algorithm. .

3. The method for visual guidance of a blocking robot as described in claim 1, characterized in that, Step 2 includes the following: Step 21: First, set a section of the robotic arm's running trajectory points, and set the trajectory of several points; Step 22: After the robotic arm moves to the designated pose state of the trajectory points in sequence, it pauses for 3 seconds and collects the image data and corresponding robotic arm TCP pose data at the specific points. Step 23: The obtained image data is used to extract the target plate and calculate the pose of the target plate using the corresponding pose estimation algorithm; Step 24: Obtain the three-dimensional coordinates of the target plate in the camera coordinate system based on the image. Based on offline TCP calibration, the three-dimensional coordinates of the target plate at each point in the coordinate system of the robotic arm base can be obtained. Equation (2) is established, and the RANSAC algorithm is used to solve the equation to obtain the transformation matrix of the camera relative to the coordinate system of the robot arm base. This is recorded as the output result of hand-eye calibration. (2)。 4. The method for visual guidance of a blocking robot as described in claim 1, characterized in that, Step 352 includes the following: After obtaining the pixel coordinates of the target points in the target plate, the category information output during target detection and recognition can be directly matched with the standard template of the target. A two-dimensional point sparse matching method based on ICP is used. During ICP matching, the PCA algorithm is first used to fit the target pixels and the main direction of the target plate. Then, the target image is rotated to the main direction of the target plate. Next, center point normalization is used to translate and align the target image with the target template, obtaining a coarse alignment result after rotation and translation. Further fine alignment is performed using the ICP algorithm, which iteratively selects the nearest neighbor point as the matching point, calculates the transformation matrix, calculates the error after transformation, and then transforms the entire set of points according to the transformation matrix. This nearest neighbor point search and matching process is repeated until the final transformation error change is less than a certain threshold, obtaining the final fine alignment result. After aligning the target image with the target template, the pixel coordinates of the target points in the target image are obtained according to the pre-designed world coordinates of the target template points. The one-to-one correspondence between the pixel coordinates and world coordinates of the target points in the target image is then substituted into the PNP equation to calculate the corresponding target pose. (4), Where [u,v] are pixel coordinates, K is the camera intrinsic parameter matrix, [R|t] is the pose matrix to be obtained, and [x,y,z] are world coordinates.

5. A method for visual guidance of a blocking robot as described in claim 1, characterized in that, Step 353 includes the following: After the target detection and pose calculation steps described above, the pose of the tracked target within the field of view of a single camera is obtained. When using a multiple camera system, the same target appears in the fields of view of multiple cameras. By simultaneously solving the multi-camera projection equations, the pose of the target is calculated. A point in the world coordinate system After projection imaging, the actual projection point in the j-th camera image is At this time, the corresponding points detected are The intrinsic parameter matrix of camera j is The transformation matrix relative to the reference coordinate system is ; When multiple cameras are observing simultaneously, the back projection error is used for modeling: (5), in, Let i be the observation data from cameras numbered 1 to n. This refers to the pixel value of the corresponding point within a single field of view, or the value in the world coordinate system.

6. A system for visual guidance of a blocking robot for performing the method of claim 1, characterized in that, include: Automated calibration algorithm module and target point pose tracking algorithm module; The automated calibration algorithm module calibrates the multi-camera system. The robotic arm moves along a pre-set trajectory, and at each point, the robotic arm pauses for 3 seconds. The cameras in the multi-camera system capture images and record the current TCP pose of the robotic arm's end effector until all trajectory points have been traversed. After acquiring all image data, the extrinsic parameters between the cameras and the coordinate system transformation matrix between the cameras are calculated. After completing the multi-camera calibration, based on the captured images and the TCP pose of the robotic arm's end effector, the transformation matrix between the camera coordinate system and the robotic arm's base coordinate system is calculated to obtain the hand-eye calibration parameters. The target point pose tracking algorithm module extracts target points in real time from the on-site operation images obtained by the cameras in the multi-camera system. Based on the position of the target points and the actual local coordinate system coordinates of the target points, it calculates the pose of the target plate. Based on the pose of the target plate, it obtains the pose of the end-effector coordinate system of the robotic arm, realizing real-time end-effector pose tracking. At the same time, it performs pose offset calculation with the target pose obtained by the multi-camera system. Based on the pre-calibrated hand-eye alignment results, it converts the results into motion control quantities for the robotic arm and sends them to the robotic arm actuator to execute the operation.

Citation Information

Patent Citations

  • Quick replacement visual positioning system for mechanical arm tool in nuclear environment and positioning method

    CN107590835A

  • Positioning system and method for disassembling and assembling steam generator blocking plate bolt through multi-joint mechanical arm

    CN114211494A