Device, system and method for visual guidance of block board robot

Abstract

The invention belongs to the technical field of nuclear power maintenance, and particularly relates to a device, a system and a method for visual guidance of a block board robot. Comprising an operation site camera, a target board, an industrial personal computer processor and a remote monitoring end, the target board is installed at the tail end of a mechanical arm, the operation site camera is connected with the industrial personal computer processor, the industrial personal computer processor is connected with the remote monitoring end, and the remote monitoring end is provided with a display screen. The method has the advantages that automatic hand-eye calibration and multi-camera calibration of a guide system can be achieved without manual intervention, a target detection tracking algorithm of deep learning is adopted, a high-robustness target tracking effect under the visible light shooting condition can be provided, and the target tracking accuracy is improved. On one hand, the method has the use advantages of flexibility and convenience in dark rotation of a single-camera system, and on the other hand, the multi-camera three-dimensional pose estimation precision is also obtained.

Description

Technical Field

[0001] The invention belongs to the technical field of nuclear power maintenance, and in particular relates to a device, system and method for visual guidance of a plugging robot. Background Art

[0002] The steam generator high water level plug (referred to as the plug) is used to isolate the main pipeline under high water level conditions, ensuring that other maintenance work can be carried out in the steam generator water chamber during the high water level of the nuclear power plant overhaul, saving overhaul time. At present, manual operation is generally used to install and remove the plug in the primary side water chamber of the steam generator. Since the water chamber space belongs to the high radiation dose area and the space is narrow, during the manual disassembly and assembly of the plug, there is a large dose of radiation to the personnel, and the installation quality is greatly affected by human factors. Therefore, many research units at home and abroad have begun to explore the use of automatic installation equipment to replace operators to complete the work inside the water chamber. However, due to the high technical difficulty, there is still no mature automatic installation engineering product for on-site implementation. The process planning of the installation of the steam generator high water level plug can be mainly divided into three processes: robot body installation, plug and sealing assembly installation, and bolt tightening. Some of the operations are completed by remote control outside the steam generator water chamber, and some operations are completed by the plug disassembly and assembly robot.

[0003] The installation of plugging plates and seals is mainly completed by remote control of plugging plate robots, which is mainly divided into four steps:

[0004] (1) The operator delivers the plugging plate from the plugging plate inlet, and the internal mechanical arm grabs the delivered plugging plate;

[0005] (2) The mechanical arm moves the clamped plugging plate to the inner ring flange;

[0006] (3) The center of the mechanical arm screwing assembly is aligned with the center of the inner flange;

[0007] (4) Unfold the screw assembly, move it around the center of the circle to each bolt, and tighten the bolts.

[0008] There are mainly two existing technical solutions:

[0009] The first is a single camera + target method:

[0010] A single camera is used to detect the target point pattern. The target board detection adopts infrared detection. The target point will show a strong high-brightness point in the infrared image due to its strong reflection of infrared light. The pixel coordinates of the target point in the image can be obtained by simple threshold segmentation and extraction of connected areas. Combined with the world coordinates of the target point on the target board, the PNP (Perspective-n-Point) algorithm can be used to calculate the pose of the target board in the camera coordinate system. The pose of the robot arm when the robot arm successfully grasps is calibrated in advance. In actual control, repositioning control is performed according to the calibrated target pose.

[0011] For the blocking operation scenario, the existing single-camera target visual guidance method has the following disadvantages:

[0012] (1) The infrared detection of target points can only perform the target point posture tracking task, and cannot cope with other tasks that require normal visible light image recognition. The plugging operation is designed to grab the plugging plate, locate the center of the inner flange ring, place the plugging plate, and tighten the bolts. Among them, the target cannot be set at the inner flange ring mouth. Therefore, the center position of the flange hole can only be obtained through image recognition and positioning methods. The flange mouth cannot be observed only through infrared images.

[0013] (2) The target pose solution accuracy can only be guaranteed when the target is on the same plane as the target point. When the target is not on the same plane, there will be a large deviation, especially in the estimation of the depth direction. The blocking operation involves positioning and guiding tasks in the depth direction, which requires high-precision positioning information in the depth direction, which is difficult to meet under single-camera conditions.

[0014] (3) The existing single-camera guidance system adopts a repositioning control method, which requires the robot arm to calibrate its position when it successfully grasps the object in advance, and the calibration process requires manual intervention, which is cumbersome. It cannot effectively use the requirements of automatic operation of the blocking plate.

[0015] The second is the multi-camera + target method

[0016] In the existing multi-camera target vision guidance system, the detection principle of the target point is basically the same as that of the single camera. It is also based on the special material characteristics of the target point. The threshold segmentation method is used to extract the highlight points in the infrared image and calculate the pixel coordinates of the target point in the multi-camera image. Then, based on the conversion relationship between the pre-calibrated cameras, the multi-view reprojection principle is used to calculate the three-dimensional coordinates of the target point, and then the three-dimensional posture of the target plate is obtained.

[0017] For the blocking operation scenario, the existing multi-camera target visual guidance method has the following disadvantages:

[0018] (1) As with the single-camera case, infrared detection of target points can only be used to track the position of the target points, and cannot cope with other tasks that require normal visible light image recognition. The plugging operation involves the grabbing of the plugging plate, the center positioning of the inner flange ring, the placement of the plugging plate, and the tightening of the bolts. The target cannot be set at the inner flange ring mouth, so the center position of the flange hole can only be obtained through image recognition and positioning methods. The flange mouth cannot be observed only through infrared images.

[0019] (2) The installation layout of multiple cameras needs to be designed in advance, and the external parameters between the cameras (the coordinate system transformation relationship between the cameras) need to be accurately calibrated in advance. The usual calibration method is to complete the calibration of the internal parameters and distortion parameters of each camera, and then manually move the calibration rod or calibration plate in the calibration area to cover all areas as much as possible. The transformation relationship of each camera has been calibrated to keep the calibration error in the entire scene at a small level. However, the working space for the plug installation is relatively small, and there is radiation dose. It is dangerous for personnel to enter the calibration, and the calibration process is difficult to achieve.

[0020] (3) Under multi-camera conditions, at least two cameras are required to observe the same target point at the same time. However, the blocking robot needs to cross a large area in the process of grabbing and placing the blocking board, and the use of the multi-camera mode is greatly restricted. In view of the above introduction and analysis of the working scenarios of the blocking robot and the existing methods, the present invention proposes a target plate pose estimation framework combining a single camera and multiple cameras and an automated hand-eye calibration and multi-camera calibration method. On the one hand, it makes up for the inaccurate depth estimation of a single camera and the limited field of view of multiple cameras, and on the other hand, it solves the calibration problems existing in the existing visual guidance methods and the applicability of the system. Summary of the invention

[0021] The purpose of the present invention is to provide a device, system and method for visual guidance of a plugging robot. The visual guidance method is adopted. A camera is set up to shoot the target at the end of the robot arm, the grasped object or the positioning hole position target, so as to obtain the robot arm posture and the final position and posture of the operation. The motion control instructions of the robot arm are output through the control algorithm to guide and control the plugging operation robot to complete the above-mentioned 4-step remote control operation process, replacing manual remote control operation and realizing fully automated guidance and control tasks.

[0022] The technical solution of the present invention is as follows: a device for visual guidance of a blocking robot, comprising a work site camera, a target board, an industrial computer processor and a remote monitoring terminal, wherein the target board is installed on the end of a robotic arm, the work site camera is connected to the industrial computer processor, the industrial computer processor is connected to the remote monitoring terminal, and the remote monitoring terminal has a display screen.

[0023] The work site camera adopts a multi-camera system, and the camera system includes multiple groups of cameras. At least two groups of cameras can cover the observation range, ensuring that the three-dimensional position and posture of the target within the work range can be obtained.

[0024] The industrial computer processor controls multiple cameras on site, obtains on-site image data, calculates the end posture of the robotic arm based on the acquired multi-camera image data, identifies the work target, and estimates its posture, and outputs the control signal of the robotic arm based on the posture offset between the two to control the movement of the robotic arm.

[0025] The display of the remote monitoring end is used for the operator to observe the work site image, the end position and posture of the robot arm, and the target position and posture results.

[0026] A system for visual guidance of a blocking robot, comprising: an automatic calibration algorithm module and a target point position tracking algorithm module;

[0027] The automated calibration algorithm module calibrates the multi-camera system. The robot moves along a preset trajectory point. At each point, the robot stays still for 3 seconds. The cameras in the multi-camera system capture images and record the current TCP posture of the end of the robot until all trajectory points are traversed. After all image data are obtained, the external parameters between the cameras and the coordinate system conversion matrix between the cameras are calculated. After the multi-camera calibration is completed, the conversion matrix between the camera coordinate system and the robot base coordinate system is solved according to the captured images and the TCP posture of the end of the robot to obtain the hand-eye calibration parameters.

[0028] The target point posture tracking algorithm module extracts the target point in real time according to the on-site operation image obtained by the camera in the multi-camera system, solves the posture of the target plate according to the position of the target point and the actual coordinate value of the local coordinate system of the target point, obtains the posture of the tool coordinate system of the end of the robot arm according to the posture of the target plate, realizes real-time posture tracking of the end of the robot arm, and simultaneously calculates the posture offset with the target posture obtained by the multi-camera, converts it into the motion control quantity of the robot arm according to the prior hand-eye calibration result, and sends it to the robot arm actuator to perform the operation action.

[0029] A method for visual guidance of a blocking robot, comprising the following steps:

[0030] Step 1: Perform multi-camera calibration to obtain the internal parameter matrix and distortion coefficient calibration;

[0031] Step 2: Run the preset trajectory points to perform hand-eye calibration;

[0032] Step 3: Run the deep learning-based target pose tracking algorithm to track and obtain the end pose of the robotic arm in real time:

[0033] Step 4: Use template matching to obtain the position of the work target in each image of the multi-camera. According to the matching points and multi-camera calibration parameters, calculate the three-dimensional coordinates of the target matching points in the camera coordinate system. Combined with the size data of the target, use the PNP algorithm to solve the target pose.

[0034] Step 5: Convert the work target posture to the robot base coordinate system, obtain the robot motion control parameters, control the robot action, and combine the robot end posture tracking results to determine whether the robot has reached the target position. If it has not reached the position, convert the target posture difference into motion increments, and continue to generate motion control parameters until the posture difference between the two is less than the set threshold.

[0035] The step 1 includes the following:

[0036] Step 11: Set a number of trajectory points, control the robot arm to move along the trajectory points, and obtain the target plate images taken by each camera when the robot arm reaches each trajectory point;

[0037] Step 12: Calculate and output the position and posture of the target plate in each camera image;

[0038] Step 13: Establish equations based on the three-dimensional coordinate points of the target plate in different camera coordinate systems. The coordinates of the target point in camera 1 are X C1 , the coordinate in camera 2 is X C2 , according to formula (1), establish the equation:

[0039] X C2 =T 1t2 *X C1 (1)

[0040] Step 14: The RANSAC algorithm can be used to solve the coordinate transformation matrix T from camera 1 to camera 2 1t2 .

[0041] The step 2 includes the following:

[0042] Step 21: First, set a section of the robot's running trajectory points and set the trajectories of several points;

[0043] Step 22: Let the robot arm run to the posture state of the specified point in sequence according to the trajectory points and pause for 3 seconds, and collect image data at the specific point and the corresponding robot arm TCP posture data respectively;

[0044] Step 23: The acquired image data is extracted using a target plate and a corresponding pose estimation algorithm to calculate the pose of the target plate;

[0045] Step 24: Obtain the three-dimensional coordinates X of the target plate in the camera coordinate system based on the image CAccording to the offline TCP calibration, the three-dimensional coordinates X of the target plate at each point in the robot base coordinate system can be obtained. B , establish equation (2), and use the RANSAC algorithm to solve the equation to obtain the transformation matrix T of the camera relative to the robot base coordinate system CtB , recorded as the hand-eye calibration output result

[0046] X B =T CtB *X C (2).

[0047] The step 3 includes the following:

[0048] Step 31: Dataset Generation

[0049] The standard target template image is rotated, translated, cut, perspective, and scaled, and a certain amount of grayscale changes and noise are added. At the same time, the mask frame of the target template image is also subjected to the same geometric processing as the target image, so as to obtain the true value frame of the target block, and annotate a small amount of real data sets obtained on site to obtain a real data set;

[0050] Step 32: Model training

[0051] The detection network uses the yoloV7 network and uses the simulated data set to train the detection network;

[0052] Step 33: Model Optimization

[0053] A small amount of real scene data obtained on site is used to fine-tune and optimize the initial detection network trained with simulated data;

[0054] Step 34: Target Tracking

[0055] Each frame of image data is processed by the detection network to obtain the target detection result, and the DeepSort tracking algorithm is further used to track the target in each frame;

[0056] Step 35: Pose estimation

[0057] After the target plate tracking and identification is completed, the target point in the target plate pattern is detected and extracted to obtain the pixel coordinates of the target point. Then, the target point in the target pattern is matched with the standard target template to obtain the correspondence between the pixel point in the image block and the target template point, and then the world coordinate of the target point is obtained. The world coordinate is a rectangular coordinate system established with the center of the target template as the dot, the length and width of the template as the X and Y axes respectively, and the plane where the target template is located perpendicular to the Z axis.

[0058] Step 36: The end position of the robot arm can be obtained according to the obtained target plate position and the conversion relationship between the calibrated target plate coordinate system and the TCP tool coordinate system at the end of the robot arm.

[0059] The step 35 includes the following:

[0060] Step 351: Target point extraction

[0061] A circle is used as the target point, and the target point is clearly distinguished by setting the background color of the target plate to white and the target point to black. The target point detection first performs bilateral filtering on the tracked target pattern block image to enhance the clarity of the image. At the same time, Gaussian smoothing is used to remove part of the noise. Then the MSER region extraction algorithm is used to extract the color consistency area with strong grayscale contrast. The non-maximum region suppression algorithm is used to obtain the accurate target point area. The non-maximum region suppression will perform elliptical fitting on the shape of the current region within each region according to the MSER region extraction result, fit the center of the ellipse, the major axis radius, the minor axis radius and the rotation angle of the ellipse, and use the area of ​​the fitted ellipse and the degree of proximity to the circle for weighted scoring. When suppressing the non-maximum region, each time it will be traversed from high to low according to the score. Each cycle will remove the area with the highest score of the current selection greater than a certain threshold. After the cycle traversal is completed, the area closest to the target point is retained.

[0062]

[0063] Where s1 represents the area of ​​the region, s2 is the proportion of darker areas in the region, s3 is the ratio of the short axis length to the long axis length, and r a is the minor axis length, r b is the major axis length;

[0064] Step 352: Target plate pose estimation based on ICP

[0065] After obtaining the pixel coordinates of the target point in the target plate, the category information output during target detection and recognition can be directly matched to the standard template of the target. The two-dimensional point sparse matching method based on ICP is adopted. When performing ICP matching, the PCA algorithm is first used to fit the target pixel points and the main direction of the target plate, and then the target image is rotated to the main direction of the target plate. Then, the center point normalization method is used to translate the target image and the target template to obtain the rough alignment result after rotation and translation. The further fine alignment method adopts the ICP algorithm, that is, iteratively selects the nearest point as the matching point, calculates the transformation matrix, and then calculates the error after the transformation. Then, the overall point is transformed according to the transformation matrix, and then the above nearest point search and matching process is repeated until the final conversion error change is less than a certain threshold, and the final fine alignment result is obtained.

[0066] After obtaining the alignment between the target image and the target template, the one-to-one correspondence between the pixel coordinates of the target point in the target image and the world coordinates is obtained according to the world coordinates designed in advance by the target template points, and then brought into the PNP equation to solve the corresponding target plate pose.

[0067]

[0068] Where [u,v] is the pixel coordinate, K is the camera intrinsic parameter matrix, [R|t] is the required pose matrix, and [x,y,z] is the world coordinate;

[0069] Step 353: Multi-camera pose fusion

[0070] After the above target plate detection and pose solving steps, the pose of the tracked target plate in the field of view of a single camera is obtained. When multiple camera systems are used, the same target appears in the fields of view of multiple cameras. The multi-camera projection equations are combined to solve the pose of the target plate.

[0071] A point (x i ,y i , z i ), the real projection point in the jth camera image after projection imaging is The detected homonymous points are The intrinsic parameter matrix of camera j is K j , the transformation matrix (external parameter matrix) relative to the reference coordinate system is T j ;

[0072] When there are multiple cameras observing at the same time, the back-projection error is used for modeling:

[0073]

[0074] Wherein j=1...n is the observation data in the camera from 1 to n, and i=1...m is the value of the corresponding pixel of the same name point in a single field of view or the world coordinate system.

[0075] The beneficial effect of the present invention is that the present invention can replace the existing remote control operation mode of the plugging plate robot, and is used to automatically guide the plugging plate robot to install and remove the plugging plate. In addition, the present invention can realize the automatic hand-eye calibration and multi-camera calibration of the guidance system without human intervention, and adopts the target detection and tracking algorithm of deep learning, which can provide a highly robust target tracking effect under visible light shooting conditions. The single-camera independent pose estimation and multi-camera pose fusion mode are adopted. On the one hand, the system has the advantages of single-camera system being flexible and easy to use in dark rotation, and on the other hand, the multi-camera three-dimensional pose estimation accuracy is obtained, so that the device and system have high accuracy, applicability and scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 A schematic diagram of a device for visual guidance of a plate blocking robot provided by the present invention;

[0077] Figure 2 Schematic diagram of the installation site of the plugging operation

[0078] Figure 3 A schematic diagram of a system for visual guidance of a panel blocking robot provided by the present invention;

[0079] Figure 4 Flowchart of the automated multi-camera calibration method;

[0080] Figure 5 It is a flow chart of the automated hand-eye calibration method;

[0081] Figure 6 This is a flow chart of the target pose tracking algorithm based on deep learning;

[0082] Figure 7 The target plate pattern used for the part;

[0083] Figure 8 Generate training data examples for simulation in the present invention;

[0084] Fig. 9 This is the flow chart of the Kalman filter algorithm;

[0085] Fig.10 It is the flow chart of the target point extraction algorithm;

[0086] Fig.11 The result map of the extraction of the front region for non-maximum region suppression;

[0087] Fig.12 This is the region extraction result image after non-maximum region suppression;

[0088] Fig.13 Schematic diagram of multi-camera reprojection. DETAILED DESCRIPTION

[0089] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0090] The present invention provides a device, system and method for visual guidance of a plugging robot, which adopts a visual guidance method, sets up a camera to shoot the target at the end of the robot arm, the grasped object or the positioning hole position target, obtains the robot arm posture and the final position and posture of the operation, outputs the motion control instructions of the robot arm through the control algorithm, and automatically guides and controls the plugging robot to complete the above 4-step remote control operation process, replacing manual remote control operation, and realizing a fully automated guidance and control task. The present invention does not include the original plugging robot part, but only involves the visual guidance part.

[0091] A device and system for visual guidance of a plugging robot. First, a target plate printed with a target pattern is fixed to the plugging plate, and the position of the plugging plate grabbing position and the position of the bolt hole relative to the target plate center coordinate system are measured and calibrated. The camera used for guidance is calibrated offline to obtain the intrinsic parameters and distortion parameters of each camera. The target plate is fixed at the end of the plugging robot operation, and offline TCP calibration is completed, that is, the position of the end of the robot arm operation relative to the origin of the target point is calibrated.

[0092] Then, install the calibrated guide camera and configured light source to the corresponding position, connect the power supply, link the camera image signal output end to the industrial computer controlling the blocking robot, and install the software system into the control system.

[0093] Then, the visual guidance software system of the present invention is started, and the automated hand-eye calibration and multi-camera calibration tools are started to run to complete the hand-eye calibration and multi-camera calibration.

[0094] Next, turn on the guided operation mode and follow the previous remote control robotic arm plug installation operation steps to start the robotic arm automated installation operation in sequence.

[0095] After the job is completed, exit the system and turn off the power of the device.

[0096] like Figure 1 As shown, a device for visual guidance of a blocking robot includes: a work site camera, a target board, an industrial computer processor and a remote monitoring terminal, wherein the target board is installed on the end of the robot arm, the work site camera is connected to the industrial computer processor, the industrial computer processor is connected to the remote monitoring terminal, and the remote monitoring terminal has a display screen.

[0097] The work site camera adopts a multi-camera system to ensure that at least two groups of cameras can cover the observation range and obtain the three-dimensional position of the target within the work range. Figure 7The target plate is printed or sprayed on the aluminum plate, and the size of the pattern must be guaranteed. After printing, it needs to be installed at the end of the robot arm and fixed to the end tool (air lock). The position of the tool coordinate system of the robot arm can be calculated based on the position of the target plate.

[0098] The industrial computer processor controls multiple cameras on site, obtains on-site image data, calculates the end posture of the robotic arm based on the acquired multi-camera image data, identifies the work target (flange hole, plugging plate, bolt, etc.), and estimates its posture, and outputs the control signal of the robotic arm based on the posture offset of the two to control the movement of the robotic arm.

[0099] The display of the remote monitoring terminal is used for operators to observe the results of the work site image, the end position of the robot arm, the target position, etc.

[0100] A system for visual guidance of a panel blocking robot comprises: an automatic calibration algorithm module and a target point position tracking algorithm module. After the mechanical arm and the camera system are installed on site, the system first runs the automatic calibration algorithm module.

[0101] The automated calibration algorithm module calibrates the multi-camera system. The robotic arm moves along a preset trajectory point. At each point, the robotic arm remains stationary for 3 seconds. The cameras in the multi-camera system capture images and record the current TCP posture of the end of the robotic arm until all trajectory points are traversed. After all image data are obtained, the external parameters between the cameras and the coordinate system transformation matrix between the cameras are calculated. After the multi-camera calibration is completed, the transformation matrix between the camera coordinate system and the robotic arm base coordinate system is solved according to the captured images and the TCP posture of the end of the robotic arm to obtain the hand-eye calibration parameters.

[0102] After completing the hand-eye calibration, you can enter the robotic arm operation process.

[0103] The target point posture tracking algorithm module is a target point posture tracking algorithm module based on deep learning. It extracts the target point in real time according to the on-site operation image obtained by the camera in the multi-camera system, and solves the posture of the target plate according to the position of the target point and the actual coordinate value of the local coordinate system of the target point. The posture of the tool coordinate system of the end of the robot arm is obtained according to the posture of the target plate, realizing real-time posture tracking of the end of the robot arm. At the same time, it performs posture offset calculation with the target posture obtained by multiple cameras, converts it into the motion control quantity of the robot arm according to the prior hand-eye calibration results, and sends it to the robot arm actuator to perform the operation action.

[0104] A method for visual guidance of a blocking robot, comprising the following steps:

[0105] Step 1: After the visual guidance hardware device and the robotic arm are installed, the multi-camera calibration method flow chart is as follows: Figure 4 As shown. Multi-camera calibration relies on the installation of a target at the end of the robotic arm, and each camera has been calibrated with a single camera before, and the checkerboard method can be used for calibration. To obtain accurate internal parameter matrix and distortion coefficient calibration, this part of the calibration work only needs to be completed in advance, and the subsequent processes can be used. The specific process is as follows:

[0106] Step 11: Set a number of trajectory points, control the robot arm to move along the trajectory points, and obtain the target plate images taken by each camera when the robot arm reaches each trajectory point;

[0107] Step 12: Calculate and output the pose of the target plate in each camera image according to the target pose tracking module (this algorithm can be obtained using a target pose tracking algorithm module based on deep learning);

[0108] Step 13: Establish equations based on the three-dimensional coordinate points of the target board in different camera coordinate systems. For example, the coordinates of the target point in camera 1 are X C1 , the coordinate in camera 2 is X C2 , according to formula (1), establish the equation:

[0109] X C2 =T 1t2 *X C1 (1)

[0110] Step 14: The RANSAC algorithm can be used to solve the coordinate transformation matrix T from camera 1 to camera 2 1t2 .

[0111] Note: When solving the external parameters between multiple cameras (camera coordinate system conversion matrix), one of the cameras can be used as a fixed reference coordinate system, and the conversion relationship between the other cameras and the reference camera coordinate system can be solved separately.

[0112] Step 2: Run the preset trajectory points for automated hand-eye calibration.

[0113] The TCP calibration of the robot arm mainly calibrates the coordinate conversion relationship between the origin position of the calibration plate fixed at the end of the robot arm and the TCP position set at the end of the robot arm. Usually, in order to facilitate subsequent calculations, the calibration plate will be set as the TCP point. The method flow chart is as follows Figure 5 As shown, the specific process is as follows:

[0114] Step 21: First, set a section of the robot's running trajectory points. Usually, several (such as 50) point trajectories can be set. The more trajectory points are set, the higher the subsequent calibration accuracy and reliability.

[0115] Step 22: Let the robot arm run to the posture state of the specified point in sequence according to the trajectory points and pause for 3 seconds, and collect image data at the specific point and the corresponding robot arm TCP posture data respectively;

[0116] Step 23: The obtained image data can be used to calculate the pose of the target plate by using the target plate extraction and the corresponding pose estimation algorithm (this algorithm can be obtained using the target pose tracking algorithm module based on deep learning);

[0117] Step 24: Obtain the three-dimensional coordinates X of the target plate in the camera coordinate system based on the image C According to the offline TCP calibration, the three-dimensional coordinates X of the target plate at each point in the robot base coordinate system can be obtained. B , establish equation (2), and use the RANSAC (Random Sample Consensus) algorithm to solve the equation, and we can get the transformation matrix T of the camera relative to the robot base coordinate system CtB , recorded as the hand-eye calibration output result.

[0118] X B =T CtB *X G (2)

[0119] Step 3: Run the target pose tracking algorithm based on deep learning. The algorithm flow chart is as follows: Figure 6 , real-time tracking to obtain the end position of the robotic arm, the specific process is as follows:

[0120] Step 31: Dataset Generation (Offline Phase)

[0121] An industrial scene image with steel equipment and other equipment as the background, which is close to the on-site operation scene, is used as the background image. Figure 7 After a certain amount of rotation, translation, shearing, perspective, scaling and other geometric transformations, a certain amount of grayscale changes and noise are added, and then randomly pasted to the background image. At the same time, the background image will also undergo random affine transformations, color changes, random noise additions, and corresponding random motion blur additions. At the same time, the mask frame of the target template image will also undergo the same geometric processing as the target image to obtain the true value frame of the target block. The example of generating a data set is as follows Figure 8 shown.

[0122] A small amount of real data sets obtained on-site are labeled to obtain real data sets.

[0123] Step 32: Model training (offline phase)

[0124] The detection network uses the yoloV7 network (YOLO stands for You Only Look Once) and uses the simulated data set to train the detection network.

[0125] Step 33: Model optimization (offline phase)

[0126] A small amount of real scene data obtained on site is used to fine-tune and optimize the initial detection network trained with simulated data;

[0127] Step 34: Target Tracking (Online Use)

[0128] Each frame of image data is processed by the detection network to obtain the target detection result, and the DeepSort tracking algorithm is further used to track the target in each frame.

[0129] After using the YOLO network to detect each frame of the video image to obtain the target area, the YOLO network detection result cannot absolutely guarantee the accuracy of the detection result, so it is necessary to combine the detection results of the previous frames to judge the current detection result. In addition, due to problems such as changes in lighting and occlusion, the target in some frame images cannot be successfully detected, so it is also necessary to combine the detection results of the previous moment to optimize.

[0130] The present invention introduces the DeepSort target tracking framework to further track the detection results. The DeepSort framework mainly includes Kalman prediction, deep feature matching and Hungarian algorithm.

[0131] The Kalman prediction algorithm process is as follows Fig. 9 As shown:

[0132] in is the state vector of the target at time k-1, R k-1 is the covariance matrix between the state quantities at time k-1. k-1 represents the control amount of the system at time k, z k represents the measured value at time k, A, B and H are state transformation matrices, which are adjustment coefficients in the state transformation process, and are constants here. Matrix B represents the gain of the optional control input u, and matrix H represents the state variable x k For the measured variable z k The gain of . Q is the process excitation noise covariance matrix, R is the observation noise covariance matrix, and Q and R are constants.

[0133] The deep feature uses the output vector of the BackBone main frame of the YOLO network as the deep feature of the target pattern, and uses the cosine function to calculate the similarity of the feature vector. At the same time, combined with the state information predicted by Kalman, the matching association of the detection targets of the previous and next consecutive frames is performed according to the Hungarian algorithm. In particular, for partially occluded low-confidence candidate targets, the correct low-confidence detection target is detected and output according to the similarity association degree of the previous and next frame targets.

[0134] Step 35: Pose estimation

[0135] After the target plate tracking and identification is completed, the target points in the target plate pattern need to be detected and extracted to obtain the pixel coordinates of the target points. Then, the target points in the target pattern need to be matched with the standard target template to obtain the correspondence between the pixel points in the image block and the target template points, and then the world coordinates of the target points are obtained. The world coordinates are a rectangular coordinate system with the center of the target template as the dot, the length and width of the template as the X and Y axes respectively, and the plane where the target template is located perpendicular to the Z axis. The world coordinates of the target point can be obtained in advance based on the actual size of the template and the design of the point.

[0136] After obtaining the correspondence between the world coordinates and pixel coordinates of the target point, we can use the PNP algorithm to calculate the pose of the center of the target plate in the camera coordinate system.

[0137] The above is the result of obtaining the target pose of a single camera. When there are multiple cameras arranged in the scene and the pose of the same target is calculated in the field of view of two cameras at the same time, the pose of the target plate can be further optimized according to the reprojection principle, through the reprojection equation, using the coordinate system transformation matrix calibrated between the cameras, to obtain an accurate three-dimensional spatial pose.

[0138] Step 351: Target point extraction

[0139] Since the target plate pattern will use a pattern with strong grayscale contrast in advance to significantly distinguish the target point, and in order to improve the positioning accuracy and reliability of the target point, regular geometric patterns such as corner points or circles are usually used to facilitate accurate detection and positioning.

[0140] In the present invention, we use a circle as the target point, and set the target plate background color to white and the target point to black to clearly distinguish the target point. However, due to the interference of noise, lighting changes, ambient light, etc. in the field shooting, the image is usually reduced in contrast and blurred at the edge. In addition, the change of viewing angle usually causes geometric distortion. The circle will be projected into an ellipse when not imaging orthogonally. Therefore, an ellipse extraction method is required in the detection process. The detailed target point detection and extraction process is as follows: Fig.10 shown.

[0141] The target point detection first performs bilateral filtering on the tracked target pattern block image to enhance the image clarity, and uses Gaussian smoothing to remove some noise, and then uses the MSER (Maximally Stable ExtremalRegions) region extraction algorithm to extract the color consistency region with strong grayscale contrast. Since region extraction usually results in a region being surrounded by multiple extraction frames, it is usually necessary to use a non-maximal region suppression algorithm to obtain an accurate target point region.

[0142] Among them, non-maximum region suppression will perform elliptical fitting on the current region shape within each region according to the MSER region extraction result, fit the center of the ellipse, the major axis radius, the minor axis radius and the rotation angle, and use the area of ​​the fitted ellipse and the degree of proximity to the circle for weighted scoring (see formula 3). When suppressing non-maximum regions, each time it will be traversed from high to low according to the score. In each cycle, the regions with an overlap greater than a certain threshold with the currently selected region with the highest score will be removed. After the cyclic traversal is completed, the region closest to the target point is retained.

[0143]

[0144] Where s1 represents the area of ​​the region, s2 is the proportion of the darker area in the region, and s3 is the ratio of the short axis length to the long axis length (r a is the minor axis length, r b is the major axis length).

[0145] After non-maximum area suppression, relatively accurate target point extraction results can be basically obtained. However, in some cases, erroneous points may appear, especially when the target is rotated, resulting in the identification of the target area with background scenes. The background extraction results will interfere with the extraction of target points. Therefore, it is necessary to further screen the target points. The usual method is to perform screening based on the shape constraints of the extracted target points and the similarity of the target template. The main criteria include the ratio of the major and minor axes and the area of ​​the darker target point. In addition, the target plate usually detected by the YOLO network generally has category labels. The distribution of standard target points corresponding to the category, such as the number of target points, can be used for judgment, and the erroneous target points can be eliminated to output the final detection and positioning results. Fig.11 is the result before non-maximum region suppression, Fig.12 This is the result after non-maximum region suppression. It is obvious that repeated detection results and those with slight offsets can be filtered out to obtain a stable and accurate detection frame.

[0146] Step 352: Target plate pose estimation based on ICP

[0147] After obtaining the pixel coordinates of the target points in the target plate, we can use the category information output during target detection and recognition to directly correspond to the standard template of the target. However, since the actual target pattern and the standard target template generally have geometric changes such as rotation, scaling, and misalignment, they cannot be directly corresponded to the corresponding points in the target template. At this time, further matching with the target template is required to obtain the correspondence between the target extracted pixel points and the target template points.

[0148] The present invention adopts a two-dimensional point sparse matching method based on ICP (Iterative Closest Point). When designing the target plate, in order to make the target plate have a direction-guiding characteristic, it is generally designed to be in a non-fully stacked state so that the target plate can indicate the direction of the target. Therefore, when performing ICP matching, the PCA (principle component analysis) algorithm is first used to fit the target pixel points and the main direction of the target plate, and then the target image is rotated to the main direction of the target plate, and then the center point normalization method is used to translate the target image and the target template. At this time, a rough alignment result after rotation and translation can be obtained, and the further fine alignment method is performed using the ICP algorithm, that is, iteratively selecting the nearest point as the matching point, calculating the conversion matrix, and then calculating the error after the conversion, and then converting the entire point according to the conversion matrix, and then repeating the above-mentioned nearest point search and matching process until the final conversion error change is less than a certain threshold, and the final fine alignment result is obtained.

[0149] After the target image and the target template are aligned, the one-to-one correspondence between the pixel coordinates of the target point in the target image and the world coordinates can be obtained according to the world coordinates designed in advance by the target template points, and then brought into the PNP equation (as shown in Formula 4) to solve the corresponding target plate pose.

[0150]

[0151] Where [u,v] is the pixel coordinate, K is the camera intrinsic parameter matrix, [R|t] is the required pose matrix (extrinsic parameter matrix), and [x,y,z] is the world coordinate.

[0152] Step 353: Multi-camera pose fusion

[0153] After the above target plate detection and pose solving steps, the pose of the tracked target plate within the field of view of a single camera can be obtained. When multiple camera systems are used, the same target appears in the field of view of multiple cameras. At this time, we can combine the multi-camera projection equations to solve a more accurate target plate pose. The principle diagram of multi-camera re-projection is shown in the figure below. Fig.13 shown.

[0154] A point (x i,y i , z i ), the real projection point in the jth camera image after projection imaging is The detected homonymous points are The intrinsic parameter matrix of camera j is K j , the transformation matrix (external parameter matrix) relative to the reference coordinate system is T j .

[0155] When there are multiple cameras observing at the same time, the back-projection error is used for modeling:

[0156]

[0157] Where j=1...n is the observation data in the camera from 1 to n, and i=1...m is the value of the corresponding pixel or world coordinate system of the same name point in a single field of view. The above is the multi-view reprojection error equation to be optimized. By minimizing formula 5, the accurate pose [R|t] is obtained. At this point, the three-dimensional pose result of the target plate in multiple cameras can be obtained.

[0158] Step 36: The end position of the robot arm can be obtained according to the obtained target plate position and the conversion relationship between the calibrated target plate coordinate system and the TCP tool coordinate system at the end of the robot arm.

[0159] Step 4: Use template matching to obtain the position of the work target in each image of the multi-camera. According to the matching points and multi-camera calibration parameters, the three-dimensional coordinates of the target matching points in the camera coordinate system are calculated. Combined with the target size data, the PNP algorithm is used to solve the target pose.

[0160] Step 5: Follow the hand-eye calibration parameters, convert the work target posture to the robot base coordinate system, obtain the robot motion control parameters, control the robot action, and combine the robot end posture tracking results to determine whether the robot has reached the target position. If it has not reached the position, the target posture difference is converted into motion increments, and the motion control parameters are continuously generated until the posture difference between the two is less than the set threshold.

Claims

1. A device for visual guidance of a blocking robot, characterized in that: It includes a work site camera, a target board, an industrial computer processor and a remote monitoring terminal, wherein the target board is installed on the end of the mechanical arm, the work site camera is connected to the industrial computer processor, the industrial computer processor is connected to the remote monitoring terminal, and the remote monitoring terminal has a display screen.

2. A device for visual guidance of a plate blocking robot as claimed in claim 1, characterized in that: The work site camera adopts a multi-camera system, and the camera system includes multiple groups of cameras. At least two groups of cameras can cover the observation range, ensuring that the three-dimensional position and posture of the target within the work range can be obtained.

3. The device for visual guidance of a plate blocking robot according to claim 1, characterized in that: The industrial computer processor controls multiple cameras on site, obtains on-site image data, calculates the end posture of the robotic arm based on the acquired multi-camera image data, identifies the work target, and estimates its posture, and outputs the control signal of the robotic arm based on the posture offset between the two to control the movement of the robotic arm.

4. The device for visual guidance of a plate blocking robot according to claim 1, characterized in that: The display of the remote monitoring end is used for the operator to observe the work site image, the end position and posture of the robot arm, and the target position and posture results.

5. A system for visual guidance of a blocking robot, characterized in that: include: Automatic calibration algorithm module and target point posture tracking algorithm module; The automated calibration algorithm module calibrates the multi-camera system. The robot moves along a preset trajectory point. At each point, the robot stays still for 3 seconds. The cameras in the multi-camera system capture images and record the current TCP posture of the end of the robot until all trajectory points are traversed. After all image data are obtained, the external parameters between the cameras and the coordinate system conversion matrix between the cameras are calculated. After the multi-camera calibration is completed, the conversion matrix between the camera coordinate system and the robot base coordinate system is solved according to the captured images and the TCP posture of the end of the robot to obtain the hand-eye calibration parameters. The target point posture tracking algorithm module extracts the target point in real time according to the on-site operation image obtained by the camera in the multi-camera system, solves the posture of the target plate according to the position of the target point and the actual coordinate value of the local coordinate system of the target point, obtains the posture of the tool coordinate system of the end of the robot arm according to the posture of the target plate, realizes real-time posture tracking of the end of the robot arm, and simultaneously calculates the posture offset with the target posture obtained by the multi-camera, converts it into the motion control quantity of the robot arm according to the prior hand-eye calibration result, and sends it to the robot arm actuator to perform the operation action.

6. A method for visual guidance of a blocking robot, characterized in that: The steps include: Step 1: Perform multi-camera calibration to obtain the internal parameter matrix and distortion coefficient calibration; Step 2: Run the preset trajectory points to perform hand-eye calibration; Step 3: Run the deep learning-based target pose tracking algorithm to track and obtain the end pose of the robotic arm in real time: Step 4: Use template matching to obtain the position of the work target in each image of the multi-camera. According to the matching points and multi-camera calibration parameters, calculate the three-dimensional coordinates of the target matching points in the camera coordinate system. Combined with the size data of the target, use the PNP algorithm to solve the target pose. Step 5: Convert the work target posture to the robot base coordinate system, obtain the robot motion control parameters, control the robot action, and combine the robot end posture tracking results to determine whether the robot has reached the target position. If it has not reached the position, convert the target posture difference into motion increments, and continue to generate motion control parameters until the posture difference between the two is less than the set threshold.

7. A method for visual guidance of a panel blocking robot as claimed in claim 6, characterized in that: The step 1 includes the following: Step 11: Set a number of trajectory points, control the robot arm to move along the trajectory points, and obtain the target plate images taken by each camera when the robot arm reaches each trajectory point; Step 12: Calculate and output the position and posture of the target plate in each camera image; Step 13: Establish equations based on the three-dimensional coordinate points of the target plate in different camera coordinate systems. The coordinates of the target point in camera 1 are X C1 , the coordinate in camera 2 is X C2 , according to formula (1), establish the equation: X C2 =T 1t2 *X C1 (1) Step 14: The RANSAC algorithm can be used to solve the coordinate transformation matrix T from camera 1 to camera 2 1t2 .

8. A method for visual guidance of a panel blocking robot as claimed in claim 6, characterized in that: The step 2 includes the following: Step 21: First, set a section of the robot's running trajectory points and set the trajectories of several points; Step 22: Let the robot arm run to the posture state of the specified point in sequence according to the trajectory points and pause for 3 seconds, and collect image data at the specific point and the corresponding robot arm TCP posture data respectively; Step 23: The acquired image data is extracted using a target plate and a corresponding pose estimation algorithm to calculate the pose of the target plate; Step 24: Obtain the three-dimensional coordinates X of the target plate in the camera coordinate system based on the image C According to the offline TCP calibration, the three-dimensional coordinates X of the target plate at each point in the robot base coordinate system can be obtained. B , establish equation (2), and use the RANSAC algorithm to solve the equation to obtain the transformation matrix T of the camera relative to the robot base coordinate system CtB , recorded as the hand-eye calibration output result X B =T CtB *X C (2)。 9. A method for visual guidance of a panel blocking robot as claimed in claim 6, characterized in that: The step 3 includes the following: Step 31: Dataset Generation The standard target template image is rotated, translated, cut, perspective, and scaled, and a certain amount of grayscale changes and noise are added. At the same time, the mask frame of the target template image is also subjected to the same geometric processing as the target image, so as to obtain the true value frame of the target block, and annotate a small amount of real data sets obtained on site to obtain a real data set; Step 32: Model training The detection network uses the yoloV7 network and uses the simulated data set to train the detection network; Step 33: Model Optimization A small amount of real scene data obtained on site is used to fine-tune and optimize the initial detection network trained with simulated data; Step 34: Target Tracking Each frame of image data is processed by the detection network to obtain the target detection result, and the DeepSort tracking algorithm is further used to track the target in each frame; Step 35: Pose estimation After the target plate tracking and identification is completed, the target point in the target plate pattern is detected and extracted to obtain the pixel coordinates of the target point. Then, the target point in the target pattern is matched with the standard target template to obtain the correspondence between the pixel point in the image block and the target template point, and then the world coordinate of the target point is obtained. The world coordinate is a rectangular coordinate system established with the center of the target template as the dot, the length and width of the template as the X and Y axes respectively, and the plane where the target template is located perpendicular to the Z axis. Step 36: The end position of the robot arm can be obtained according to the obtained target plate position and the conversion relationship between the calibrated target plate coordinate system and the TCP tool coordinate system at the end of the robot arm.

10. A method for visual guidance of a panel blocking robot as claimed in claim 9, characterized in that: The step 35 includes the following: Step 351: Target point extraction A circle is used as the target point, and the target point is clearly distinguished by setting the background color of the target plate to white and the target point to black. The target point detection first performs bilateral filtering on the tracked target pattern block image to enhance the clarity of the image. At the same time, Gaussian smoothing is used to remove part of the noise. Then the MSER region extraction algorithm is used to extract the color consistency area with strong grayscale contrast. The non-maximum region suppression algorithm is used to obtain the accurate target point area. The non-maximum region suppression will perform elliptical fitting on the shape of the current region within each region according to the MSER region extraction result, fit the center of the ellipse, the major axis radius, the minor axis radius and the rotation angle of the ellipse, and use the area of ​​the fitted ellipse and the degree of proximity to the circle for weighted scoring. When suppressing the non-maximum region, each time it will be traversed from high to low according to the score. Each cycle will remove the area with the highest score of the current selection greater than a certain threshold. After the cycle traversal is completed, the area closest to the target point is retained. Where s1 represents the area of ​​the region, s2 is the proportion of darker areas in the region, s3 is the ratio of the short axis length to the long axis length, and r a is the minor axis length, r b is the major axis length; Step 352: Target plate pose estimation based on ICP After obtaining the pixel coordinates of the target point in the target plate, the category information output during target detection and recognition can be directly matched to the standard template of the target. The two-dimensional point sparse matching method based on ICP is adopted. When performing ICP matching, the PCA algorithm is first used to fit the target pixel points and the main direction of the target plate, and then the target image is rotated to the main direction of the target plate. Then, the center point normalization method is used to translate the target image and the target template to obtain the rough alignment result after rotation and translation. The further fine alignment method adopts the ICP algorithm, that is, iteratively selects the nearest point as the matching point, calculates the transformation matrix, and then calculates the error after the transformation. Then, the overall point is transformed according to the transformation matrix, and then the above nearest point search and matching process is repeated until the final conversion error change is less than a certain threshold, and the final fine alignment result is obtained. After obtaining the alignment between the target image and the target template, the one-to-one correspondence between the pixel coordinates of the target point in the target image and the world coordinates is obtained according to the world coordinates designed in advance by the target template points, and then brought into the PNP equation to solve the corresponding target plate pose. Where [u,v] is the pixel coordinate, K is the camera intrinsic parameter matrix, [R|t] is the required pose matrix, and [x,y,z] is the world coordinate; Step 353: Multi-camera pose fusion After the above target plate detection and pose solving steps, the pose of the tracked target plate in the field of view of a single camera is obtained. When multiple camera systems are used, the same target appears in the fields of view of multiple cameras. The multi-camera projection equations are combined to solve the pose of the target plate. A point (x i ,y i , z i ), the real projection point in the jth camera image after projection imaging is The detected homonymous points are The intrinsic parameter matrix of camera j is K j , the transformation matrix relative to the reference coordinate system is T j ; When there are multiple cameras observing simultaneously, the back-projection error is used for modeling: Among them, j=1...n is the observation data in the camera 1-n, and i=1...m is the value of the pixel or world coordinate system corresponding to the same name point in a single field of view.

Images