A method and device for identifying and positioning parts of a visual robot

Abstract

This invention provides a method and apparatus for identifying and locating parts in a visual robot. It uses a binocular camera, a YOLOv4-tiny network, and a PSPNet image segmentation network to capture and analyze the location of the parts. By establishing a transformation relationship between the world coordinate system and the pixel plane coordinate system, the left and right images are corrected, and the region of interest is extracted from the corrected left image. Coarse and fine template matching is performed using YOLOv4-tiny and PSPNet, and sub-pixel disparity and final object depth information are calculated. The spatial coordinates of the parts are calculated using the inscribed circle center and the principle of triangle similarity, thereby improving the detection accuracy, real-time performance, and robustness of the identification and localization.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, specifically to a method and apparatus for identifying and locating visual robot parts. Background Technology

[0002] The key issue in part recognition and localization lies in depth distance. Depth distance based on binocular vision is obtained using stereo matching methods. There are currently five main stereo matching methods: 1) Local matching methods, which are calculated based on windows and use a cost function (or similarity metric) to determine disparity by comparing image patches of the same size in the left and right views. Its basic process is generally cost calculation, cost aggregation, disparity calculation, and disparity refinement; 2) Global matching methods, which utilize global constraint information of the image and are not sensitive to blurring of local images, but have a high computational cost. Global matching algorithms construct a global energy function and then minimize it using optimization methods to obtain a dense disparity map; 3) Semi-Global Matching (SGM) adopts the idea of ​​minimizing the energy function, but unlike global matching methods, SGM transforms the optimization problem of a two-dimensional image into a one-dimensional optimization problem (i.e., scanline optimization) with multiple paths, aggregating path costs from multiple directions, and using the winner-take-all (WTA) algorithm to calculate disparity, achieving a better balance between matching accuracy and computational cost; 4) Deep learning-based non-end-to-end matching methods learn a module from traditional stereo matching methods using a convolutional neural network (CNN), achieving more robust performance than traditional methods; 5) Deep learning-based end-to-end matching methods first extract features from the input image using a convolutional neural network (CNN). Then, the matching status is determined by comparing the image's feature representation with image features in the database. Compared with traditional image matching methods, it can automatically learn feature representations between images without requiring manual design of feature extraction algorithms. Secondly, it can handle complex images, including those with different lighting conditions and poses.

[0003] The technical disadvantages of the existing solution are as follows:

[0004] 1. Currently used stereo matching techniques suffer from problems such as image occlusion and image reflection.

[0005] 2. The SGBM (Semi-Global Block Matching) algorithm is relatively slow.

[0006] 3. The BM (Block-matching) algorithm is easily affected by factors such as lighting and weak textures, and has a relatively high probability of mismatches.

[0007] Therefore, there is a need to provide a method and apparatus for the identification and localization of visual robot parts with higher detection accuracy, real-time performance, and robustness. Summary of the Invention

[0008] In view of this, in order to solve the above-mentioned problems in the prior art, this application proposes a method and apparatus for the identification and positioning of visual robot parts.

[0009] This application solves the above problems through the following technical means:

[0010] This application proposes a method for identifying and locating visual robot parts, characterized by the following steps:

[0011] Step S100: Establish the transformation relationship between the world coordinate system and the pixel plane coordinate system of the stereo camera, and calibrate the stereo camera;

[0012] Step S200: Use a binocular camera to acquire left and right images of the part, and correct image distortion to generate a corrected left image and a corrected right image;

[0013] Step S300: Use the YOLOv4-tiny network to extract the rectangular region of interest (ROI) left image of the part from the corrected left image, and send it into the PSPNet image segmentation network to further extract the ROI. Then, perform grayscale processing on the extracted ROI and the corrected right image to obtain the grayscale ROI of the corrected left image and the grayscale image of the corrected right image.

[0014] Step S400: Perform coarse template matching between the grayscale region of interest (ROI) of the corrected left image and the grayscale image of the corrected right image. Extract the region corresponding to the grayscale ROI of the corrected left image from the grayscale image of the corrected right image, and feed it into the PSPNet image segmentation network to further extract the ROI, thus obtaining the grayscale ROI of the corrected right image. Perform binarization and AND operation on the grayscale ROI of the corrected left image and the grayscale ROI of the corrected right image to obtain a new left image and a new right image of the ROI.

[0015] Step S500: Perform fine template matching on the new left image of the region of interest and the new right image of the region of interest, calculate the sub-pixel disparity between the new left image of the region of interest and the new right image of the region of interest, and calculate the object depth information based on the binocular camera ranging principle;

[0016] Step S600: Calculate the largest inscribed circle in the new region of interest (ROI) left image region, with the center of the circle being the pixel center of the part. Then, calculate the spatial coordinates of the part's location based on the principle of triangle similarity.

[0017] Preferably, the transformation relationship between the world coordinate system and the pixel plane coordinate system of the binocular camera is expressed as follows:

[0018]

[0019] In the formula: S is the scaling factor, (u, v) are the coordinates of point p in the pixel coordinate system, (X... W Y W Let (p) be the coordinates of point p in the image coordinate system, dX and dY represent the physical dimensions of a single pixel in each row and column, respectively, and (u0, v0) be the coordinates of the center of the camera sensor in the pixel coordinate system; f is the camera focal length; R and 0 T Let X be the rotation matrix and identity matrix, respectively, with a shape of (3, 3); t is the translation matrix, with a shape of (3, 1); (X... W Y W Z W () represents the world coordinates of the target object;

[0020] The above formula simplifies to:

[0021]

[0022] In the formula: K is the camera intrinsic parameter matrix, The rotation matrix R and translation matrix t are obtained through camera calibration.

[0023] Preferably, the calibration of the binocular camera includes the following steps:

[0024] The stereo camera was calibrated using Zhang Dingyou's checkerboard calibration method. Fifteen images of a 5mm×5mm checkerboard pattern were taken and imported into the MATLAB toolbox to obtain the stereo camera's distortion parameters, rotation matrix, and translation matrix. The stereo camera's distortion parameters, considering radial and tangential distortion, can be expressed as follows:

[0025]

[0026] In the formula: (X d y d (x, y) represents the normalized image point coordinates after distortion, (x, y) represents the ideal, distortion-free normalized image coordinates, and r represents the distance from the image pixel to the image center point (r0). 2 =X 2 +y 2 k1, k2, and k3 are radial distortion parameters, and p1 and p2 are tangential distortion parameters.

[0027] Preferably, the step of extracting the rectangular ROI left image containing the part from the corrected left image using the YOLOv4-tiny network includes:

[0028] The YOLOv4-tiny network contains two YOLO heads;

[0029] The first YOLO head divides the corrected left image into a 26×26 grid, with pre-defined prior bounding boxes of (23,27), (37,58), and (50,50).

[0030] The second YOLO head divides the corrected left image into a 13×13 grid, with pre-defined prior bounding boxes of (50,50), (81,82), and (135,169).

[0031] Preferably, the step of performing fine template matching on the new left image of the region of interest and the new right image of the region of interest includes the following steps:

[0032] Based on the epipolar constraint characteristic of the image after camera calibration, stereo matching of the left and right images of the new region of interest is achieved by sliding the left and right regions. Subpixel disparity between the two images is calculated, and the SAD function is used for similarity evaluation during the matching process. The SAD function is expressed as follows:

[0033] C SAD (p, d) = ∑q∈N p |I l (p)-I r (pd)|

[0034] In the formula: C SAD (p, d) represents the absolute difference between the pixel values ​​of the two images, N p q represents the matching window; q represents a pixel to be matched within the matching window; I l (p) represents the region to be matched in the left image, I r (pd) represents the region to be matched in the right image with a disparity of d.

[0035] Preferably, the step of calculating the object depth information based on the binocular camera ranging principle includes:

[0036] After obtaining the disparity and similarity curve through precise template matching, the three coordinate points with the highest similarity (x1, y1), (x2, y2), and (x3, y3) are extracted and substituted into the quadratic function y = ax 2 The fitted function is obtained by adding bx and c. The maximum and minimum values ​​of the fitted function are then calculated to obtain the coordinates of the highest point (x). max y max), x max That is, the optimal parallax, y max The similarity is the optimal parallax.

[0037] Based on the sub-pixel disparity values ​​mentioned above and according to the principle of binocular vision, the depth information expression of an object is as follows:

[0038]

[0039] In the formula: z is the object depth, f is the camera focal length, b is the baseline; X L X is the distance from the left camera's imaging point to the left imaging plane. R These represent the distances from the right camera's imaging point to the right imaging plane.

[0040] Preferably, the step of calculating the largest inscribed circle in the left image region of the new region of interest, with the center of the circle being the pixel center of the target object, and then calculating the spatial coordinates of the part's location based on the principle of triangle similarity, includes:

[0041] Contour detection is performed on the segmented object image to obtain the contour matrix; the largest contour Q is then extracted. max The distance between the coordinates of a point within the contour and a point on the contour is calculated to find the center of the largest inscribed circle within the contour. The expression is as follows:

[0042]

[0043] In the formula: R is the radius of the inscribed circle, (x i y i ) is the maximum contour Q max The pixels on (x) j y j () represents the pixels within the outline;

[0044] The center of the largest inscribed circle and the optimal gripping point (u, v) are obtained from the expression, and the spatial coordinates (X, Y, Z) of the part are calculated by substituting them into the similarity principle.

[0045] A second aspect of this application provides a device for identifying and locating visual robot parts, comprising:

[0046] The first processing module establishes the transformation relationship between the world coordinate system and the pixel plane coordinate system of the stereo camera and calibrates the stereo camera.

[0047] The second processing module uses a binocular camera to acquire left and right images of the part, corrects image distortion, and generates a corrected left image and a corrected right image.

[0048] The first extraction module uses the YOLOv4-tiny network to extract the rectangular region of interest (ROI) left image of the part from the corrected left image, and sends it to the PSPNet image segmentation network to further extract the ROI. The extracted ROI and the corrected right image are then grayscaled to obtain the grayscale ROI of the corrected left image and the grayscale image of the corrected right image.

[0049] The first matching module performs coarse template matching between the grayscale region of interest (ROI) of the corrected left image and the grayscale image of the corrected right image. It then extracts the region corresponding to the grayscale ROI of the corrected left image from the grayscale image of the corrected right image and feeds it into the PSPNet image segmentation network to further extract the ROI, thus obtaining the grayscale ROI of the corrected right image. Finally, it performs a binarization and AND operation on the grayscale ROIs of the corrected left and right images to obtain a new left image and a new right image with a new ROI.

[0050] The second matching module performs fine template matching on the new region of interest left image and the new region of interest right image, calculates the sub-pixel disparity between the new region of interest left image and the new region of interest right image, and calculates the object depth information based on the binocular camera ranging principle.

[0051] The calculation module calculates the maximum inscribed circle in the new region of interest (ROI) left image region, with the center of the circle being the pixel center of the part. Then, it calculates the spatial coordinates of the part's location based on the principle of triangle similarity.

[0052] A third aspect of this application provides a computer device including a processor and a memory, the memory being used to store executable instructions for controlling the processor to perform the method described in any of the first aspects of this application.

[0053] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any of the first aspects of this application.

[0054] Compared with the prior art, the beneficial effects of the present invention include at least the following:

[0055] 1. By adding epipolar constraints through camera calibration, the range of the matching search task can be reduced and the matching accuracy improved;

[0056] 2. The object detection algorithm YOLO-tiny lightweight network, which adopts deep learning, automatically adjusts the optimal detection accuracy through prior boxes and training, thereby improving the real-time performance of detection.

[0057] 3. The PSPNet image segmentation algorithm based on deep learning is adopted to enhance feature extraction, improve the capture of global information, and accurately identify pixels belonging to the target object itself, thereby improving detection accuracy.

[0058] 4. Using the region of interest from the target detection results for matching eliminates the influence of irrelevant pixels, reducing the possibility of false matching;

[0059] 5. The structure is simple and reliable, and the error is low. Attached Figure Description

[0060] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0061] Figure 1 This is a schematic diagram of the structure of a visual robot part identification and positioning device provided by the present invention;

[0062] Figure 2 This is a schematic diagram illustrating the positional relationship between a camera and a component provided by the present invention;

[0063] Figure 3 This is a schematic diagram of a fine template matching process for left and right images provided by the present invention;

[0064] Figure 4 This is a schematic diagram of a binocular ranging principle provided by the present invention;

[0065] Figure 5 This is a schematic diagram of an optimal gripping point for a part provided by the present invention;

[0066] Figure 6 This is a flowchart of a visual robot part identification and positioning method provided by the present invention;

[0067] Figure 7 This is a schematic diagram of the structure of a visual robot part identification and positioning device provided by the present invention;

[0068] Figure 8 This is a schematic diagram of a computer device structure for the identification and positioning of visual robot parts provided by the present invention. Detailed Implementation

[0069] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the technical solutions of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the described embodiments are merely some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0070] In this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0071] Technical terms used in this application:

[0072] YOLOv4-tiny network: Its full name is You Only Look Once version 4-tiny. This is a lightweight version of the fourth-generation You Only Look Once (YOLO) object detection algorithm proposed by Joseph Redmon et al. The YOLOv4-tiny network is a deep learning network commonly used for object detection tasks. It is a lightweight version of the YOLOv4 network, with fewer parameters and faster execution speed. The YOLOv4-tiny network has a simple structure, effectively saving computational resources while maintaining detection accuracy.

[0073] PSPNet, short for Pyramid Scene Parsing Network, is a deep learning algorithm in computer vision, commonly used for semantic segmentation tasks. It's a panoramic convolutional neural network model proposed by Kaiming He and his team. PSPNet primarily adds pyramid pooling layers to the basic FCN network to better integrate multi-scale information and generate segmentation results in a multi-scale manner. PSPNet effectively captures local details and global contextual information, contributing to improved segmentation accuracy and performance. PSPNet has been widely applied to semantic segmentation tasks in various scenarios, such as natural image segmentation, medical image segmentation, and remote sensing image segmentation.

[0074] ROI: Short for Region of Interest, in computer vision and image processing, ROI refers to the region of image we are interested in or the part of the image that needs to be analyzed and processed. ROI defines the specific area we want to analyze or process, allowing us to focus our attention on the region of interest rather than the entire image. ROI plays a crucial role in fields such as object detection, image segmentation, and image analysis. ROI can help us save computational resources, avoid unnecessary calculations, and improve the efficiency of algorithms.

[0075] In one possible implementation, the device structure is as follows: Figure 1 As shown. The positions of the camera and parts are as follows. Figure 2 As shown in the diagram, the device includes a palletizing robot, a binocular camera, and a calibration plate. The palletizing robot is fixedly connected to a base, and a suction cup is attached to its end. The binocular camera is mounted on a camera bracket, which is fixedly connected to the base. The working platform is mounted above the base, and the calibration plate is placed flat above the base. The device uses the cooperation of the palletizing robot, the binocular camera, and the suction cup to locate and grasp small parts.

[0076] Example 1

[0077] This application proposes a method for identifying and locating visual robot parts, characterized by the following steps:

[0078] Step S100: Establish the transformation relationship between the world coordinate system and the pixel plane coordinate system of the stereo camera, and calibrate the stereo camera.

[0079] Preferably, the transformation relationship between the world coordinate system and the pixel plane coordinate system of the binocular camera is expressed as follows:

[0080]

[0081] In the formula: S is the scaling factor, (u, v) are the coordinates of point p in the pixel coordinate system, (X... W Y W Let (p) be the coordinates of point p in the image coordinate system, dX and dY represent the physical dimensions of a single pixel in each row and column, respectively, and (u0, v0) be the coordinates of the center of the camera sensor in the pixel coordinate system; f is the camera focal length; R and 0 T Let X be the rotation matrix and identity matrix, respectively, with a shape of (3, 3); t is the translation matrix, with a shape of (3, 1); (X... W Y W Z W () represents the world coordinates of the target object;

[0082] The above formula simplifies to:

[0083]

[0084] In the formula: K is the camera intrinsic parameter matrix, The rotation matrix R and translation matrix t are obtained through camera calibration.

[0085] Preferably, the calibration of the binocular camera includes the following steps:

[0086] The stereo camera was calibrated using Zhang Dingyou's checkerboard calibration method. Fifteen images of a 5mm×5mm checkerboard pattern were taken and imported into the MATLAB toolbox to obtain the stereo camera's distortion parameters, rotation matrix, and translation matrix. The stereo camera's distortion parameters, considering radial and tangential distortion, can be expressed as follows:

[0087]

[0088] In the formula: (x d y d (x, y) represents the normalized image point coordinates after distortion, (x, y) represents the ideal, distortion-free normalized image coordinates, and r represents the distance from the image pixel to the image center point (r0). 2 =x 2 +y 2 k1, k2, and k3 are radial distortion parameters, and p1 and p2 are tangential distortion parameters.

[0089] Step S200: Use a binocular camera to acquire left and right images of the part and correct image distortion to generate a corrected left image and a corrected right image;

[0090] Step S300: Use the YOLOv4-tiny network to extract the rectangular region of interest (ROI) left image of the part from the corrected left image, and send it into the PSPNet image segmentation network to further extract the ROI. Then, perform grayscale processing on the extracted ROI and the corrected right image to obtain the grayscale ROI of the corrected left image and the grayscale image of the corrected right image.

[0091] Preferably, the step of extracting the rectangular region of interest (ROI) left image containing the part from the corrected left image using the YOLOv4-tiny network includes:

[0092] The YOLOv4-tiny network contains two YOLO heads;

[0093] The first YOLO head divides the corrected left image into a 26×26 grid, with pre-defined prior bounding boxes of (23,27), (37,58), and (50,50).

[0094] The second YOLO head divides the corrected left image into a 13×13 grid, with pre-defined prior bounding boxes of (50,50), (81,82), and (135,169).

[0095] The PSPNet network model replaces the traditional residual network with MobileNetV2 as the backbone for feature extraction, employing a pyramid pooling structure to enhance feature extraction. The first layer of the pyramid pooling uses global pooling to generate a single feature output. The remaining three layers divide the input feature map into 2×2, 3×3, and 6×6 sub-regions, pooling each sub-region. Finally, the pooled single feature maps, including location information, are combined. To maintain the weights of the global features, a 1×1 convolutional kernel is used after each pyramid level. When the dimension of a certain level is n, the dimensionality of the semantic features is reduced to 1 / n of the original features. Then, bilinear interpolation is used to directly upsample the low-dimensional feature map, making it the same scale as the original feature map. Finally, the feature maps from different levels are concatenated to form the final pyramid pooled global feature.

[0096] Step S400: Perform coarse template matching between the grayscale region of interest (ROI) of the corrected left image and the grayscale image of the corrected right image. Extract the region corresponding to the grayscale ROI of the corrected left image from the grayscale image of the corrected right image, and feed it into the PSPNet image segmentation network to further extract the ROI, thus obtaining the grayscale ROI of the corrected right image. Perform binarization and AND operation on the grayscale ROI of the corrected left image and the grayscale ROI of the corrected right image to obtain a new left image and a new right image of the ROI.

[0097] Step S500: Perform fine template matching on the new left image of the region of interest and the new right image of the region of interest, calculate the sub-pixel disparity between the new left image of the region of interest and the new right image of the region of interest, and calculate the object depth information based on the binocular camera ranging principle;

[0098] refer to Figure 3 The diagram illustrates the fine template matching (movement step size of 1 pixel) between the new region of interest (left image) and the new region of interest (right image); P l P rLet (x0, y0) and (x0+d, y0) represent the new regions of interest (ROIs) in the left and right images, respectively. (x0, y0) and (x0+d, y0) represent the center points of the left and right regions, respectively, and d represents the disparity. Because the images after camera calibration exhibit epipolar constraints—meaning the target objects on the left and right cameras are on the same horizontal line—matching only requires searching along the same horizontal line. Stereo matching is achieved by sliding the left and right regions, and sub-pixel disparity is calculated. During the matching process, the sum of absolute differences (SAD) is used for similarity evaluation. This involves taking the absolute value of the difference between the corresponding grayscale values ​​of the matching points in the left and right images, summing the results, and then evaluating the similarity between the left and right image patches.

[0099] Preferably, the step of performing fine template matching on the new left image of the region of interest and the new right image of the region of interest includes the following steps:

[0100] Based on the epipolar constraint characteristic of the image after camera calibration, stereo matching of the left and right images of the new region of interest is achieved by sliding the left and right regions. Subpixel disparity between the two images is calculated, and the SAD function is used for similarity evaluation during the matching process. The SAD function is expressed as follows:

[0101] C sAD (p, d) = ∑q∈N p |I l (p)-I r (pd)|

[0102] In the formula: C SAD (p, d) represents the absolute difference between the pixel values ​​of the two images, N p q represents the matching window; q represents a pixel to be matched within the matching window; I l (p) represents the region to be matched in the left image, I r (pd) represents the region to be matched in the right image with a disparity of d.

[0103] As a preferred option, refer to Figure 4 As shown, the calculation of object depth information based on the binocular camera ranging principle includes:

[0104] After obtaining the disparity and similarity curve through precise template matching, the three coordinate points with the highest similarity (x1, y1), (x2, y2), and (x3, y3) are extracted and substituted into the quadratic function y = ax 2 The fitted function is obtained by adding bx and c. The maximum and minimum values ​​of the fitted function are then calculated to obtain the coordinates of the highest point (x). max y max ), x max That is, the optimal parallax, y maxThe similarity is the optimal parallax.

[0105] Based on the sub-pixel disparity values ​​mentioned above and according to the principle of binocular vision, the depth information expression of an object is as follows:

[0106]

[0107] In the formula: z is the object depth, f is the camera focal length, b is the baseline; X L X is the distance from the left camera's imaging point to the left imaging plane. R Here, O represents the distance from the right camera's imaging point to the right imaging plane, P is a point on the object being measured, and O is the distance from the right camera's imaging point to the right imaging plane. L and O R These are the optical centers of the two cameras, Z is the object depth, and P is the optical center of the object. L and P R These are the projections of the same pixel onto the left and right imaging planes, respectively.

[0108] Step S600: Calculate the largest inscribed circle in the new region of interest (ROI) left image region, with the center of the circle being the pixel center of the part. Then, calculate the spatial coordinates of the part's location based on the principle of triangle similarity.

[0109] Preferably, the step of calculating the maximum inscribed circle in the left image region of the new ROI, with the center of the circle being the pixel center of the target object, and then calculating the spatial coordinates of the part's location based on the principle of triangle similarity, includes:

[0110] Contour detection is performed on the segmented object image to obtain the contour matrix; the largest contour Q is then extracted. max The distance between the coordinates of a point within the contour and a point on the contour is calculated to find the center of the largest inscribed circle within the contour. The expression is as follows:

[0111]

[0112] In the formula: R is the radius of the inscribed circle, (x i y i ) is the maximum contour Q max The pixels on (x) j y j () represents the pixels within the outline;

[0113] The center of the largest inscribed circle and the optimal gripping point (u, v) are obtained from the expression, and the spatial coordinates (X, Y, Z) of the part are calculated by substituting them into the similarity principle.

[0114] A second aspect of this application provides a device for identifying and locating visual robot parts, comprising:

[0115] The first processing module establishes the transformation relationship between the world coordinate system and the pixel plane coordinate system of the stereo camera and calibrates the stereo camera.

[0116] The second processing module uses a binocular camera to acquire left and right images of the part, corrects image distortion, and generates a corrected left image and a corrected right image.

[0117] The first extraction module uses the YOLOv4-tiny network to extract the rectangular region of interest (ROI) left image of the part from the corrected left image, and sends it to the PSPNet image segmentation network to further extract the ROI. The extracted ROI and the corrected right image are then grayscaled to obtain the grayscale ROI of the corrected left image and the grayscale image of the corrected right image.

[0118] The first matching module performs coarse template matching between the grayscale region of interest (ROI) of the corrected left image and the grayscale image of the corrected right image. It then extracts the region corresponding to the grayscale ROI of the corrected left image from the grayscale image of the corrected right image and feeds it into the PSPNet image segmentation network to further extract the ROI, thus obtaining the grayscale ROI of the corrected right image. Finally, it performs a binarization and AND operation on the grayscale ROIs of the corrected left and right images to obtain a new left image and a new right image with a new ROI.

[0119] The second matching module performs fine template matching on the new region of interest left image and the new region of interest right image, calculates the sub-pixel disparity between the new region of interest left image and the new region of interest right image, and calculates the object depth information based on the binocular camera ranging principle.

[0120] The calculation module calculates the maximum inscribed circle in the new region of interest (ROI) left image region, with the center of the circle being the pixel center of the part. Then, it calculates the spatial coordinates of the part's location based on the principle of triangle similarity.

[0121] Example 3

[0122] This invention also provides a computer device for the identification and positioning of visual robot parts, the device comprising:

[0123] At least one processor;

[0124] At least one memory for storing at least one program;

[0125] When the at least one program is executed by the at least one processor, the at least one processor implements the method for identifying and locating visual robot parts as described in any of the above embodiments.

[0126] The content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0127] Example 4

[0128] The present invention also provides a computer-readable storage medium storing a visual robot part identification and positioning program, wherein when the visual robot part identification and positioning program is executed by a processor, the visual robot part identification and positioning method as described in any of the above embodiments is implemented.

[0129] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the vision robot part's identification and positioning device, connecting various parts of the entire vision robot part's identification and positioning operable device via various interfaces and lines.

[0130] The memory can be used to store the computer programs and / or modules. The processor, by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory, realizes various functions of the visual robot part recognition and positioning system. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created based on the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0131] Although the description of this application has been quite detailed and particularly focused on several of the described embodiments, it is not intended to limit itself to any of these details or embodiments or any particular embodiment. Rather, it should be considered as effectively covering the intended scope of this application by referring to the appended claims and taking into account the prior art, which provides for a broad possible interpretation of these claims. Furthermore, the foregoing description of this application with respect to embodiments foreseeable by the inventors is intended to provide a useful description, and non-substantial modifications to this application that have not yet been foreseen may still represent equivalent modifications.

[0132] Compared with the prior art, the beneficial effects of the present invention include at least the following:

[0133] 1. By adding epipolar constraints through camera calibration, the range of the matching search task can be reduced and the matching accuracy improved;

[0134] 2. The object detection algorithm YOLO-tiny lightweight network, which adopts deep learning, automatically adjusts the optimal detection accuracy through prior boxes and training, thereby improving the real-time performance of detection.

[0135] 3. The PSPNet image segmentation algorithm based on deep learning is adopted to enhance feature extraction, improve the capture of global information, and accurately identify pixels belonging to the target object itself, thereby improving detection accuracy.

[0136] 4. Using the region of interest from the target detection results for matching eliminates the influence of irrelevant pixels, reducing the possibility of false matching;

[0137] 5. The structure is simple and reliable, and the error is low.

[0138] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A method for identifying and locating visual robot parts, characterized in that, Includes the following steps: Step S100: Establish the transformation relationship between the world coordinate system and the pixel plane coordinate system of the stereo camera, and calibrate the stereo camera; Step S200: Use a binocular camera to acquire left and right images of the part, and correct image distortion to generate a corrected left image and a corrected right image; Step S300: Use the YOLOv4-tiny network to extract the rectangular region of interest (ROI) left image of the part from the corrected left image, and send it into the PSPNet image segmentation network to further extract the ROI. Then, perform grayscale processing on the extracted ROI and the corrected right image to obtain the grayscale ROI of the corrected left image and the grayscale image of the corrected right image. Step S400: Perform coarse template matching between the grayscale region of interest (ROI) of the corrected left image and the grayscale image of the corrected right image. Extract the region corresponding to the grayscale ROI of the corrected left image from the grayscale image of the corrected right image, and feed it into the PSPNet image segmentation network to further extract the ROI, thus obtaining the grayscale ROI of the corrected right image. Perform binarization and AND operation on the grayscale ROI of the corrected left image and the grayscale ROI of the corrected right image to obtain a new left image and a new right image of the ROI. Step S500: Perform fine template matching on the new left image of the region of interest and the new right image of the region of interest, calculate the sub-pixel disparity between the new left image of the region of interest and the new right image of the region of interest, and calculate the object depth information based on the binocular camera ranging principle; The step of performing fine template matching between the new left image of the region of interest and the new right image of the region of interest includes the following steps: Based on the epipolar constraint characteristic of the image after camera calibration, stereo matching of the left and right images of the new region of interest is achieved by sliding the left and right regions. Subpixel disparity between the two images is calculated, and the SAD function is used for similarity evaluation during the matching process. The SAD function is expressed as follows: C SAD (p,d)=∑q∈N p |I l (p)-I r (p-d)| In the formula: C SAD (p, d) represents the absolute difference between the pixel values ​​of the two images, N p q represents the matching window; q represents a pixel to be matched within the matching window; I l (p) represents the region to be matched in the left image, I r (pd) represents the region to be matched in the right image with a disparity of d; The calculation of object depth information based on the binocular camera ranging principle includes: The disparity-similarity curve is obtained through precise template matching. The three coordinate points with the highest similarity (x1, y1), (x2, y2), and (x3, y3) are extracted and substituted into the quadratic function y = ax 2 The fitted function is obtained by adding bx and c. The maximum and minimum values ​​of the fitted function are then calculated to obtain the coordinates of the highest point (x). max ,y max ), x max For optimal parallax, y max The similarity is the optimal parallax. Step S600: Calculate the largest inscribed circle in the new region of interest (ROI) left image region, with the center of the circle being the pixel center of the part. Then, calculate the spatial coordinates of the part's location based on the principle of triangle similarity.

2. The method for identifying and locating visual robot parts according to claim 1, characterized in that, The transformation relationship between the world coordinate system and the pixel plane coordinate system of the stereo camera is established by the following expression: In the formula: S is the scaling factor, (u, v) are the coordinates of point p in the pixel coordinate system, dX and dY represent the physical dimensions of a single pixel in each row and column, respectively, (u0, v0) are the coordinates of the center of the camera sensor in the pixel coordinate system; f is the camera focal length; R and 0 T Let X be the rotation matrix and identity matrix, respectively, with a shape of (3, 3); t is the translation matrix, with a shape of (3, 1); (X... W Y W Z W () represents the world coordinates of the target object; The above formula simplifies to: In the formula: K is the camera intrinsic parameter matrix, The rotation matrix R and translation matrix t are obtained through camera calibration.

3. The method for identifying and locating visual robot parts according to claim 1, characterized in that, The calibration of the binocular camera includes the following steps: The stereo camera was calibrated using Zhang Dingyou's checkerboard calibration method. Fifteen images of a 5mm×5mm checkerboard pattern were taken and imported into the MATLAB toolbox to obtain the stereo camera's distortion parameters, rotation matrix, and translation matrix. The stereo camera's distortion parameters, considering radial and tangential distortion, can be expressed as follows: In the formula: (x d ,y d (x, y) represents the normalized image point coordinates after distortion, (x, y) represents the ideal, distortion-free normalized image coordinates, and r represents the distance from the image pixel to the image center point (r0). 2 =x 2 +y 2 k1, k2, and k3 are radial distortion parameters, and p1 and p2 are tangential distortion parameters.

4. The method for identifying and locating visual robot parts according to claim 1, characterized in that, The step of extracting the rectangular region of interest (ROI) containing the part from the corrected left image using the YOLOv4-tiny network includes: The YOLOv4-tiny network contains two YOLO heads; The first YOLO head divides the corrected left image into a 26×26 grid, with pre-defined prior bounding boxes of (23,27), (37,58), and (50,50). The second YOLO head divides the corrected left image into a 13×13 grid, with pre-defined prior bounding boxes of (50,50), (81,82), and (135,169).

5. The method for identifying and locating visual robot parts according to claim 1, characterized in that, Based on the optimal disparity value mentioned above and according to the principle of binocular vision, the depth information expression of an object is as follows: In the formula: z is the object depth, f is the camera focal length, b is the baseline; X L X is the distance from the left camera's imaging point to the left imaging plane. R These represent the distances from the right camera's imaging point to the right imaging plane.

6. The method for identifying and locating visual robot parts according to claim 1, characterized in that, The process of calculating the maximum inscribed circle in the left image region of the new region of interest, with the center of the circle being the pixel center of the target object, and then calculating the spatial coordinates of the part's location based on the principle of triangle similarity, includes: Contour detection is performed on the segmented object image to obtain the contour matrix; the largest contour Q is then extracted. max The distance between the coordinates of a point within the contour and a point on the contour is calculated to find the center of the largest inscribed circle within the contour. The expression is as follows: In the formula: R is the radius of the inscribed circle, (x i ,y i ) is the maximum contour Q max The pixels on (x) j ,y j () represents the pixels within the outline; The center of the largest inscribed circle and the optimal gripping point (u, v) are obtained from the expression, and the spatial coordinates (X, Y, Z) of the part are calculated by substituting them into the similarity principle.

7. A device for identifying and locating visual robot parts, characterized in that, Include: The first processing module establishes the transformation relationship between the world coordinate system and the pixel plane coordinate system of the stereo camera and calibrates the stereo camera. The second processing module uses a binocular camera to acquire left and right images of the part, corrects image distortion, and generates a corrected left image and a corrected right image. The first extraction module uses the YOLOv4-tiny network to extract the rectangular region of interest (ROI) left image of the part from the corrected left image, and sends it to the PSPNet image segmentation network to further extract the ROI. The extracted ROI and the corrected right image are then grayscaled to obtain the grayscale ROI of the corrected left image and the grayscale image of the corrected right image. The first matching module performs coarse template matching between the grayscale region of interest (ROI) of the corrected left image and the grayscale image of the corrected right image. It then extracts the region corresponding to the grayscale ROI of the corrected left image from the grayscale image of the corrected right image and feeds it into the PSPNet image segmentation network to further extract the ROI, thus obtaining the grayscale ROI of the corrected right image. Finally, it performs a binarization and AND operation on the grayscale ROIs of the corrected left and right images to obtain a new left image and a new right image with a new ROI. The second matching module performs fine template matching on the new region of interest left image and the new region of interest right image, calculates the sub-pixel disparity between the new region of interest left image and the new region of interest right image, and calculates the object depth information based on the binocular camera ranging principle. The step of performing fine template matching between the new left image of the region of interest and the new right image of the region of interest includes: Based on the epipolar constraint characteristic of the image after camera calibration, stereo matching of the left and right images of the new region of interest is achieved by sliding the left and right regions. Subpixel disparity between the two images is calculated, and the SAD function is used for similarity evaluation during the matching process. The SAD function is expressed as follows: C sAD (p,d)=∑q∈N p |I l (p)-I r (p-d)| In the formula: C SAD (p, d) represents the absolute difference between the pixel values ​​of the two images, N p q represents the matching window; q represents a pixel to be matched within the matching window; I l (p) represents the region to be matched in the left image, I r (pd) represents the region to be matched in the right image with a disparity of d; The calculation of object depth information based on the binocular camera ranging principle includes: The disparity-similarity curve is obtained through precise template matching. The three coordinate points with the highest similarity (x1, y1), (x2, y2), and (x3, y3) are extracted and substituted into the quadratic function y = ax 2 The fitted function is obtained by adding bx and c. The maximum and minimum values ​​of the fitted function are then calculated to obtain the coordinates of the highest point (x). max ,y max ), x max For optimal parallax, y max The similarity is the optimal parallax. The calculation module calculates the maximum inscribed circle in the new region of interest (ROI) left image region, with the center of the circle being the pixel center of the part. Then, based on the principle of triangle similarity, it calculates the spatial coordinates of the part's location.

8. A computer device, characterized in that... It includes a processor and a memory, the memory being used to store executable instructions for controlling the processor to perform the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that... It stores a computer program thereon, which, when executed by a processor, implements the method as described in any one of claims 1 to 6.

Images