A binocular vision apple recognition and accurate positioning method

Abstract

The application discloses a binocular vision apple recognition and accurate positioning method, and comprises the following steps: calibrating a binocular camera composed of two monocular cameras; establishing a model for apple recognition; optimizing parameters in an SGBM stereo vision matching algorithm; and calculating an actual centroid position of an apple; the method can accurately obtain three-dimensional coordinates of the apple in space under the condition that the apple is not blocked, and can still accurately obtain the three-dimensional coordinates of the apple in space under the condition that the apple is blocked by leaves and branches, thereby improving the efficiency of the algorithm while ensuring the positioning accuracy of the apple.

Description

Technical Field

[0001] This invention relates to a method for identifying and locating ripe red apples under binocular vision, and more particularly to a method for accurately locating ripe red apples when they are obscured by leaves or other objects. Background Technology

[0002] In apple recognition, many methods rely on the apple's color and shape features. However, these algorithms, based on texture and color characteristics, have significant limitations, lacking universal models and exhibiting low robustness. Deep learning, on the other hand, has developed rapidly, offering higher accuracy and advantages compared to traditional methods. After obtaining the rectangular bounding box of the apple's location using deep learning, accurately determining the apple's 3D coordinates is crucial for driving a robotic arm to harvest apples. This is a critical technology affecting the accuracy and stability of apple harvesting. Due to factors such as lighting, invalid matching points and random error points inevitably occur in stereo vision matching. Furthermore, occlusion by leaves and other elements means that simply using the location information of the apple's centroid on the detected image plane instead of the apple's location will result in the acquisition of the 3D location information of the leaves, leading to apple location failure. Therefore, simply using partial points on the rectangular bounding box of the apple's image coordinate system as the apple's location information is highly unreliable. Summary of the Invention

[0003] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a method for identifying and accurately locating ripe red apples, so that the apples still have high positioning accuracy even under the influence of foliage occlusion and invalid matching points and random error points in stereo vision matching.

[0004] The present invention provides a method for binocular vision apple recognition and precise positioning, comprising the following steps:

[0005] A method for binocular vision apple recognition and precise positioning, characterized by the following steps:

[0006] Step 1: Calibrate the stereo camera consisting of the left and right cameras to obtain the intrinsic parameter matrices and distortion vectors of the left and right cameras, as well as the translation, rotation, and pose transformation matrices between the left and right cameras.

[0007] The first step is to connect the binocular camera to the computer, and then use the left and right cameras to take pictures of the pre-prepared checkerboard pattern calibration board. During the shooting process, rotate the calibration board so that it is in different positions in the field of view of the binocular camera and the corner points of the checkerboard on the calibration board can be fully displayed. This will result in multiple calibration board images taken by the left camera and multiple calibration board images taken by the right camera. At the same time, the calibration board images taken by the left camera and the calibration board images taken by the right camera form a pair of calibration board images.

[0008] The second step involves importing the collected images of multiple calibration plates into Python. Using the Zhang Zhengyou calibration method in Python's OpenCV library, single-target calibration is performed on the left and right cameras, and binocular calibration is performed on the stereo cameras. This yields the intrinsic parameter matrices and distortion vectors of the left and right cameras, as well as the rotation matrix and translation vector between them.

[0009] Step 2: Build a model for apple recognition:

[0010] Step 3: Optimize the parameters in the SGBM stereo vision matching algorithm. When the calibration board is clear in the disparity map of each pair of calibration board images in Step 1, record the corresponding parameters of the SGBM algorithm at this time as optimized parameters when converting the distortion-corrected 2D apple image of the left camera into a 3D apple image during the depth measurement of the apple.

[0011] Step 4: Prepare an apple without any leaves obscuring it as the first apple. Calculate the actual centroid position of the first apple. The specific calculation process is as follows:

[0012] Step 1: Place the first apple in the field of view of the left camera, so that the z-coordinate of the estimated centroid of the first apple in the left camera coordinate system is close to the z-coordinate of the center of the calibration board in the left camera coordinate system when taking the image of the calibration board in Step 1.

[0013] The second step is to take a picture of the first apple using both the left and right cameras, and then perform distortion correction on the pictures of the first apple taken by the left and right cameras.

[0014] The third step is to import the OpenVINO model inference core library IECore into Python to perform model inference on the apple recognition model obtained in step two, obtain the network structure and weight parameters of the deep learning neural network, and then input the first apple image taken by the distortion-corrected left camera into the YOLOv4 object detection algorithm to identify the apple in the image and generate a rectangular bounding box tangent to the outer contour of the first apple.

[0015] Step 4: Using the SGBM stereo vision matching algorithm with optimized parameters from Step 3, perform stereo vision matching on the first apple image taken by the left camera after distortion correction and the first apple image taken by the right camera after distortion correction, expanding the 2D image of the first apple into a 3D depth image of the first apple:

[0016] Step 5: Cropping the first apple image taken by the left camera after distortion correction in step 2 along the rectangular bounding box to obtain the color image within the rectangular bounding box of the first apple image, and extracting the red area in the color image to obtain the binarized image after the red area is extracted.

[0017] Step 6: Use the randint function in Python's random library to randomly coarsely sample the 3D points within the rectangular bounding box of the first apple's 3D depth image to obtain a 3D coarse sample point set. The 3D coarse sample point set is then filtered for fitting the apple's spherical surface.

[0018] Step 7: Perform fine screening on the coarse sample point set obtained in step 6. The specific steps are as follows:

[0019] Step 701: Calculate the mean and variance of the z-axis coordinates of the coarse sample points in the coarse sample point set:

[0020] Step 702: For the coarse sample point set {P} i}, each coarse sample point P in i = 1…n i (x i ,y i ,z i Set the filter range [z] i -σ z ,z i +σ z Determine whether the z-coordinate value of each coarse sample point in the coarse sample point set falls on coarse sample point P. i (x i ,y i ,z i Within the selection interval of ), let the total number of sample points falling within the interval be score[i], where score[i] is the score of the i-th point; where σ z —The variance of the z-axis coordinate values ​​of the coarse sample points in the coarse sample point set; z i —The z-coordinate value of the i-th coarse sample point in the coarse sample point set;

[0021] Step 703: Compare the scores of all coarse sample points. The set of points selected from the coarse sample points with the highest scores is the new set of sample points {P} obtained through fine screening. j}(j=1…N), denoted as I k N is I k The number of new sample points;

[0022] Step 8: Use the RANSAC algorithm to analyze I. k Further screening yielded the optimal interior points for apple spherical fitting;

[0023] Step 9: Randomly select 8 samples from the optimal interior point set, and use the least squares matrix method to fit the first apple sphere to obtain the coordinates of the centroid of the first apple in the left camera coordinate system and the radius of the first apple sphere model.

[0024] Compared with other existing technologies, this method has the following advantages:

[0025] 1. Wide applicability: This method can accurately obtain the three-dimensional coordinates of an apple in space when the apple is unobstructed. In more general cases, such as when there are obstructions such as leaves and branches, it can still accurately obtain the three-dimensional coordinates of the apple. This makes it more universal and practical in complex orchard environments. Moreover, as long as the position of the rectangular bounding box of the apple in the image can be determined by apple recognition, this method can be used to accurately locate the apple, without being limited to using the YOLOv4 object detection algorithm as the method for apple recognition.

[0026] 2. High efficiency and accuracy: This method does not extract all 3D points on the apple to draw a point cloud for apple fitting. Instead, it selects a small number of 3D points on the apple and, through algorithm screening and optimization, eliminates erroneous points that failed to match in stereo vision matching and erroneous points caused by foreground occlusion and background interference. At the same time, in apple segmentation, it uses the rectangular box obtained by apple recognition for pseudo-segmentation instead of completely segmenting the apple, which improves the efficiency of the algorithm while ensuring the accuracy of apple positioning. Attached Figure Description

[0027] Figure 1 The image of the first apple without leaves obscuring it is identified in the distortion-corrected left camera image.

[0028] Figure 2 The image shows a second apple obscured by leaves in the left camera image after distortion correction.

[0029] Figure 3 The binary image obtained after extracting the red area from the rectangular bounding box of the first cropped apple.

[0030] Figure 4 The binary image obtained after extracting the red area from the rectangular bounding box of the cropped second apple.

[0031] Figure 5 A flowchart of the process for identifying and accurately locating Apple devices.

[0032] Figure 6This is the flowchart for step seven of step four above.

[0033] Figure 7 This is the flowchart for step eight of step four above. Detailed Implementation

[0034] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0035] The present invention provides a method for binocular vision apple recognition and precise positioning, comprising the following steps:

[0036] Step 1: Calibrate the stereo camera consisting of the left and right cameras to obtain the intrinsic parameter matrices and distortion vectors of the left and right cameras, as well as the translation, rotation, and pose transformation matrices between the left and right cameras.

[0037] The first step is to connect the binocular camera to the computer, and then use the left and right cameras to take pictures of the pre-prepared checkerboard pattern calibration board. During the shooting process, rotate the calibration board so that it is in different positions in the field of view of the binocular camera and the corner points of the checkerboard on the calibration board can be fully displayed. This will result in multiple calibration board images taken by the left camera and multiple calibration board images taken by the right camera. At the same time, the calibration board images taken by the left camera and the calibration board images taken by the right camera form a pair of calibration board images. For example, 15 pairs of calibration board images can be taken.

[0038] The hardware devices used in the specific implementation of this invention include an industrial binocular camera, a computer, and two apples for identification and positioning.

[0039] The pre-prepared chessboard calibration plate can be manufactured as follows:

[0040] The checkerboard pattern paper is flatly adhered to the acrylic substrate to obtain a checkerboard pattern calibration plate for monocular and binocular camera calibration:

[0041] The size of the checkerboard grid and the size of each square on the calibration board can be selected according to the application scenario. The selection principle is to first determine the working distance from the optical center of the left camera to the set point on the apple stem in the camera coordinate system of the left camera, and then calculate the X-axis and Y-axis lengths of the left camera's field of view at that distance. The size of the calibration board can be selected as 1 / 4 to 1 / 3 of the X-axis and Y-axis lengths. For example, if the checkerboard grid size is selected as 7 rows x 10 columns, and the size of each square is 25mm x 25mm, it can be printed on A4 paper with a printing resolution of 300 DPI.

[0042] The second step involves importing the collected calibration board images into Python. Using the OpenCV library in Python (OpenCV is a cross-platform computer vision and machine learning software library released under the Apache 2.0 license (open source), providing an interface for the Python language and containing many general algorithms for image processing and computer vision), the Zhang Zhengyou calibration method (see Zhang ZA Flexible New Technique for Camera Calibration[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2000,22(11):1330-1334. (A flexible new technique for camera calibration)) is employed to perform single-target calibration on the left and right cameras and dual-target calibration on the stereo camera, obtaining the intrinsic parameter matrices and distortion vectors of the left and right cameras, as well as the rotation matrix and translation vector between the left and right cameras:

[0043] The calibration of both monocular and binocular cameras is implemented using the OpenCV library in Python (see pages 291-309 of "OpenCV4 Explained Based on Python" published by Posts & Telecom Press in September 2021). OpenCV provides a rich set of APIs for checkerboard corner detection and monocular / binocular camera calibration, making camera calibration very convenient. If the following equipment is selected: the industrial binocular camera is from Blue Sky Technology, model LT-USB1080P, with a lens model LT-C146MM-3MP and a focal length of 6mm; the computer used has an AMD Ryzen 7 5800H CPU and an NVIDIA GeForce GTX 1650 graphics card; and the apple radius is approximately 40mm. The software mainly includes an apple recognition program, an image processing program, and an SGBM algorithm stereo vision matching program.

[0044] The intrinsic parameter matrices and distortion vectors of the two cameras, as well as the rotation matrix and translation vector between the two cameras, obtained after single-target and dual-target positioning are as follows:

[0045] Intrinsic parameter matrix of the left camera:

[0046] Intrinsic parameter matrix of the right camera:

[0047] The distortion vector of the left camera: dist1 = [-0.52 0.5974 0 0 0]

[0048] The distortion vector of the right camera: dist2 = [-0.5178 0.4581 0 0 0]

[0049] Rotation matrix between the two cameras:

[0050] Translation vector between the two cameras: T = [-129.2953 -0.6675 1.0062]

[0051] Step 2: Build a model for apple recognition:

[0052] The apple recognition model can be any existing publicly available model. This model was created by AlexeyAB and is publicly available in his GitHub repository, specifically at [URL missing]. https: / / github.com / AlexeyAB / darknet / releases .

[0053] Step 3: Optimize the parameters in the SGBM stereo vision matching algorithm (for details of the SGBM algorithm, please refer to the method described in Hirschmuller H. Stereo processing by semiglobal matching and mutual information[J].IEEE Transactions on pattern analysis and machine intelligence,2007,30(2):328-341. (Stereo processing based on semiglobal matching and mutual information)). When the calibration board in the disparity map of each pair of calibration board images in Step 1 is clear, record the parameters corresponding to the SGBM algorithm at this time as optimized parameters for converting the distortion-corrected two-dimensional apple image of the left camera into a three-dimensional apple image during the depth measurement of the apple.

[0054] As one embodiment of the present invention, the specific process of optimizing the parameters in the SGBM stereo vision matching algorithm may include:

[0055] Step 301: Read the images of each pair of calibration plates taken in Step 1;

[0056] Step 302: Perform distortion correction on each pair of calibration board images (for details on the distortion correction method, please refer to page 298 of "OpenCV4 Explained Based on Python" published by Posts & Telecom Press in September 2021) to obtain the distortion-free calibration board images after correction. The distortion correction of the calibration board images is implemented through OpenCV. Using the intrinsic parameter matrices and distortion vectors of the left and right cameras obtained in Step 1, the mapping matrix required for image correction is calculated through the API provided by OpenCV, and then the distortion in the original image is removed using the mapping matrix.

[0057] Step 303: Initialize the SGBM algorithm, perform disparity and depth calculations on the calibration board images taken by the left camera and the right camera in each pair of calibration board images after distortion correction in Step 302, and obtain multiple disparity maps;

[0058] Step 304: Continuously change the parameters of the SGBM algorithm and observe the effect of each disparity map. When all disparity maps are clear, record the parameters of the SGBM algorithm at this time for subsequent depth measurement.

[0059] Step 4, the specific process is as follows: Figure 5 As shown, an apple without leaves obscuring it is used as the first apple. The actual centroid position of the first apple is calculated. The specific calculation process is as follows:

[0060] Step 1: Place the first apple in the field of view of the left camera, so that the z-coordinate of the estimated centroid of the first apple in the left camera coordinate system is close to the z-coordinate of the center of the calibration board in the left camera coordinate system when taking the image of the calibration board in Step 1. Usually, the difference in z-coordinate is within 200mm. For example, place the first apple in the left camera coordinate system according to the estimated centroid position. The coordinates of the estimated centroid position in the left camera coordinate system are (0mm, 0mm, 1000mm). The estimated position of the centroid of the apple can be half of the length, width and height of the apple.

[0061] The second step is to take a picture of the first apple using both the left and right cameras, and then perform distortion correction on the pictures of the first apple taken by the left and right cameras.

[0062] Step 3: Import the OpenVINO model inference core library IECore into Python to perform model inference on the apple recognition model obtained in Step 2, obtain the network structure and weight parameters of the deep learning neural network, and then input the first apple image taken by the distortion-corrected left camera into the YOLOv4 object detection algorithm (for details on the YOLOv4 algorithm, please refer to Bochkovskiy A, Wang CY, Liao HY M. YOLOv4: Optimal speed and accuracy of object detection[J]. arXiv preprint arXiv:2004.10934,2020. (YoLOv4: Optimal speed and accuracy of object detection)). This algorithm identifies the apple in the image and generates a rectangular bounding box tangent to the outer contour of the first apple. The identified first apple image is shown below. Figure 1 As shown;

[0063] Step 4: Using the SGBM stereo vision matching algorithm with optimized parameters from Step 3, perform stereo vision matching on the first apple image taken by the left camera after distortion correction and the first apple image taken by the right camera after distortion correction, expanding the 2D image of the first apple into a 3D depth image of the first apple:

[0064] In this step, the SGBM algorithm is used to perform stereo matching on the corrected left and right camera images. As a semi-global matching algorithm, the SGBM algorithm has a significantly better stereo matching effect than the local matching algorithm BM algorithm, and can achieve high accuracy, meeting the requirements for precise apple positioning. Since the SGBM algorithm does not obtain the optimal result, the parameters need to be adjusted to obtain a sufficiently good algorithm effect. The parameters used here are the optimized parameters obtained in step three. The disparity is calculated on the first apple images taken by the left and right cameras after distortion correction in step four, and a three-dimensional depth map can be obtained. The z-axis in the three-dimensional depth map is the depth information.

[0065] Step 5: Cropping the first apple image taken by the left camera after distortion correction in step 2 along the rectangular bounding box to obtain the color image within the rectangular bounding box of the first apple image, and extracting the red area in the color image to obtain the binarized image after the red area is extracted.

[0066] As one embodiment of the present invention, the specific process of obtaining the binarized image is as follows:

[0067] Step 501: Crop the color image within the rectangular bounding box and convert the color image to the HSV color space;

[0068] Step 502: In the HSV color space described in step 501, the H component (hue) is less affected by light. Therefore, under the complex light conditions in the orchard, the method of setting a threshold for the H component to segment the red area in the color image is more reliable. The H component corresponding to the red area has two ranges, 0-10 and 156-180. After setting the threshold, two binary images are obtained by filtering.

[0069] Step 503: Overlay the two binary images obtained in Step 502 to obtain a new binary image. The white (pixel value of 255) in the new binary image corresponds to the red in the cropped color image in Step 501, and the black (pixel value of 0) in the binary image corresponds to the other colors in the cropped color image in Step 501. Figure 1 The binary image after extracting the red region from the rectangular bounding box of the first apple identified in the image is shown below. Figure 3 As shown.

[0070] Step 6: Use the randint function in Python's random library to perform random coarse sampling on the 3D points within the rectangular bounding box of the first apple 3D depth image to obtain a 3D coarse sample point set. The 3D coarse sample point set is then filtered for fitting the apple sphere.

[0071] The specific steps of the random coarse sampling can be as follows:

[0072] Step 601: From Figure 3 The x and y coordinates of the sampling points in the pixel coordinate system are obtained by randomly sampling from the binary image after the red region is extracted.

[0073] Step 602: Determine the pixel value of the binary image corresponding to the coordinates of the sampling point. If it is 255, the selected area is red. Continue to determine whether the z-axis coordinate value of the pixel in the 3D depth image obtained by stereo vision matching is within a set range. The selection of this set range is determined by the working distance of the Apple recognition. For example, if the working distance of the Apple recognition is 500-2000mm, then determine whether the z-axis coordinate value of the pixel in the 3D depth image obtained by stereo vision matching is between 500-2000mm. If the condition is met, save the 3D coordinate point as the 3D coarse sample point P of the Apple spherical fitting. i Otherwise, discard the sample point and proceed to the next step;

[0074] Step 603: Return to step 601 to obtain the two-dimensional coordinates in the pixel coordinate system, and then continue to execute steps 602 to 603 to obtain the three-dimensional coarse sample point P. i (x i ,y i ,z i Sampling continues until a sufficient number of coarse sample points are collected, at which point the sampling is stopped, resulting in a three-dimensional coarse sample point set {P}. i}(i=1…n), considering the accuracy and efficiency of the algorithm, let's take an example: In this example, after sampling 100 coarse sample points, we stop sampling and obtain the coarse sample point set {P}. i}(i=1…100); The sample points obtained by coarse sampling can effectively avoid sampling the 3D points corresponding to the non-red areas within the rectangular bounding box and the erroneous 3D points (the z value of the 3D coordinates is infinite) in the holes caused by mismatch in the SGBM algorithm.

[0075] Step 7: Perform fine screening on the coarse sample point set obtained in Step 6:

[0076] The fine-tuning of the coarse sample point set involves using algorithms to filter out 3D points in the red areas on the background at the four corners of the cropped rectangular image. These red areas, after color segmentation, will become white areas in the binary image. Therefore, the coarse sample point set may contain such erroneous sample points. These erroneous sample points exhibit significant outlier characteristics in their z-axis coordinates compared to their x and y-axis coordinates, unlike the correct sample points. Therefore, filtering based on the z-axis coordinates can effectively remove these erroneous sample points. The specific steps are as follows:

[0077] Step 701: Calculate the mean and variance of the z-axis coordinates of the coarse sample points in the coarse sample point set:

[0078]

[0079] μ z —The mean of the z-axis coordinates of the coarse sample points in the coarse sample point set

[0080] σ z — Variance of the z-axis coordinates of coarse sample points in the coarse sample set

[0081] z i —The z-coordinate of the i-th coarse sample point in the coarse sample point set

[0082] n — the number of coarse sample points in the coarse sample point set

[0083] Step 702: For the coarse sample point set {P} i}, i = 1…n (e.g., {P i Each coarse sample point P in}(i=1…100) i (x i ,y i ,z i Set the filter range [z] i -σ z ,z i +σ z Determine whether the z-coordinate value of each coarse sample point in the coarse sample point set falls on coarse sample point P. i (x i ,y i ,z i Within the filtering interval of ), the total number of sample points falling within the interval is denoted as score[i], and score[i] is the score of the i-th point;

[0084] Step 703: Compare the scores of all coarse sample points. The set of points selected from the coarse sample points with the highest scores is the new set of sample points {P} obtained through fine screening. j}(j=1…N), denoted as I k N is I kThe number of new sample points;

[0085] Step 8: Use the RANSAC algorithm (see Choi S, Kim T, Yu W. Performance evaluation of RANSAC family[J]. Journal of Computer Vision, 1997, 24(3):271-300.) to evaluate I. k Further screening yielded the optimal interior points for apple spherical fitting;

[0086] As one embodiment of the present invention, the specific steps for obtaining the optimal interior point using the RANSAC algorithm are as follows:

[0087] Step 801: Obtain I from step 7 k Four sample points are randomly selected. Since the model to be solved has four unknown parameters, the minimum number of sample points required to initialize the model is four. The matrix form of the least squares method is used to fit the sphere of the first apple to obtain the centroid coordinates (a,b,c) and radius r of the first apple.

[0088] Step 802: Traverse I k For all sample points in the sphere, calculate the nearest distance from each sample point to the sphere to obtain d. j (j=1…N);

[0089] Step 803: Make a judgment; if d j If the value is ≤10 (mm), then the corresponding sample point P will be... j It is stored as an interior point in the sample consensus set S;

[0090] Step 804: The number of interior points in the sample consensus set S obtained in step 803 is l; if l > 0.9N, then the sample consensus set S is obtained, in which the interior points {T} are... i} (i = 1…l) is the best-fit interior point set; otherwise, return to step 801 and randomly select 4 sample points to continue iterating until a consistent sample set S is obtained.

[0091] Step 9: Considering accuracy and efficiency, randomly select 8 samples from the optimal set of interior points, and use the least squares matrix method to fit the spherical surface of the first apple to obtain the coordinates of the centroid of the first apple in the left camera coordinate system and the radius of the spherical model of the first apple.

[0092] The specific principle of using the least squares matrix form for spherical fitting is as follows:

[0093] Assuming the first apple model is a spatial sphere, its equation is:

[0094] (xa) 2 +(yb) 2 +(zc) 2 =r 2

[0095] (x,y,z)——Coordinates of a point on the sphere;

[0096] (a,b,c) – Coordinates of the centroid of the first apple in the left camera coordinate system

[0097] r — the radius of the first apple spherical model;

[0098] Expanding on it, we get:

[0099] -2xa-2yb-2zc+(a 2 +b 2 +c 2 -r 2 )=-x 2 -y 2 -z 2

[0100] Converting to matrix form yields:

[0101]

[0102] A1=-2x, A2=-2y, A3=-2z, A4=1, d=a 2 +b 2 +c 2 -r 2 e = -x 2 -y 2 -z 2 ;

[0103] When there are 8 samples (x) i ,y i ,z i When performing spherical fitting on (i = 1…8), it is transformed into the form Aβ = e, where:

[0104] Further multiply A by both left and right sides simultaneously. T have to:

[0105] A T Aβ=A T e

[0106] Finally, multiply both sides by A on the left. T A) -1 have to:

[0107] β=(A T A) -1 *(AT e)

[0108] Based on the example in step 603: After step seven, the mean and variance of the sample points of the first apple are calculated to be μ. z =964.0626mmσ z =10.8990mm, and the number of refined sample points after screening is 68. The optimal interior point set of the first apple obtained through step eight has 66 sample points. Step nine is used to perform apple spherical fitting. When the estimated centroid of the first apple is placed at (0mm 0mm 1000mm) in the left camera coordinate system, the calculated centroid coordinates of the first apple are (2.3428mm-5.7377mm 995.1694mm), and the radius is r=44.4725mm. The positioning accuracy can meet the requirements of apple picking and can achieve precise positioning of the apple centroid.

[0109] Verification Example:

[0110] Replace the first apple in step one with an apple partially obscured by leaves. Label the apple partially obscured by leaves as the second apple. Repeat steps two through nine of step four. The image of the second apple identified in step three is shown below. Figure 2 As shown, the binary image obtained in step 5 is as follows. Figure 4 As shown, the mean and variance of the second apple sample points in step seven are μ. z =977.8539mmσ z =19.0748mm, and the final centroid coordinates of the second apple are (6.48mm-1.14mm997.9085mm), with a radius of r=44.3907mm. It can also achieve accurate positioning even when there are leaves blocking the view, which can meet the accuracy requirements for apple picking.

Claims

1. A method for binocular vision apple recognition and precise positioning, characterized in that... Includes the following steps: Step 1: Calibrate the stereo camera consisting of the left and right cameras to obtain the intrinsic parameter matrices and distortion vectors of the left and right cameras, as well as the translation, rotation, and pose transformation matrices between the left and right cameras. The first step is to connect the binocular camera to the computer, and then use the left and right cameras to take pictures of the pre-prepared checkerboard pattern calibration board. During the shooting process, rotate the calibration board so that it is in different positions in the field of view of the binocular camera and the corner points of the checkerboard on the calibration board can be fully displayed. This will result in multiple calibration board images taken by the left camera and multiple calibration board images taken by the right camera. At the same time, the calibration board images taken by the left camera and the calibration board images taken by the right camera form a pair of calibration board images. The second step involves importing the collected images of multiple calibration plates into Python. Using the Zhang Zhengyou calibration method in Python's OpenCV library, single-target calibration is performed on the left and right cameras, and binocular calibration is performed on the stereo cameras. This yields the intrinsic parameter matrices and distortion vectors of the left and right cameras, as well as the rotation matrix and translation vector between them. Step 2: Build a model for apple recognition: Step 3: Optimize the parameters in the SGBM stereo vision matching algorithm. When the calibration board is clear in the disparity map of each pair of calibration board images in Step 1, record the corresponding parameters of the SGBM algorithm at this time as optimized parameters when converting the distortion-corrected 2D apple image of the left camera into a 3D apple image during the depth measurement of the apple. Step 4: Prepare an apple without any leaves obscuring it as the first apple. Calculate the actual centroid position of the first apple. The specific calculation process is as follows: Step 1: Place the first apple in the field of view of the left camera, so that the z-coordinate of the estimated centroid of the first apple in the left camera coordinate system is close to the z-coordinate of the center of the calibration board in the left camera coordinate system when taking the image of the calibration board in Step 1. The z-coordinates are close to each other, meaning that the difference in z-coordinates is within 200mm. The second step is to take a picture of the first apple using both the left and right cameras, and then perform distortion correction on the pictures of the first apple taken by the left and right cameras. The third step is to import the OpenVINO model inference core library IECore into Python to perform model inference on the apple recognition model obtained in step two, obtain the network structure and weight parameters of the deep learning neural network, and then input the first apple image taken by the distortion-corrected left camera into the YOLOv4 object detection algorithm to identify the apple in the image and generate a rectangular bounding box tangent to the outer contour of the first apple. Step 4: Using the SGBM stereo vision matching algorithm with optimized parameters from Step 3, perform stereo vision matching on the first apple image taken by the left camera after distortion correction and the first apple image taken by the right camera after distortion correction, expanding the 2D image of the first apple into a 3D depth image of the first apple: Step 5: Cropping the first apple image taken by the left camera after distortion correction in step 2 along the rectangular bounding box to obtain the color image within the rectangular bounding box of the first apple image, and extracting the red area in the color image to obtain the binarized image after the red area is extracted. Step 6: Use the randint function in Python's random library to randomly coarsely sample the 3D points within the rectangular bounding box of the first apple's 3D depth image to obtain a 3D coarse sample point set. The 3D coarse sample point set is then filtered for fitting the apple's spherical surface. Step 7: Perform fine screening on the coarse sample point set obtained in step 6. The specific steps are as follows: Step 701: Calculate the mean and variance of the z-axis coordinates of the coarse sample points in the coarse sample point set: Step 702: For the coarse sample point set Each coarse sample point Set the filter range Determine whether the z-coordinate value of each coarse sample point in the coarse sample point set falls within the coarse sample point set. Within the selection interval, the total number of sample points falling within the interval is denoted as . , That is, the first The score is 1 point; among which... —The variance of the z-axis coordinate values ​​of the coarse sample points in the coarse sample point set; —The z-coordinate of the i-th coarse sample point in the coarse sample point set. —The number of coarse sample points in the coarse sample point set; Step 703: Compare the scores of all coarse sample points. The set of points selected from the coarse sample points with the highest scores is the new set of sample points obtained through fine screening. , recorded as , for The number of new sample points; Step 8: Use the RANSAC algorithm to... Further screening yielded the optimal interior points for apple spherical fitting; Step 9: Randomly select 8 samples from the optimal interior point set, and use the least squares matrix method to fit the first apple sphere to obtain the coordinates of the centroid of the first apple in the left camera coordinate system and the radius of the first apple sphere model.

2. The method for binocular vision apple recognition and precise positioning according to claim 1, characterized in that, The specific process of optimizing the parameters in the SGBM stereo vision matching algorithm includes: Step 301: Read the images of each pair of calibration plates taken in Step 1; Step 302: Using the intrinsic parameter matrices and distortion vectors of the left and right cameras obtained in Step 1, calculate the mapping matrix required for image correction using the API provided by OpenCV, and then use the mapping matrix to remove the distortion in each pair of calibration board images to obtain the corrected distortion-free calibration board images. Step 303: Initialize the SGBM algorithm, perform disparity and depth calculations on the calibration board images taken by the left camera and the right camera in each pair of calibration board images after distortion correction in Step 302, and obtain multiple disparity maps; Step 304: Continuously change the parameters of the SGBM algorithm and observe the effect of each disparity map. When all disparity maps are clear, record the parameters of the SGBM algorithm at this time.

3. The method for binocular vision apple recognition and precise positioning according to claim 1 or 2, characterized in that, The specific process of obtaining the binarized image is as follows: Step 501: Crop the color image within the rectangular bounding box and convert the color image to the HSV color space; Step 502: In the HSV color space described in step 501, the threshold of the H component is set to segment the red region in the color image. The H component corresponding to the red region has two ranges, 0-10 and 156-180. Then, two binary images are obtained by filtering. Step 503: Overlay the two binary images obtained in step 502 to obtain a new binary image. The white in the new binary image corresponds to the red in the cropped color image in step 501, and the black in the binary image corresponds to other colors other than red in the cropped color image in step 501.

4. The method for binocular vision apple recognition and precise positioning according to claim 3, characterized in that: The specific steps of the random coarse sampling are as follows: Step 601: Randomly sample the x and y coordinates of the sampling points in the pixel coordinate system from the binary image obtained from the red region; Step 602: Determine the pixel value of the binary image corresponding to the coordinates of the sampling point. If it is 255, the selected area is red. Continue to determine whether the z-axis coordinate value of the sampling point on the 3D depth image obtained by stereo vision matching is within the set interval. If it is within the set interval, save the 3D coordinate point as a 3D coarse sample point for apple spherical fitting. Otherwise, discard the sample point and proceed to the next step; Step 603: Return to step 601 to obtain the two-dimensional coordinates in the pixel coordinate system, and then continue to execute steps 602 to 603 to obtain the three-dimensional coarse sample points. Sampling stops after collecting a sufficient number of 100 coarse sample points, resulting in a three-dimensional coarse sample point set. .

5. The method for binocular vision apple recognition and precise positioning according to claim 4, characterized in that: The specific steps for obtaining the optimal interior points using the RANSAC algorithm are as follows: Step 801: Obtained from step 4, step 7 Four sample points were randomly selected, and the spherical surface of the first apple was fitted using the least squares matrix method to obtain the centroid coordinates of the first apple. and the radius of the first apple ; Step 802: Traversal For all sample points, calculate the nearest distance from each sample point to the sphere, and obtain... ; Step 803: Make a judgment, if Then the corresponding sample points As an interior point, it is stored in the sample consensus set. middle; Step 804: The sample consensus set obtained in step 803 The number of interior points in the array is ;like Then a consistent set of samples is obtained. interior points If the set of interior points is the best-fit, then return to step 801 and randomly select 4 more sample points to continue the iteration until a consistent set of samples is obtained. ; Step 9: Randomly select 8 samples from the optimal interior point set and use the least squares matrix form to fit the first apple sphere.

Images