Tomato picking method and system based on binocular vision and machine readable storage medium

Abstract

The embodiment of the invention provides a tomato picking method and system based on binocular vision and a machine readable storage medium, and belongs to the technical field of binocular vision detection. The picking method comprises the following steps: acquiring binocular images of tomatoes, and identifying the tomatoes based on a target detection model to obtain position coordinates of a detection frame, tomato shielding conditions and maturity; when the tomato is a mature tomato and is not shielded, obtaining a disparity map of the binocular image based on a binocular detection model; determining the two-dimensional coordinates of the tomato according to the position coordinates of the detection frame, and calculating the three-dimensional coordinates of the tomato in combination with the parallax value; and picking the tomatoes by using a mechanical arm according to the three-dimensional coordinates of the tomatoes. According to the tomato picking method provided by the invention, the growth condition and the shielding condition of the tomatoes can be comprehensively considered, and accurate positioning and accurate picking of the tomatoes are realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of binocular vision detection, in particular to a tomato picking method and system based on binocular vision and a machine readable storage medium. BACKGROUND

[0002] At present, some researches have used basic target detection models such as YOLOv5 and YOLOv7 to detect tomatoes and obtain the pixel coordinates of the tomatoes, and then used binocular ranging algorithms to obtain the depth of the tomatoes, so as to obtain the spatial position information of the tomatoes by combining the pixel coordinates and the depth of the tomatoes. Based on this position information, a mechanical arm is guided to pick the tomatoes.

[0003] Although the target detection model based on YOLO and the binocular matching model can realize tomato detection and positioning, they are not good at detecting difficult targets. For example, when the tomatoes are detected while being blocked by branches or other tomatoes, the detection accuracy will decrease. Moreover, guiding the mechanical arm according to this detection result will also cause inaccurate picking of the mechanical arm, and even cause damage to the tomatoes.

[0004] Therefore, the existing tomato picking method can only be used for picking tomatoes in a scene where the tomatoes are not blocked and grow not densely, and it is difficult to meet the demand in a limited complex scene. However, in the actual growth scene of tomatoes, such as a greenhouse, it is inevitable that the tomatoes will grow in overlapping. When picking in such a complex situation, if only the position of the tomatoes is considered and the blocking of the tomatoes is ignored, it is easy to cause damage to the fruits during picking, affecting the overall yield and quality. SUMMARY

[0005] The purpose of the embodiments of the present application is to provide a tomato picking method and system based on binocular vision and a machine readable storage medium, which can solve the problem that the existing tomato picking method has poor picking effect when the tomatoes are blocked.

[0006] To achieve the above object, the embodiment of the present application provides a tomato picking method based on binocular vision, which comprises the following steps: acquiring a binocular image of a tomato, wherein the binocular image comprises a left image and a right image; identifying the tomato in the left image based on a target detection model, and identifying the identified tomato through a detection box to obtain the position coordinates of the detection box, whether the tomato is blocked, and a maturity category; when the tomato is a mature tomato and is not blocked, the following operations are performed: acquiring a disparity map of the binocular image based on a binocular detection model, wherein the disparity map comprises disparity values of all pixel points in the left image and the right image, and the disparity values are obtained by calculating the horizontal coordinate difference of the same pixel point in the left image and the right image; determining the two-dimensional coordinates of the center point of the tomato according to the position coordinates of the detection box, and combining the disparity value corresponding to the center point to calculate the three-dimensional coordinates of the center point of the tomato; and picking the tomato by using a mechanical arm according to the three-dimensional coordinates of the center point of the tomato.

[0007] Optionally, when the tomato is a mature tomato and is blocked, the category of the blocking object is determined, specifically comprising: calculating the pixel value of the green region in the detection box, if the pixel value of the green region meets the pixel threshold interval, it is determined that the blocking object is a leaf, and if it does not meet the pixel threshold interval, it is determined that the blocking object is another tomato.

[0008] Optionally, when the blocking object is a leaf, the area size of the green region in the detection box is calculated, and when the area size is less than 30% of the detection box, the tomato is picked by using the mechanical arm according to the three-dimensional coordinates of the center point of the tomato; when the blocking object is another tomato, a tomato is reselected, and the maturity category, whether the tomato is blocked, and the blocking condition of the reselected tomato are determined, and when the reselected tomato is a mature tomato and is not blocked, or the blocking object is a leaf and the area size of the leaf is less than 30% of the detection box, the reselected tomato is picked by using the mechanical arm according to the three-dimensional coordinates of the center point of the tomato.

[0009] Optionally, when the tomato is picked by using the mechanical arm, if the tomato is picked, the three-dimensional coordinates of the picked tomato are recorded, and if the tomato is not picked after picking three times, a picking failure information is returned.

[0010] Optionally, the target detection model is an improved YOLOv8n model, which comprises an improved Head module, including two added output channels, for outputting a value indicating whether there is a block; when the improved YOLOv8n model is trained, the blocking loss function is a BCE loss function, and the expression is as follows:

[0011] wherein,N represents the number of samples in each training batch, represents the occlusion label, 1 for occlusion, 0 for no occlusion, is the occlusion probability predicted by the improved YOLOv8n model.

[0012] Optionally, the binocular detection model is an improved IGEV model, which specifically comprises: a feature extraction module, configured to perform feature extraction on left and right images of the binocular image to obtain left and right feature maps of 1 / 4 resolution size; a cost volume module, comprising a group correlation unit and a channel shuffling unit, the group correlation unit combines the left and right feature maps, and the combined feature maps are rearranged in channels based on the channel shuffling unit; a 3D convolution module, comprising a 3D U-Net and a 3D Conv, configured to perform convolution operation on the channel rearranged feature maps to obtain an initial disparity of 1 / 4 resolution size; and a recurrent convolution module, configured to perform convolution calculation on the initial disparity to output a disparity map.

[0013] Optionally, according to the position coordinates of the detection frame, the two-dimensional coordinates of the center point of the tomato are determined, and the calculation formula is:

[0014]

[0015] wherein the position coordinates of the detection frame are , and the two-dimensional coordinates of the center point of the tomato are .

[0016] Optionally, the three-dimensional coordinates of the center point of the tomato are calculated by combining the disparity value corresponding to the center point, comprising: according to the disparity value corresponding to the center point, the distance of the center point of the tomato from the binocular camera is calculated, and the calculation formula is:

[0017] wherein, is the focal length of the binocular camera, is the baseline of the binocular camera, is the disparity value; According to the two-dimensional coordinates of the center point of the tomato and the distance of the center point of the tomato from the binocular camera, the three-dimensional coordinates of the center point of the tomato are calculated, and the calculation formula is:

[0018]

[0019] wherein, is the intrinsic parameter of the binocular camera, x and y are two-dimensional horizontal and vertical coordinates of the center point of the tomato respectively, and z is the distance between the center point of the tomato and the binocular camera.

[0020] In a second aspect, the present application provides a tomato picking system based on binocular vision, comprising a memory and a processor, the memory has a program stored thereon, and the processor is configured to execute the program, wherein the program, when executed, is configured to perform the tomato picking method based on binocular vision.

[0021] In a third aspect, the present application provides a machine readable storage medium, the machine readable storage medium has instructions stored thereon, the instructions are used to make the machine execute the tomato picking method based on binocular vision.

[0022] Through the above technical solution, the target detection model is used to identify the tomato in the left image of the binocular image, and the position coordinates of the detection frame, the occlusion condition of the tomato and the maturity category are obtained. The two-dimensional coordinate information of the center point of the tomato can be calculated through the position coordinates of the detection frame, and the target detection model used in the present application can be used to detect whether the tomato is occluded, so as to solve the problem that the occlusion condition is ignored in the existing tomato picking method.

[0023] When it is judged that the tomato is a mature tomato and there is no occlusion according to the identified maturity category and occlusion condition of the tomato, the parallax map of the tomato is calculated through the binocular detection model according to the binocular image of the tomato, and the three-dimensional coordinates of the tomato are calculated by combining the two-dimensional coordinate information of the center point of the tomato and the parallax value corresponding to the center point. Finally, the three-dimensional coordinates are used to guide the mechanical arm to pick the tomato. The picking method provided by the present application integrates the target detection technology and the binocular vision technology, can comprehensively consider the growth condition and the occlusion condition of the tomato, and is used to optimize the picking strategy, so as to realize the accurate positioning and accurate picking of the tomato.

[0024] Other features and advantages of the present application will be described in detail in the following specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0025] The accompanying drawings are included to provide a further understanding of the present application and constitute a part of the specification, and are used together with the following specific embodiments to explain the present application, but do not constitute a limitation on the present application. In the drawings: Figure 1 is a flowchart of a tomato picking method based on binocular vision provided by the present application; Figure 2 is a schematic diagram of a binocular camera used in the present application; Figure 3is a structural schematic diagram of a YOLOv8n model provided by an embodiment of the present application; Figure 4 is a structural schematic diagram of a Head module provided by an embodiment of the present application; Figure 5 is a structural schematic diagram of an improved Head module provided by an embodiment of the present application; Figure 6 is a tomato labeled image schematic diagram provided by an embodiment of the present application; Figure 7 is a tomato labeled data schematic diagram provided by an embodiment of the present application; Figure 8 is a structural schematic diagram of an improved IGEV model provided by an embodiment of the present application; Figure 9 is a structural schematic diagram of a feature extraction module provided by an embodiment of the present application; Figure 10 is a structural schematic diagram of a channel shuffling unit provided by an embodiment of the present application; Figure 11 is a parallax schematic diagram provided by an embodiment of the present application; Figure 12 is a RealSense D435i schematic diagram provided by an embodiment of the present application; Figure 13 (a) part in is an RGB image of a tomato collected by a RealSense D435i provided by an embodiment of the present application, and (b) part is a depth map of the tomato provided by an embodiment of the present application; Figure 14 is a tomato planning picking flow schematic diagram provided by an embodiment of the present application; Figure 15 is a flow framework diagram of a tomato positioning and picking method provided by an embodiment of the present application; Figure 16 is another tomato picking method flow schematic diagram based on binocular vision provided by an embodiment of the present application. DETAILED DESCRIPTION

[0026] The specific embodiments of the embodiments of the present application are described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to illustrate and explain the embodiments of the present application, and are not used to limit the embodiments of the present application.

[0027] It should be noted that the acquisition, transmission, storage, use, processing, etc. of data in the technical solutions of the present application comply with the relevant provisions of laws and regulations. In the embodiments of the present application, some existing industry solutions such as software, components, models, etc. may be mentioned, which should be considered as exemplary, and the purpose is only to illustrate the feasibility of the implementation of the technical solutions of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0028] Tomato is a high-nutrition agricultural product. The traditional tomato planting and picking relies on manual labor, which is slow and inefficient. In recent years, the combination of computer vision and automation has provided technical support for the development of agricultural intelligence. The YOLO (You Only Look Once) series model has been applied to tomato detection, realizing real-time high-precision tomato detection.

[0029] When using the YOLO model to detect tomatoes, the tomato fruit features are extracted based on the backbone network, and then the image coordinates (x, y) of the tomato fruit are inferred through the detection head. Existing researches use YOLOv5, YOLOv7, etc. as the basis model, and improve the tomato detection rate through a series of improvement methods, such as replacing the backbone network with MobileNetv3, and adding attention mechanisms such as SA and CBAM to the network. The experimental results show that the detection accuracy reaches 92.1%, and the speed reaches 26.5FPS (Frame Per Second).

[0030] Binocular distance measurement is a distance measurement technology based on computer vision, which uses two cameras to simulate human eyes, determines the depth of the target through the triangular relationship, and then uses camera parameters to determine the position of the target. In addition, there are also depth cameras developed specifically for distance measurement, such as the RealSense series and the Kinect series. This camera integrates two ordinary cameras, an infrared camera and a receiver. The binocular camera is responsible for shooting normal binocular images, the infrared camera is responsible for projecting infrared waves, and the receiver receives the reflected infrared waves. By measuring the time difference between the emission and reception of infrared waves, the distance can be converted. There are also solutions that use two industrial cameras to build a system instead of a depth camera. The entire technical process includes target positioning, binocular distance measurement, and target positioning, etc. The key technologies are binocular distance measurement and positioning.

[0031] The binocular distance measurement algorithm is divided into traditional feature-based matching algorithm and deep learning-based distance measurement algorithm. The traditional feature-based binocular distance measurement algorithm has low accuracy and real-time performance. The deep learning-based binocular distance measurement algorithm has become the mainstream choice. Through the binocular images collected by the binocular camera, the depth of the tomato can be obtained by the deep learning binocular distance measurement algorithm. Existing researches use RAFT-Stereo and other models to infer the depth of the tomato, and use YOLOv5 to detect the tomato, with a positioning error of about 5mm. Depth cameras can also be used for tomato distance measurement. Depth cameras can directly return the depth of the entire scene through distance sensors, and the position of the tomato can be detected through the YOLO algorithm, so as to locate the tomato. Nowadays, using depth cameras for tomato positioning has gradually become a common method.

[0032] In agricultural production, tomato picking is a complex and delicate process, especially when tomatoes grow in clusters. The mutual occlusion between fruits significantly increases the difficulty of picking. Existing picking methods often fail to accurately determine the state of occluded tomatoes, leading to fruit damage during picking and affecting overall yield and quality. Therefore, the present application provides a tomato picking method based on binocular vision, which comprehensively considers the growth and occlusion of tomatoes to achieve accurate positioning and precise picking. The flowchart of the picking method is shown in Figure 1

[0033] Step S1: Obtain binocular images of tomatoes; Step S2: Based on the target detection model, identify the tomatoes in the left image, and identify the identified tomatoes through the detection box to obtain the position coordinates of the detection box, whether the tomato is occluded, and the maturity category; Step S3: Determine the maturity category and occlusion of the tomato. When the tomato is mature and there is no occlusion, proceed to step S4; Step S4: Based on the binocular detection model, obtain the disparity map of the binocular image; Step S5: Determine the two-dimensional coordinates of the center point of the tomato according to the position coordinates of the detection box, and calculate the three-dimensional coordinates of the center point of the tomato in combination with the disparity value corresponding to the center point; Step S6: Use a mechanical arm to pick the tomato according to the three-dimensional coordinates of the center point of the tomato.

[0034] Specifically, in step S1, the binocular camera is used to obtain the binocular images of the tomatoes. The collected binocular images include left and right images. Figure 2 is a schematic diagram of the binocular camera used in the present application.

[0035] ​In step S2, the target detection model used by the embodiment of the present application is an improved YOLOv8n model. Using the original YOLOv8n model can obtain the maturity and position of the tomatoes, but cannot obtain whether the tomatoes are blocked. Therefore, the embodiment of the present application improves the detection head structure of the YOLOv8n model, that is, the Head module, including two additional output channels for outputting the values of whether there is blocking. Figure 3 is a structural diagram of a YOLOv8n model provided by the embodiment of the present application. Figure 4 is a structural diagram of a Head module provided by the embodiment of the present application. Figure 5 is a structural diagram of an improved Head module provided by the embodiment of the present application.

[0036] As shown in Figure 5 , the Head module in the embodiment outputs a prediction result with a width: height: number of channels of 20:20:30. The number of channels is 30, indicating that there are 30 numbers in the prediction result output by the Head, which represent the detection results of the tomatoes, as shown by the yellow part in Figure 5 .

[0037] Among them, the 1 number corresponding to conf represents the confidence, that is, the probability that the detected object is a tomato, and the value range is 0-1. The 4 numbers corresponding to Bbox represent the detected position, represented by xywh, x and y are the center coordinates of the detection box, and w and h are the width and height of the detection box.

[0038] The 3 numbers corresponding to Cls represent the maturity prediction result of the detected tomato. In the embodiment, the maturity categories set include: immature, semi-mature and mature, which correspond to 3 channels respectively. Each channel uses a decimal number of 0-1 to represent the probability of being this category, and the values of the three channels add up to 1. The category represented by the largest value in the three channels is the predicted maturity category. For example, the values of the three channels of cls are (0.1, 0.1, 0.8) according to the position (immature, semi-mature, mature). The largest value 0.8 corresponds to the category “mature”, and the output maturity category is “mature”.

[0039] The 2 numbers corresponding to Obs represent the prediction result of whether there is blocking, which is the improved part of the Head module in the embodiment of the present application. Two channels are added to output the prediction result value of “blocked / unblocked”. The values output by the two channels are both decimal numbers of 0-1, and the sum is 1. The category represented by the largest value in the two channels is the prediction result of whether there is blocking. For example, the values of the two channels corresponding to obs are (0.2, 0.8) according to the position (unblocked, blocked), and 0.8 corresponds to the category “blocked”. Therefore, the prediction result of blocking is output as “blocked”.

[0040] The YOLOv8n model uses three different bounding boxes to better adapt to the detection of objects of different shapes. Since the number of channels corresponding to Conf, bbox, cls, and obs in each bounding box is 10 (1+4+3+2), the total number of channels for the three bounding boxes is 30.

[0041] The training phase of the improved YOLOv8n model involves collecting image data of greenhouse-grown tomatoes from a picking garden and performing data augmentation on the image data. This is achieved by using random rotation, random flipping, random scaling, random cropping, and Gaussian noise to transform the features of the original images, thereby expanding the dataset.

[0042] The acquired tomato images were divided into two categories: "occluded" and "unoccluded," and labeled using the `labelimg` tag. The labeled images are shown below. Figure 6 As shown, the labeled data is as follows Figure 7 As shown. Figure 7 In the diagram, the values ​​within the red boxes represent maturity levels: 0 for immature, 1 for semi-mature, and 2 for mature. The values ​​within the yellow boxes indicate the location of the detection box. In addition to the original YOLOv8n labels, blue boxes are added to represent occlusion: 0 for unoccluded and 1 for occluded.

[0043] The improved YOLOv8n model was trained using a labeled tomato image dataset. The original YOLOv8n model's loss function consists of three parts: confidence loss, bounding box loss, and classification loss. The confidence loss uses the BCE loss, calculated as follows:

[0044] in, N This indicates the number of samples in each training batch. This represents the sample label: 1 for a tomato and 0 for no tomato. This represents the confidence level of the model's prediction.

[0045] The bounding box loss uses the CIoU (Complete IoU) loss, which is calculated using the following formula:

[0046] in, IoU The intersection-union ratio (IUU) of the predicted bounding box and the ground truth bounding box. Indicates the center point location of the prediction box. This indicates the center point of the actual bounding box. Indicates calculation and Euler distance, Diagonal distance of the minimum bounding rectangle containing the predicted box and the real box. For the aspect ratio consistency measure, is a weight coefficient, and the calculation formula is:

[0047]

[0048] wherein, and represent the width and height of the real bounding box, w and h represent the width and height of the predicted box.

[0049] The calculation formula of the classification loss cls is:

[0050] wherein, N represents the number of samples in each batch, C represents the number of categories, which is 3 (immature, semi-mature, and mature) in this embodiment, represents whether the i-th sample belongs to the category i , and C represents the probability that the i-th sample predicted by the model belongs to the category . i C

[0051] After improving the detection head, the occlusion part is trained using the BCE loss function, and the calculation formula of the occlusion loss function is:

[0052] wherein, N represents the number of samples in each batch, represents the occlusion label, which is 1 if occluded and 0 if not occluded, is the occlusion probability predicted by the model.

[0053] The sum of the confidence loss , the detection box loss , the classification loss , and the occlusion loss is used as the final loss function of the improved YOLOv8n model, and the calculation formula is:

[0054] ​​The trained model can identify the position of the tomato in the image and distinguish between the occluded and non-occluded categories of tomatoes. The maturity of the tomato is determined using a color threshold. When the tomato detection box indicates the position of the tomato, it can be limited to a small range in the image, and the color threshold can be used to determine the maturity of the tomato at this time. After the overlapping tomatoes are identified, the fully exposed tomato is identified as non-occluded, and the subsequent tomato is identified as occluded, thus reflecting the occlusion relationship of the tomatoes. The input and output of the improved YOLOv8n model are shown in Table 1.

[0055] Table 1 Input and output of target detection model

[0056] When a tomato is detected, the improved YOLOv8n model outputs its maturity (unripe, semi-ripe, ripe), the detection box position x1, y1 indicating the top-left corner coordinates of the box, x2, y2 indicating the bottom-right corner coordinates of the box, and whether the tomato is occluded.

[0057] In step S3, the maturity and occlusion of the tomato identified by the target detection model are determined. When the maturity of the identified tomato is shown as ripe and the occlusion is non-occlusion, it is determined that the tomato can be picked, and step S4 is entered.

[0058] The binocular detection model used in step S4 is an improved IGEV model. The existing IGEV model is a binocular algorithm model that has achieved leading results in the field of binoculars. However, due to the complex growth conditions of tomatoes in the greenhouse, the existing IGEV model is not suitable for adaptation, so the existing IGEV model needs to be improved to enhance the model's feature extraction ability for tomato images in complex situations. Figure 8 is a structural diagram of the improved IGEV model provided by the embodiment of the present application.

[0059] As Figure 8 shown, the improved IGEV model includes a feature extraction module, a cost volume module, a 3D convolution module, and a recurrent convolution module. Specifically, in the embodiment of the present application, the feature extraction module (Feature Extractor, Figure 8 indicated by the green box) and the cost volume module (Cost Volume, Figure 8 indicated by the blue box) are improved. For the feature extraction module, the original MobileNetv2 network structure is replaced with a network structure based on YOLOv8n, and the pre-trained partial convolution block parameters of YOLOv8n are used for transfer learning. The structural diagram of the feature extraction module is shown in Figure 9 , wherein Conv( Figure 9The middle yellow part) and SPPF ( Figure 9 The middle blue part) is a transpose convolution, Conv( Figure 9 The middle blue part) is a transpose convolution, Conv( Figure 9 The middle green part) is a convolution block, which is initialized by a random initialization method. C represents a channel concatenation operation.

[0060] For the cost volume module, only the group-wise correlation (Gwc) unit is included in the original IGEV model. In this embodiment, a channel shuffle unit is added to combine the group-wise correlation unit and the channel shuffle unit to construct the cost volume module. By adding the channel shuffle unit, the channels are rearranged to solve the problem of isolation between convolution feature channels and increase the feature correlation and representation. The structure diagram of the channel shuffle unit (channel shuffle) is shown in FIG. 6. After the group-wise correlation unit (Gwc), a channel shuffle unit is added to shuffle the output of the Gwc by channel and perform channel reordering to increase the correlation between different channels. Finally, the Concatnate operation is used to combine the feature maps by channel. Figure 10

[0061] The tomato binocular images collected by the binocular camera are input into the improved IGEV model. The left and right images in the binocular images are feature extracted by the feature extraction module to obtain left and right feature maps with a size of 1 / 4 resolution. Then, the left and right feature maps are combined by the cost volume module, and the combined feature maps are rearranged by channel based on the channel shuffle unit.

[0062] The feature maps processed by the cost volume module are input into the 3D convolution module. The 3D U-Net and 3D Conv in the 3D convolution module maintain the original 3D convolution operation of the IGEV model unchanged, and output an initial disparity (InitialDisparity) with a size of 1 / 4 resolution. The recurrent convolution module also maintains the original IGEV model unchanged. The initial disparity is input into the recurrent convolution module (ConvGRU) to output a final disparity map (Final Disparity). The input and output of the improved IGEV model are shown in Table 2.

[0063]

[0064] Specifically, the disparity value is the horizontal coordinate difference value of the same pixel point between the right image and the left image based on the left image in the binocular image. The disparity values of all pixel points in the left image and the right image form a disparity map. Figure 11 ​is a parallax schematic diagram, wherein the position of the tomato is shown as point P, the left image and the right image captured by the binocular camera image the tomato respectively, and the imaging points are P1 and P2 respectively, and the parallax is the difference between the x-axis coordinates of P2 and P1, i.e. disp=x2-x1. The improved IGEV model will predict the parallax values of all pixels in the left image, thereby forming a parallax map.

[0065] In step S5, the improved YOLOv8n tomato detection result and the binocular model are used together for tomato positioning. Since YOLOv8 will output the detection frame (x1, y1, x2, y2) of the tomato, representing the left upper corner coordinates and the right lower corner coordinates of the detection frame, the two-dimensional coordinates of the center point of the tomato can be obtained using x2-x1 and y2-y1 :

[0066]

[0067] The obtained center point two-dimensional coordinates The parallax value is obtained from the corresponding position in the parallax map, i.e. the parallax value of the center point of the tomato. According to the parallax value corresponding to the center point, the distance of the center point of the tomato from the binocular camera is calculated z , and the calculation formula is:

[0068] wherein, is the focal length of the binocular camera, is the baseline of the binocular camera, and are obtained by binocular calibration, and the binocular calibration can use the existing matlab program for calibration, is the parallax value.

[0069] According to the two-dimensional coordinates of the center point of the tomato and the distance of the center point of the tomato from the binocular camera, the three-dimensional coordinates of the center point of the tomato are calculated, and the calculation formula is:

[0070]

[0071] wherein, is the intrinsic parameter of the binocular camera, which can also be obtained by matlab calibration, are the two-dimensional horizontal coordinates and vertical coordinates of the center point of the tomato, z is the distance of the center point of the tomato from the binocular camera. The obtained is the three-dimensional coordinates of the center point of the tomato, thereby realizing the fusion positioning of the improved YOLOv8n model and the improved IGEV model.

[0072] In practical applications, the ranging results of RealSense D435i and the improved IGEV model can be combined. Figure 12 is a schematic diagram of RealSense D435i. RealSense D435i is a distance sensor based on light propagation algorithm, which can measure distance, but its disadvantage is that it is affected by the light environment. Figure 13 Part (a) of is an RGB image of a tomato taken by RealSense D435i, and part (b) is a depth map of the RGB image of the tomato after depth visualization. As can be seen, there is a black invalid area at the edge of the object in the depth map, so this invalid area can be supplemented using a binocular vision algorithm model to realize fusion ranging. That is, the distance measured by RealSense D435i is directly provided to the binocular detection model. Thus, the problem of RealSense D435i being sensitive to the environment is overcome, and the problem of binocular vision algorithm being inefficient in reasoning about textureless areas is also remedied.

[0073] In the above tomato picking method, the operation process when the tomato is a mature tomato and there is no occlusion is given, that is, when the tomato is mature and there is no occlusion, it can be directly picked up. In some embodiments, when the detection result of the tomato according to the target detection model is that the tomato is a mature tomato and there is an occlusion, the category of the occlusion is determined, specifically including: The pixel value of the green region in the detection frame is calculated, and if the pixel value of the green region satisfies the pixel threshold interval, it is determined that the occlusion is a leaf, and if it does not satisfy, it is determined that the occlusion is another tomato.

[0074] Wherein, the pixel threshold interval is set to [35, 50, 50]~[85, 255, 255], and if the pixel value of the green region in the detection frame satisfies this threshold interval, it is considered to be in the leaf category.

[0075] When the occlusion is a leaf, the size of the region occupied by the green region in the detection frame is calculated, and when the size of the region is less than 30% of the detection frame, the robot arm is used to pick the tomato according to the three-dimensional coordinates of the center point of the tomato. When the occlusion is another tomato, the tomato is reselected, and the maturity category, whether it is occluded, and the occlusion situation of the reselected tomato are determined, and when the reselected tomato is a mature tomato and there is no occlusion, or the occlusion is a leaf and the size of the leaf is less than 30% of the detection frame, the robot arm is used to pick the reselected tomato according to the three-dimensional coordinates of the center point of the tomato.

[0076] Wherein, the size of the tomato can be calculated according to the size of the detection frame output by the target detection model, and the size of the leaf region can be obtained by calculating the size of the green region in the detection frame.

[0077] When the mechanical arm is used to pick the tomato, if the tomato is picked, the three-dimensional coordinates of the picked tomato are recorded, and if the tomato is not obtained after picking 3 times, the picking failure information is returned. It should be noted that when the maturity category of the tomato identified by the target detection model is unripe or semi-ripe, the tomato target is replaced for calculation, and the tomato is not picked.

[0078] In the embodiment of the application, the RRT* algorithm is used to plan the motion path of the mechanical arm. RRT* (Rapidly-exploring Random Trees*) is a sampling-based path planning algorithm, which belongs to a probability complete algorithm (i.e. the probability of finding a feasible path tends to 1 as the number of samples increases). It introduces the "cost optimization" and "rewiring" mechanism on the basis of the basic RRT algorithm, which can gradually approach the optimal path and is suitable for path planning problems in high-dimensional space and complex obstacle environment (such as robot navigation, mechanical arm motion planning, etc.).

[0079] The algorithm randomly samples a space point in space If there is an obstacle collision, the space point is discarded , and a new random sampling is performed. When the sampled space position has no collision, a space point nearby is detected with r as the radius x ', if the cost from the initial position to x ' is less than , then x ' is used instead of , which is saved as a legal path until the target picking point is reached. Since the occlusion information can provide information about whether there is an obstacle in front of the target tomato, and the tomato can be relocated when the target tomato is occluded, until there is no occlusion in front of the tomato, i.e. no obstacle, therefore, the occlusion information is helpful for the RRT* algorithm to select the tomato that is easy to pick, so as to improve the efficiency of the RRT* algorithm. Figure 14 A flowchart for planning the picking of tomatoes.

[0080] Figure 15 is a flowchart of a tomato positioning and picking method provided by the embodiment of the application. As Figure 15As shown, in the data preparation stage, this method acquires tomato images using a camera and labels them for both tomatoes and occlusion, providing a dataset for the tomato detection model. A stereo camera is used to capture tomato images. In the model preparation stage, an improved YOLOv8n model is used to detect tomatoes and occlusion, while an improved IGEV model is used to infer distances to tomatoes, achieving tomato distance measurement. The detection results, occlusion relationships, and distance measurement results are combined to enable the two models to jointly locate tomatoes. During harvesting, harvesting is based on maturity and occlusion levels. If there is no occlusion, harvesting is performed directly; if occlusion exists, harvesting is planned in layers.

[0081] Figure 16 This is a schematic diagram of another tomato harvesting method based on binocular vision provided in this embodiment of the invention. This method designs the harvesting process according to the layering and occlusion relationships of the tomatoes. After harvesting the outer layer of tomatoes, the occluded tomatoes are exposed, and then the tomatoes that are occluded behind are harvested. This method can improve the success rate of harvesting and reduce harvesting damage. Figure 16 As shown, firstly, the YOLO object detection model and the IGEV binocular ranging model are used to locate the tomato. Then, the ripeness of the tomato is determined. If it is ripe, it is determined whether it is occluded; if the tomato is not ripe, another tomato target is selected for relocation. If the tomato is ripe and there is no obstruction, then attempt to pick the tomato. If there is obstruction, determine the type of obstruction: leaves or other tomatoes. If it is leaves, calculate the obstruction coverage. If it is other tomatoes, perform tomato relocation and select other tomato targets. If the leaf coverage is less than 30%, you can try to harvest it. If it is not less than 30%, you should estimate the shape of the shaded object and fit the shape of the shaded tomato. When a tomato is successfully picked, its location and distribution information are provided in real time. If no tomato is picked, the picking is considered unsuccessful, and three more picking attempts are made. If no tomato is picked after three attempts, the picking is considered a failure, the failed tomato is marked, and its location information is recorded. Once the location and distribution information of the successfully harvested tomatoes are obtained, it is determined whether the harvesting is over. If not, the tomatoes are located again, and the above harvesting method is repeated.

[0082] Secondly, the present invention provides a tomato harvesting system based on binocular vision, the system comprising a memory and a processor, wherein the memory stores a program and the processor is used to run the program, wherein the program, when run, is used to execute: the tomato harvesting method based on binocular vision described in any one of the above.

[0083] In a third aspect, the present application provides a machine readable storage medium having instructions stored thereon for causing a machine to perform any of the above tomato picking methods based on binocular vision.

[0084] Those skilled in the art will understand that embodiments of the present application can be provided as methods, systems, or computer program products. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable program code.

[0085] The present application is described in reference to the flowchart and / or block diagrams of the methods, apparatus (systems) and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagrams, and combinations of flows and / or blocks in the flowchart and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart and / or block diagrams block or blocks. Figure 1 The flowchart and / or block diagrams can also be implemented by one or more of the following: Figure 1 An apparatus with the specified functions of one or more of the flow or blocks.

[0086] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart and / or block diagrams block or blocks. Figure 1 The flowchart and / or block diagrams can also be implemented by one or more of the following: Figure 1 An apparatus with the specified functions of one or more of the flow or blocks.

[0087] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart and / or block diagrams block or blocks. Figure 1 The flowchart and / or block diagrams can also be implemented by one or more of the following: Figure 1 An apparatus with the specified functions of one or more of the flow or blocks.

[0088] In one typical configuration, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0089] Memory can include non-persistent memory, Random Access Memory (RAM), and / or non-volatile memory, such as Read Only Memory (ROM) or flash memory, in a computer readable medium. Memory is an example of computer readable media.

[0090] Computer readable media includes permanent and non-permanent, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital versatile disc (DVD), or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer readable media does not include transitory media, such as modulated data signals and carrier waves.

[0091] It should also be noted that the terms "comprising", "containing", or any other variant thereof, are intended to encompass a non-exclusive inclusion, such that a process, method, article or apparatus that comprises a list of elements does not include only those elements in the list, but can also include other elements not expressly listed or inherent to such process, method, article or apparatus. Without further limitation, an element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0092] The above merely provides an example of the present application and is not intended to limit the present application. The present application can have various modifications and changes for those skilled in the art. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the scope of claims of the present application.

Claims

1. A method for picking tomatoes based on binocular vision, characterized in that, The picking method comprises: acquiring a binocular image of the tomato, the binocular image comprising a left image and a right image; based on a target detection model, identifying the tomato in the left image, and identifying the identified tomato through a detection box to obtain the position coordinates of the detection box, whether the tomato is occluded, and the maturity category; when the tomato is a mature tomato and there is no occlusion, the following operations are performed: based on a binocular detection model, acquiring a disparity map of the binocular image, the disparity map comprising disparity values of all pixel points in the left image and the right image, the disparity values being obtained by calculating the horizontal coordinate difference of the same pixel point in the left image and the right image; determining the two-dimensional coordinates of the center point of the tomato according to the position coordinates of the detection box, and combining the disparity value corresponding to the center point to calculate the three-dimensional coordinates of the center point of the tomato; and using a mechanical arm to pick the tomato according to the three-dimensional coordinates of the center point of the tomato.

2. The picking method according to claim 1, characterized in that, when the tomato is a mature tomato and there is occlusion, the category of the occlusion is determined, specifically comprising: calculating the pixel value of the green region in the detection box, if the pixel value of the green region satisfies the pixel threshold interval, it is judged that the occlusion is a leaf, if it does not satisfy, it is judged that the occlusion is other tomatoes.

3. The picking method according to claim 2, characterized in that, when the occlusion is a leaf, the area size of the green region in the detection box is calculated, when the area size is less than 30% of the detection box, the mechanical arm is used to pick the tomato according to the three-dimensional coordinates of the center point of the tomato; when the occlusion is other tomatoes, the maturity category of the reselected tomato, whether it is occluded, and the occlusion are determined, when the reselected tomato is a mature tomato and there is no occlusion, or the occlusion is a leaf and the area size of the leaf is less than 30% of the detection box, the mechanical arm is used to pick the reselected tomato according to the three-dimensional coordinates of the center point of the tomato.

4. The picking method according to claim 3, characterized in that, when the mechanical arm is used to pick the tomato, if the tomato is picked, the three-dimensional coordinates of the picked tomato are recorded, if the tomato is not obtained after picking for 3 times, a picking failure information is returned.

5. The picking method according to claim 1, characterized in that, The target detection model is an improved YOLOv8n model, comprising: an improved Head module, including two added output channels for outputting the value of whether there is occlusion; when training the improved YOLOv8n model, the occlusion loss function is a BCE loss function, and its expression is: wherein, N represents the number of samples of each training batch, represents an occlusion label, 1 for occlusion and 0 for no occlusion, is the occlusion probability predicted by the improved YOLOv8n model.

6. The picking method according to claim 1, characterized in that, The binocular detection model is an improved IGEV model, specifically comprising: a feature extraction module for feature extraction of the left and right images of the binocular image to obtain left and right feature maps of 1 / 4 resolution size; a cost volume module comprising a group correlation unit and a channel shuffle unit, the group correlation unit combines the left and right feature maps, and the combined feature maps are rearranged in channels based on the channel shuffle unit; a 3D convolution module comprising a 3D U-Net and a 3D Conv, for convolution operation on the rearranged feature maps to obtain an initial disparity of 1 / 4 resolution size; and A cyclic convolution module is configured to perform convolution calculation on the initial parallax to output a parallax map.

7. The picking method according to claim 1, characterized in that, According to the position coordinates of the detection frame, two-dimensional coordinates of the center point of the tomato are determined, and a calculation formula is as follows: The position coordinates of the detection frame are The two-dimensional coordinates of the center point of the tomato are .

8. The picking method according to claim 7, characterized in that, According to the parallax value corresponding to the center point, three-dimensional coordinates of the center point of the tomato are calculated, including: According to the parallax value corresponding to the center point, a distance between the center point of the tomato and the binocular camera is calculated, and a calculation formula is as follows: wherein, is the focal length of the binocular camera, is the baseline of the binocular camera, is the parallax value; According to the two-dimensional coordinates of the center point of the tomato and the distance between the center point of the tomato and the binocular camera, three-dimensional coordinates of the center point of the tomato are calculated, and a calculation formula is as follows: wherein, is an intrinsic parameter of the binocular camera, are two-dimensional horizontal and vertical coordinates of the center point of the tomato, respectively, and z is a distance from the binocular camera to the center point of the tomato.

9. A binocular vision-based tomato picking system, the system comprising a memory and a processor, characterized in that, The memory stores a program, and the processor is configured to execute the program, and when the program is executed, the program is configured to perform the tomato picking method based on binocular vision according to any one of claims 1-8. 10.A machine readable storage medium, storing instructions for causing a machine to perform any one of the tomato picking methods based on binocular vision.

Images