A method for estimating a string tomato picking posture based on semantic segmentation

Abstract

This invention discloses a method for estimating the picking posture of bunch tomatoes based on semantic segmentation. First, a bunch tomato image segmentation dataset is constructed. Then, an image semantic segmentation model is trained using this dataset to obtain the bunch tomato image segmentation model. Next, the RGB images of the bunch tomatoes to be processed and their corresponding depth maps are acquired. The RGB images are then input into the bunch tomato image segmentation model, which outputs the corresponding bunch tomato segmentation results. Finally, based on the bunch tomato segmentation results and the corresponding depth maps, the picking posture of all fruit stems in the bunch tomato segmentation results is estimated to obtain a set of bunch tomato picking postures. This method can accurately locate the picking direction, improving the accuracy of automated picking and the fruit integrity rate.

Description

Technical Field

[0001] This invention relates to a tomato pose estimation method in the fields of automated fruit and vegetable harvesting robots and computer vision technology, specifically a string tomato harvesting pose estimation method based on semantic segmentation. Background Technology

[0002] Cherry tomatoes have thin skin and tender flesh, and their respiration rate increases dramatically after ripening, requiring timely harvesting. Timely harvesting not only preserves the fresh taste and flavor of cherry tomatoes but also reduces economic losses caused by pests and diseases by shortening the fruiting period. Furthermore, scientific and regular harvesting can break the apical dominance of the plant, stimulate the sprouting of lateral branches and flower bud differentiation, increasing the fruit yield per plant by 15%-20%, effectively extending the harvest period and increasing overall yield, thus forming a sustainable and virtuous cycle of production.

[0003] The development of tomato harvesting robots not only fills the technological gap in manpower but also represents a significant breakthrough in achieving precision agriculture. In intelligent tomato harvesting scenarios, fruit posture estimation is a core technological element for achieving efficient and damage-free harvesting. Due to the randomness of tomato growth postures and the complexities of vine entanglement and overlapping fruits in real-world scenarios, without accurate posture estimation, the end effector of the harvesting robot will struggle to determine the optimal grasping angle, easily causing stem tearing or skin damage. Therefore, it is necessary to propose a precise method for estimating the harvesting posture of bunch tomatoes. Summary of the Invention

[0004] To address the challenges of precise harvesting in existing automated tomato harvesting methods due to random fruit postures and interference from the main stem, this invention proposes a method for estimating the harvesting posture of bunch tomatoes. This method integrates fruit stalk skeleton extraction and spatial vector calculation techniques. By extracting the stalk skeleton lines and combining them with depth information to determine three-dimensional coordinates, a spatial vector is constructed using the geometric relationship between the fruit stalk and the main stem contours to accurately determine the relative position of the fruit and the main stem. This invention ensures that the robotic arm avoids interference from the main stem along the optimal path, preventing premature rotting of highly mature fruits due to collisions or scratches with the main stem during harvesting. It provides technical support for large-scale, standardized intelligent harvesting, solving the technical difficulties of complex growth postures of bunch tomatoes and susceptibility to obstruction during robotic arm grasping in facility agriculture, significantly improving the success rate of automated harvesting and fruit integrity.

[0005] The technical solution of the present invention is as follows:

[0006] I. A Semantic Segmentation-Based Method for Estimating Tomato Picking Posture

[0007] Step 1: Construct a string of tomato images segmentation dataset, and then use the string of tomato images segmentation dataset to train the image semantic segmentation model to obtain the string of tomato images segmentation model.

[0008] Step 2: Obtain the RGB image of the tomato string to be processed and the corresponding depth map. Input the RGB image of the tomato string to be processed into the tomato string image segmentation model, and the model outputs the corresponding tomato string segmentation result.

[0009] Step 3: Based on the segmentation results of the bunch tomatoes and the corresponding depth map, estimate the picking posture of all fruit stems in the segmentation results of the bunch tomatoes to obtain the picking posture set of the bunch tomatoes.

[0010] In step 2, the segmentation result of the bunch tomato includes the outlines of all bunch tomato stems and main stems within the field of view of the bunch tomato RGB image.

[0011] Step 3 specifically involves:

[0012] Step 3.1: Based on the segmentation results of the bunch tomatoes and the corresponding depth map, extract the skeleton lines of all fruit stalks and main stems within the field of view to obtain the skeleton lines corresponding to all fruit stalks and main stems;

[0013] Step 3.2: Calculate the intersection of the skeleton line of each fruit stalk with the skeleton line of the main stem and denote it as the main connection point B and the midpoint A of the skeleton line of the fruit stalk. Denote the vector from the main connection point B to the midpoint A of the skeleton line of the current fruit stalk as the posture vector of the current fruit stalk. Construct a plane with the posture vector of the current fruit stalk as the normal vector and passing through the midpoint A of the skeleton line of the current fruit stalk and denote it as the picking plane of the tomato bunch corresponding to the current fruit stalk.

[0014] Step 3.3: Project the current fruit stalk's pose vector onto the horizontal plane and denote the projected vector as the first projection vector; find a vector within the current picking plane and calculate its projection onto the horizontal plane, denote it as the second projection vector; denote the vector corresponding to the second projection vector perpendicular to the first projection vector within the current picking plane as the candidate picking vector; denote the projection direction of the camera's optical axis on the depth as the positive direction, and denote the positive vector among the candidate picking vectors as the final picking vector;

[0015] Step 3.4: Adjust the final picking vector according to the current depth relationship between the fruit stalk and the main stem to obtain the picking posture of the bunch of tomatoes;

[0016] Step 3.5: Repeat steps 3.2-3.4 to calculate the picking posture of the remaining fruit stems of the bunch tomatoes, thereby obtaining the picking posture set of bunch tomatoes.

[0017] In step 3.1, the extracted skeletal lines of the fruit stalk and main stem are further subjected to subpixel-level thinning and smoothing by Gaussian convolution kernels, and the processed skeletal lines of the fruit stalk and main stem are used as the corresponding final skeletal lines.

[0018] In step 3.2, the intersection of the skeleton line of each fruit stalk and the skeleton line of the main stem is calculated, specifically including:

[0019] Connect the two endpoints of the skeleton line of each fruit stalk and then extend it towards the main stem. The intersection of the extended line and the skeleton line of the main stem is taken as the main connection point B.

[0020] In step 3.2, the depth values ​​of the points in the neighborhood of each endpoint of the skeleton line of each fruit stalk are filtered by median, and the average depth value of the filtered points in the neighborhood of that endpoint is taken as the depth value of that endpoint.

[0021] In step 3.2, the posture vector of the fruit stalk is a unit vector.

[0022] Step 3.4 specifically involves:

[0023] If the midpoint A of the current fruit stalk's skeleton line is behind the skeleton line of the main stem, then the vector perpendicular to the final picking vector is taken as the picking posture for the bunch of tomatoes; if the midpoint A of the current fruit stalk's skeleton line is in front of the skeleton line of the main stem, then the inverse vector of the final picking vector is taken as the picking posture for the bunch of tomatoes.

[0024] II. A computer device

[0025] The device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the semantic segmentation-based string tomato picking posture estimation method.

[0026] III. A computer-readable storage medium

[0027] The medium stores a computer program, which, when executed by a processor, implements the steps of the semantic segmentation-based string tomato picking posture estimation method.

[0028] This invention proposes a method for estimating the pose of cluster tomatoes based on 3D coordinate modeling and spatial vector analysis. By integrating semantic segmentation and skeleton extraction techniques with RGB-D multimodal data processing, it achieves accurate judgment of fruit pose and harvesting vector planning. This method addresses problems in traditional pose estimation such as multi-endpoint misjudgment, main stem interference, and depth loss by constructing a comprehensive optimization system from image acquisition to vector generation, effectively improving the stability and reliability of automated harvesting.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] 1. When extracting the skeleton line of the fruit stalk, this invention determines the midpoint depth by calculating the average depth value of the two ends of the skeleton curve, avoiding errors caused by fluctuations in a single depth value, and providing more reliable depth data support for subsequent posture judgment.

[0031] 2. This invention utilizes a semantic-geometric fusion-based tomato pose estimation framework to achieve cross-modal mapping from two-dimensional images to three-dimensional spatial vectors. By constructing spatial vectors and determining picking vectors, the relative positions of the fruit and the main stem are transformed into quantifiable mathematical expressions. Through precise vector calculation and projection analysis, this invention improves the accuracy of pose estimation and effectively reduces the risk of picking failure caused by interference from the main stem.

[0032] 3. This invention determines the relative positions of the fruit and the main stem based on the positive and negative directions of a spatial vector, and adaptively selects the corresponding picking vector. This mechanism can intelligently handle bunches of tomatoes in different postures, greatly improving the practicality and stability of the automated picking system in complex environments. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of the overall process of the method of the present invention.

[0034] Figure 2 This is a schematic diagram illustrating the specific implementation process of the method of the present invention.

[0035] Figure 3 This is a schematic diagram of the attitude estimation process of the method of the present invention. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the invention.

[0037] Because the growth posture of tomatoes is random, and complex situations such as vine entanglement and fruit overlap occur in real-world scenarios, this invention provides a method for locating picking points of clustered tomatoes based on semantic segmentation. This method improves the positioning accuracy of picking points for clustered tomatoes. The overall process is as follows: Figure 1 As shown. The specific execution steps are as follows:

[0038] Step 1: Construct a string of tomato image segmentation dataset, such as Figure 2 As shown, the image semantic segmentation model is then trained using the string tomato image segmentation dataset to obtain the string tomato image segmentation model.

[0039] Step 1 is as follows:

[0040] Step 1.1: Acquire target images; Data collection was conducted at an agricultural tomato planting base in a certain area. The subjects collected were cherry tomatoes grown on trellises, and each tomato fruit was captured from five angles. RGB images of bunch tomatoes were collected under different growth stages, lighting conditions, and occlusion scenarios. The Labelme tool was then used to annotate the tomato stems and main stems with polygonal outlines. The stem was defined as "stem," referring to the visible part of the stalk connecting the main stem and the fruit, and the main stem as "mainstem," referring to the entire main stem containing the bunch tomatoes. The quality of the collected dataset was screened during annotation. Data with excessively short stems or overlapping stems was not suitable for automated harvesting by robots and was therefore not annotated. This resulted in a multi-class segmentation dataset containing stems, main stems, and fruits. Enhancement methods such as random rotation, brightness adjustment, and depth noise injection were used to expand the multi-class segmentation dataset, enhancing the model's adaptability to the environment. Simultaneously, during image enhancement, it was necessary to ensure that the stem and main stem structures were clearly visible in the images, thus obtaining the bunch tomato image segmentation dataset. The dataset is divided according to a ratio of 8:1:1 for training set: validation set: test set.

[0041] Step 1.2: Train the YOLOv5-SEG semantic segmentation model using the tomato cluster image segmentation dataset, so that the model can initially identify the outlines of all tomato cluster stems and main stems within the field of vision, until the training is completed, and obtain the trained YOLOv5-SEG semantic segmentation model as the tomato cluster image segmentation model.

[0042] During model training, a server was used as the training platform, with the following specifications: an NVIDIA GeForce RTX 4090 graphics card with 24209 MiB of video memory, weight decay set to 0.0005, a batch size of 16, an image input size of 640*360, 100 epochs, and a momentum of 0.937. After training, the best.pt file from the training results was used for inference, and a depth camera was used for real-time testing.

[0043] Step 2: Acquire the RGB image of the bunch of tomatoes to be processed and the corresponding depth map. Input the RGB image of the bunch of tomatoes to be processed into the bunch of tomatoes image segmentation model. The model outputs the corresponding bunch of tomatoes segmentation result. The bunch of tomatoes segmentation result includes the outline of all bunch of tomatoes' stems and main stems within the field of view of the bunch of tomatoes' RGB image.

[0044] Step 3: Based on the segmentation results of the bunch tomatoes and the corresponding depth map, estimate the picking posture of all fruit stems in the segmentation results of the bunch tomatoes to obtain the picking posture set of the bunch tomatoes, and then output it to the picking robot to pick the bunch tomatoes.

[0045] Step 3 specifically involves:

[0046] Step 3.1: Based on the tomato segmentation results and the corresponding depth map, the Zhang-Suen algorithm is used to extract the skeleton lines of all fruit stalks and main stems within the field of view, obtaining the skeleton lines corresponding to all fruit stalks and main stems, such as... Figure 3 As shown, the fruit stalk skeleton extraction can employ an iterative refinement strategy. The extracted fruit stalk and main stem skeleton lines are subjected to sub-pixel-level refinement and smoothing using a Gaussian convolution kernel (kernel size 3×3, standard deviation 0.8), with the processed skeleton lines of the fruit stalk and main stem serving as the final skeleton lines. Specifically, the sub-pixel-level refinement includes: secondary refinement of regions with a skeleton width greater than 3 pixels in the initial skeleton; and precise localization of skeleton endpoints using Harris corner detection, improving coordinate accuracy to the sub-pixel level.

[0047] Step 3.2: Calculate the intersection of the skeleton line of each fruit stalk with the skeleton line of the main stem, and denote it as the main connection point B and the midpoint A of the skeleton line of the fruit stalk. Both the main connection point B and the midpoint A of the skeleton line are three-dimensional coordinates. Since the main stem is relatively thick and its depth is relatively stable, the depth values ​​of the 3*3 pixel area around the endpoint of the main stem are filtered by median, and the average depth value of the 3*3 pixel area around the endpoint is taken as the depth value of the endpoint. The vector pointing from the main connection point B to the midpoint A of the current fruit stalk skeleton line is denoted as the current fruit stalk's posture vector; the fruit stalk's posture vector is a unit vector used to represent the posture direction of the bunch of tomatoes relative to the main stem. Since cutting the fruit stalk perpendicularly is the easiest way to achieve successful cutting, the picking vector of the bunch of tomatoes is a vector perpendicular to the fruit stalk posture vector at the picking point. Therefore, this invention uses the current fruit stalk posture vector as the normal vector and constructs a plane passing through the midpoint A of the current fruit stalk skeleton line, and denotes it as the picking plane of the bunch of tomatoes corresponding to the current fruit stalk.

[0048] To address the issue of multiple start and end points potentially appearing in a skeleton line, this invention employs an endpoint classification and priority sorting strategy. Specifically:

[0049] Based on the morphological characteristics of each skeleton line, the local curvature, branching angle, and distance to the nearest fruit corresponding to multiple endpoints are calculated. A curvature threshold of 0.8 is set, prioritizing endpoints with curvature greater than this threshold. Simultaneously, a weighted evaluation is performed based on the Euclidean distance from the endpoint to the centroid of the nearest fruit and a weighting coefficient of 0.4, while constraining the branching angle at the endpoints to be close to 180°. The specific formula is as follows:

[0050] Score=0.4×Curvature+0.4×(1 / Distance)+0.2×AngleConsistency

[0051] Wherein, Score is the endpoint probability score of each endpoint, Curvature is the local curvature of each endpoint, AngleConsistency is the branch angle of each endpoint, and Distance is the Euclidean distance from each endpoint to the nearest centroid of the fruit.

[0052] All endpoints are comprehensively scored, and the endpoint with the highest probability score is selected as the final start and end points, thereby ensuring the uniqueness and accuracy of the fruit stalk skeleton curve and providing a reliable foundation for subsequent spatial vector calculation and attitude determination.

[0053] Specifically, calculating the intersection points of the scaffold line of each fruit stalk with the scaffold line of the main stem includes:

[0054] Connect the two endpoints of the skeleton line of each fruit stalk and then extend it towards the main stem. The intersection of the extended line and the skeleton line of the main stem is taken as the main connection point B.

[0055] After median filtering of the depth values ​​of the neighborhood points of each endpoint of the skeleton line of each fruit stalk, the mean of the filtered depth values ​​of the neighborhood points of that endpoint is taken as the depth value of that endpoint. The average of the depth values ​​of the two endpoints of each fruit stalk is taken as the depth value of point A at the midpoint of the skeleton line of that fruit stalk.

[0056] For the depth values ​​of the endpoints, a depth loss compensation mechanism can be used, specifically: in the temporal dimension, weighted fusion is performed using depth data from three consecutive frames; in the spatial dimension, bilateral filtering is used to smooth the missing regions, with a filtering radius of 15 pixels and a standard deviation of 0.8. This yields the depth value for each point.

[0057] Step 3.3: Since the bunch tomatoes grow randomly around the main stem, interference between the end effector and the main stem must be avoided during harvesting. Project the current fruit stalk's attitude vector onto the horizontal plane and denote the projected vector as the first projection vector; find a vector within the current harvesting plane and calculate its projection onto the horizontal plane, denote it as the second projection vector; denote the vector corresponding to the second projection vector perpendicular to the first projection vector within the current harvesting plane as the candidate harvesting vector; since there are two vectors that meet the conditions in the plane, the projection direction of the camera's optical axis in the depth plane (i.e., the projection direction of the depth direction in the camera coordinate system in the depth plane) is taken as the positive direction, and the positive vector among the candidate harvesting vectors is denoteed as the final harvesting vector; this harvesting vector passes through the midpoint of the fruit stalk's skeleton curve and forms a reasonable avoidance relationship with the main stem in the horizontal direction, that is, the movement direction of this harvesting vector will not directly face the main stem in the horizontal dimension, effectively avoiding interference between the end effector and the main stem during harvesting, thereby greatly reducing the risk of interference between the end effector and the main stem and ensuring the smooth progress of the harvesting action.

[0058] Step 3.4: Adjust the final picking vector according to the current depth relationship between the fruit stalk and the main stem to obtain the picking posture of the bunch of tomatoes;

[0059] Step 3.4 specifically involves:

[0060] To better adapt to the growth posture of the bunch tomatoes, the actuator posture vector is dynamically adjusted based on the position of the bunch tomatoes relative to the main stem. If the midpoint A of the current fruit stalk's skeleton line is behind the main stem's skeleton line, it indicates that the bunch tomatoes are behind the main stem and are easily interfered with by the main stem during harvesting. In this case, the vector perpendicular to the final harvesting vector is taken as the bunch tomatoes' harvesting posture. If the midpoint A of the current fruit stalk's skeleton line is in front of the main stem's skeleton line, it indicates that the bunch tomatoes are in front of the main stem. In this case, there is no obstruction or interference from the main stem, and they are not interfered with by the main stem. Therefore, the inverse vector of the final harvesting vector is taken as the bunch tomatoes' harvesting posture.

[0061] Step 3.5: Repeat steps 3.2-3.4 to calculate the picking posture of the remaining fruit stems of the bunch tomatoes, thereby obtaining the set of picking postures of bunch tomatoes in the current field of view.

[0062] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0063] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0064] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for estimating the picking posture of tomatoes based on semantic segmentation, characterized in that, Includes the following steps: Step 1: Construct a string of tomato images segmentation dataset, and then use the string of tomato images segmentation dataset to train the image semantic segmentation model to obtain the string of tomato images segmentation model. Step 2: Obtain the RGB image of the tomato string to be processed and the corresponding depth map. Input the RGB image of the tomato string to be processed into the tomato string image segmentation model, and the model outputs the corresponding tomato string segmentation result. Step 3: Based on the segmentation results of the bunch tomatoes and the corresponding depth map, estimate the picking posture of all fruit stems in the bunch tomato segmentation results to obtain the picking posture set of bunch tomatoes. Step 3 specifically involves: Step 3.1: Based on the segmentation results of the bunch tomatoes and the corresponding depth map, extract the skeleton lines of all fruit stalks and main stems within the field of view to obtain the skeleton lines corresponding to all fruit stalks and main stems; Step 3.2: Calculate the intersection of the skeleton line of each fruit stalk with the skeleton line of the main stem and denote it as the main connection point B and the midpoint A of the skeleton line of the fruit stalk. Denote the vector from the main connection point B to the midpoint A of the skeleton line of the current fruit stalk as the posture vector of the current fruit stalk. Construct a plane with the posture vector of the current fruit stalk as the normal vector and passing through the midpoint A of the skeleton line of the current fruit stalk and denote it as the picking plane of the tomato bunch corresponding to the current fruit stalk. Step 3.3: Project the current fruit stalk's pose vector onto the horizontal plane and denote the projected vector as the first projection vector; find a vector within the current picking plane and calculate its projection onto the horizontal plane, denote it as the second projection vector; denote the vector corresponding to the second projection vector perpendicular to the first projection vector within the current picking plane as the candidate picking vector; denote the projection direction of the camera's optical axis on the depth as the positive direction, and denote the positive vector among the candidate picking vectors as the final picking vector; Step 3.4: Adjust the final picking vector according to the current depth relationship between the fruit stalk and the main stem to obtain the picking posture of the bunch of tomatoes; Step 3.5: Repeat steps 3.2-3.4 to calculate the picking posture of the remaining fruit stems of the bunch tomatoes, thereby obtaining the picking posture set of bunch tomatoes.

2. The method for estimating the picking posture of tomatoes based on semantic segmentation according to claim 1, characterized in that, In step 2, the segmentation result of the bunch tomato includes the outlines of all bunch tomato stems and main stems within the field of view of the bunch tomato RGB image.

3. The method for estimating the picking posture of tomatoes based on semantic segmentation according to claim 1, characterized in that, In step 3.1, the extracted skeletal lines of the fruit stalk and main stem are further subjected to subpixel-level thinning and smoothing by Gaussian convolution kernels, and the processed skeletal lines of the fruit stalk and main stem are used as the corresponding final skeletal lines.

4. The method for estimating the picking posture of tomatoes based on semantic segmentation according to claim 1, characterized in that, In step 3.2, the intersection of the skeleton line of each fruit stalk and the skeleton line of the main stem is calculated, specifically including: Connect the two endpoints of the skeleton line of each fruit stalk and then extend it towards the main stem. The intersection of the extended line and the skeleton line of the main stem is taken as the main connection point B.

5. The method for estimating the picking posture of tomatoes based on semantic segmentation according to claim 1, characterized in that, In step 3.2, the depth values ​​of the points in the neighborhood of each endpoint of the skeleton line of each fruit stalk are filtered by median, and the average depth value of the filtered points in the neighborhood of that endpoint is taken as the depth value of that endpoint.

6. A method for estimating the picking posture of tomatoes based on semantic segmentation according to claim 1, characterized in that, In step 3.2, the posture vector of the fruit stalk is a unit vector.

7. The method for estimating the picking posture of tomatoes based on semantic segmentation according to claim 1, characterized in that, Step 3.4 specifically involves: If the midpoint A of the current fruit stalk's skeleton line is behind the skeleton line of the main stem, then the vector perpendicular to the final picking vector is taken as the picking posture for the bunch of tomatoes; if the midpoint A of the current fruit stalk's skeleton line is in front of the skeleton line of the main stem, then the inverse vector of the final picking vector is taken as the picking posture for the bunch of tomatoes.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the semantic segmentation-based method for estimating the picking posture of tomatoes as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the semantic segmentation-based method for estimating the picking posture of tomatoes as described in any one of claims 1 to 7.

Images