Tomato picking property judgment method based on double-factor shielding evaluation

Abstract

The invention discloses a tomato picking property judgment method based on double-factor shielding evaluation. Comprising the following steps: firstly, collecting an original tomato plant image and extracting a region of interest of a tomato fruit so as to obtain a tomato fruit main body image; thirdly, inputting the tomato fruit main body image into a pre-trained two-factor shielding evaluation model, and outputting a shielding type and a shielding rate of the tomato fruit by the model; and finally, according to the shielding type and the shielding rate of the tomato fruits, judging the picking property of the tomato fruits by adopting a hierarchical decision-making strategy. According to the method, similar shielding and heterogeneous shielding are distinguished, and the dynamic threshold value is introduced under the heterogeneous shielding, so that the problem that the picking risk cannot be accurately evaluated only by depending on a single shielding rate in the prior art is solved, and the operation safety and the picking success rate of the robot in an unstructured environment are effectively improved; and meanwhile, the data annotation cost of model training is remarkably reduced.

Description

Technical Field

[0001] This invention belongs to the field of agricultural automation and robotics technology, and specifically relates to a method for determining the harvestability of tomatoes based on two-factor occlusion assessment. Background Technology

[0002] With the development of smart agriculture, agricultural robots are increasingly being used in greenhouse operations. However, the greenhouse environment is unstructured and highly variable, and tomato fruits are often obscured by leaves, stems, or adjacent fruits. Severe occlusion can prevent the vision system from accurately perceiving the target's grasping posture, leading to operational malfunctions such as robotic arm collisions or end effector entanglement.

[0003] Existing visual algorithms mostly focus on object detection (such as outputting bounding boxes) and lack quantitative analysis of occlusion states. While some studies have attempted to use occlusion rate to assess harvesting difficulty, they typically rely on a single occlusion rate threshold, ignoring the physical properties of the occluded object (such as the difference between stems and ripe fruit). Furthermore, existing methods for restoring occluded regions often require expensive pixel-level manual annotation or computationally intensive image inpainting techniques, which are insufficient for the real-time needs of robotic operations.

[0004] Therefore, developing a lightweight harvestability determination method that can simultaneously sense the type and degree of occlusion and does not require extensive manual labeling is of great significance for enabling autonomous decision-making in agricultural robots. Summary of the Invention

[0005] In view of the problems and needs existing in the background technology, the purpose of this invention is to provide a method for determining the pickability of tomatoes based on two-factor occlusion assessment.

[0006] The technical solution provided by this invention is as follows:

[0007] I. A Method for Determining the Harvestability of Tomatoes Based on Two-Factor Occlusion Assessment

[0008] Step S1: Acquire original tomato plant images and extract the region of interest of tomato fruits to obtain the main image of the tomato fruits;

[0009] Step S2: Input the main image of the tomato fruit into the pre-trained two-factor occlusion evaluation model. The model outputs the occlusion type and occlusion rate of the tomato fruit.

[0010] Step S3: Based on the type and rate of shading of the tomato fruit, a stratified decision-making strategy is used to determine the harvestability of the tomato fruit.

[0011] If there are multiple images of the main body of the tomato fruit, repeat steps S2-S3 to generate the pickability of each tomato fruit.

[0012] In step S1, extracting the region of interest (ROI) of the tomato fruit includes:

[0013] First, a semantic segmentation model is used to globally locate and segment the tomato fruit in the original tomato plant image to obtain the initial mask of the tomato. Then, the initial mask of the tomato is subjected to background suppression processing to obtain the mask after background suppression. Then, the fruit region boundary is expanded on the mask after background suppression to obtain the main image of the tomato fruit.

[0014] In step S2, the two-factor occlusion assessment model includes a two-factor shared backbone network, a classification branch, and a regression branch. The two-factor shared backbone network is connected to the classification branch and the regression branch. The classification branch is used to predict the occlusion type, and the regression branch is used to predict the occlusion rate.

[0015] The dataset used in the pre-training process of the two-factor occlusion evaluation model includes ground truth occlusion rates. These ground truth occlusion rates are generated using a weakly supervised method based on biogeometric priors, as detailed below:

[0016] First, obtain the main mask of the tomato fruit; then, calculate the convex hull of its contour point set based on the main mask of the tomato fruit; next, perform ellipse fitting with symmetry constraints based on the convex hull to obtain the fitted ellipse of the tomato fruit; finally, calculate the true value of the occlusion rate based on the area of ​​the main mask of the tomato fruit and the fitted ellipse.

[0017] The symmetry constraints include axial symmetry or central symmetry.

[0018] The method of using a hierarchical decision-making strategy to determine the harvestability of tomato fruits includes:

[0019] If each tomato fruit is unobstructed or has the same type of obstruction, it is considered ready to be picked.

[0020] If the occlusion type of each tomato fruit is heterogeneous occlusion, then its occlusion rate is determined. If the occlusion rate is less than the preset safety threshold, it is determined to attempt to pick; if the occlusion rate is greater than the preset safety threshold, it is determined to be unharvestable.

[0021] II. A Tomato Harvest Determination System Based on Two-Factor Occlusion Assessment

[0022] The image acquisition unit is used to acquire raw images of tomato plants;

[0023] The image extraction unit is used to extract the region of interest of the tomato fruit from the original tomato plant image to obtain the main image of the tomato fruit;

[0024] The tomato parameter calculation unit is used to store the pre-trained two-factor occlusion evaluation model and to predict the occlusion type and occlusion rate of the tomato fruit based on the tomato fruit subject image.

[0025] The harvestability determination unit is used to determine the harvestability of tomato fruits based on the type and rate of shading, using a stratified decision-making strategy.

[0026] III. A computer device

[0027] The computer device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the method for determining the pickability of tomatoes based on two-factor occlusion assessment.

[0028] IV. A computer-readable storage medium

[0029] The storage medium stores a computer program, which, when executed by a processor, implements the steps of the method for determining the pickability of tomatoes based on two-factor occlusion assessment.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] 1. More Reliable Two-Factor Decision-Making: The tomato fruit pickability determination process of this invention does not rely solely on the shading rate, but combines the shading type. For similar shading (soft shading), even if the shading rate is high, it is still determined that the fruit is ready for picking (because the shading can be removed by removing the fruit in front); for different types of shading (hard shading), the risk is strictly controlled. This strategy, which conforms to the logic of actual operation, significantly improves the accuracy of picking determination in field tests.

[0032] 2. Weakly supervised training reduces costs: The weakly supervised method based on biogeometric prior proposed in this invention solves the problem of difficult labeling of occluded areas. It eliminates the need for manual pixel-level labeling of invisible areas (i.e., background), greatly reducing data preparation costs.

[0033] 3. Lightweight and Real-time Performance: Using MobileNetV3 as the backbone network and combining it with a multi-task learning architecture, the inference speed is fast, meeting the embedded real-time control requirements of agricultural robots. Attached Figure Description

[0034] Figure 1 This is an overall flowchart of the method of the present invention.

[0035] Figure 2 The effect after expanding the mask by 20%.

[0036] Figure 3 This is a schematic diagram of a weakly supervised truth generation process based on biogeometric priors.

[0037] Figure 4 This is a schematic diagram of the structure of a two-factor occlusion evaluation network.

[0038] Figure 5 This is a schematic diagram of the decision-making logic for stratified harvestability.

[0039] Figure 6 These are the regression results for occlusion categories and occlusion rates in the embodiments of the present invention. Detailed Implementation

[0040] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.

[0041] like Figure 1 As shown, the tomato harvestability determination method based on two-factor occlusion assessment proposed in this invention includes the following steps:

[0042] Step S1: Acquire original tomato plant images (i.e., original images of the tomato operation scene) using image acquisition equipment and extract the region of interest of the tomato fruit. The aim is to extract a clean fruit image from the complex greenhouse background, thereby obtaining the main image of the tomato fruit.

[0043] In practice, an Intel RealSense D405 camera mounted on the end effector of the harvesting robot is used to acquire RGB-D images of tomato fruits. To ensure image quality, the acquisition distance is maintained within the optimal operating range of 0.2-0.5 meters, which guarantees both the camera's depth measurement accuracy and covers a suitable field of view. The RGB color images and depth images output by the camera are registered and aligned using hardware or software to ensure a one-to-one correspondence between pixels, providing a foundation for subsequent spatial positioning. The acquisition process covers different lighting and shading conditions in the greenhouse to ensure the robustness of the algorithm in actual operations.

[0044] In one feasible implementation, extracting the region of interest (ROI) of a tomato fruit includes:

[0045] First, a semantic segmentation model is used to globally locate and segment the tomato fruit in the original tomato plant image, obtaining the tomato's bounding box and initial mask. Then, the initial mask of the tomato is subjected to background suppression processing, that is, only the main fruit area is retained, to obtain the mask after background suppression processing. Then, the mask after background suppression processing is expanded by a preset ratio (such as 20%) of the fruit area boundary to retain some environmental context information, thereby obtaining the main image of the tomato fruit.

[0046] In one feasible implementation, the semantic segmentation model is the YOLOv8-seg instance segmentation network. This network is a pre-trained model, specifically, the acquired raw RGB image and labels are input into the YOLOv8-seg instance segmentation network for training, enabling the network to quickly output the bounding box coordinates and pixel-level binary masks of all detected tomato fruits within the field of view.

[0047] In one feasible implementation, background suppression specifically involves performing a Hadamard product operation between the generated binary mask and the original RGB image. Specifically, pixel values ​​within the mask area remain unchanged, while background pixel values ​​outside the mask area are set to black. This operation effectively removes environmental background noise, allowing subsequent networks to focus on the fruit itself and its contact areas.

[0048] This invention specifically expands the fruit region boundary of the mask after background suppression processing because occlusion usually occurs at the fruit edge. To preserve the interaction features between the occluding object (such as the stem, main stem, red tomato, green tomato, etc.) and the fruit boundary, this embodiment expands the cropped mask region. If only the fruit mask is used to regress the occlusion type and occlusion rate, the contextual information around the fruit will be lost, causing the network to be unable to accurately distinguish the occlusion type. For example, red and green tomato occlusions will create similar concave contours on the tomato; without contextual information, misjudgment is very likely. Therefore, this embodiment sets the expansion ratio to 20% (i.e., magnifying the length and width by a factor of 1.2 with the mask center as the origin), thereby introducing a local receptive field containing the features of the occluding object, such as... Figure 2 As shown in the figure. Experimental data indicate that this moderate background introduction helps the network to more accurately determine the occlusion type by utilizing the texture abruptness and geometric discontinuity of the edges, significantly improving the network's accuracy in distinguishing between similar occlusions and dissimilar occlusions.

[0049] Step S2: Input the main image of the tomato fruit into a pre-trained two-factor occlusion evaluation model. The model outputs the occlusion type and occlusion rate of each tomato fruit in the main image, such as... Figure 6 As shown.

[0050] One feasible implementation method is, for example Figure 4 As shown, the two-factor occlusion assessment model includes a two-factor shared backbone network, a classification branch, and a regression branch. The two-factor shared backbone network is connected to the classification and regression branches. The classification branch is used to predict the occlusion type. Occlusion types include no occlusion, occlusion of the same type, and occlusion of different types. Occlusion of red tomatoes is categorized as occlusion of the same type. Occlusion of green tomatoes and other objects (such as stems and leaves) is categorized as occlusion of different types. The regression branch is used to predict the occlusion rate.

[0051] In one feasible implementation, the two-factor shared backbone network includes a connected MobileNetV3 network and a global average pooling layer. The MobileNetV3 network is used to extract image features; the global average pooling layer outputs a feature vector.

[0052] In one feasible implementation, the classification branch includes a fully connected layer.

[0053] In one feasible implementation, the regression branch includes a connected fully connected layer and an activation layer. The activation function of the activation layer is a sigmoid function. The occlusion rate is a value between 0 and 1.

[0054] The dataset used in the pre-training process of the two-factor occlusion evaluation model includes ground truth occlusion rates. To address the problem of difficulty in labeling ground truth values ​​due to the inability to directly observe occluded regions, the ground truth occlusion rates in this invention are generated using a weakly supervised method based on biogeometric priors. Figure 3 As shown, the specific process is as follows:

[0055] First, a main mask of the tomato fruit is obtained, which is manually labeled. Next, based on the prior knowledge that the natural growth form of a mature tomato is approximately a convex polygon on a 2D projection, the convex hull of its contour point set is calculated based on the main mask. This aims to automatically fill in the unnatural contour depressions and gaps caused by occlusion using mathematical morphology methods, "supporting" the broken visible area back to an approximate convex shape, providing a reliable geometric basis for subsequent fitting. Then, an ellipse fitting with symmetry constraints is performed based on this convex hull to obtain the fitted ellipse of the tomato fruit as its complete geometric shape. Finally, the true occlusion rate (i.e., the ratio of the two areas) is calculated based on the areas of the main mask and the fitted ellipse. Specifically, the following formula is satisfied:

[0056]

[0057] in, This is the true value of the occlusion rate; Area visible Area is the area of ​​the main mask (i.e., the red part in the figure). fitted The area of ​​the fitted ellipse.

[0058] Because the visible area can be very irregular under severe occlusion (e.g., occlusion rate > 50%), direct fitting can easily lead to severe distortion of the generated ellipse (e.g., excessive flattening or tilting). Therefore, this invention utilizes the axial symmetry or centrosymmetry of tomatoes to apply constraints, preventing the fitting result from overfitting to local edges, thereby reconstructing a complete fruit model that conforms to physical laws even under partial observation conditions.

[0059] The method proposed in this invention eliminates the need for manual, subjective annotation of invisible areas, thus ensuring the objectivity and accuracy of the training data.

[0060] Loss function L during the training process of the two-factor occlusion evaluation model total Classification loss L cls and regression loss L reg The weighted composition is calculated using the following formula:

[0061]

[0062] Among them, the classification regression loss L cls Using the cross-entropy loss function, the regression loss L reg The Huber loss function is used. λ1 and λ2 are weight parameters that balance the gradients of the two tasks.

[0063] Step S3: Based on the occlusion type and occlusion rate of the tomato fruits, a hierarchical decision-making strategy is adopted to determine the harvestability of each tomato fruit. Then, the collected depth map is combined to generate operation instructions and send them to the harvesting robot for the purpose of harvesting the tomato fruits.

[0064] In one feasible implementation, a hierarchical decision-making strategy is used to determine the harvestability of each tomato fruit, such as... Figure 5 As shown, it includes:

[0065] If each tomato fruit is unobstructed or has the same type of obstruction, it is considered ready to be picked.

[0066] If the occlusion type of each tomato fruit is heterogeneous occlusion, then its occlusion rate is determined. If the occlusion rate is less than the preset safety threshold, it is determined to attempt to pick; if the occlusion rate is greater than the preset safety threshold, it is determined to be unharvestable.

[0067] The specific logic for determining the harvestability of this invention is as follows: When the classification result is no obstruction or obstruction of the same type, it is determined to be harvestable and set as a high priority. This is because the obstruction of the same type is itself a harvesting target, and the robot can plan a sequence to prioritize harvesting the fruit in front of the obstruction to remove the obstruction. When the classification result is obstruction of a different type and the regression obstruction rate is less than a preset threshold (set to 30% in this embodiment), it is determined to attempt harvesting. At this time, the obstacle is usually only located at the edge of the fruit, and the end effector such as the vacuum suction cup still has a chance to successfully adsorb it. When the classification result is obstruction of a different type and the regression obstruction rate is greater than or equal to the preset threshold, it is determined to be unharvestable to avoid forced harvesting that could cause the robotic arm to collide with the hard stem or damage the crop.

[0068] If there are multiple images of the tomato fruit (i.e., there are multiple regions of interest for tomato fruits in the original images of the tomato plants), then repeat steps S2-S3 to generate the pickability of each tomato fruit.

[0069] The present invention proposes a tomato harvestability determination system based on two-factor occlusion assessment, comprising:

[0070] The image acquisition unit is used to acquire raw images of tomato plants;

[0071] The image extraction unit is used to extract the region of interest of the tomato fruit from the original tomato plant image to obtain the main image of the tomato fruit;

[0072] The tomato parameter calculation unit is used to store the pre-trained two-factor occlusion evaluation model and to predict the occlusion type and occlusion rate of the tomato fruit based on the tomato fruit subject image.

[0073] The harvestability determination unit is used to determine the harvestability of each tomato fruit based on the type and rate of shading, using a stratified decision-making strategy.

[0074] The present invention proposes a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method.

[0075] This invention proposes a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of a method for determining the pickability of tomatoes based on a two-factor occlusion assessment.

[0076] This invention proposes a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of a method for determining the pickability of tomatoes based on a two-factor occlusion assessment.

[0077] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0078] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for determining the harvestability of tomatoes based on a two-factor occlusion assessment, characterized in that, Includes the following steps: Step S1: Acquire original tomato plant images and extract the region of interest of tomato fruits to obtain the main image of the tomato fruits; Step S2: Input the main image of the tomato fruit into the pre-trained two-factor occlusion evaluation model. The model outputs the occlusion type and occlusion rate of the tomato fruit. Step S3: Based on the type and rate of shading of the tomato fruit, a stratified decision-making strategy is used to determine the harvestability of the tomato fruit.

2. The method for determining the harvestability of tomatoes based on two-factor occlusion assessment according to claim 1, characterized in that, If there are multiple images of the main body of the tomato fruit, repeat steps S2-S3 to generate the pickability of each tomato fruit.

3. The method for determining the harvestability of tomatoes based on two-factor occlusion assessment according to claim 1, characterized in that, In step S1, extracting the region of interest (ROI) of the tomato fruit includes: First, a semantic segmentation model is used to globally locate and segment the tomato fruit in the original tomato plant image to obtain the initial mask of the tomato. Then, the initial mask of the tomato is subjected to background suppression processing to obtain the mask after background suppression. Then, the fruit region boundary is expanded on the mask after background suppression to obtain the main image of the tomato fruit.

4. The method for determining the harvestability of tomatoes based on two-factor occlusion assessment according to claim 1, characterized in that, In step S2, the two-factor occlusion assessment model includes a two-factor shared backbone network, a classification branch, and a regression branch. The two-factor shared backbone network is connected to the classification branch and the regression branch. The classification branch is used to predict the occlusion type, and the regression branch is used to predict the occlusion rate.

5. The method for determining the harvestability of tomatoes based on two-factor occlusion assessment according to claim 1, characterized in that, The dataset used in the pre-training process of the two-factor occlusion evaluation model includes ground truth occlusion rates. These ground truth occlusion rates are generated using a weakly supervised method based on biogeometric priors, as detailed below: First, obtain the main mask of the tomato fruit; then, calculate the convex hull of its contour point set based on the main mask of the tomato fruit; next, perform ellipse fitting with symmetry constraints based on the convex hull to obtain the fitted ellipse of the tomato fruit; finally, calculate the true value of the occlusion rate based on the area of ​​the main mask of the tomato fruit and the fitted ellipse.

6. The method for determining the harvestability of tomatoes based on two-factor occlusion assessment according to claim 5, characterized in that, The symmetry constraints include axial symmetry or central symmetry.

7. The method for determining the harvestability of tomatoes based on two-factor occlusion assessment according to claim 1, characterized in that, The method of using a hierarchical decision-making strategy to determine the harvestability of tomato fruits includes: If each tomato fruit is unobstructed or has the same type of obstruction, it is considered ready to be picked. If the occlusion type of each tomato fruit is heterogeneous occlusion, then its occlusion rate is determined. If the occlusion rate is less than the preset safety threshold, it is determined to attempt to pick; if the occlusion rate is greater than the preset safety threshold, it is determined to be unharvestable.

8. A tomato harvestability determination system based on two-factor occlusion assessment, characterized in that, include: The image acquisition unit is used to acquire raw images of tomato plants; The image extraction unit is used to extract the region of interest of the tomato fruit from the original tomato plant image to obtain the main image of the tomato fruit; The tomato parameter calculation unit is used to store the pre-trained two-factor occlusion evaluation model and to predict the occlusion type and occlusion rate of the tomato fruit based on the main image of the tomato fruit. The harvestability determination unit is used to determine the harvestability of tomato fruits based on the type and rate of shading, using a stratified decision-making strategy.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the tomato harvestability determination method based on two-factor occlusion assessment as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the tomato pickability determination method based on two-factor occlusion assessment as described in any one of claims 1 to 7.

Images