Stacked mature strawberry fruit stem posture detection method based on YOLOv10-pose
Through the stacked ripe strawberry shank posture detection method based on YOLOv10-pose, the problem that intelligent strawberry picking robots are difficult to quickly and accurately obtain strawberry postures is solved, and fast and accurate strawberry picking is achieved, reducing the damage rate and improving the picking success rate.
Patent Information
- Application Number
- CN202410938545.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-13
- Publication Date
- 2024-11-26
AI Technical Summary
The prior art is difficult to quickly and accurately obtain the posture of stacking ripe strawberry fruit shanks, resulting in the slow picking speed of smart strawberry picking robots and high damage rate, which cannot meet the needs of smart picking.
Using the stacked ripe strawberry shank posture detection method based on YOLOv10-pose, data enhancement processing is performed by acquiring and labeling the greenhouse strawberry image dataset, and a YOLOv10-pose model is established, using this model for real-time detection, output bounding boxes and key points. , calculate the posture and picking points of the fruit handle to achieve end-to-end key point detection.
The detection speed and accuracy are improved, the precise picking of strawberries is achieved, the damage rate during the robot picking process is reduced, and the picking success rate is improved by manually assisting the treatment of difficult-to-picture strawberries.
Smart Images

Figure CN119027936A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the field of integration of machine vision and greenhouse fruits, and specifically to a method for detecting the posture of stacked ripe strawberry stalks based on YOLOv10-pose. Background Art
[0002] Strawberry is a nutritious and flavorful fruit that can be grown in the field or in greenhouses and is widely distributed around the world. The strawberry picking season is relatively long each year, and the picking cost accounts for a quarter of its production cost. Currently, strawberry picking still relies on manual labor and lacks intelligent strawberry picking robots. However, due to the impact of an aging population and the loss of agricultural labor, the demand for intelligent strawberry picking robots is becoming increasingly urgent.
[0003] One of the challenges in developing an intelligent strawberry picking robot is the inability to quickly and accurately obtain the posture of the stacked ripe strawberry stems. Strawberries are easily damaged, and in order to reduce the damage rate during machine picking, they are generally picked by shearing + clamping the stems. Quickly and accurately obtaining the posture of the stacked ripe strawberry stems can maximize the success rate of picking and achieve precise picking.
[0004] Many researchers have devoted themselves to fruit posture detection. Some researchers obtain fruit masks through instance segmentation methods and obtain fruit postures based on fruit biological characteristics. Other researchers use a two-stage deep learning model to obtain detection boxes and then detect key points based on heat maps to achieve posture detection. Although the above research has completed the task of fruit posture detection, the speed is still relatively slow.
[0005] In the past two years, with the introduction of key point detection in the YOLO series, this type of method has been widely used in agricultural tasks to complete posture detection or picking point positioning. Some researchers have made improvements based on the YOLOv5 model and YOLOv8 to enable it to obtain key point and bounding box detection capabilities. The end-to-end posture detection method based on YOLO-pose has a certain speed advantage over other algorithms, but the accuracy can still be further improved. Summary of the invention
[0006] The purpose of the present invention is to overcome the shortcomings of the above-mentioned background technology and provide a method for detecting the posture of stacked ripe strawberry stalks based on YOLOv10-pose, which should have the characteristics of fast detection speed and high accuracy.
[0007] The technical solution of the present invention is:
[0008] A method for detecting the posture of stacked ripe strawberry stalks based on YOLOv10-pose comprises the following steps:
[0009] S1: Obtain and annotate greenhouse strawberry image dataset;
[0010] S2: Enhance the data set;
[0011] S3: Build the YOLOv10-pose model;
[0012] S4: Train the YOLOv10-pose model;
[0013] S5: Use the YOLOv10-pose model to detect greenhouse strawberries in real time and obtain bounding boxes and key points;
[0014] S6: Calculate the fruit stalk posture and picking point;
[0015] S7: End effector picking strawberries.
[0016] The step S1 comprises: using a camera to obtain greenhouse strawberry images in the morning, noon, evening, cloudy days, rainy days and sunny days respectively; marking the bounding box and key points of the ripe strawberry images; the key points include calyx key points and stem key points; the key points are divided into visible and blocked.
[0017] The step S2 includes: adjusting the hue, saturation and brightness of the image; scaling, flipping, splicing, erasing and cropping the image.
[0018] The YOLOv10-pose model of step S3 includes: adding a key point detection branch on the basis of the YOLOv10 model, and reconstructing the loss function of the YOLOv10-pose model;
[0019] The YOLOv10-pose model uses a dual label assignment strategy, combining multi-label assignment with single-label assignment during training and single-label assignment during prediction to improve post-processing speed.
[0020] The key point detection branch includes a Conv module, a Conv module, a Conv2d module, and a key point loss calculation; the Conv module includes a convolution layer, a batch normalization layer, and an activation function layer.
[0021] The loss function includes:
[0022] Loss Yolov10-pose =Loss one2many +Loss one2one
[0023] Loss one2many =λ kobj Loss kobj +λ pose Loss pose +λ boxLoss box +λ cls Loss cls
[0024] Loss one2one =λ kobj Loss kobj +λ pose Loss pose +λ box Loss box +λ cls Loss cls
[0025] Where: kobj , pose , box , cls is a constant;
[0026]
[0027] Where: y i is the predicted value of sample i, p i is the label value of sample i, and N is the number of samples;
[0028]
[0029] Among them: kpt_mask is the category of the key point; x pred and pred is the predicted coordinate of the key point; x gt and gt is the label coordinate of the key point; area is the area of the bounding box; sigmas is the scaling factor;
[0030] Loss box =Loss CIOU +Loss DFL
[0031] Loss DFL (S i ,S i+1 )=-((y i+1 -y)log(S i )+(yy i )log(S i+1 ))
[0032] Where: S i , S i+1 are the predicted value and near-predicted value output by the network respectively; y, y i ,y i+1 They are the actual label value, label integral value, and near label integral value respectively.
[0033] The step S5 includes: collecting greenhouse strawberry images, inputting the images into the YOLOv10-pose model for processing, and outputting bounding boxes and key points.
[0034] The step S6 includes: when the strawberry has a calyx key point and a stem key point, the line connecting the two key points is the posture of the strawberry stalk, and the midpoint of the line is the picking point of the strawberry.
[0035] The step S7 includes: adjusting the posture of the end effector body according to the posture of the fruit stalk, and picking the strawberries according to the picking point.
[0036] The beneficial effects of the present invention are:
[0037] (1) The present invention proposes a YOLOv10-pose model, which is an end-to-end key point detection model. It does not require non-maximum suppression in the post-processing stage and has a fast detection speed. The backbone network of the model also provides a high detection accuracy.
[0038] (2) The YOLOv10-pose model of the present invention can output a bounding box for real-time detection of the maturity of strawberries, and judge whether the bounding box is effectively detected according to the confidence level; in actual operation, only easy-to-pick strawberries are picked, and difficult-to-pick strawberries are left for manual processing, thereby realizing collaborative operation of robots and humans;
[0039] (3) The YOLOv10-pose model of the present invention can output two key points for detecting the posture and picking point of the strawberry stalk without any other auxiliary operation. The method is simple and effective, which can effectively avoid the errors introduced by multiple steps and improve the speed of posture detection. The picking point is then used to determine the position of strawberry shearing and clamping, which improves the picking success rate and realizes the precise picking of stacked ripe strawberries. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1 It is a flow chart of the present invention.
[0041] Figure 2 It is an architecture diagram of the YOLOv10-pose model of the present invention.
[0042] Figure 3 It is an architectural diagram of the C2f module of the present invention.
[0043] Figure 4 It is an architecture diagram of the Conv module of the present invention.
[0044] Figure 5 is an architecture diagram of the V10Pose module of the present invention.
[0045] Figure 6It is an architecture diagram of the C2fCIB module and the SCDown module of the present invention.
[0046] Figure 7 is a schematic diagram of the greenhouse strawberry image.
[0047] Figure 8 This is a schematic diagram of labeling greenhouse strawberry images.
[0048] Fig. 9 This is a diagram of the bounding boxes and key points output by the YOLOv10-pose model.
[0049] Fig.10 It is a schematic diagram of the picking points of the greenhouse strawberry image. DETAILED DESCRIPTION
[0050] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0051] like Figure 1 As shown, a method for detecting the posture of stacked ripe strawberry stalks based on YOLOv10-pose includes the following steps:
[0052] S1: Obtain and annotate greenhouse strawberry image dataset;
[0053] (1) Use a camera to obtain a greenhouse strawberry image dataset in the morning, noon, evening, cloudy, rainy, and sunny days;
[0054] The camera is an Intel Realsense L515 camera, and the image resolution is set to 640 × 480. To reduce the impact of stems and leaves on strawberries, the main optical axis of the camera is at an angle of about 45° to the strawberry ridge surface and is about 0.5m to 0.8m away from the ridge surface.
[0055] The greenhouse strawberry images collected by the camera are as follows: Figure 7 As shown in FIG. 1 , strawberries are picked using a robot, which includes a controller, a camera for acquiring images, and an end effector for picking strawberries.
[0056] (2) Annotate the bounding boxes and key points of the ripe strawberry image;
[0057] The labeling software is Labelme and the labeling file is in JSON format.
[0058] Since this method involves two parts: strawberry bounding box and key point detection, the mature strawberry fruits and key points need to be labeled separately during annotation; each strawberry has two key points, the first key point is the strawberry calyx, that is, the calyx key point, and the second key point is located on the stem, which is a certain distance away from the calyx, that is, the stem key point.
[0059] Since key points may be blocked, they have two states: visible and blocked. When marking, set the key point visibility parameter, 1 for blocked and 2 for visible.
[0060] Image annotation such as Figure 8 As shown, the calyx key point is key point 1 in the figure, and the stem key point is key point 2 in the figure.
[0061] S2: Enhance the data set;
[0062] Enhancement processing includes adjusting the hue, saturation and brightness of the image, as well as scaling, flipping, splicing, erasing and cropping the image to enhance the generalization ability and robustness of the model.
[0063] (1) The hue, saturation, and brightness of the image are adjusted by applying HSV transformation. The hue transformation parameter is 0.015, the saturation transformation parameter is 0.7, and the brightness transformation parameter is 0.4, so that the model can better adapt to different lighting and color conditions.
[0064] (2) Randomly scale the image with a parameter value of 0.5 to help the model learn features at different scales;
[0065] (3) The image is horizontally flipped with a probability of 50% to help the model learn symmetric features and improve recognition accuracy;
[0066] (4) Use mosaic data enhancement to stitch multiple different images into one. The parameter value is 1.0, which means that stitching is always performed. This not only increases the diversity of the data, but also enables the model to better handle objects in different scenes.
[0067] (5) Random erasing is performed with a probability of 0.4 to solve the problem of partially blocked or missing images.
[0068] (6) The parameter value of the cropping operation is 1.0, which means that the image will be cropped proportionally. Through the cropping operation, the model can focus on different parts of the image and enhance its sensitivity to local features.
[0069] S3: Build the YOLOv10-pose model;
[0070] The establishment process of the YOLOv10-pose model includes: adding a key point detection branch based on the YOLOv10 model, retaining the original YOLOv10 model's feature of not requiring non-maximum suppression during post-processing; and reconstructing the loss function of the YOLOv10-pose model.
[0071] The YOLOv10-pose model uses a dual label assignment strategy, combining multi-label assignment with single-label assignment during training and single-label assignment during prediction to improve post-processing speed.
[0072] The keypoint detection branch is similar to the bounding box detection branch and is also added at different feature layers to predict objects of different sizes.
[0073] The key point detection branch includes Conv module, Conv module, Conv2d module, and key point loss calculation.
[0074] The Conv module includes convolutional layers, batch normalization layers, and activation function layers, which are used to extract features and enhance the nonlinear expression capabilities of the model;
[0075] The Conv2d module is used to output the coordinates and visibility of key points. The output quantity is 6, representing the x and y coordinates and visibility of two key points. This improvement enables the YOLOv10-pose model to be used for object detection and pose estimation.
[0076] The loss function of the YOLOv10-pose model includes Loss one2many 、Loss one2one .
[0077] Loss one2many With Loss one2one The loss functions of all include key point loss (Loss kobj 、Loss pose )、Loss box 、Loss cls .
[0078] Loss pose is calculated by taking the L2 loss between the predicted and true coordinates of each keypoint, dividing it by the area of the true bounding box, and finally dividing it by the scaling factor.
[0079] Loss kobj The sum of the binary cross entropy losses of N keypoints is used and then the average is taken.
[0080] Loss cls is the binary cross entropy loss of N targets, added and then averaged.
[0081] Lossbox Use Loss CIOU and Loss DFL Loss CIOU For the existing formula.
[0082] Specifically include:
[0083] Loss Yolov10-pose =Loss one2many +Loss one2one
[0084] Loss one2many =λ kobj Loss kobj +λ pose Loss pose +λ box Loss box +λ cls Loss cls
[0085] Loss one2one =λ kobj Loss kobj +λ pose Loss pose +λ box Loss box +λ cls Loss cls
[0086] Where: kobj , pose , box , cls are constants, taking values of 1, 12, 7.5, and 0.5 respectively;
[0087]
[0088] Where: y i is the predicted value of sample i, p i is the label value of sample i, and N is the number of samples;
[0089]
[0090] Among them: kpt_mask is the category of the key point, x pred and pred is the predicted coordinate of the key point, x gt and gt is the label coordinate of the key point, area is the area of the bounding box, and sigmas is the scaling factor;
[0091] Loss box =LossCIOU +Loss DFL
[0092] Loss DFL (S i ,S i+1 )=-((y i+1 -y)log(S i )+(yy i )log(S i+1 ))
[0093] Where: S i , S i+1 are the predicted value and near-predicted value output by the network respectively; y, y i ,y i+1 They are the actual label value, label integral value, and near label integral value respectively.
[0094] S4: Train the YOLOv10-pose model;
[0095] Input the dataset into the YOLOv10-pose model for training and optimize the model hyperparameters.
[0096] The training environment is: Ubuntu 20.04 operating system, equipped with 32GB memory, NVIDIA RTX2080Ti graphics card, Xeon CPU E5-2680 v3@2.50GHz processor.
[0097] The deep learning framework is PyTorch 1.10.0, and the CUDA and CUDNN versions are 11.0 and 8.0.5, respectively.
[0098] The network training parameters are set as follows: the input image size is 640×640, the batch size is 8, and the epoch is 700.
[0099] The optimizer is AdamW, the initial learning rate is 0.01, the momentum factor is 0.937, and the weight decay coefficient is 0.0005.
[0100] S5: Use the YOLOv10-pose model to detect greenhouse strawberries in real time and obtain bounding boxes and key points;
[0101] Deploy the trained YOLOv10-pose model, collect greenhouse strawberry images, input the images into the YOLOv10-pose model for processing, and output the bounding box and key points:
[0102] (1) Input the strawberry color image to be detected into the YOLOv10-pose model, identify the ripe strawberries in the image and generate the bounding box of each strawberry;
[0103] (2) Detect the key point position of each identified ripe strawberry.
[0104] Bounding boxes and key points are filtered according to the predicted confidence: for bounding boxes, only those with confidence higher than the preset threshold will be retained; for key points, the number of detected key points may be 0, 1 or 2, depending on their confidence; if the key point confidence is lower than the preset threshold, the key point will be excluded.
[0105] (3) Output the filtered ripe strawberry bounding box and key points.
[0106] The present invention can effectively detect and locate ripe strawberries in an image, and even if there is occlusion, can reasonably filter out inaccurate detection results according to the confidence level, thereby ensuring the accuracy and reliability of the output results.
[0107] S6: Calculate the fruit stalk posture and picking point;
[0108] When a strawberry has both a calyx key point and a stem key point, connect these two key points. The line connecting the two points is the posture of the strawberry stalk, and the midpoint of the line is the picking point of the strawberry.
[0109] When a strawberry has only one key point or no key point, it is impossible to obtain the strawberry stalk posture and picking point, and these strawberries will be left for manual picking.
[0110] Intelligent mechanized strawberry picking does not need to complete the task of picking all strawberries, but to do its best to pick all easy-to-pick strawberries to achieve the highest picking success rate. For strawberries that are difficult to pick, manual assistance will be provided.
[0111] like Fig. 9 As shown, the key point of the calyx is the key point 1 in the figure, the key point of the stem is the key point 2 in the figure, and the line connecting the key point 1 and the key point 2 is the posture of the fruit stalk.
[0112] like Fig.10 As shown, the midpoint of the line is the picking point.
[0113] S7: end effector picking strawberries;
[0114] The end effector starts picking, first adjusting the end effector posture according to the posture of the fruit stalk, and then picking the strawberry according to the picking point:
[0115] (1) The robot controller analyzes the posture of the fruit stem and the picking point, and calculates the optimal cutting position and angle of the end effector.
[0116] (2) The robot adjusts the posture of the end effector to match the position and angle of the strawberry stalk and align it with the picking point;
[0117] (3) The end effector performs shearing and clamping actions, cutting the strawberry from the stalk while clamping the strawberry, completing the harvest.
[0118] The following is a further description of the YOLOv10-pose model.
[0119] The model consists of three parts: Backbone network, Neck network and Head network.
[0120] The Backbone network includes a Conv module, a Conv module, a C2f module, a Conv module, a C2f module, a SCDown module, a C2f module, a SCDown module, a C2fCIB module, a SPPF module, and a PSA module which are connected in sequence.
[0121] The Neck network includes an Upsample module, a Concat module, a C2f module, an Upsample module, a Concat module, a C2f module, a Conv module, a Concat module, a C2fCIB module, a SCDown module, a Concat module, and a C2fCIB module, which are connected in sequence.
[0122] The Neck network consists of three V10Pose modules.
[0123] The connection relationship among Backbone network, Neck network and Head network is: the second C2f module of Backbone network is connected to the second Concat module of Neck network; the third C2f module of Backbone network is connected to the first Concat module of Neck network; the PSA module of Backbone network is respectively connected to the first Upsample module and the fourth Concat module of Neck network; the first C2f module of Neck network is connected to the third Concat module; the second C2fCIB module and two C2fCIB modules of Neck network are respectively connected to the three V10Pose modules of Head network.
[0124] like Figure 5 As shown in the figure, the V10Pose module includes the V10Detect branch and the key point detection branch. The key point detection branch includes the Conv module, the Conv module, the Conv2d module, and the key point loss calculation (Keypoint Loss).
[0125] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.
Claims
1. A method for detecting the posture of stacked ripe strawberry stalks based on YOLOv10-pose, comprising the following steps: S1: Obtain and annotate greenhouse strawberry image dataset; S2: Enhance the data set; S3: Build the YOLOv10-pose model; S4: Train the YOLOv10-pose model; S5: Use the YOLOv10-pose model to detect greenhouse strawberries in real time and obtain bounding boxes and key points; S6: Calculate the fruit stalk posture and picking point; S7: End effector picking strawberries.
2. According to a YOLOv10-pose based stacked ripe strawberry stalk posture detection method according to claim 1, it is characterized in that: The step S1 comprises: using a camera to obtain greenhouse strawberry images in the morning, noon, evening, cloudy days, rainy days and sunny days respectively; marking the bounding box and key points of the ripe strawberry images; the key points include calyx key points and stem key points; the key points are divided into visible and blocked.
3. A method for detecting the posture of stacked ripe strawberry stalks based on YOLOv10-pose according to claim 2, characterized in that: The step S2 includes: adjusting the hue, saturation and brightness of the image; scaling, flipping, splicing, erasing and cropping the image.
4. A method for detecting the posture of stacked ripe strawberry stalks based on YOLOv10-pose according to claim 3, characterized in that: The YOLOv10-pose model of step S3 includes: adding a key point detection branch on the basis of the YOLOv10 model, and reconstructing the loss function of the YOLOv10-pose model; The YOLOv10-pose model uses a dual label assignment strategy, combining multi-label assignment with single-label assignment during training and single-label assignment during prediction to improve post-processing speed.
5. A method for detecting the posture of stacked ripe strawberry stalks based on YOLOv10-pose according to claim 4, characterized in that: The key point detection branch includes Con v Module, Con v Module, Con v 2d module, key point loss calculation; the Con v The module includes convolution layer, batch normalization layer, and activation function layer.
6. A method for detecting the posture of stacked ripe strawberry stalks based on YOLOv10-pose according to claim 5, characterized in that: The loss function includes: Loss Yo1ov10-pose =Loss one2many +Loss one2one Loss one2many =λ kooj Loss kobj +λ pose Loss pose +λ box Loss box +λ cls Loss cls Loss one2one =λ kobj Lossk obj +λ pose Loss pose +λ box Loss box +λ cls Loss cls Where: kobj , pose , box , cls is a constant; Where: y i is the predicted value of sample i, p i is the label value of sample i, and N is the number of samples; Among them: kpt_mask is the category of the key point; x pred and pred is the predicted coordinate of the key point; x gt and gt is the label coordinate of the key point; area is the area of the bounding box; sigmas is the scaling factor; Loss box =Loss CIOU +Loss DFL Loss DFL (S i ,S i+1 )=-((y i+1 -y)log(S i )+(y-y i )log(S i+1 )) Where: S i , S i+1 are the predicted value and near-predicted value output by the network respectively; y, y i ,y i+1 They are the actual label value, label integral value, and near label integral value respectively.
7. The method for detecting the posture of stacked ripe strawberry stalks based on YOLOv10-pose according to claim 6, characterized in that: The step S5 includes: collecting greenhouse strawberry images, inputting the images into the YOLOv10-pose model for processing, and outputting bounding boxes and key points.
8. The method for detecting the posture of stacked ripe strawberry stalks based on YOLOv10-pose according to claim 7, characterized in that: The step S6 includes: when the strawberry has a calyx key point and a stem key point, the line connecting the two key points is the posture of the strawberry stalk, and the midpoint of the line is the picking point of the strawberry.
9. The method for detecting the posture of stacked ripe strawberry stalks based on YOLOv10-pose according to claim 8, characterized in that: The step S7 includes: adjusting the posture of the end effector body according to the posture of the fruit stalk, and picking the strawberries according to the picking point.
Citation Information
Cited By
Fruit target and picking point synchronous detection method and system
CN119851266A
Medical action training method and device based on machine vision model and storage medium
CN120048004A
