A method for positioning grape picking points by combining deep learning and RGB-D information fusion
By combining deep learning and RGB-D information fusion, the YOLOv8 and Fast SCNN models were improved, achieving high-precision positioning of grape picking points. This solved the problem of low recognition accuracy of traditional machine vision methods under environmental changes, and improved picking efficiency and accuracy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHEJIANG UNIV
- Filing Date
- 2024-06-04
- Publication Date
- 2026-07-31
AI Technical Summary
Traditional machine vision methods are sensitive to environmental changes when identifying and locating grape stems, resulting in low recognition accuracy and a tendency to misclassify, which affects the positioning accuracy of the picking point.
By combining deep learning and RGB-D information fusion, the YOLOv8 model is improved by adding an SPD layer and an attention module LSKA. Combined with the Fast SCNN model, a depth camera is used to acquire RGB color stream and depth stream, and fruit stem target detection and mask generation are performed to calculate 3D coordinates and pose. Finally, a robotic arm is used for harvesting.
It improves the accuracy of grape picking point identification and positioning, can adapt to fruit and stem identification in different environments, reduces background interference, and enhances the detection performance of small targets and blurred images.
Smart Images
Figure CN118521643B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image recognition technology, and in particular to a method for locating grape picking points by combining deep learning and RGB-D information fusion. Background Technology
[0002] Traditional grape harvesting typically relies on manual labor, requiring a large workforce and exhibiting low efficiency. Furthermore, manual harvesting is susceptible to seasonal labor shortages and rising costs, posing challenges to the grape industry in terms of human resources and cost pressures.
[0003] To address the problems associated with traditional manual harvesting, automated harvesting technology has gradually become a trend in the grape industry. Automated harvesting technology utilizes advanced machine vision and deep learning algorithms to achieve rapid positioning and precise harvesting of grape bunches, thereby improving harvesting efficiency and quality while reducing labor costs and human error.
[0004] Due to the clustered growth pattern of grapes, the key to automated harvesting lies in the identification and localization of the picking point on the grape stem. Traditional machine vision methods, such as K-means clustering, color space transformation, and threshold segmentation, are used to segment the grape berries. Then, the stem region is determined based on the statistical relative positional relationship between the grape berries and the stem. Finally, methods such as Hough line detection are used to further refine the stem region and pinpoint the picking point on the stem. However, traditional machine vision methods are highly sensitive to changes in environmental conditions such as lighting, making them unsuitable for the complex and variable orchard environment.
[0005] With the development of deep learning, its application in agricultural environments has shown that deep learning methods have higher recognition accuracy and are less sensitive to environmental changes compared to traditional machine vision methods. Although progress has been made in picking point localization, accuracy has not yet reached optimal levels. Due to the small size of grape stems, the accuracy of segmenting grape stems using instance segmentation algorithms is relatively low, thus affecting the accuracy of picking point localization. Furthermore, background objects, such as grapevines and farm tools, which have high similarity to grape stems, are often misclassified as grape stems, leading to low detection recall. Summary of the Invention
[0006] The technical problem to be solved by this invention is to provide a method for locating grape picking points by combining deep learning and RGB-D information fusion, which can improve the accuracy of identification.
[0007] The technical solution adopted in this invention is: a method for locating grape picking points by combining deep learning and RGB-D information fusion, characterized by comprising the following steps:
[0008] S1. Acquire images of grapes;
[0009] S2. Preprocess the grape images acquired in step S1;
[0010] S3. Use annotation software to label grape fruits and stems;
[0011] S4. Based on the YOLOv8 model, an SPD layer and an attention module LSKA are added to obtain the YOLOv8s model. Then, the images labeled in step S3 are used for training to obtain the trained YOLOv8s model.
[0012] S5. Crop the image according to the target frame of the fruit stem marked in step S3, and scale it uniformly to the set pixels;
[0013] S6. Use annotation software to annotate the fruit stem mask on the image that has been cropped and scaled in step S5;
[0014] S7. Train the Fast SCNN model using the images labeled in step S6 to obtain the trained Fast SCNN model;
[0015] S8. Use a depth camera to simultaneously acquire the RGB color stream and depth stream of the grapes to be picked, and align the color stream and depth stream;
[0016] S9. Load the trained YOLOv8s model and perform bounding box detection on grape fruits and stems in the color stream;
[0017] S10. The color stream is cropped frame by frame according to the detected fruit stem target box, and the result is fed into the trained FastSCNN model to generate a fruit stem mask for the cropped image.
[0018] S11. Binarize the generated fruit stem mask image to obtain the three-dimensional coordinates of the picking point and the pose of the picking point.
[0019] S12. Based on the three-dimensional coordinates of the picking point and the pose at the picking point obtained in step S11, pick the fruit.
[0020] The preprocessing in step S2 includes using data augmentation tools to randomly rotate, add noise, apply Gaussian blur, randomly crop, and stretch the acquired grape images to expand the dataset.
[0021] Preferably, step S3 specifically includes the following steps:
[0022] S3.1. Mark the bounding box surrounding the grapes as "cluster", which is defined as the smallest bounding rectangle that surrounds all grapes belonging to the same bunch;
[0023] S3.2 Mark the bounding box surrounding the grape stem as "stem", which is defined as the visible part of the stem connecting the grape stalk and the grape berries.
[0024] Preferably, in step S3, the intermediate stem visible due to missing grape berries and images where more than A% of the grape berries are obscured are not marked.
[0025] Preferably, in step S4, an SPD layer is added after each convolutional layer with a stride of 2 in the YOLOv8 model.
[0026] Preferably, in step S4, an attention mechanism LSKA is added to the last two layers of the backbone network of the YOLOv8 model.
[0027] Preferably, step S11 also requires obtaining the fruit stalk length L, and step S12 requires determining whether harvesting can proceed based on the obtained fruit stalk length L and the calculated safe harvesting length L'.
[0028] Preferably, step S11 specifically includes:
[0029] S11.1. Use the skeleton extraction algorithm in the OpenCV library to extract the peduncle skeleton and obtain a series of peduncle skeleton points. In the aligned depth flow, obtain the three-dimensional point set of the skeleton point. Use the space curve fitting algorithm to fit the three-dimensional point set to obtain the three-dimensional curve of the peduncle skeleton. Use the integral function to calculate the spatial length of the curve segment, denoted as L.
[0030] S11.2 Calculate the centroid of the fruit stalk region using the centroid calculation formula, and use it as the picking point. Obtain the three-dimensional coordinates of the picking point in the aligned depth flow.
[0031] S11.3 At the picking point, calculate the tangent of the three-dimensional curve, which represents the current pose of the fruit stalk at the picking point.
[0032] Preferably, step S12 specifically includes the following steps:
[0033] Step 12.1: Calculate the safe picking length L' based on the structure and width of the end effector and the statistically determined safe picking distance range. The reference value is: L' = (L clipper +2*L statistical ), where L clipper L is the clamping width of the end effector. statistical To determine the safe distance that should be maintained near the harvesting point, the height L of the top of the end effector from the clamping shear is... height The distance from the picking point to the grape trellis is L. picking ;
[0034] Step 12.2: If L>L', and L height<L picking Send the picking point coordinates and the pose of the fruit stalk to the robotic arm control unit, perform path planning, adjust the angle and position of the end effector, and carry out picking;
[0035] Step 12.3: If L < L', or L height > L picking , abandon picking.
[0036] Compared with the prior art by using the above method, the present invention has the following advantages:
[0037] 1. The deep learning model is more robust in practical applications and can adapt to the grape fruit and fruit stalk recognition requirements under different environments. The deep learning model can learn and extract features from a large amount of data, and can identify grape fruits and fruit stalks of various shapes, colors and sizes, thus improving the recognition accuracy;
[0038] 2. The proposed cascade model developed based on the YOLOv8 object detection model and the Fast SCNN semantic segmentation model can exclude the interference of most backgrounds and improve the accuracy of fruit stalk recognition;
[0039] 3. In the proposed cascade model, the original YOLOv8 network is improved, the small target recognition ability and the detection performance of blurred images are enhanced, and the overall model accuracy is improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1 is a flowchart of a method for locating grape picking points by combining deep learning and RGB-D information fusion according to the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS
[0041] The embodiments of the present invention will be described in detail below. The examples of the embodiments are shown in the drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary and are intended to explain the present invention and should not be construed as a limitation to the present invention.
[0042] Embodiment 1:
[0043] A method for locating grape picking points by combining deep learning and RGB-D information fusion includes the following steps:
[0044] S1. Use devices such as ordinary cameras, depth cameras, mobile phones, etc. to collect grape images, and the collected images are images of multiple varieties of grapes under different lighting, viewing angles, and weather conditions;
[0045] S2. Use data augmentation tools to process the collected grape images by random rotation, adding noise, Gaussian blur, random cropping, stretching, etc., to expand the dataset;
[0046] S3. Use annotation software to annotate the target boxes for grape fruits and stems, specifically including:
[0047] S3.1. Mark the bounding box surrounding the grapes as "cluster", which is defined as the smallest bounding rectangle that surrounds all grapes belonging to the same bunch;
[0048] S3.2 Mark the bounding box surrounding the grape stem as “stem”, which is defined as the visible part of the stem connecting the grape stalk and the grape berries. This excludes the middle stem that is visible due to the absence of grape berries, and the stem of grape berries that are mostly obscured (more than 70% obscuration is considered mostly obscured), indicating that the berries cannot be harvested.
[0049] S4. Based on the original YOLOv8 deep learning object detection network, improvements are made by adding an SPD layer and an LSKA module to obtain the YOLOv8s model, which specifically includes:
[0050] S4.1. After each convolution with a stride of 2 in the YOLOv8 model, an SPD layer is added. The SPD layer slices the feature map X of size S×S×C1 into a series of sub-feature maps according to a certain scale. These sub-feature maps can be represented as:
[0051] f 0,0 =X[0:S:scale,0:S:scale],…,f scale-1,0 =X[scale-1:S:scale,0:S:scale];
[0052] …,
[0053] f 0,scale-1 =X[0:S:scale,scale-1:S:scale],…,f scale-1,scale-1
[0054] =X[scale-1:S:scale,scale-1:S:sclae]
[0055] For any feature map X, subgraph f x,y Given all X such that (i+x) and (j+y) are divisible by scale. i,j Therefore, each sub-feature map is equivalent to downsampling the feature map X by scale. Next, these sub-feature maps are concatenated along the channel dimension to form a new feature map of size X. By adding an SPD layer, feature maps can be sampled without using strided convolution and pooling operations, avoiding the loss of feature information and helping to improve the detection capabilities of blurry images and small targets;
[0056] S4.2. An attention mechanism, LSKA, is added to the last two layers of the YOLOv8 model's backbone network. LSKA is based on LKA. LKA is a novel attention mechanism that decomposes the convolutional kernel into three parts: a depthwise separable convolution (DW-Conv) of size (2d-1) × (2d-1), a depthwise separable convolution (DW-Conv) of size (2d-1) × (2d-1), and a depthwise separable convolution (DW-Conv) of size (2d-1) × (2d-1). The depthwise separable dilated convolution (DW-D-Conv) and the pointwise convolution (1×1Conv) of size (1×1) are used in LSKA. To address the issue that computational and memory usage in the depthwise separable convolutional layers of the LKA module increases quadratically with the increase of the convolution kernel size, the 2D convolution kernel of the depthwise separable convolutional layers is decomposed into cascaded horizontal and vertical 1D kernels with sizes of [missing information]. It leverages the advantages of self-attention, such as long-range dependency, and also benefits from the advantages of convolution, such as utilizing local contextual information, thereby improving overall recognition accuracy. At the same time, computational complexity and memory usage remain at a low level.
[0057] S5. Divide the dataset labeled in step S3 into training set, validation set and test set in a ratio of 7:2:1, and train the YOLOv8s model.
[0058] S6. Crop the image according to the target frame of the fruit stem marked in step S3, and scale it uniformly to 900×900 pixels.
[0059] S7. Use annotation software to annotate the fruit stem mask on the cropped and scaled image;
[0060] S8. Train the Fast SCNN model using the images labeled in step S7 to obtain the trained Fast SCNN model. The Fast SCNN model is a conventional model in existing technology, so it is not elaborated on here.
[0061] S9. Use a depth camera to photograph the grapes to be picked, obtain the RGB color stream and depth stream of the target image, and align the color stream and depth stream, that is, unify the image coordinates of the depth image and the image coordinate system of the RGB image, so that the pixel value in the depth image is the depth value of the corresponding pixel in the RGB image. The color stream is sent to the cascaded model to identify the stem and picking point of the grapes to be picked, and the depth stream is used to obtain the three-dimensional coordinates.
[0062] S10. Load the trained YOLOv8s model and perform bounding box detection on grape fruits and stems in the color stream.
[0063] S11. The color stream is cropped frame by frame according to the detected fruit stem target box, and the result is fed into the trained FastSCNN model to generate a fruit stem mask for the cropped image.
[0064] S12. Binarize the generated fruit stalk mask image, specifically including:
[0065] Step 12.1: Use the skeleton extraction algorithm in the OpenCV library to extract the peduncle skeleton and obtain a series of peduncle skeleton points. In the aligned depth flow, obtain the three-dimensional point set of the skeleton points. Use the space curve fitting algorithm to fit the three-dimensional point set to obtain the three-dimensional curve of the peduncle skeleton. Use the integral function to calculate the spatial length of the curve segment, denoted as L.
[0066] Step 12.2: Calculate the centroid of the fruit stalk region using the centroid calculation formula, and use it as the picking point. In the aligned depth flow, obtain the three-dimensional coordinates of the picking point.
[0067] Step 12.3: At the picking point, calculate the tangent of the three-dimensional curve, which represents the pose of the current fruit stalk at the picking point;
[0068] S13. Determine the relationship between the fruit stalk length L and the calculated safe picking length L':
[0069] Step 13.1: Calculate the safe picking length L' based on the structure and width of the end effector and the statistically determined safe picking distance range. The reference value is: L' = (L clipper +2*L statistical ), where L clipper L is the clamping width of the end effector. statistical To determine the safe distance that should be maintained near the harvesting point, the height L of the top of the end effector from the clamping shear is... height The distance from the picking point to the grape trellis is L. picking In this embodiment, a fixed end effector is used, so its L clipper It is a constant value, L statistical In actual calculations, this is also a constant value, L. height It is also a constant value, and since the three-dimensional coordinates of the picking point have been calculated, and the grape trellis itself is fixed, L can be obtained through calculation. picking ;
[0070] Step 13.1: If L>L', and L height <L picking This indicates that the length of the fruit stalk has reached the condition for safe picking. At this time, the picking action can be performed. The coordinates of the picking point and the pose of the fruit stalk are sent to the robotic arm control unit to perform path planning, adjust the angle and position of the end effector, and pick the fruit.
[0071] Step 13.2: If L < L’, it indicates that picking is likely to cause a collision between the end effector and the fruit, which may result in fruit damage, and picking is abandoned; if L height >L picking , it indicates that picking is likely to cause a collision between the end effector and the grape trellis, and picking is abandoned.
[0072] Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limitations on the present invention. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.
[0073] For those skilled in the art, various changes and modifications will undoubtedly be obvious after reading the above description. Therefore, the appended claims should be regarded as covering all changes and modifications that embrace the true spirit and scope of the present invention. Any and all equivalent ranges and contents within the scope of the claims should be considered to still fall within the spirit and scope of the present invention.
Claims
1. A method for positioning grape picking points by combining deep learning and RGB-D information fusion, characterized in that, It includes the following steps: S1. Acquire images of grapes; S2. Preprocess the grape images acquired in step S1; S3. Use annotation software to annotate grape fruits and stems; S4. Improve the YOLOv8 model by adding an SPD layer after each convolutional layer with a stride of 2 in the YOLOv8 model, and adding an attention mechanism LSKA to the last two layers of the backbone network of the YOLOv8 model to obtain the YOLOv8s model. Then, train the model using the images labeled in step S3 to obtain the trained YOLOv8s model. S5. Crop the image according to the target frame of the fruit stem marked in step S3, and scale it uniformly to the set pixels; S6. Use annotation software to annotate the fruit stem mask on the image that has been cropped and scaled in step S5; S7. Train the Fast SCNN model using the images labeled in step S6 to obtain the trained Fast SCNN model; S8. Simultaneously acquire the RGB color stream and depth stream of the grapes to be picked using a depth camera, and align the color stream and depth stream; S9. Load the trained YOLOv8s model and perform bounding box detection on grape fruits and stems in the color stream; S10. The color stream is cropped frame by frame according to the detected fruit stem target box, and the result is fed into the trained Fast SCNN model to generate a fruit stem mask for the cropped image. S11. Binarize the generated fruit stalk mask image to obtain the three-dimensional coordinates of the picking point and the pose at the picking point. The length L of the fruit stalk also needs to be obtained. S12. Based on the three-dimensional coordinates of the picking point and the pose of the picking point obtained in step S11, pick the fruit and determine whether picking is possible based on the obtained stem length L and the calculated safe picking length L'.
2. The method for locating grape picking points by combining deep learning and RGB-D information fusion according to claim 1, characterized in that: The preprocessing in step S2 includes using data augmentation tools to randomly rotate, add noise, Gaussian blur, randomly crop, and stretch the acquired grape images to expand the dataset.
3. The method for locating grape picking points by combining deep learning and RGB-D information fusion according to claim 1, characterized in that: Step S3 specifically includes the following steps: S3.
1. Mark the bounding box surrounding the grapes as "cluster", which is defined as the smallest bounding rectangle that encloses all grapes belonging to the same bunch; S3.2 Mark the bounding box surrounding the grape stem as "stem", which is defined as the visible part of the stem connecting the grape stalk and the grape berries.
4. The method for locating grape picking points by combining deep learning and RGB-D information fusion according to claim 3, characterized in that: In step S3, the intermediate stems that are visible due to the absence of grape berries and images where more than A% of the grape berries are obscured are not marked.
5. The method for locating grape picking points by combining deep learning and RGB-D information fusion according to claim 1, characterized in that: Step S11 specifically includes: S11.
1. Use the skeleton extraction algorithm in the OpenCV library to extract the peduncle skeleton and obtain a series of peduncle skeleton points. In the aligned depth flow, obtain the three-dimensional point set of the skeleton point. Use the space curve fitting algorithm to fit the three-dimensional point set to obtain the three-dimensional curve of the peduncle skeleton. Use the integral function to calculate the spatial length of the curve segment, denoted as L. S11.2 Calculate the centroid of the fruit stalk region using the centroid calculation formula, and use it as the picking point. Obtain the three-dimensional coordinates of the picking point in the aligned depth flow. S11.
3. At the picking point, calculate the tangent of the three-dimensional curve, which represents the pose of the current fruit stalk at the picking point.
6. The method for locating grape picking points by combining deep learning and RGB-D information fusion according to claim 1, characterized in that, Step S12 specifically includes the following steps: Step 12.1: Calculate the safe picking length L' according to the structure and width of the end effector and the statistical safe picking distance range. The reference size is: L'=(Lclipper + 2 * Lstatistical), where Lclipper is the width of the end effector's clip, Lstatistical is the safe distance that should be reserved near the picking point statistically, the height of the top of the entire end effector from the clip is Lheight, and the distance from the picking point to the grape trellis is Lpicking; Step 12.2: If L > L', and Lheight < Lpicking, send the picking point coordinates and the fruit stalk pose to the robotic arm control unit, execute path planning, adjust the angle and position of the end effector, and perform picking; Step 12.3: If L < L', or Lheight > Lpicking, abandon the picking.