A single-stage localization method for grape picking point positions in images combining detection network and point regression

Abstract

The present invention relates to a single-stage method for locating the picking point in grape images, combining a detection network and point regression. Compared to existing technologies, this method overcomes the difficulty in locating the stem position in grape images. The method includes the following steps: acquiring and preprocessing grape images; constructing a grape picking point location model; training the grape picking point location model; acquiring images of grapes to be located; and locating the picking point in the grape images. The method utilizes a detection network with point regression to detect grape stems and simultaneously determine the picking point, achieving excellent and accurate picking point location. Its simplicity, deployability, and operability far surpass those of two-stage methods, providing practical and reliable technical support for grape harvesting.

Description

Technical Field

[0001] The present invention relates to the field of agricultural picking technology, and in particular to a single-stage positioning method for grape picking points based on a grape image, combining a detection network and point regression. Background Art

[0002] Currently, grape harvesting relies primarily on manual labor, which suffers from low automation, poor timeliness, and high costs. With the application of machine vision, artificial intelligence, and robotics in horticulture, automated harvesting has become an emerging trend. Grape harvesting is typically performed by cutting the stem to avoid damage to the soft skin. Measuring the cut location within the stem, known as the picking point, is an essential and critical step in this automated harvesting process.

[0003] The location of the picking point is usually detected using color and morphological threshold methods, and then the picking point is obtained by statistical calculation. The Otsu threshold is used to separate the fruit and the stem, and then the Harris angle is obtained on the stem. The Otsu threshold and color histogram separation are used to identify tomatoes, and then morphological and geometric calculations are combined to locate the picking point. The ability to generalize and be susceptible to noise is one of the weaknesses of the fruit detection threshold method, which leads to poor robustness and low accuracy in complex environments. With the rapid development of deep learning (DL), detection networks such as region-based convolutional neural networks (MASK R-CNN), SwinTransformer and You Only Look Once (YOLO) have been applied to fruit detection and achieved good performance.

[0004] Although deep learning has improved the accuracy of fruit detection, picking point determination based on statistical calculations is still susceptible to abnormal data and has difficulty handling variability due to the need to design specific patterns for variable scenarios. When the main axis and the stem are not aligned, the picking point is difficult to locate on the stem. The picking point is obtained by calculating the curvature of the trunk, taking the midpoint if the trunk is straight, and the point with the largest image grayscale value if the trunk is curved. During the grape picking process, changes in the shape and orientation of the grape cluster, such as inverted or parallel to the gravity vector, as well as complex lighting and occlusion, limit the success of statistical calculations. At the same time, the two-stage solution is cumbersome and inefficient when revisiting the picking point location.

[0005] YOLOv5 is one of the most popular detection networks due to its excellent accuracy and real-time detection capabilities. YOLOv5s, combined with point regression, can simultaneously determine grape stem detection and picking point candidates. Furthermore, the strided convolution or pooling used by YOLOv5s can lead to loss of information about small objects, making it difficult to locate the small size of grape stems and picking points. Spatial-to-depth (SPD) demonstrates superior performance in focusing on small objects, as SPD replaces strided convolution and pooling to preserve channel information during downsampling. Summary of the Invention

[0006] The purpose of the present invention is to solve the defect in the prior art that it is difficult to locate the stem position in the grape image, and to provide a single-stage positioning method for the grape picking point position combining a detection network and point regression to solve the above problem.

[0007] In order to achieve the above object, the technical solution of the present invention is as follows:

[0008] A single-stage localization method for grape picking point positions in grape images combining a detection network and point regression includes the following steps:

[0009] Acquisition and preprocessing of grape images: Acquisition and preprocessing of grape images in the vineyard;

[0010] Construction of grape picking point positioning model: Based on the combination of YOLOv5s and point regression, a grape picking point positioning model is constructed;

[0011] Training of grape picking point positioning model: The pre-processed grape images are input into the grape picking point positioning model for training;

[0012] Acquisition of grape images to be located: Acquisition of grape images to be located and preprocessing;

[0013] Positioning of grape picking point in image: The pre-processed grape image to be positioned is input into the trained grape picking point positioning model to obtain the position of the grape picking point in the image.

[0014] The construction of the grape picking point positioning model includes the following steps:

[0015] The grape picking point positioning model is divided into a trunk module, a neck module and a head module;

[0016] Set the backbone module to CSPDarknet53;

[0017] The neck module is set to include feature pyramid network FPN and pixel aggregation network PAN. FPN combines low-level feature maps and high-level feature maps to convey deep semantic information, and PAN enables the feature map to contain image semantics and position information.

[0018] The head module is set to include a three-point head structure, in which non-maximum suppression is used to eliminate redundant bounding boxes and retain the most relevant bounding boxes for target detection. The three-point head structure provides three different scales to detect objects of different sizes, namely 20×20, 40×40 and 80×80. The head output labels with 12-dimensional vectors include confidence, classification, bounding box and three points; additional points enable the detector to output candidate picking points.

[0019] The training of the grape picking point positioning model includes the following steps:

[0020] The loss function of the grape picking point positioning model is composed of three parts: bounding box loss, classification loss, and confidence loss;

[0021] Among them, the loss function L yolo Expressed as:

[0022] L yolo =α•L conf +β·L class +γ•L bbox

[0023] Among them, α, β, and γ are the weight factors of confidence loss, classification loss, and bounding box loss; Wing-loss is used to calculate the integral loss:

[0024]

[0025] The non-negative w sets the range of the nonlinear part to (-w, w), and e limits the curvature of the nonlinear region, and C = ww • ln(1 + w / e) is a constant that smoothly connects the linear and nonlinear parts of the piecewise definition;

[0026] The loss function of the point vector s = {s i}, its basic truth value s'={s i '}, defined as:

[0027]

[0028] where i = 1, 2, ..., 6;

[0029] The new loss function of the grape picking point positioning model is set as:

[0030] Loss(s)=L yolo +λgL wing (s),

[0031] Where λ is the weight factor of the integral loss;

[0032] The backbone module obtains grape images and extracts multi-scale features of grape images;

[0033] The neck module integrates multi-scale features of grape images to locate semantic and position information;

[0034] The head module outputs labels based on the semantic and location information of the grape image and locates candidate picking points.

[0035] Beneficial effects

[0036] The present invention proposes a single-stage grape picking point positioning method that combines a detection network and point regression. Compared with the existing technology, it uses a detection network with point regression to detect grape stems and simultaneously determine the picking points, achieving good and accurate picking point positioning. Its simplicity, deployability and operability are far superior to the two-stage method, providing practical and reliable technical support for grape fruit picking.

[0037] This paper builds YOLO by combining YOLOv5s with point regression to achieve one-stage picking point localization. By integrating SPD into YOLO-R, SYOLO-R addresses the loss of small objects, enabling accurate localization in complex lighting and occlusion environments. Feature visualization also demonstrates that SYOLO-R captures dense and relevant features of the area surrounding the picking point. SYOLO-R achieves excellent and accurate picking point localization, significantly outperforming two-stage approaches in terms of simplicity, deployability, and operability. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a method sequence diagram of the present invention;

[0039] Figure 2 This is a framework diagram of the grape picking point positioning model of the present invention;

[0040] Figure 3 This is a grape image annotated by preprocessing Labelme. DETAILED DESCRIPTION

[0041] In order to provide a further understanding and appreciation of the structural features and effects achieved by the present invention, a detailed description is provided with reference to preferred embodiments and accompanying drawings as follows:

[0042] like Figure 1 As shown, the single-stage positioning method for grape picking point positions in grape images combining a detection network and point regression according to the present invention includes the following steps:

[0043] The first step is to obtain and preprocess grape images: obtain grape images in the garden and preprocess them.

[0044] Here, grapes from Daxu Vineyard in Hefei, Anhui Province, China are selected as data samples. Images obtained in different real-world scenes using a handheld digital camera (MV-CE060-10UC, Hikvision) are used as the dataset for this example. The image dataset consists of 688 images with a uniform size of 1024*1024 pixels.

[0045] Table 1 Dataset details

[0046]

[0047] The image dataset can be divided into three categories, with normal, exposed and dim light, as well as single cluster and multi-cluster grapes. 75% of the 688 images were divided into training sets and 25% were divided into validation sets. The specific types and numbers of images are shown in Table 1. Labelme was used to mark the detection area of the grape stems and candidate picking points, and the bounding box was marked with the width, height and center coordinates of the detection box. Because the grape stems are closely connected to the fruit, the area of the bounding box will contain part of the grapes. These three points are marked as candidate picking points, where the midpoint of the stem is the optimal picking point, and the other two points are the boundaries of the picking area to prevent damage to the main rootstock and fruit, as shown in Figure 1. Figure 3 shown.

[0048] The second step is to build a grape picking point positioning model: Based on the combination of YOLOv5s and point regression, a grape picking point positioning model is built, such as Figure 2 shown.

[0049] (1) The grape picking point positioning model is divided into a trunk module, a neck module and a head module.

[0050] (2) Set the backbone module to CSPDarknet53 to extract rich features of the image.

[0051] (3) The neck module is set to include feature pyramid network FPN and pixel aggregation network PAN. FPN combines low-level feature maps and high-level feature maps to convey deep semantic information, and PAN enables the feature map to contain image semantics and position information.

[0052] (4) The head module is set to include a three-point head structure, in which non-maximum suppression is used to eliminate redundant bounding boxes and retain the most relevant bounding boxes for target detection. The three-point head structure provides three different scales to detect objects of different sizes, namely 20×20, 40×40 and 80×80. The head output labels with 12-dimensional vectors include confidence, classification, bounding box and three points; the additional points enable the detector to output candidate picking points.

[0053] The third step is to train the grape picking point positioning model: input the preprocessed grape image into the grape picking point positioning model for training.

[0054] (1) The loss function of the grape picking point positioning model is set to consist of three parts: bounding box loss, classification loss, and confidence loss;

[0055] Among them, the loss function L yolo Expressed as:

[0056] L yolo =α•L conf +β·L class +γ·L bbox

[0057] Among them, α, β, and γ are the weight factors of confidence loss, classification loss, and bounding box loss; Wing-loss is used to calculate the integral loss:

[0058]

[0059] The non-negative w sets the range of the nonlinear part to (-w, w), and e limits the curvature of the nonlinear region, and C = ww • ln(1 + w / e) is a constant that smoothly connects the linear and nonlinear parts of the piecewise definition;

[0060] The loss function of the point vector s = {s i}, its basic truth value s'={s i '}, defined as:

[0061]

[0062] where i = 1, 2, ..., 6;

[0063] The new loss function of the grape picking point positioning model is set as:

[0064] Loss(s)=L yolo +λgL wing (s),

[0065] Where λ is the weight factor of the integral loss.

[0066] (2) The backbone module obtains grape images and extracts multi-scale features of grape images.

[0067] (3) The neck module integrates the multi-scale features of the grape image to locate the semantic and position information.

[0068] (4) The head module outputs labels based on the semantic and position information of the grape image and locates candidate picking points.

[0069] The fourth step is to obtain the image of the grapes to be located: obtain the image of the grapes to be located and perform preprocessing.

[0070] Step 5: Localization of grape picking point positions: Input the pre-processed grape image to be located into the trained grape picking point localization model to obtain the grape picking point positions.

[0071] The experiments were conducted on a 64-bit Ubuntu 20.4 machine with an Intel Core i7 8700K CPU @ 3.70GHz and a GeForce GTX 3090 (24GB) graphics card. The experiments were conducted using the PyTorch 1.12, CUDA 11.7, and Python 3.8 deep learning frameworks, with a batch size of 32, an input size of 512×512, and 150 training epochs.

[0072] IoU (intersection over union) is used to evaluate the correlation between the predicted bounding box and the ground truth. If the IoU is greater than or equal to 0.5, the bounding box is considered to be correctly predicted.

[0073]

[0074] Among them, Apb is the area of the predicted bounding box, and Agt is the ground truth.

[0075] Average precision (AP) is the most commonly used evaluation metric because it provides a comprehensive model evaluation. Therefore, AP is used to evaluate the accuracy of grape stem detection and picking point candidate determination. The AP value is the area under the two axes of the precision (P)-recall (R) curve. The calculation of P, R and AP is shown in the figure below. Among them, AP_0.5 and AP_0.5:0.95 mean AP of IoU = 0.5, and the average AP of IoU ranges from 0.5 to 0.95, with a step of 0.05. AP_0.5:0.95 is usually lower than AP_0.5 because the former uses a higher IoU threshold to calculate AP, that is, a stricter standard.

[0076]

[0077]

[0078] In the calculation of P, R, and AP, AP_0.5 and AP_0.5:0.95 represent AP at IoU = 0.5, and the average AP at IoU ranges from 0.5 to 0.95, with a step of 0.05. AP_0.5:0.95 is usually lower than AP_0.5 because the former uses a higher IoU threshold to calculate AP, which is a stricter standard.

[0079] In addition to accuracy, parameters and floating-point operations (FLOPs) are also used to evaluate the complexity and computational power of the network.

[0080] Parameter = KH ×K W ×C in ×C out

[0081] FLOPs = (2 × K H ×K W ×C in -1)×C out ×H out ×W out

[0082] Among them, KH and KW are the height and width of the convolution kernel, Cin and Cout are the number of input and output channels, and Hout and Wout are the height and width of the feature map.

[0083] The experiment was conducted in an open orchard with complex lighting conditions, including dim, normal, and overexposed light, various disturbances including disordered leaves and branches blocking the stems, and different stem positions, including vertical, left-bent, and right-bent. Some stems were partially obscured by leaves, making it difficult to accurately locate the picking point. Completely ambiguous picking point positioning is impossible to achieve in practical applications, so this situation was ignored in this study. In addition, dim and overexposed light may cause the stem to blend with the background, causing the stem to rearrange like other stems.

[0084] The present invention combines a detection network with point regression to develop a one-stage visual method for grape picking point positioning. Using YOLOv5s and point regression methods, YOLO-R can simultaneously detect grape stems and picking point candidates, with an AP_0.5 of 80.27% and an AP_0.5:0.95 of 42.20%. By integrating SPD into YOLO-R, SYOLO-R solves the above problems and obtains accurate and reliable first-level picking point positioning, with an AP_0.5 of 84.81% and an AP_0.5:0.95 of 45.1%. In addition, SYOLO-R's feature extraction capability for small area Darea around the picking point is verified by Grad-CAM. The method described in the present invention provides a feasible solution for the positioning of picking points in grapes and has broad application prospects in automatic grape picking.

[0085] The above shows and describes the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions merely illustrate the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the appended claims and their equivalents.

Claims

1. A single-stage localization method for grape picking points in images combining detection networks and point regression, characterized in that: The following steps are involved: 11) Acquisition and preprocessing of grape images: Acquisition and preprocessing of grape images in the vineyard; 12) Construction of grape picking point positioning model: A grape picking point positioning model was constructed based on the combination of YOLOv5s and point regression; The construction of the grape picking point positioning model includes the following steps: 121) Setting the grape picking point positioning model includes a trunk module, a neck module and a head module; 122) Set the backbone module to CSPDarknet53; 123) The neck module is set to include a feature pyramid network FPN and a pixel aggregation network PAN. FPN combines low-level feature maps and high-level feature maps to convey deep semantic information, and PAN enables the feature map to contain image semantics and position information. 124) The head module is set to include a three-point head structure, in which non-maximum suppression is used to eliminate redundant bounding boxes and retain the most relevant bounding boxes for target detection. The three-point head structure provides three different scales to detect objects of different sizes, namely 20×20, 40×40 and 80×80, and the head output label with a 12-dimensional vector includes confidence, classification, bounding box and three points; the additional point enables the detector to output candidate picking points; 13) Training the grape picking point positioning model: The pre-processed grape image is input into the grape picking point positioning model for training; the training of the grape picking point positioning model includes the following steps: 131) The loss function of the grape picking point positioning model is set to consist of three parts: bounding box loss, classification loss, and confidence loss; 132) The backbone module obtains a grape image and extracts multi-scale features of the grape image; 133) The neck module integrates the multi-scale features of grape images to locate semantic and position information; 134) The head module outputs labels based on the semantic and position information of the grape image and locates candidate picking points; 14) Acquiring the image of grapes to be located: Acquiring the image of grapes to be located and performing preprocessing; 15) Positioning of grape picking points in images: The pre-processed grape images to be positioned are input into the trained grape picking point positioning model to obtain the positions of the grape picking points in the images.

2. The single-stage positioning method for grape picking point positions based on a combination of a detection network and point regression according to claim 1 is characterized in that: Loss function L yolo Expressed as: L yolo =α·L conf +β·L class +γ·L bbox Among them, α, β, and γ are the weight factors of confidence loss, classification loss, and bounding box loss; Wing-loss is used to calculate the integral loss: The non-negative w sets the range of the nonlinear part to (-w, w), and e limits the curvature of the nonlinear region, and C = ww ln(1 + w / e) is a constant that smoothly connects the linear and nonlinear parts of the piecewise definition; The loss function of the point vector s = {s i }, its basic truth value s'={s i '}, defined as: where i = 1, 2, ..., 6; The new loss function of the grape picking point positioning model is set as: Loss(s)=L yolo +λ·L wing (s), Where λ is the weight factor of the integral loss.

Images