A WGB-YOLO-based multi-class pitaya detection method in a picking line

Abstract

The application discloses a kind of target picking row multi-class pitaya detection method based on WGB-YOLO, comprising the following steps: step one, using depth camera to shoot pitaya RGB image and depth image;Step two, mark image;Step three, using data enhancement to expand the pitaya dataset marked, and it is divided into training set and verification set;Step four, construct WGB-YOLO network model;Step five, using training set to train WGB-YOLO network model, obtains WGB-YOLO pitaya detection model;Step six, using verification set to verify the detection precision of WGB-YOLO pitaya detection model;Step seven, WGB-YOLO pitaya detection model meeting the requirement of detection precision is deployed to the mobile terminal system of densely planted orchard fruit picking robot, for detecting the multi-class pitaya of target picking row.The application improves the precision of pitaya multi-class detection and identification method to detect target picking row fruits, and can distinguish whether fruit is the fruit of target picking row.

Description

Technical Field

[0001] This invention relates to the fields of target recognition technology and smart agriculture technology, specifically a method for detecting multiple types of dragon fruit in target picking rows based on WGB-YOLO. Background Technology

[0002] With the rapid development of smart agriculture and deep learning, dragon fruit detection and identification, as well as robotic harvesting, have become research hotspots. Accurate detection and identification of dragon fruit is of great significance for advancing robotic harvesting and automated orchard management.

[0003] Because dragon fruit is densely planted in commercial orchards, there are two detection requirements for the detection and positioning of dragon fruit for robotic harvesting: one is to identify only the dragon fruit in the current target harvesting row; the other is to classify the dragon fruit according to the occlusion situation to facilitate the selection of harvesting strategies.

[0004] To achieve intelligent detection of dragon fruit in the current picking row, the existing YOLOv4-LITE network model detection and recognition technology can accurately detect all dragon fruit in the captured image with high accuracy. However, this network cannot distinguish whether the fruit belongs to the current target picking row, nor can it classify the detected fruit, which will bring huge obstacles to the fruit robot's dragon fruit picking. Summary of the Invention

[0005] Purpose of the invention: To provide a method for detecting multiple types of dragon fruit in target picking rows based on WGB-YOLO, which can effectively solve the problems of low accuracy, large detection model size and high computational complexity of existing dragon fruit multi-class detection and recognition methods, and promote the deployment and application of dragon fruit multi-class detection and recognition models on mobile robots.

[0006] Technical solution: A method for detecting multiple types of dragon fruit in a target harvesting row based on WGB-YOLO, comprising the following steps:

[0007] Step 1: In the orchard, use a depth camera to capture RGB and depth images of the dragon fruit;

[0008] Step 2, Image Labeling: Label the RGB images of dragon fruit taken in Step 1, and distinguish the various types of dragon fruit in the target picking row into fruits in the target picking row and fruits in the non-target picking row.

[0009] Step 3: Augment the labeled dragon fruit dataset using data augmentation and divide it into training and validation sets;

[0010] Step 4: Construct the WGB-YOLO network model;

[0011] Step 5: Train the WGB-YOLO network model using the training set to obtain the WGB-YOLO dragon fruit detection model;

[0012] Step 6: Verify the detection accuracy of the WGB-YOLO dragon fruit detection model using the validation set. If the detection accuracy requirement is met, proceed to the next step; otherwise, repeat steps 1 to 6 until a WGB-YOLO dragon fruit detection model that meets the detection accuracy requirement is obtained.

[0013] Step 7: Deploy the WGB-YOLO dragon fruit detection model, which meets the detection accuracy requirements, onto the mobile terminal system of the fruit picking robot in the densely planted orchard for detecting multiple types of dragon fruit in the target picking row.

[0014] Furthermore, in step one, the depth camera used is a D435i depth camera.

[0015] Furthermore, step two, specifically labeling the RGB image, includes the following steps:

[0016] (1) Based on the on-site picking needs of the robot orchard and the shading of dragon fruit, the fruit categories are divided into ON, FCC, OB and NF; among them, ON represents unshaded fruit, FCC represents connected fruit, OB represents shaded fruit, and NF represents fruit in the non-target picking row.

[0017] (2) Use labelIMG software to label all fruits in the RGB image, classify the fruit categories as ON, FCC or OB, and generate an XML file;

[0018] (3) Based on the depth pixel unit I of the depth camera and the depth image pixel P M Calculate the distance D between image pixels. M :

[0019] D M =I·P M ;

[0020] (4) Set the threshold distance for non-target picking rows according to the row spacing of dragon fruit in the orchard; if the row spacing of dragon fruit in the orchard is L, then the threshold distance for non-target picking rows is set to L / 2.

[0021] (5) Read all fruit bounding boxes from the XML file of an RGB image, and calculate the distance D from the corresponding image pixel. M Read the distance information of each pixel within the bounding box (bbox) i :

[0022]

[0023] In the formula, i is the index of the fruit's bounding box in an RGB image. These are the coordinates of the top left corner of the annotation box. These are the coordinates of the bottom right corner of the annotation box;

[0024] (6) Calculate the proportion r of pixels within each bounding box whose distance is greater than the threshold distance. i If the ratio r i If the percentage is greater than 50%, then the label of the fruit's annotation box in the XML file of that image should be changed to NF.

[0025]

[0026]

[0027] In the formula, n and m are the horizontal and vertical position indices of the pixel distance information within the i-th bounding box, respectively; bbox i It represents the distance information of pixels within the i-th bounding box; Ibbox i It's bbox i Compare the calculated binarized result with L / 2;

[0028] (7) Perform steps (5) to (6) on each RGB image until all are completed.

[0029] Furthermore, in step four, the method for constructing the WGB-YOLO network model includes the following steps:

[0030] 1) Construct the backbone network of the WGB-YOLO network model using Conv and WFE-C4, and sequentially compress the feature map scale by 1 / 2 using 5 sets of Conv and WFE-C4 to obtain feature maps P1 to P5 at different scales;

[0031] 2) A weighted bidirectional feature pyramid network is used to build a Head network based on Conv, GF-SPPF and WFE-C4 to achieve efficient extraction of image feature layers;

[0032] 3) The WGB-YOLO network constructed in step 2) uses the same configuration for its detection head and loss function as the YOLOv3 network.

[0033] 5. A method for detecting multiple types of dragon fruit in a target picking row based on WGB-YOLO according to claim 4, characterized in that the WFE-C4 is built on the Bottleneck network and mainly includes the following steps:

[0034] a) Construct the upper wing network structure;

[0035] The upper wing enhances features from the perspective of feature channels. First, global average pooling is performed to compress the feature map size from C×H×W to C×1×1, and conv1d is calculated. Then, it is multiplied with the input feature map to obtain the upper wing channel enhanced feature W. up :

[0036]

[0037] Conv k,c (X)=SeLU(BN(conv2d k,c (X)));

[0038]

[0039] Where X is the feature map, GAP is global average pooling, h is the height of the feature map, w is the width of the feature map, k is the kernel size, SeLU is the scaling exponential linear unit, BN is batch normalization, and C, H and W are the number of channels, height and width of the feature map, respectively.

[0040] b) Construct the lower wing network structure;

[0041] The lower wing enhances features from a spatial perspective. First, the feature map is averaged across its channels. Then, a conv2d kernel of size 7x7 is used to calculate the channel attention features. This channel attention feature is multiplied by the input feature map to obtain the lower wing spatial enhancement feature W. down :

[0042]

[0043]

[0044] c) The two wing enhancement features are concatenated with the main channel Bottleneck network in one-dimensional concatenation, then batch normalization and MetaAconC activation function calculation are performed, followed by a convolution calculation to obtain the output feature F. WFE -C4:

[0045]

[0046]

[0047] F WFE-C4 =Conv k＝1 (MetaAconC(BN(X c )));

[0048] Where || is the concatenation operator, Btn is the Bottleneck network, W1 and W2 are the weights to be calculated, p1 and p2 are the learning parameters to be adjusted, and β is the adaptive function.

[0049] Furthermore, in step four, GF-SPPF is obtained by improving upon SPP, and mainly includes the following steps:

[0050] (a) In the GF-SPP network, average pooling with a kernel size of 5x5 is used instead of max pooling with a kernel size of 5x5 in the SPP network;

[0051] (b) Global average pooling is used instead of max pooling with a kernel size of 13x13. After two convolution calculations with a kernel size of 1x1 and sigmoid activation function calculations, the global enhanced feature G is obtained. F :

[0052] G F (X)=Conv k＝1,c＝c1 (δ(Conv k＝1,c＝c1 / 2 (GAP(X))));

[0053] (c) Feature maps after average pooling and max pooling, and global enhancement feature G F Perform dot product calculations to obtain the enhanced feature maps (AGs) after average pooling and max pooling. F With MG F :

[0054]

[0055]

[0056] (d) Original input feature map, AG F With MG F The output feature map F is obtained after concatenation along dimension 1. GF -SPP:

[0057]

[0058] Beneficial effects:

[0059] This invention improves the accuracy of multi-class dragon fruit detection in target picking rows based on WGB-YOLO, enabling the identification of dragon fruit in target picking rows and distinguishing whether a fruit belongs to the target picking row. This promotes the deployment and application of dragon fruit detection models on orchard robots. Attached Figure Description

[0060] Figure 1 This is a flowchart of the method for detecting multiple types of dragon fruit in the target harvesting row based on WGB-YOLO, as described in this invention;

[0061] Figure 2 This is a flowchart illustrating the various types of dragon fruit in the current target picking row in the image labeled in this invention.

[0062] Figure 3 This is an architecture diagram of the WGB-YOLO network model of the present invention;

[0063] Figure 4 This is a structural diagram of the WFE-C4 network of the present invention;

[0064] Figure 5 This is a structural diagram of the GF-SPP network of the present invention;

[0065] Figure 6 The figure shows the comparison results of the AP performance of WGB-YOLO and eight other deep learning models in detecting multiple types of dragon fruit in commercial densely planted orchards.

[0066] Figure 7 The figure shows the comparison results of the mAP performance of WGB-YOLO and eight other deep learning models in detecting multiple types of dragon fruit in commercial densely planted orchards.

[0067] Figure 8 This image shows a comparison between the WGB-YOLO model of this invention and eight other deep learning models for detecting multiple types of dragon fruit in commercial densely planted orchards. Detailed Implementation

[0068] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings, but the scope of protection of the present invention is not limited to the embodiments.

[0069] This invention proposes a method for detecting multiple types of dragon fruit in target picking rows based on WGB-YOLO, such as... Figure 1 As shown, the main steps are as follows:

[0070] Step 1: In the orchard, use a D435i depth camera to capture RGB and depth images of dragon fruit.

[0071] Step 2: Label the RGB images: Label the dragon fruit RGB images taken in Step 1 to distinguish the various types of dragon fruit in the target picking row into fruits in the target picking row and fruits in the non-target picking row.

[0072] like Figure 2 As shown, step two, labeling the RGB image, specifically includes the following steps:

[0073] (1) Based on the on-site picking needs of the robot orchard and the shading situation of dragon fruit, the fruit categories are divided into ON (unshaded fruit), FCC (connected fruit), OB (shaded fruit) and NF (non-target picking row fruit).

[0074] (2) LabelIMG software was used to label all fruits in the RGB image, classifying the fruits into ON, FCC and OB categories, and generating an XML file;

[0075] Since the distance between the fruit and the camera cannot be determined from the RGB image, it is impossible to determine whether the fruit is from the target picking row. Therefore, this step does not distinguish whether the fruit is an NF fruit.

[0076] (3) Based on the depth pixel unit I and depth image pixel P of the D435i depth camera M Calculate the distance D between image pixels. M :

[0077] D M =I·P M

[0078] (4) Set the threshold distance for non-target picking rows based on the row spacing of dragon fruit planted in the orchard;

[0079] If the row spacing for planting dragon fruit in the orchard is L (L=2.1m), then the threshold distance for non-target picking rows is set to L / 2;

[0080] (5) Read all fruit bounding boxes from the XML file of an RGB image, and calculate the distance D from the corresponding image pixel. M Read the distance information of each pixel within the bounding box (bbox) i :

[0081]

[0082] In the formula, i is the index of the fruit's bounding box in an RGB image. These are the coordinates of the top left corner of the annotation box. These are the coordinates of the bottom right corner of the annotation box;

[0083] (6) Calculate the proportion r of pixels within each bounding box whose distance is greater than the threshold distance. i If the ratio r i If the percentage is greater than 50%, then the label of the fruit's annotation box in the XML file of the RGB image should be changed to NF.

[0084]

[0085]

[0086] In the formula, n and m are the horizontal and vertical position indices of the pixel distance information within the i-th bounding box, respectively; bbox i It represents the distance information of pixels within the i-th bounding box; Ibbox i It's bbox i Compare the calculated binarized result with L / 2;

[0087] (7) Perform steps (5) to (6) on each RGB image until all are completed.

[0088] Step 3: Augment the labeled dragon fruit dataset using data augmentation and divide it into training and validation sets; specifically, the training and validation sets can be divided in an 8:2 ratio.

[0089] Step 4: Construct the WGB-YOLO network model;

[0090] like Figure 3 As shown, the method for constructing the WGB-YOLO network model mainly includes the following steps:

[0091] 1) Construct the backbone network of the WGB-YOLO network model using Conv and WFE-C4, and sequentially compress the feature map scale by 1 / 2 using 5 sets of Conv and WFE-C4 to obtain feature maps P1 to P5 at different scales;

[0092] 2) Based on Conv, GF-SPPF and WFE-C4, a weighted bidirectional feature pyramid network (Bi-FPN) structure is used to build a head network to achieve efficient extraction of image feature layers;

[0093] 3) The WGB-YOLO network constructed in step 2) uses the same configuration for its detection head and loss function as the YOLOv3 network.

[0094] In order to be suitable for use on mobile terminals, the WGB-YOLO-s network model can also be adopted. WGB-YOLO-s is a lightweight model of WGB-YOLO. The network architecture is the same as the WGB-YOLO network model, except that the number of network channels is reduced by half and the number of WFE-C4 is reduced to 2 / 3 of the original.

[0095] like Figure 4 As shown, the WFE-C4 is built on the Bottleneck network and mainly includes the following steps:

[0096] a) Construct the upper wing network structure;

[0097] The upper wing enhances features from the perspective of feature channels. First, global average pooling is performed to compress the feature map size from C×H×W to C×1×1, and conv1d is calculated. Then, it is multiplied with the input feature map to obtain the upper wing channel enhanced feature W. up .

[0098]

[0099] Conv k,c (X)=SeLU(BN(conv2d k,c (X)))

[0100]

[0101] Here, X is the feature map, GAP is global average pooling, h is the height of the feature map, w is the width of the feature map, k is the kernel size, SeLU is scaled exponential linear unit, BN is batch normalization, and C, H and W are the number of channels, height and width of the feature map, respectively.

[0102] b) Construct the lower wing network structure;

[0103] The lower wing enhances features from a spatial perspective. First, the feature map is averaged across its channels. Then, a conv2d kernel of size 7x7 is used to calculate the channel attention features. This channel attention feature is multiplied by the input feature map to obtain the lower wing spatial enhancement feature W. down :

[0104]

[0105]

[0106] c) The two wing enhancement features are concatenated with the main channel Bottleneck network in one-dimensional concatenation, then batch normalization and MetaAconC activation function calculation are performed, followed by a convolution calculation to obtain the output feature F. WFE-C4 .

[0107]

[0108]

[0109] F WFE-C4 =Conv k＝1 (MetaAconC(BN(X c )))

[0110] Where || is the concatenation operator, Btn is the Bottleneck network, W1 and W2 are the weights to be calculated, p1 and p2 are the learning parameters to be adjusted, and β is the adaptive function.

[0111] like Figure 5 As shown, the GF-SPPF is an improvement on SPP, and mainly includes the following steps:

[0112] (a) In the GF-SPP network, average pooling with a kernel size of 5x5 is used instead of max pooling with a kernel size of 5x5 in the SPP network;

[0113] (b) Global average pooling is used instead of max pooling with a kernel size of 13x13. After two convolution calculations with a kernel size of 1x1 and sigmoid activation function calculations, the global enhanced feature G is obtained. F :

[0114] G F (X)=Conv k＝1,c＝c1 (δ(Conv k＝1,c＝c1 / 2 (GAP(X))))

[0115] (c) Feature maps after average pooling and max pooling, and global enhancement feature G F Perform dot product calculations to obtain the enhanced feature maps (AGs) after average pooling and max pooling. F With MG F :

[0116]

[0117]

[0118] (d) Original input feature map, AG F With MG F The output feature map F is obtained after concatenation along dimension 1. GF-SPP :

[0119]

[0120] Step 5: Train the WGB-YOLO network model using the training set to obtain the WGB-YOLO dragon fruit detection model.

[0121] Step 6: Verify the detection accuracy of the WGB-YOLO dragon fruit detection model using the validation set. If the detection accuracy requirement is met, proceed to the next step; otherwise, repeat steps 1 to 6 until a WGB-YOLO dragon fruit detection model that meets the detection accuracy requirement is obtained.

[0122] Step 7: Deploy the WGB-YOLO dragon fruit detection model, which meets the detection accuracy requirements, onto the mobile terminal system of the fruit picking robot in the densely planted orchard for detecting multiple types of dragon fruit in the target picking row.

[0123] The performance comparison of the WGB-YOLO dragon fruit detection model constructed using the method of this invention with dragon fruit detection models constructed using eight other deep learning models (Faster RCNN, VFnet, SSD, FCOS, YOLOv4, YOLOv5-1, YOLOX-1, and YOLOv7) in detecting multiple types of dragon fruit in commercial densely planted orchards is as follows: Figures 6-7As shown, the WGB-YOLO dragon fruit detection model achieved the highest average precision (AP) for detecting ON, FCC, and OB fruits. The mean average precision (mAP) for detecting multiple types of dragon fruit also achieved the highest value, demonstrating the superior detection performance of the WGB-YOLO dragon fruit detection model.

[0124] The WGB-YOLO dragon fruit detection model, built with WGB-YOLO and eight other deep learning models, is compared in the following example image: [Image showing comparison of multiple dragon fruit varieties detected in a commercially densely planted orchard]. Figure 8 As shown in the comparison example, it can be seen that the other 8 deep learning models all made mistakes in detecting dragon fruit (circles indicate detection errors, solid squares represent ON fruits, dotted-line squares represent OB fruits, and double-dash squares represent FCC fruits), while the WGB-YOLO model could correctly detect all fruits.

[0125] As described above, although the invention has been shown and described with reference to specific preferred embodiments, it should not be construed as limiting the invention itself. Various changes in form and detail may be made without departing from the spirit and scope of the invention as defined in the appended claims.

Claims

1. A method for detecting multiple types of dragon fruit in a target picking row based on WGB-YOLO, characterized in that, Includes the following steps: Step 1: In the orchard, use a depth camera to capture RGB and depth images of the dragon fruit; Step 2, Image Labeling: Label the RGB images of dragon fruit taken in Step 1, and distinguish the various types of dragon fruit in the target picking row into fruits in the target picking row and fruits in the non-target picking row. Step 3: Augment the labeled dragon fruit dataset using data augmentation and divide it into training and validation sets; Step 4: Construct the WGB-YOLO network model; Step 5: Train the WGB-YOLO network model using the training set to obtain the WGB-YOLO dragon fruit detection model; Step 6: Verify the detection accuracy of the WGB-YOLO dragon fruit detection model using the validation set. If the detection accuracy requirement is met, proceed to the next step; otherwise, repeat steps 1 to 6 until a WGB-YOLO dragon fruit detection model that meets the detection accuracy requirement is obtained. Step 7: Deploy the WGB-YOLO dragon fruit detection model that meets the detection accuracy requirements onto the mobile terminal system of the fruit picking robot in the densely planted orchard for detecting multiple types of dragon fruit in the target picking row; Step four, the method for constructing the WGB-YOLO network model, includes the following steps: 1) Construct the backbone network of the WGB-YOLO network model using Conv and WFE-C4, and sequentially compress the feature map scale by 1 / 2 using 5 sets of Conv and WFE-C4 to obtain feature maps P1~P5 at different scales; 2) A weighted bidirectional feature pyramid network is used to build a Head network based on Conv, GF-SPP and WFE-C4 to achieve efficient extraction of image feature layers; 3) The WGB-YOLO network constructed in step 2) uses the same configuration for its detection head and loss function as the YOLOv3 network; The WFE-C4 is built on the Bottleneck network and includes the following steps: a) Construct the upper wing network structure; The upper wing enhances features from the perspective of feature channels. First, global average pooling is performed to compress the feature map size from C×H×W to C×1×1, and conv1d is calculated. Then, it is multiplied with the input feature map to obtain the upper wing channel enhanced feature W. up : ； ； ； Where X is the feature map, GAP is global average pooling, h is the height of the feature map, w is the width of the feature map, k is the kernel size, SeLU is the scaling exponential linear unit, BN is batch normalization, and C, H and W are the number of channels, height and width of the feature map, respectively. b) Construct the lower wing network structure; The lower wing enhances features from a spatial perspective. First, the feature map is averaged across its channels. Then, a conv2d kernel of size 7x7 is used to calculate the channel attention features. This channel attention feature is multiplied by the input feature map to obtain the lower wing spatial enhancement feature W. down : ； ； c) The two wing enhancement features are concatenated with the main channel Bottleneck network in one-dimensional concatenation, then batch normalization and MetaAconC activation function calculation are performed, followed by a convolution calculation to obtain the output feature F. WFE-C4 : ； ； ； Where || is the concatenation operator, Btn is the Bottleneck network, W1 and W2 are the weights to be calculated, and p1 and p2 are the learning parameters to be adjusted; In step four, GF-SPP is obtained by improving upon SPP, and includes the following steps: (a) In the GF-SPP network, average pooling with a kernel size of 5x5 is used instead of max pooling with a kernel size of 5x5 in the SPP network; (b) Global average pooling is used instead of max pooling with a kernel size of 13x13. Then, after two convolution calculations with a kernel size of 1x1 and sigmoid activation function calculations, the global enhanced feature G is obtained. F : ； (c) Feature maps after average pooling and max pooling and global enhancement feature G F Perform dot product calculations to obtain the enhanced feature maps (AGs) after average pooling and max pooling. F With MG F : ； ； (d) Original input feature map, AG F With MG F The output feature map F is obtained after concatenation along dimension 1. GF-SPP : 。 2. The method for detecting multiple types of dragon fruit in a target picking row based on WGB-YOLO according to claim 1, characterized in that, In step one, the depth camera used is a D435i depth camera.

3. The method for detecting multiple types of dragon fruit in a target picking row based on WGB-YOLO according to claim 1, characterized in that, Step two, specifically, includes the following steps for labeling the RGB image: (1) Based on the on-site picking needs of the robot orchard and the shading of dragon fruit, the fruit categories are divided into ON, FCC, OB and NF; among them, ON represents unshaded fruit, FCC represents connected fruit, OB represents shaded fruit, and NF represents fruit in the non-target picking row. (2) Use labelIMG software to label all fruits in the RGB image, classify the fruit categories as ON, FCC or OB, and generate an XML file; (3) Based on the depth pixel unit I of the depth camera and the depth image pixel P M Calculate the distance D between image pixels. M : ； (4) Set the threshold distance for non-target picking rows according to the row spacing of dragon fruit in the orchard; if the row spacing of dragon fruit in the orchard is L, then the threshold distance for non-target picking rows is set to L / 2. (5) Read all fruit bounding boxes from an XML file of an RGB image, and calculate the distance D from the corresponding image pixel. M Read the distance information of each pixel within the bounding box (bbox) i : ； In the formula, i is the index of the fruit's bounding box in an RGB image. These are the coordinates of the top left corner of the annotation box. These are the coordinates of the bottom right corner of the annotation box; (6) Calculate the proportion r of pixels within each bounding box whose distance is greater than the threshold distance. i If the ratio r i If the percentage is greater than 50%, then the label of the fruit's bounding box in the image's XML file should be changed to NF. ； ； In the formula, n and m are the horizontal and vertical position indices of the pixel distance information within the i-th bounding box, respectively; bbox i It represents the distance information of pixels within the i-th bounding box; Ibbox i It's bbox i Compare the calculated binarized result with L / 2; (7) Perform steps (5) to (6) on each RGB image until all are completed.

Images