A pitaya intelligent detection and identification method in a target picking row of a dense planting orchard

Abstract

The application discloses a kind of compact planting orchard target picking line pitaya intelligent detection identification method, including the parameter of depth camera is set and gathers compact planting orchard pitaya image, target picking line pitaya fruit target frame in image is marked and data set is made, adopts MBSF network training target picking line pitaya detection model, whether the MBSF model trained satisfies the requirement of target picking line pitaya detection precision: if not satisfied, increase the number of image in data set and adjust MBSF model parameter, re-execute step S2 and S3;If satisfied, then execute next step, deploy the MBSF model trained to pitaya picking robot detection system, detect and identify the pitaya of target picking line in orchard, greatly reduce the malfunction that fruit picking robot executes picking non-target pitaya.

Description

Technical Field

[0001] This invention relates to the fields of target recognition technology and smart agriculture technology, specifically to a method for intelligent detection and recognition of dragon fruit in the target picking rows of densely planted orchards. Background Technology

[0002] With the rapid development of smart agriculture and deep learning, dragon fruit detection and identification, as well as robotic harvesting, have become research hotspots. Accurate detection and identification of dragon fruit is of great significance for advancing robotic harvesting and automated orchard management.

[0003] Because dragon fruit is densely planted in commercial orchards, dragon fruit in non-target picking rows severely interferes with the detection of dragon fruit in the target picking rows. These non-target dragon fruit rows are unsuitable for harvesting robots. Previous studies detecting dragon fruit in images identified all dragon fruit in the image, without distinguishing whether a particular dragon fruit was within the target row. However, if the harvesting robot identifies dragon fruit in non-target picking rows as harvesting targets, it can cause the robot to malfunction and fail to perform the harvesting operation, or lead to a collision between the robot's end effector and the dragon fruit trees in the target picking row.

[0004] To achieve intelligent detection of dragon fruit, the existing YOLOv4-LITE network model technology for detecting and recognizing dragon fruit can accurately detect all dragon fruit in the captured image with high accuracy. However, this detection network model cannot distinguish dragon fruit that is not in the target picking row, which will bring huge obstacles to the harvesting of dragon fruit by fruit robots. Summary of the Invention

[0005] This invention provides an intelligent detection and identification method for dragon fruit in the target picking row of a densely planted orchard. It solves the problem of existing dragon fruit detection and identification methods erroneously detecting fruits in non-target picking rows, thereby effectively solving the problem of fruit picking robots erroneously picking fruits in non-target rows, resulting in failure to complete the picking, as well as the problem of collision between the end effector and the dragon fruit tree in the target picking row.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a method for intelligent detection and identification of dragon fruit in the target picking row of a densely planted orchard, comprising the following steps:

[0007] S1. Set the parameters of the depth camera and acquire images of dragon fruit in the densely planted orchard;

[0008] S2. Label the target bounding boxes of dragon fruit in the picking row in the image, and at the same time create the dataset;

[0009] S3. A dragon fruit detection model for target picking rows is trained using the MBSF network;

[0010] S4. Check whether the trained MBSF model meets the accuracy requirements for dragon fruit detection in the target picking row. If it does not meet the requirements, increase the number of images in the dataset and adjust the MBSF model parameters. Repeat steps S2 and S3. If it meets the requirements, proceed to the next step.

[0011] S5. Deploy the trained MBSF model to the detection system of the dragon fruit picking robot to detect and identify the dragon fruit in the target picking row in the orchard.

[0012] Preferably, step S1 includes the following sub-steps:

[0013] 1) Place the depth camera in the middle of the adjacent rows of dragon fruit, set the spacing between adjacent rows of dragon fruit to l, the height of the camera above the ground to be half the height h of the dragon fruit tree, and place the camera vertically so that the depth camera's field of view covers the height range of the entire dragon fruit tree, so that all fruits are in the image, and the dragon fruit fruits are fixed to the dragon fruit tree support stakes 5 on both sides by the provided support line 6.

[0014] 2) Align the depth image of the depth camera with the pixels of the RGB image;

[0015] 3) The depth camera captures images between the rows of dragon fruit, simultaneously acquiring RGB images of the dragon fruit and their corresponding depth images;

[0016] 4) The depth camera moves a certain distance along the row direction between adjacent rows of dragon fruit to ensure that images taken in two adjacent shots do not overlap;

[0017] 5) If the number of dragon fruit images collected meets the requirements for training the deep learning model, stop collecting images; otherwise, repeat steps 3) and 4).

[0018] Preferably, step S2 includes the following sub-steps:

[0019] 1) Combining the depth image imgD with the depth pixel unit I of the depth camera d The depth distance D corresponding to the pixel in the RGB image is obtained. RGB Its formula is:

[0020] D RGB (i,j)=imgD(i,j)·I d , (i=1,2,…,H; j=1,2,…,W)

[0021] In the formula, D RGB It represents the depth distance of a pixel in the RGB image from the depth camera in the Z direction; imgD represents the number of pixels in the depth image; I dis the distance unit of the depth image pixel value, i is the pixel index number in the height direction of the image, j is the pixel index number in the width direction of the image, and H and W are the height and width of the image, respectively.

[0022] 2) Use depth-distance threshold D t Background information from non-target picking rows is removed, including dragon fruit trees, dragon fruit, and other interfering backgrounds. The removed background information in the RGB image is filled with gray, using the following formula:

[0023]

[0024] In the formula, Rb_RGB is the RGB image after removing the background; Ori_RGB is the RGB image; gray_color is the gray pixel value; D t It is the depth distance threshold, which is half the spacing between the planting rows of dragon fruit plants;

[0025] 3) Label the dragon fruit on the background-removed RGB image and generate an Rb_XML file;

[0026] 4) Modify the filename attribute in the Rb_XML file to the RGB image name to generate the target row fruit annotation file XML for the RGB image;

[0027] 5) Create a dragon fruit dataset for the target picking rows in the orchard using RGB images and their XML files in a 7:1.5:1.5 ratio.

[0028] Preferably, in step S3, the MBSF network architecture construction step includes the following sub-steps:

[0029] 1) Scale the images to a uniform size of 640×640×3;

[0030] 2) The scaled image is processed by the CBL layer to obtain the feature map C1, where the CBL is composed of a concatenated convolutional layer, a batch normalization layer and a Leaky ReLU;

[0031] 3) Feature map C1 is calculated in MBlock layer three times in a row to obtain feature maps C2 to C4 in sequence, where MBlock layer is Blocknet of MobilenetV3;

[0032] 4) Feature map C4 is processed sequentially using the MBlock layer and the spatial pyramid network to obtain feature map C5;

[0033] 5) Input feature maps C3 to C5 into the feature pyramid network that incorporates the SEnet attention mechanism to further obtain enhanced feature maps;

[0034] 6) Enhance the feature map input to the Anchor-free detector to predict the category and location of dragon fruit; and output the dragon fruit detection results.

[0035] Compared with the prior art, the beneficial effects of the present invention are as follows: The present invention provides an intelligent detection and identification method for dragon fruit in the target picking row of a densely planted orchard. It can intelligently detect and identify dragon fruit in the target picking row of a densely planted orchard, and can prevent dragon fruit in non-target picking rows from being mistakenly identified as dragon fruit in the target picking row. This greatly reduces the chance of malfunctions caused by fruit picking robots picking non-target dragon fruit, or the robot's end effector hitting the dragon fruit tree in the target picking row. Attached Figure Description

[0036] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0037] In the attached diagram:

[0038] Figure 1 This is an overall flowchart of the intelligent detection and identification method for dragon fruit in the target picking row of a densely planted orchard according to the present invention;

[0039] Figure 2 This is a schematic diagram of the process for marking the target picking row of dragon fruit in the image of the present invention;

[0040] Figure 3 This is a schematic diagram of the shooting position of the depth camera of the present invention;

[0041] Figure 4 This is a schematic diagram of the target harvesting rows and non-target harvesting rows of dragon fruit in a densely planted orchard according to the present invention;

[0042] Figure 5 This is a flowchart of the depth camera parameter setting and dragon fruit image acquisition in a densely planted orchard according to the present invention;

[0043] Figure 6 This is the MBSF network architecture diagram of the present invention;

[0044] The labels in the image are: 1. Dragon fruit tree; 2. Dragon fruit; 3. Depth camera; 4. Camera bracket; 5. Dragon fruit tree support stake; 6. Support line; 7. Target picking row; 8. Non-target picking row. Detailed Implementation

[0045] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0046] Example: Figure 1As shown, a method for intelligent detection and identification of dragon fruit in the target picking row of a densely planted orchard mainly includes the following steps:

[0047] S1. Set the parameters of depth camera 3 and acquire images of dragon fruit in a densely planted orchard, such as... Figure 2 As shown, the specific steps include the following:

[0048] 1) such as Figure 3 and Figure 4 As shown, the depth camera 3 is placed in the middle of the adjacent rows of dragon fruit trees and mounted on the camera bracket 4. The spacing between the adjacent rows of dragon fruit trees is set to l. The height of the depth camera 3 above the ground is half the height h of the dragon fruit tree 1. The depth camera 1 is placed vertically so that the field of view of the depth camera 3 covers the height range of the entire dragon fruit tree 1, so that all fruits are in the image. The dragon fruit fruits are fixed to the dragon fruit tree support stakes 5 on both sides by the support line 6.

[0049] 2) Align the depth image of depth camera 3 with the RGB image pixels;

[0050] 3) Depth camera 3 captures images between the rows of dragon fruit, and simultaneously acquires the RGB images of the dragon fruit and their corresponding depth images;

[0051] 4) The depth camera 3 moves a certain distance along the row direction between adjacent rows of dragon fruit to ensure that the images captured by two adjacent shots do not overlap;

[0052] 5) If the number of dragon fruit images collected meets the training requirements of the deep learning model, stop collecting images; otherwise, repeat steps 3) and 4).

[0053] S2. Label the target bounding boxes for dragon fruit (row 7) in the image, and simultaneously create the dataset, such as... Figure 5 As shown, the specific steps include the following:

[0054] 1) Combining the depth image imgD with the depth pixel unit I of the depth camera 3 d The depth distance D corresponding to the pixel in the RGB image is obtained. RGB Its formula is:

[0055] D RGB (i,j)=imgD(i,j)·I d , (i=1,2,…,H; j=1,2,…,W)

[0056] In the formula, D RGB It represents the depth distance of a pixel in the RGB image from the depth camera 3 in the Z direction; imgD represents the number of pixels in the depth image; I dIt is the distance unit for pixel values ​​in the depth image. i is the pixel index number in the image height direction, and j is the pixel index number in the image width direction. H and W are the image height and width, respectively;

[0057] 2) Use depth-distance threshold D t Background information from non-target picking row 8 is removed, including the dragon fruit tree 1 (non-target picking row 8), the dragon fruit, and other interfering background. The removed background information in the RGB image is filled with gray, using the following formula:

[0058]

[0059] In the formula, Rb_RGB is the RGB image after removing the background; Ori_RGB is the RGB image; gray_color is the gray pixel value; D t It is the depth distance threshold, which is half the spacing between the planting rows of dragon fruit plants;

[0060] 3) Label the dragon fruit 3 on the background-removed RGB image and generate an Rb_XML file;

[0061] 4) Modify the filename attribute in the Rb_XML file to the RGB image name to generate the target row fruit annotation file XML for the RGB image;

[0062] 5) Create a dragon fruit dataset for row 7 of the target picking area in the orchard using RGB images and their XML files in a 7:1.5:1.5 ratio;

[0063] S3. A dragon fruit detection model for target picking row 7 is trained using the MBSF network, such as... Figure 6 As shown, the construction steps of the MBSF network architecture include the following sub-steps:

[0064] 1) Scale the images to a uniform size of 640×640×3;

[0065] 2) The scaled image is processed by the CBL layer to obtain the feature map C1, where the CBL is composed of a concatenated convolutional layer, a batch normalization layer and a Leaky ReLU;

[0066] 3) Feature map C1 is calculated in MBlock layer three times in a row to obtain feature maps C2 to C4 in sequence, where MBlock layer is Blocknet of MobilenetV3;

[0067] 4) Feature map C4 is processed sequentially using the MBlock layer and the Spatial Pyramid Network (SPPF) to obtain feature map C5;

[0068] 5) Input feature maps C3 to C5 into the feature pyramid network (SE-FPN) that incorporates the SEnet attention mechanism to further obtain enhanced feature maps;

[0069] 6) Enhance the feature map input to the anchor-free detector to predict the category and location of dragon fruit; and output the dragon fruit detection results;

[0070] S4. Check whether the trained MBSF model meets the accuracy requirements for dragon fruit detection in target picking row 7. If it does not meet the requirements, increase the number of images in the dataset and adjust the MBSF model parameters. Repeat steps S2 and S3. If it meets the requirements, proceed to the next step.

[0071] S5. Deploy the trained MBSF model to the detection system of the dragon fruit picking robot to detect and identify the dragon fruit in the target picking row 7 in the orchard.

[0072] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for intelligent detection and identification of dragon fruit in the target picking row of a densely planted orchard, characterized in that, Includes the following steps: S1. Set the parameters of the depth camera and acquire images of dragon fruit in the densely planted orchard; S2. Label the target bounding boxes of dragon fruit in the picking row in the image, and at the same time create the dataset; S3. A dragon fruit detection model for target picking rows is trained using the MBSF network; S4. Check whether the trained MBSF model meets the accuracy requirements for dragon fruit detection in the target picking row. If it does not meet the requirements, increase the number of images in the dataset and adjust the MBSF model parameters. Repeat steps S2 and S3. If it meets the requirements, proceed to the next step. S5. Deploy the trained MBSF model to the detection system of the dragon fruit picking robot to detect and identify dragon fruit in the target picking row in the orchard. Step S3, specifically the MBSF network architecture construction step, includes the following sub-steps: 1) Scale the images to a uniform size of 640×640×3; 2) The scaled image is processed by the CBL layer to obtain the feature map C1, where the CBL is composed of a concatenated convolutional layer, a batch normalization layer and a Leaky ReLU; 3) Feature map C1 is calculated in MBlock layer three times in a row to obtain feature maps C2 to C4 in sequence, where MBlock layer is the Block net of MobilenetV3; 4) Feature map C4 is processed sequentially using the MBlock layer and the spatial pyramid network to obtain feature map C5; 5) Input feature maps C3 to C5 into the feature pyramid network that incorporates the SEnet attention mechanism to further obtain enhanced feature maps; 6) Enhance the feature map input to the Anchor-free detector to predict the category and location of dragon fruit; and output the dragon fruit detection results.

2. The intelligent detection and identification method for dragon fruit in the target picking row of a densely planted orchard according to claim 1, characterized in that, Step S1 includes the following sub-steps: 1) Place the depth camera in the middle of the adjacent rows of dragon fruit trees, set the spacing between adjacent rows of dragon fruit trees to l, and set the height of the camera above the ground to half the height h of the dragon fruit tree. Place the camera vertically so that the depth camera's field of view covers the entire height range of the dragon fruit tree, so that all fruits are in the image. 2) Align the depth image of the depth camera with the pixels of the RGB image; 3) The depth camera captures images between the rows of dragon fruit, simultaneously acquiring RGB images of the dragon fruit and their corresponding depth images; 4) The depth camera moves a certain distance along the row direction between adjacent rows of dragon fruit to ensure that images taken in two adjacent shots do not overlap; 5) If the number of dragon fruit images collected meets the requirements for training the deep learning model, stop collecting images; otherwise, repeat steps 3) and 4).

3. The intelligent detection and identification method for dragon fruit in the target picking row of a densely planted orchard according to claim 1, characterized in that, Step S2 includes the following sub-steps: 1) Combining the depth image imgD with the depth pixel unit I of the depth camera d The depth distance D corresponding to the pixel in the RGB image is obtained. RGB Its formula is: D RGB (i,j)=imgD(i,j)·I d ,i=1,2,…,H;j=1,2,…,W In the formula, D RGB It represents the depth distance of a pixel in the RGB image from the depth camera in the Z direction; imgD represents the number of pixels in the depth image; I d It is the distance unit of the depth image pixel value, i is the pixel index number in the height direction of the image, j is the pixel index number in the width direction of the image, and H and W are the height and width of the image, respectively. 2) Use depth-distance threshold D t Background information from non-target picking rows is removed, including dragon fruit trees, dragon fruit, and other interfering backgrounds. The removed background information in the RGB image is filled with gray, using the following formula: In the formula, Rb_RGB is the RGB image after removing the background; Ori_RGB is the RGB image; gray_color is the gray pixel value; D t It is the depth distance threshold, which is half the spacing between the planting rows of dragon fruit plants; 3) Label the dragon fruit on the background-removed RGB image and generate an Rb_XML file; 4) Modify the filename attribute in the Rb_XML file to the RGB image name to generate the target row fruit annotation file XML for the RGB image; 5) Create a dragon fruit dataset for the target picking rows in the orchard using RGB images and their XML files in a 7:1.5:1.5 ratio.

Images