A fruit recognition and positioning method for crop fruit picking

Abstract

The application discloses a fruit recognition and positioning method for crop fruit picking. The application introduces a Ghostnet lightweight module to obtain a GN-YOLOv5s model, replaces original convolution layers with depth separable convolution to extract features and reduce network calculation amount; secondly, to further reduce the model size, a regularization term about a scaling coefficient is introduced in the BN layer for sparse training, channels with a scaling factor gamma of 0-0.005 are screened out, pruning processing is performed to obtain a GS-YOLOv5s model; finally, to make the pruned model maintain high detection accuracy, a knowledge distillation method is adopted, a teacher network is used to assist the fine-tuning of the pruned model to obtain a PD-yolov5s, the hardware cost is reduced, and the model is convenient to deploy on a low-computing platform. The application can reduce network complexity, improve detection efficiency, make it be deployed on an embedded platform with small computing power, and obtain positioning information of dense crop fruits, which lays a foundation for subsequent action planning of a picking robot.

Description

Technical Field

[0001] This invention relates to a method for fruit identification and positioning, and more particularly to a method for fruit identification and positioning during the harvesting of agricultural crops. Background Technology

[0002] Harvesting robots are currently widely used in agriculture. As a key technology for harvesting robots, researchers have conducted extensive research on the detection and localization of crop fruits. The function of target detection and localization enables harvesting robots to perceive the environment and guide the movement of the robot's end effector, thereby completing the harvesting task.

[0003] However, existing object detection and localization methods have the following drawbacks in agricultural applications. First, existing object detection models are complex, with a large number of parameters and high computational demands. Therefore, these models have high hardware requirements, poor portability, and are difficult to deploy on embedded platforms with limited computing power. Furthermore, the harvesting efficiency of current harvesting robots is not high enough, mainly due to insufficient image processing speed during object detection. Second, in natural environments, most crop fruits are spherical and densely packed, often obscured by branches and overlapping. Conventional fruit recognition algorithms use the fruit's color space to distinguish the fruit from the background, and after further processing, identify complete fruits. However, this algorithm cannot accurately identify densely packed fruits. Third, current object detection algorithms applied in agriculture lack the ability to simultaneously detect and localize.

[0004] Therefore, in response to the above technical problems, this invention proposes a method for fruit identification and positioning during crop harvesting. Summary of the Invention

[0005] The purpose of this invention is to provide a method for fruit identification and localization during crop harvesting. This invention features reduced network complexity, improved detection efficiency, deployment on embedded platforms with lower computing power, and the ability to obtain localization information of densely packed crop fruits, laying the foundation for subsequent motion planning of harvesting robots.

[0006] The technical solution of the present invention: a method for fruit identification and positioning during crop harvesting, comprising the following steps:

[0007] Step 1: Collect images of crop fruits and establish a fruit dataset to lay the foundation for subsequent crop fruit identification.

[0008] Step 2: Replace the C3_x structure in the original YOLOv5s model with the Ghostnet module to obtain the GN-YOLOv5s model, reducing the number of model parameters and computational cost.

[0009] Step 3: Perform sparse training on the GN-YOLOv5s model obtained in Step 2 to obtain the GS-YOLOv5s model. The sparse training first uses the scaling factor γ of the BN layer as an evaluation index of channel importance, providing a basis for pruning the GS-YOLOv5s model.

[0010] Step 4: Set the pruning rate. Prune the channels with γ values ​​of 0-0.005 in the GS-YOLOv5s model obtained in Step 3, remove the input and output of the pruned channels, adjust the weights of the pruned model, and finally adjust the pruning rate to train and repeat this process to obtain the GP-YOLOv5s model, which further reduces the number of model parameters.

[0011] Step 5: Use the original YOLOv5s model as the teacher model and the GP-YOLOv5s model obtained in Step 4 as the student model. Through the complete output of the original YOLOv5s model, the GP-YOLOv5s model obtained in Step 4 can master the reasoning method of the original YOLOv5s model. The classification loss function of the GP-YOLOv5s model obtained in Step 4 is shown in Equation (4), the coordinate frame loss function is shown in Equation (5), and the final distillation overall loss function is shown in Equation (6).

[0012]

[0013] In Equation 4 The target, class probabilities, and bounding boxes corresponding to the GP-YOLOv5s model obtained in step four. For their respective true values, f obj f class f bb These are the loss functions for the target, the classification, and the bounding box, respectively.

[0014] In Equation 5 Loss to the target For distillation loss, λ D These are the weighting coefficients. Predict the target for the teacher model;

[0015] In Equation 6: Corresponding to the original classification loss, For distillation classification loss, λ D These are the weighting coefficients. To predict the target for the updated teacher model, we obtain

[0016] The PD-YOLOv5s model makes the final PD-YOLOv5s model more accurate than the pruned GP-YOLOv5s model, reduces the impact of replacing the C3_x structure and pruning in the original YOLOv5s model on the accuracy of recognition, and the obtained GP-YOLOv5s model can be deployed on embedded platforms with lower computing power.

[0017] Step 6; Camera Calibration: Fix the binocular camera horizontally, ensuring the calibration board is within the camera's field of view. Continuously change the position and orientation of the calibration board, collect 40-50 sets of calibration images, select 14-20 sets of images from different angles, and use the MATLAB calibration toolbox to perform calibration, obtaining the binocular camera parameters, providing hardware support for accurate positioning of crop fruits.

[0018] Step 7: After completing the camera calibration in Step 6, input the PD-YOLOv5s model obtained in Step 5 into the stereo camera. This allows the left and right cameras of the stereo camera to identify crop and fruit detection boxes. The center point of each crop detection box is taken as the target point, denoted as point P. The coordinates (X, Y, X) of target point P in the pixel coordinate system of the left camera are then recorded. L ,Y L ), denoted as P L (X L ,Y L The coordinates (X, X) of the target point P in the pixel coordinate system of the right camera. R ,Y R ), denoted as P R (X R ,Y R );

[0019] Step 8: Using the SGBM stereo matching algorithm, match the two-dimensional coordinates of the crop fruit target point P obtained in Step 7 line by line with the images acquired by the left and right cameras of the stereo camera; the pixel coordinates of point P obtained in Step 7 on the projection planes of the left and right cameras are respectively P L (X L ,Y L ) and P R (X R ,Y R The image planes of the two cameras are on the same horizontal plane, and the coordinates of point P obtained in step seven are Y = Y L =Y R From the triangular relationship, we can obtain the following equation (9):

[0020]

[0021] Where: f is the camera focal length, b is the distance between the optical centers of the left and right cameras, (X C ,Y C Z C() represents the coordinates of point P in the world coordinate system;

[0022] The disparity X of point P obtained in step seven. L -X R Let d be the coordinate system of the left camera. Using the coordinate system of the left camera as the world coordinate system, the coordinates of point P in the left camera coordinate system are calculated using the parallax value, as shown in equation (10):

[0023]

[0024] Where: b is the distance between the optical centers of the left and right cameras, d is the parallax of point P, and X L Y is the x-coordinate of target point P in the left camera in the pixel coordinate system, and Y is the y-coordinate of target point P in the left and right cameras in the pixel coordinate system.

[0025] The coordinates (Xc, Yc, Zc) calculated according to formula (10) are the three-dimensional coordinates of point P obtained in step seven, thus solving the problem of lack of spatial positioning ability in the study of crop fruits.

[0026] In the aforementioned method for fruit identification and localization during crop harvesting, in step one, after acquiring images of the crop fruits, the fruit images are manually annotated, the bounding rectangle of the fruit target is drawn, and the two categories of occluded fruits and unoccluded fruits are labeled. After the annotation is completed, a fruit dataset is established.

[0027] In the aforementioned method for fruit identification and localization during crop harvesting, in step two, the Ghostnet module first performs regular convolution on the fruit dataset to obtain a regular convolution feature map. Then, it performs a linear transformation on the generated regular convolution feature map to generate a new similar feature map. Finally, it combines the information from the two sets of feature maps to obtain all the feature information.

[0028] In the aforementioned method for fruit identification and localization during crop harvesting, in step three, sparse training first uses the scaling factor γ of the BN layer as an indicator of channel importance. The magnitude of γ is positively correlated with channel importance. The calculation formula for the BN layer is as follows:

[0029] Z out =γ·Z ^ +β (1)

[0030]

[0031] Where γ and β are the normalized parameters of the BN layer, Z m Z out δ and μ represent the input and output of the BN layer, respectively; μ and δ represent the mean and variance of the BN layer, respectively; and ε represents a constant ranging from 0 to 0.001.

[0032] Then, the regularization penalty term ∑ in equation (3) is introduced into the γ parameter of the BN layer. γ g(γ) is used to construct a sparse training loss function, which is shown below:

[0033] L=∑ (x,y) l(f(x,W),y)+λ∑ γ g(γ) (3)

[0034] Where ∑ (x,y) l(f(x,W),y) is the loss function defined in step two for the GN-YOLOv5s model, where x represents the input matrix, y represents the input labels, and W represents the model weights; λ∑ γ g(γ) is the regularization penalty used to constrain γ, where λ = 0.01 is used to balance the losses of the two terms and is called the sparsity rate.

[0035] In the aforementioned method for fruit identification and localization during crop harvesting, in step four, a pruning rate of 65-75% is set to prune the channels with γ values ​​of 0-0.005 in the GS-YOLOv5s model obtained in step three, remove the input and output of the pruned channels, adjust the weights of the pruned model, and finally adjust the pruning rate to train and repeat this process 5-10 times to obtain the GP-YOLOv5s model.

[0036] In the aforementioned method for fruit identification and positioning during crop harvesting, in step four, the detection accuracy of the GP-YOLOv5s model is no less than 0.98 times that of the GN-YOLOv5s model in step two, and the model size is no more than 0.5 times that of the GN-YOLOv5s model in step two.

[0037] In the aforementioned method for fruit identification and positioning during crop harvesting, in step seven, the optical center of the lens of the left camera of the binocular camera is defined as O. L The optical center of the lens of the right camera is defined as O. R O L O is used as the origin of the camera coordinate system for the left camera. R Used as the origin of the camera coordinate system for the right camera; the distance between the left and right optical centers is denoted by b as the baseline distance, and the focal length of the two cameras is denoted by f.

[0038] In the aforementioned method for fruit identification and positioning during crop harvesting, in step seven, the coordinates (X, X, Y) of the target point P in the pixel coordinate system of the left camera are calculated using equation (7). L ,Y L ), denoted as P L (X L ,Y L ),

[0039]

[0040] Where: (X) min-L ,Y min-L () represents the top-left corner image coordinates of the fruit detection box in the left camera in the pixel coordinate system; (X) max-L ,Y max-L () indicates the image coordinates of the lower right corner of the fruit detection box in the left camera in the pixel coordinate system;

[0041] The coordinates (X, X, Y) of the target point P in the pixel coordinate system of the right camera are calculated using equation (8). R ,Y R ), denoted as P R (X R ,Y R );

[0042]

[0043] Where: (X) min-R ,Y min-R () represents the top-left corner image coordinates of the fruit detection box in the right camera in the pixel coordinate system; (X) max-R ,Y max-R The value represents the image coordinates of the lower right corner of the fruit detection box in the right camera in the pixel coordinate system.

[0044] In the aforementioned method for fruit identification and localization during crop harvesting, step eight of the SGBM stereo matching algorithm consists of four main parts. First, preprocessing is performed on the input image to obtain gradient information for subsequent cost calculation. Then, the original image and the preprocessed information are fused together for cost calculation. Next, one-dimensional dynamic programming is performed in 8-16 directions, with solutions obtained for each direction. Energy accumulation is then performed, and the costs of each direction are added together to obtain the final cost. Finally, optimization is performed, with the minimum value among the calculated final costs used as the matching cost, the disparity with the matching cost used as the final disparity, and the obtained depth information is normalized to obtain a disparity map.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] Crop fruit recognition algorithms are one of the core technologies of harvesting robots. However, in natural environments, most crops grow densely, and existing target detection algorithms cannot distinguish between crops obscured by branches and leaves or other target fruits. If these algorithms are directly applied to harvesting robots, the robots will be unable to accurately collect crops and may even damage the robot's robotic arm or end effector. Therefore, this invention constructs a dense crop fruit dataset to lay the foundation for subsequent crop fruit recognition.

[0047] Secondly, existing ground target detection algorithms are relatively complex, with a large number of parameters and high computational cost. Therefore, existing models have high hardware requirements, poor portability, and are difficult to deploy on embedded platforms with limited computing power. Therefore, this invention introduces a lightweight Ghostnet module to obtain the GN-YOLOv5s model, which uses depthwise separable convolutions instead of the original convolutional layers to extract features, reducing network computation. Thirdly, to further reduce the model size, a regularization term related to the scaling factor is introduced into the BN layer for sparse training, selecting channels with a scaling factor γ of 0-0.005 for pruning, resulting in the GS-YOLOv5s model. Finally, to maintain high detection accuracy after pruning, a knowledge distillation method is used, with a teacher network assisting in fine-tuning the pruned model to obtain PD-YOLOv5s, reducing hardware costs and facilitating deployment on low-computing-power platforms.

[0048] This invention uses knowledge distillation and improves the loss function to obtain the PD-YOLOv5s model, which, after being lightweighted, improves the accuracy of PD-YOLOv5s. It also uses binocular stereo localization to address the problem of insufficient localization information for densely packed crop fruits.

[0049] In summary, this invention can reduce network complexity, improve detection efficiency, enable deployment on embedded platforms with lower computing power, and obtain the location information of densely packed fruits of crops, thus laying the foundation for the subsequent action planning of harvesting robots. Attached Figure Description

[0050] Figure 1 These are partial images from the crop fruit (chili pepper) dataset of this invention;

[0051] Figure 2 This is a schematic diagram of the structure of the GN-YOLOv5s model of the present invention;

[0052] Figure 3 This is a schematic diagram of the pruning of the GS-YOLOv5s model according to the present invention, where C i1 C i2 , ..., C i2 The inputs and outputs of the GS-YOLOv5s model;

[0053] Figure 4 This is a schematic diagram of the PD-YOLOv5s model of the present invention;

[0054] Figure 5 This is a flowchart illustrating the crop fruit positioning process of the present invention.

[0055] Figure 6 The two-dimensional coordinates P of the fruit target point in the pixel coordinate system of the left camera of the binocular camera of this invention. L (XL ,Y L );

[0056] Figure 7 This is a diagram showing the transformation of the binocular camera of the present invention from two-dimensional coordinates in the pixel coordinate system to three-dimensional coordinates in the world coordinate system;

[0057] Figure 8 This is a scaling factor distribution diagram when the sparsity rate is 0.005 during sparse training of the GN-YOLOv5s model according to this invention.

[0058] Figure 9 This is a scaling factor distribution diagram when the sparsity rate is 0.01 during sparse training of the GN-YOLOv5s model according to this invention.

[0059] Figure 10 This is the average accuracy curve of the PD-YOLOv5s of this invention when identifying crop fruits (chili peppers);

[0060] Figure 11 This is a real-time detection image of crop fruit (chili pepper) identified by the PD-YOLOv5s of this invention;

[0061] Figure 12 This is a real-time detection image of crop fruit (chili pepper) from the original YOLOv5s of this invention;

[0062] Figure 13 This is a real-time detection image of crop fruit (chili pepper) using the PD-YOLOv5 model of this invention. Detailed Implementation

[0063] The present invention will be further described below with reference to the accompanying drawings and embodiments, but this should not be construed as limiting the present invention.

[0064] Example 1: A method for fruit identification and location during crop harvesting, comprising the following steps:

[0065] Step 1: After acquiring images of crop fruits, manually annotate the fruit images, draw the bounding rectangle of the fruit targets, and label them as two categories: occluded fruits and unoccluded fruits. After the annotation is completed, establish a fruit dataset.

[0066] Step 2: Replace the C3_x structure in the original YOLOv5s model with the Ghostnet module to obtain the GN-YOLOv5s model;

[0067] The Ghostnet module first performs regular convolution on the fruit dataset to obtain regular convolution feature maps. Then, it performs a linear transformation on the generated regular convolution feature maps to generate new similar feature maps. Finally, it combines the information from the two sets of feature maps to obtain all the feature information.

[0068] Step 3: Perform sparse training on the GN-YOLOv5s model obtained in Step 2 to obtain the GS-YOLOv5s model. The sparse training first uses the scaling factor γ of the BN layer as an evaluation index of channel importance. The magnitude of γ is positively correlated with channel importance. The calculation formula for the BN layer is shown below:

[0069] Z out =γ·Z ^ +β (1)

[0070]

[0071] Where γ and β are the normalized parameters of the BN layer, Z m Z out δ and μ represent the input and output of the BN layer, respectively; μ and δ represent the mean and variance of the BN layer, respectively; and ε represents a constant ranging from 0 to 0.001.

[0072] Then, the regularization penalty term ∑ in equation (3) is introduced into the γ parameter of the BN layer. γ g(γ) is used to construct a sparse training loss function, which is shown below:

[0073] L=∑ (x,y) l(f(x,W),y)+λ∑ γ g(γ) (3)

[0074] Where ∑ (x,y) l(f(x,W),y) is the loss function defined in step two for the GN-YOLOv5s model, where x represents the input matrix, y represents the input labels, and W represents the model weights; λ∑ γ g(γ) is the regularization penalty used to constrain γ, where λ = 0.01 is used to balance the losses of the two terms and is called the sparsity rate.

[0075] Step 4: Set a pruning rate of 65-75% to prune the channels with γ values ​​of 0-0.005 in the GS-YOLOv5s model obtained in Step 3, remove the input and output of the pruned channels, adjust the weights of the pruned model, and finally adjust the pruning rate to train and repeat this process 5-10 times to obtain the GP-YOLOv5s model. The detection accuracy of the GP-YOLOv5s model should not be less than 0.98 times that of the GN-YOLOv5s model in Step 2, and the model size should not be more than 0.5 times that of the GN-YOLOv5s model in Step 2.

[0076] Step 5: Use the original YOLOv5s model as the teacher model and the GP-YOLOv5s model obtained in Step 4 as the student model. Through the complete output of the original YOLOv5s model, the GP-YOLOv5s model obtained in Step 4 can master the reasoning method of the original YOLOv5s model. The classification loss function of the GP-YOLOv5s model obtained in Step 4 is shown in Equation (4), the coordinate frame loss function is shown in Equation (5), and the final distillation overall loss function is shown in Equation (6).

[0077]

[0078] In Equation 4 The target, class probabilities, and bounding boxes corresponding to the GP-YOLOv5s model obtained in step four. For their respective true values, f obj f class f bb These are the loss functions for the target, the classification, and the bounding box, respectively.

[0079] In Equation 5 Loss to the target For distillation loss, λ D These are the weighting coefficients. Predict the target for the teacher model;

[0080] In Equation 6: Corresponding to the original classification loss, For distillation classification loss, λ D These are the weighting coefficients. To predict the target for the updated teacher model, we obtain

[0081] PD-YOLOv5s model;

[0082] Step 6; Camera Calibration: Fix the binocular camera horizontally, ensuring the calibration board is within the camera's field of view. Continuously change the position and orientation of the calibration board, collect 40-50 sets of calibration images, select 14-20 sets of images from different angles, and use the MATLAB calibration toolbox to perform calibration and obtain the binocular camera parameters.

[0083] Step 7: After completing the camera calibration in Step 6, input the PD-YOLOv5s model obtained in Step 5 into the stereo camera. This allows the left and right cameras of the stereo camera to identify crop and fruit detection boxes. The center point of each crop detection box is taken as the target point, denoted as point P. The coordinates (X, Y, X) of target point P in the pixel coordinate system of the left camera are then recorded. L ,Y L ), denoted as P L (X L ,Y LThe coordinates (X, X) of the target point P in the pixel coordinate system of the right camera. R ,Y R ), denoted as P R (X R ,Y R );

[0084] The optical center of the lens of the left camera in a binocular camera is defined as O. L The optical center of the lens of the right camera is defined as O. R O L O is used as the origin of the camera coordinate system for the left camera. R Used as the origin of the camera coordinate system for the right camera; the distance between the left and right optical centers is denoted by b as the baseline distance, and the focal length of the two cameras is denoted by f;

[0085] The coordinates (X, X, Y) of the target point P in the pixel coordinate system of the left camera are calculated using equation (7). L ,Y L ), denoted as P L (X L ,Y L ),

[0086]

[0087] Where: (X) min-L ,Y min-L () represents the top-left corner image coordinates of the fruit detection box in the left camera in the pixel coordinate system; (X) max-L ,Y max-L () indicates the image coordinates of the lower right corner of the fruit detection box in the left camera in the pixel coordinate system;

[0088] The coordinates (X, X, Y) of the target point P in the pixel coordinate system of the right camera are calculated using equation (8). R ,Y R ), denoted as P R (X R ,Y R );

[0089]

[0090] Where: (X) min-R ,Y min-R () represents the top-left corner image coordinates of the fruit detection box in the right camera in the pixel coordinate system; (X) max-R ,Y max-R The value represents the image coordinates of the lower right corner of the fruit detection box in the right camera in the pixel coordinate system.

[0091] Step 8: Using the SGBM stereo matching algorithm, the two-dimensional coordinates of the target point P of the crop fruit obtained in Step 7 are matched line by line with the images acquired by the left and right cameras of the stereo camera. The main process of the SGBM stereo matching algorithm is divided into four parts. First, preprocessing is performed on the input image to obtain gradient information for subsequent cost calculation. Then, the original image and the preprocessed information are fused for cost. Next, one-dimensional dynamic programming is performed from 8 to 16 directions to solve the problem. A solution is obtained for each direction. Then, energy is accumulated and the costs of each direction are added together to obtain the final cost. Finally, optimization is performed. The minimum value of the calculated final cost is taken as the matching cost, and the disparity with the matching cost is taken as the final disparity. The obtained depth information is normalized to obtain the disparity map.

[0092] The pixel coordinates of point P obtained in step seven on the projection planes of the left and right cameras are P1, P2, and P3, respectively. L (X L ,Y L ) and P R (X R ,Y R The image planes of the two cameras are on the same horizontal plane, and the coordinates of point P obtained in step seven are Y = Y L =Y R From the triangular relationship, we can obtain the following equation (9):

[0093]

[0094] Where: f is the camera focal length, b is the distance between the optical centers of the left and right cameras, (X C ,Y C Z C () represents the coordinates of point P in the world coordinate system;

[0095] The disparity X of point P obtained in step seven. L -X R Let d be the coordinate system of the left camera. Using the coordinate system of the left camera as the world coordinate system, the coordinates of point P in the left camera coordinate system are calculated using the parallax value, as shown in equation (10):

[0096]

[0097] Where: b is the distance between the optical centers of the left and right cameras, d is the parallax of point P, and X L Y is the x-coordinate of target point P in the left camera in the pixel coordinate system, and Y is the y-coordinate of target point P in the left and right cameras in the pixel coordinate system.

[0098] The coordinates (Xc, Yc, Zc) calculated according to equation (10) are the three-dimensional coordinates of point P obtained in step seven.

[0099] Experiments have shown that:

[0100] The training process of this invention was carried out in an Ubuntu 18.0 and CUDA 11.0 environment, with a GPU configuration of NVIDIA GeForce RTX 3090 and 24GB of video memory, and the GPU was used for training.

[0101] All experimental training parameters were set as follows: input image size was 640×640, the optimizer used was the momentum-driven SGD optimizer, the initial learning rate was set to 0.001, the batch size was 16, and a total of 200 training epochs were conducted. To objectively evaluate the performance of the improved lightweight model, its performance was assessed from two aspects: model complexity and accuracy. Evaluation metrics included model parameter count, floating-point operations, model size, mean AP, and inference speed (FPS).

[0102] 1. All images required for the experiment were collected from the crop planting base of the Guizhou Academy of Agricultural Sciences. The crops were in the green-ripe stage. To accurately reflect the growth posture of the crops and the complexity of the environment, images were collected under different lighting and angle conditions, including: unobstructed, non-overlapping crop fruit images; partially overlapping fruit images; images of crop fruit (pepper) obscured by leaves; images of crop fruit obscured by branches; and images of crop fruit obscured by a mixture of leaves and branches. The acquisition device was the built-in camera of a mobile phone, with contrast, saturation, and sharpness set to standard mode. A total of 2000 images were collected, including 785 top-view images (462 under direct light, 323 under backlight) and 1215 eye-view images (871 under direct light, 344 under backlight). Some original images are shown below. Figure 1 As shown, using the image annotation software "Labelling", bounding rectangles of crop fruit targets are drawn, enabling manual annotation of crop fruits and constructing a crop fruit dataset.

[0103] 2. Figure 2 The diagram shows the GN-YOLOv5s model from step two. The backbone network of the GN-YOLOv5s model is a combination of Ghost modules, Conv convolutional structures, and SPP structures, while the head network consists of FPN and PAN structures. Table 1 shows a comparison between the GN-YOLOv5s model from step two and the original YOLOv5s model from step two.

[0104] Table 1

[0105]

[0106] As shown in Table 1, replacing the C3_x structure in the original YOLOv5s model with the Ghostnet module reduces the number of parameters in the resulting GN-YOLOv5s model compared to the original YOLOv5s model.

[0107] The YOLOv5s model reduced the number of parameters and computational load by 6.8%, the number of floating-point operations by 15.1%, and the model size by 12.7%.

[0108] 3. Perform sparse training on the GN-YOLOv5s model to obtain the GS-YOLOv5s model. The distribution of the sparse training scaling factor is shown in the figure below. Figure 8 and Figure 9 As shown, Figure 8 To show the distribution of scaling factors in the BN layer when the sparsity is set to 0.005, Figure 9 To show the distribution of scaling factors in the BN layer when the sparsity is set to 0.01, from... Figure 8 and 9 As can be seen, when the sparsity rate is 0.005, there are relatively few scaling factor values ​​distributed between 0 and 0.005, which does not achieve the effect of sparse training. When the sparsity rate is 0.01, more scaling factor values ​​are distributed between 0 and 0.005, which means that more channels with lower importance can be pruned in the model, thus achieving the purpose of sparse training. Considering both the λ-sparseness and accuracy of the model, the sparsity rate was ultimately set to 0.01.

[0109] 4. Figure 3 This is a schematic diagram of pruning the GS-YOLOv5s model, where C i1 C i2 , ..., C in As input to the GS-YOLOv5s model, C j1 C j1 Table 2 shows the comparison between the GP-YOLOv5s model and the original YOLOv5s model after removing the input and output channels with scaling factors γ values ​​of 0-0.005 from the BN layer of the GS-YOLOv5s model:

[0110] Table 2

[0111]

[0112] As shown in Table 2, the GP-YOLOv5s model reduces the number of parameters by 4,952,219 and the number of floating-point operations by 9.4 GFLOPS compared to the original YOLOv5s model. The model size is reduced by 6.9 MB, indicating that the model is effectively compressed.

[0113] 5. Figure 4This is a diagram of the PD-YOLOv5s model. The original YOLOv5s detection model is used as the teacher model, and the GP-YOLOv5s model is used as the student model. Through the complete output of the original YOLOv5s model, the GP-YOLOv5s model learns the inference method of the original YOLOv5s model. The performance comparison between the GP-YOLOv5s model and the original YOLOv5s model is shown in Table 3.

[0114] Table 3

[0115]

[0116] Figure 10 The figure shows the average accuracy curve of the PD-YOLOv5s model when identifying crop fruits (chili peppers). After 150 model iterations, the average accuracy is 92.9%, which effectively improves the accuracy of the model. Even after the model is lightweighted, the accuracy is still no less than 98% of the accuracy of the original YOLOv5s model.

[0117] Figure 11 The image shows the detection of PD-YOLOv5s when identifying crop fruits (chili peppers). As can be seen from the image, PD-YOLOv5s can accurately identify crop fruits (chili peppers). Figure 12 and 13 They respectively showed the original

[0118] A comparison of the YOLOV5s crop fruit recognition model and the PD-YOLO5s model. Figure 12 This is the result of the original YOLOv5s model recognizing crop fruit (chili pepper). Figure 13 The results of the PD-YOLOv5s model in recognizing crop fruits (peppers) are shown below. Figure 12 and Figure 13 It can be seen that the compressed PD-YOLOv5s model can still maintain high recognition accuracy.

[0119] 6. Figure 6 The two-dimensional coordinates P of the fruit target point in the pixel coordinate system for the left camera of the stereo camera. L (X L ,Y L ), representing the two-dimensional coordinates (X, X) of the target point P of the crop fruit in the left camera pixel coordinate system. L ,Y L ). Figure 7 This is a transformation diagram of the target point of a crop fruit from two-dimensional coordinates in the pixel coordinate system to three-dimensional coordinates in the world coordinate system. In the diagram, O... L O is the optical center of the lens of the left camera. R O is the optical center of the lens of the right camera. L O is used as the origin of the camera coordinate system for the left camera.R P is used as the origin of the camera coordinate system for the right camera. L (X L ,Y L ) represents the coordinates (X, X) of the target point P in the pixel coordinate system of the left camera. L ,Y L ), P R (X R ,Y R ) represents the coordinates (X, X) of the target point P in the pixel coordinate system of the right camera. R ,Y R The algorithm was used to locate the crop fruit. During the experiment, the crop fruit moved 20mm away from the camera each time, and the location was determined once, mainly based on the spatial depth of the (chili pepper). The initial position of the (chili pepper) was (100mm, 100mm, 100mm). The results of the 8 positioning tests are shown in Table 4.

[0120] Table 4

[0121]

[0122]

[0123] As can be seen from Table 4, the positioning error did not exceed 1.84 during the 8 positioning processes, and fluctuated around 0 multiple times, proving that the positioning method of the present invention has a certain degree of accuracy and stability.

Claims

1. A fruit recognition and positioning method for crop fruit picking, characterized by: Comprise the following steps: Step one, image acquisition of crop fruits, and establish fruit data set; Step two, replace the C3_x structure in the original YOLOv5s model with a Ghostnet module to obtain a GN-YOLOv5s model; Step three, sparse training of the GN-YOLOv5s model obtained in step two to obtain a GS-YOLOv5s model, and the sparse training first takes the scaling factor gamma of the BN layer as an evaluation channel importance indicator; Step four, set the pruning rate, prune the channels with a gamma value of 0-0.005 in the GS-YOLOv5s model obtained in step three, remove the input and output of the pruned channels, then adjust the model weights after pruning, finally adjust the pruning rate training and repeat the process to obtain a GP-YOLOv5s model; Step five, use the original YOLOv5s model as the teacher model and the GP-YOLOv5s model obtained in step four as the student model, let the GP-YOLOv5s model master the inference method of the original YOLOv5s model through the complete output of the original YOLOv5s model, the classification loss function of the GP-YOLOv5s model obtained in step four is shown in formula (4), the coordinate frame loss function is shown in formula (5), and the final overall loss function is shown in formula (6); wherein, in formula 4 are the target, class probability, and coordinate frame corresponding to the GP-YOLOv5s model obtained in step four, are the respective true values, f obj , f class , f bb are the loss functions of the target, classification, and coordinate frame, respectively; wherein, in formula 5 is a target loss, is a distillation loss, λ D is a weight coefficient, is a target predicted by the teacher model; Wherein, in formula 6: Corresponding to the original classification loss, Distillation classification loss, λ D Weight coefficient, The updated teacher model prediction target is obtained, and the PD-YOLOv5s model is obtained. Step six, camera calibration: fix the binocular camera horizontally, ensure that the calibration board is within the camera's field of view, constantly change the position and attitude of the calibration board, collect 40-50 groups of calibration pictures, select 14-20 groups of pictures at different angles, use the MATLAB calibration toolbox for calibration, and obtain the binocular camera parameters; Step seven, after completing the camera calibration of step six, input the binocular camera into the PD-YOLOv5s model obtained in step five, so that the crop fruit detection frame recognized by the left and right cameras of the binocular camera respectively, the center point of the crop detection frame is taken as the target point, denoted as P point, the coordinates of the target point P in the pixel coordinate system of the left camera (X L ,Y L ) is denoted as P L (X L ,Y L );the coordinates of the target point P in the pixel coordinate system of the right camera (X R ,Y R ) is denoted as P R (X R ,Y R ); Step eight, the two-dimensional coordinates of the target point P of the crop fruit obtained in step seven are matched row by row using the SGBM stereo matching algorithm on the pictures obtained by the left and right cameras of the binocular camera. L (X L ,Y L ) and P R (X R ,Y R ), the image planes of the two cameras are on the same horizontal plane, Y in the coordinates of the P point obtained in step seven = Y L = Y R , and the triangular relationship can be obtained as shown in formula (9): where: f is the camera focal length, b is the distance between the left and right camera optical centers, (X C ,Y C ,Z C ) are the coordinates of point P in the world coordinate system; The disparity X of point P obtained in step seven. L -X R Let d be the coordinate system of the left camera. Using the coordinate system of the left camera as the world coordinate system, the coordinates of point P in the left camera coordinate system are calculated using the parallax value, as shown in equation (10): wherein: b is the distance between the optical centers of the left and right cameras, d is the parallax of point P, X L is the horizontal coordinate of the target point P in the pixel coordinate system of the left camera, and Y is the vertical coordinate of the target point P in the pixel coordinate system of the left and right cameras. The coordinates (Xc, Yc, Zc) calculated according to formula (10) are the three-dimensional coordinates of point P obtained in step seven.

2. The fruit recognition and positioning method for crop fruit harvesting according to claim 1, characterized in that: In step one, after image acquisition of crop fruits, the fruit images are manually annotated, the bounding box of the fruit target is drawn, and the occluded and non-occluded fruits are labeled. After annotation, the fruit data set is established.

3. The fruit recognition and positioning method for crop fruit harvesting according to claim 1, characterized in that: In step two, the Ghostnet module first obtains a regular convolution feature map through regular convolution of the fruit data set, then generates a new similar feature map through linear transformation of the generated regular convolution feature map, and finally combines the information in the two groups of feature maps as the overall feature information.

4. The fruit recognition and positioning method for crop fruit harvesting according to claim 1, characterized in that: In step three, the sparse training first takes the scaling factor gamma of the BN layer as an evaluation channel importance indicator, and the size of gamma is positively correlated with the importance of the channel. The calculation formula of the BN layer is as follows: Z out = γ · Z ^ + β (1) where γ, β are the normalization parameters of the BN layer, Z m , Z out denote the input and output of the BN layer, μ, δ represent the mean and variance of the BN layer, respectively, and ε represents a constant ranging from 0 to 0.001; Then, the regularized penalty term ∑ γ g(γ) is introduced into the γ parameter of the BN layer in formula (3), and a sparse training loss function is constructed as follows: L = ∑ (x,y) l(f(x,W),y) + λ∑ γ g(γ) (3) where∑ (x,y) l(f(x,W),y) is the loss function defined by the GN-YOLOv5s model in step two, x represents the input matrix, y represents the input label, and W represents the weight of the model; λ∑ γ g(y) is a regular term penalty used to constrain y, where λ = 0.01 is used to balance the following two losses, referred to as the sparsity rate.

5. The fruit recognition and positioning method for crop fruit harvesting according to claim 1, characterized in that: In step four, set the pruning rate to 65-75%, prune the channels with a gamma value of 0-0.005 in the GS-YOLOv5s model obtained in step three, remove the input and output of the pruned channels, then adjust the model weights after pruning, finally adjust the pruning rate training and repeat the process 5-10 times to obtain a GP-YOLOv5s model.

6. The fruit recognition and positioning method for crop fruit harvesting according to claim 5, characterized in that: In the fourth step, the detection accuracy of the GP-YOLOv5s model is not less than 0.98 times of the GN-YOLOv5s model in the second step, and the model size is not higher than 0.5 times of the GN-YOLOv5s model size in the second step.

7. The fruit recognition and positioning method for crop fruit harvesting according to claim 1, characterized in that: In the step seven, the lens optical center of the left camera of the binocular camera is defined as O L , the lens optical center of the right camera is defined as O R , O L is used as the origin of the camera coordinate system of the left camera, O R is used as the origin of the camera coordinate system of the right camera; the distance between the left and right optical centers is represented by b as the baseline distance, and the focal length of the two cameras is represented by f.

8. The fruit recognition and positioning method for crop fruit harvesting according to claim 1, characterized in that: In step seven, the coordinates (X, X, Y) of the target point P of the left camera in the pixel coordinate system are calculated using equation (7). L ,Y L ), denoted as P L (X L ,Y L ), wherein: (X min-L ,Y min-L ) represents the left upper corner image coordinates of the fruit detection frame in the left camera under the pixel coordinate system; (X max-L ,Y max-L ) represents the right lower corner image coordinates of the fruit detection frame in the left camera under the pixel coordinate system; The coordinates (X, X, Y) of the target point P in the pixel coordinate system of the right camera are calculated using equation (8). R ,Y R ), denoted as P R (X R ,Y R ); wherein: (X min-R ,Y min-R ) represents the left upper corner image coordinates of the fruit detection frame in the right camera in the pixel coordinate system; (X max-R ,Y max-R ) represents the right lower corner image coordinates of the fruit detection frame in the right camera in the pixel coordinate system.

9. The fruit recognition and positioning method for crop fruit harvesting according to claim 1, characterized in that: In the eighth step, the main process of the SGBM stereo matching algorithm is divided into four parts: first, preprocessing, preprocessing the input image to obtain gradient information for subsequent cost calculation; second, cost fusion of the original image and the information obtained by preprocessing; third, one-dimensional dynamic programming solution from 8-16 directions, each direction is solved; fourth, energy accumulation, adding each direction to obtain the final cost; and finally, optimization processing, taking the smallest value in the calculated final cost as the matching cost, taking the disparity with the matching cost as the final disparity, and normalizing the obtained depth information to obtain a disparity map.

Images