An orchard young fruit detection and growth angle prediction method and system

Abstract

The present application relates to the technical field of image detection, and more particularly to a method and system for detecting young fruits in an orchard and predicting growth angles, comprising collecting young fruit image data; obtaining pseudo-label data; the backbone of the YOLOv8n-seg network adopts a MobileNetv3 structure, uses a Conv_BN_HSwish layer to reduce the dimensionality of the image, and improves ten bottleneck modules; the neck adopts a CARAFE up-sampling feature extraction module; the boundary box loss is optimized; the young fruit detection model is trained using labeled data and pseudo-label data, and the growth angle of the young fruit image is estimated. The present application uses semi-supervised learning to improve the generalization ability of the model through a small amount of labeled data and a large amount of unlabeled data; improves the recognition rate of young fruits by using the improved YOLOv8n-seg network; and solves the problem of insufficient precision of existing bagging methods.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method and system for detecting young fruit in orchards and predicting their growth angle. Background Technology

[0002] Bagging technology is an important technique for producing green, high-quality fruits and vegetables. Bagging effectively reduces damage from birds and insects, prevents pesticide contamination, sunburn, wind and rain damage, scratches and deformation, and improves the color of fruits and vegetables. It is also an indispensable step in the cultivation and production of high-quality fruits and vegetables. However, like harvesting ripe fruit, fruit and vegetable bagging is a highly seasonal and labor-intensive task, currently mainly done manually or with simple machinery. This is time-consuming, labor-intensive, and the quality of bagging varies greatly. Furthermore, the aging and shortage of agricultural labor is becoming increasingly prominent, and the cost of manual bagging labor is rising year by year, further increasing production costs. In view of the above, research into intelligent fruit and vegetable bagging robot technology is an inevitable trend.

[0003] Although research on intelligent bagging robots is still in its early stages, the gradual replacement of manual bagging with intelligent mechanical bagging is an inevitable trend. Regarding the bagging of young fruit, the bag must be inserted from the bottom up. To avoid damage to the fruit peel during the bagging process and to ensure smooth bagging, the primary task is target identification and acquisition of growth posture angle information. Only then can the bagging robot optimally bag young fruit in different growth postures. Among these challenges, obtaining the corresponding growth angle of the young peach fruit while detecting it during bagging is a problem that intelligent bagging robots must currently solve.

[0004] Hao Weijie et al.'s patent describes a method and system for automatically bagging apples based on deep learning. It uses the YOLOv4 algorithm to detect apples and obtain their position information, then coordinates with a robotic arm for automatic bagging. While this method can bag apples, it has some shortcomings: 1. Obtaining only the center position information of the apple cannot accurately obtain the growth angle information of the apple, which causes the rotating arm of the bagging robot to be unable to complete the bagging of the apple at the optimal angle; this may damage the peel of the young fruit during bagging, affecting the yield and quality of the apple. 2. Improvements to the YOLOv4 model result in increased weights; 3. YOLOv4 upsampling determines the upsampling kernel solely based on the spatial location of pixels, without utilizing the semantic information of the feature map. It is a "uniform" upsampling method, which cannot accurately capture the global features of an image when the receptive field is small. Summary of the Invention

[0005] To address the shortcomings of existing methods, this invention utilizes semi-supervised learning to improve the generalization ability of the model through a small amount of labeled data and a large amount of unlabeled data; it improves the recognition rate of young fruits by using an improved YOLOv8n-seg network; and it solves the problem of insufficient accuracy in bagging of existing methods.

[0006] The technical solution adopted in this invention is: a method for detecting young fruit in orchards and predicting their growth angle, comprising the following steps: Step 1: Collect image data of young fruit and preprocess the images; As a preferred embodiment of the present invention, the preprocessing includes: contrast enhancement, brightness enhancement, shearing, flipping, rotation, adding Gaussian noise, and annotation.

[0007] Step 2: Input a portion of the labeled data into the supervised network and obtain pseudo-label data by setting a confidence threshold; As a preferred embodiment of the present invention, the supervisory network includes: Deeplab (resnet50), unet, Mask-rcnn, YOLOv5-seg, YOLOv7-seg and YOLOv8n-seg.

[0008] Step 3: The backbone of the YOLOv8n-seg network adopts the MobileNetv3 structure. It uses the Conv_BN_HSwish layer to reduce the dimensionality of the image and improves the ten Bottleneck modules of the backbone. By optimizing the convolutional layers, the network parameters are reduced while improving the computation speed, making the network more focused on useful channel information and optimizing the weights of each channel.

[0009] As a preferred embodiment of the present invention, the improvement of the Bottleneck module includes: in the first Bottleneck module, features are first extracted using DW convolution, then BN normalization and h-swish activation function and SE attention mechanism are introduced; finally, dimensionality reduction is performed using 1×1 convolutional layer and BN normalization.

[0010] As a preferred embodiment of the present invention, the improvement of the Bottleneck module further includes: in the second to tenth Bottleneck modules, dimensionality expansion is first performed using 1×1 convolution, BN normalization and h-swish activation function, then DW convolution and BN normalization feature extraction are performed, and SE attention mechanism is introduced in combination with h-swish activation function to enhance the ability to capture local channel information. Then, 1×1 convolution layer and BN are used for dimensionality reduction output, and the dimensionality reduction output is then residually connected with the input features.

[0011] Step 4: Neck uses the CARAFE upsampling feature extraction module to enhance the global feature extraction capability of the image; Step 5: Introduce the vector angle between the ground truth bounding box and the predicted bounding box to optimize the bounding box loss; In a preferred embodiment of the present invention, the optimized bounding box loss includes: distance loss and angle loss.

[0012] Step 6: Train the young fruit detection model using labeled data and pseudo-labeled data, and estimate the growth angle using images of young fruits.

[0013] In a preferred embodiment of the present invention, the growth angle estimation includes: The coordinate information of the target region of the young fruit was obtained by improving the YOLOv8n-seg network model; Extract the outline coordinates of the young fruit and calculate the straight-line distance between any two points on the outline coordinates. Take the two coordinates with the longest straight-line distance as the coordinates of the two ends of the fruit axis, and calculate the angle between the straight line of the fruit axis and the horizontal line to obtain the growth angle of the young fruit.

[0014] In a preferred embodiment of the present invention, the formula for the growth angle of the young fruit is: (4) in,( x i , y i )and( x j , y j ) represents the coordinates of the two ends of the fruit axis.

[0015] As a preferred embodiment of the present invention, an orchard young fruit detection and growth angle prediction system includes: a memory for storing instructions executable by a processor; and a processor for executing the instructions to implement the orchard young fruit detection and growth angle prediction method.

[0016] In a preferred embodiment of the present invention, a computer-readable medium storing computer program code is provided, which, when executed by a processor, implements a method for detecting young fruit in an orchard and predicting its growth angle.

[0017] The beneficial effects of this invention are: 1. To address the problem of estimating the growth angle of young fruit in orchards, this method obtains both the location information and the growth angle information of the young fruit, thus solving the problem that existing methods cannot accurately obtain the growth angle of young fruit in orchards. 2. To address the issues of high manual costs and heavy workload in existing instance segmentation, we utilize semi-supervised learning and data augmentation to generate pseudo-labels using a supervised model and optimize the model through self-training methods, thereby reducing dependence on labeled data and lowering the cost and workload of data labeling. 3. In order to achieve real-time testing on mobile devices, the lightweight concept of MobileNetv3 network is incorporated, which significantly reduces the computational burden; 4. The SIOU loss function is used to introduce the vector angle between the ground truth box and the predicted box, which makes the predicted box adjust towards the direction of the ground truth box during training, thus speeding up the convergence speed. 5. The CARAFE upsampling feature extraction module is adopted to enhance the network's ability to extract global features from orchard young fruit images. The kernel prediction module and content-aware reconstruction module of the CARAFE upsampling module work together to enable the upsampling process to better capture global semantic information and improve the quality and accuracy of feature maps. Attached Figure Description

[0018] Figure 1 This is a flowchart of the pseudo-label-based semi-supervised target detection process of the present invention; Figure 2 This invention relates to a network model for detecting young fruit in orchards and estimating growth angles. Figure 3 This invention relates to the improved Bootteneck structure based on MobileNetv3. Figure 4 This is a structural model diagram of the CARAFE attention mechanism of the present invention; Figure 5 This is a schematic diagram illustrating the SIOU loss function calculation principle of the present invention; Figure 6 This is a schematic diagram illustrating the estimation of the growth angle of young fruit in an orchard according to the present invention; Figure 7 This is a visualization of the growth angle of young peach fruit according to the present invention. Detailed Implementation

[0019] The present invention will be further described below with reference to the accompanying drawings and embodiments. The drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0020] like Figure 1 As shown, a method for detecting young fruit and predicting growth angle in orchards includes the following steps: Step 1: Collect images of young fruit in the orchard, preprocess the images using data augmentation techniques, and label a small portion of the young fruit images; Using young peaches from an orchard as the research object, images of young peaches were used as the dataset. The cameras used for shooting had a resolution of 3024×4032 pixels. To ensure sample diversity, images were taken in the orchard under different weather conditions, including sunny days, cloudy days, and different times of day (morning and afternoon). To enhance the robustness and generalization of the algorithm, image enhancement processing was performed, including setting the contrast enhancement factor to 1.5, the brightness enhancement factor to 1.5, shearing, flipping, rotating, and adding Gaussian noise to increase the diversity of the dataset. Then, a random portion of the data was selected for image annotation. The annotation tool used was Labelme, and the annotation information included the key point coordinates of the young peach outline.

[0021] Step 2: Employ the self-training method in semi-supervised learning, feed a small portion of labeled data into the supervised network for self-training, and set a confidence threshold to obtain efficient pseudo-label data; like Figure 1 As shown, labeled data is fed into a supervised network for training, and the trained supervised model is used to generate pseudo-labels for semi-supervised learning. To obtain efficient pseudo-label data, the network sets a confidence threshold of 0.8 and uses the prediction results of unlabeled data as pseudo-labels. Simultaneously, data augmentation is performed on the unlabeled data, and pseudo-labels are generated concurrently. Next, labeled data and unlabeled data with pseudo-labels are merged to form an expanded training dataset, which is then input into the network for semi-supervised recurrent learning. Finally, considering both average accuracy and F-score, the optimal model is selected to generate efficient pseudo-label data. The supervised network can employ Deeplab (resnet50), UNET, Mask-RCNN, YOLOv5-seg, YOLOv7-seg, YOLOv8n-seg, etc.

[0022] Step 3: Construct a lightweight orchard young fruit detection and growth angle estimation network model. The backbone network of YOLOv8n-seg is simplified by adopting the improved MobileNetv3 structure, which reduces the number of network parameters, computational cost and model weight size, thereby improving the detection speed. like Figure 2 The lightweight orchard young fruit detection and growth angle estimation network model consists of four parts: input, backbone, neck, and prediction segment. The input uses mosaic data augmentation, adaptive anchor box calculation, and adaptive grayscale filling. The backbone uses Conv_BN_HSwish, Bottleneck, and SPPF structures. The neck module uses a PAN (path aggregation network) structure. The prediction segment obtains feature vectors of three different scales to predict the final result.

[0023] An improved MobileNetv3 architecture is used to simplify the backbone network, thereby reducing the number of network parameters, computational cost, and model weights, thus improving detection speed.

[0024] like Figure 3 As shown in the left figure, the Bottleneck layer of MobileNetv3 is used to extract features from each feature layer. Improvements to the Bottleneck layer include: First, dimensionality reduction of the input image is performed using the Conv_BN_HSwish layer, where the Conv_BN_HSwish function consists of a Conv layer, BN normalization, and an h-swish function. Next, in the first Bottleneck layer, DW convolution is used to extract features, and BN normalization, h-swish activation functions, and a squeeze-and-excitation attention mechanism are introduced to enhance the extraction of key features from small targets. Then, a 1×1 convolutional layer and BN are used for dimensionality reduction, and the output is then residually connected to the input features. The improved scheme further simplifies the structure of the original Bottleneck layer. By optimizing the convolutional layers, the network parameters are reduced while the network's computational speed is improved. This change allows the network to focus more on useful channel information and optimizes the weights of each channel.

[0025] like Figure 3 As shown in the right figure, the remaining Bottleneck layer scheme includes: first, using 1×1 convolution, BN normalization, and h-swish activation function for dimensionality expansion; then performing DW convolution and BN normalization feature extraction; and introducing SE attention mechanism combined with h-swish activation function to enhance the ability to capture local channel information and suppress some feature information that is useless to the current task; finally, using 1×1 convolution layer again for dimensionality reduction to greatly reduce model parameters and computational cost; the third and eighth Bottleneck layers in the backbone are connected to the two concat modules in the neck, respectively.

[0026] The formula for the h-swish activation function is: (1) Step 4: The Neck network uses the CARAFE upsampling feature extraction module to enhance the network's ability to extract global features from orchard young fruit images; like Figure 4 As shown, CARAFE consists of two key components: the upsampling kernel prediction module and the content-aware reassembly module.

[0027] First, the feature map is fed into the kernel prediction module, which uses a 1×1 convolution kernel to compress the feature map, reducing the original number of channels C to Cm; where Cm represents the number of channels in the compressed feature layer. The compression process is to reduce the amount of computation and the number of parameters, so as to process the feature map more efficiently in subsequent operations, as shown in Equation (10): (2) in, This is a multiple of the upsampling factor, typically 2. The size of the predicted upsampling kernel.

[0028] To use the CARAFE upsampling method, the height, width, and number of channels of the feature map are first reshaped according to certain rules; this reshaping process can be implemented using the pixel shuffle method. Specifically, the height, width, and number of channels of the feature map are reshaped as follows: , , The predicted upsampling kernel is obtained, and the prediction result is normalized using Softmax. Then, the feature image is passed to the reconstruction module, and the original feature map is passed to the reconstruction module. On each layer of the feature map, the features are multiplied element-wise with the predicted upsampling kernel to obtain the upsampling result.

[0029] Step 5: Use the SIOU loss function to further optimize the bounding box loss by introducing the vector angle between the ground truth box and the predicted box; The SIOU loss function introduces the vector angle between the ground truth bounding box and the predicted bounding box; the loss function includes angle loss and distance loss, such as... Figure 5 As shown; Figure 5 The left figure shows the angle loss, which is the minimum angle between the connection between the true target center point and the predicted target center point and the xy-axis. for When the angle is 0, the angle loss is 0. During training, if... Then minimize Otherwise minimize .

[0030] Figure 5 The left figure shows the distance loss, which is the distance between the true target center point and the predicted target center point. Its penalty cost is positively correlated with the angle cost. When the distance reaches 0, the contribution of distance cost decreases significantly; conversely, The closer The greater the contribution of distance cost.

[0031] Step 6: Train and validate the orchard young fruit detection and growth angle estimation model using labeled data and pseudo-labeled data; Lightweight training for orchard young fruit detection and growth angle estimation: The model for orchard young fruit detection and growth angle estimation was trained using labeled data and pseudo-labeled data, and then validated.

[0032] The network was jointly trained end-to-end using the Stochastic Gradient Descent (SGD) algorithm. To improve the model's detection accuracy, all input images were resized to 640×640 pixels. The network parameters were optimized using the SGD optimizer. The initial learning rate was set to 0.001, the weight decay rate to 0.005, and the momentum factor to 0.9. The validation period was set to 20, meaning the accuracy of the trained model was tested on the validation set every 20 iterations. The training and validation sets were split in an 8:2 ratio. Training stopped when the model accuracy converged. After training, the trained model was retained and validated using a test set of 300 images.

[0033] By using a trained orchard young fruit detection and growth angle estimation model, the orchard young fruit image is predicted, and the coordinate information of the target area of ​​the orchard young fruit is saved, thereby calculating the growth angle of the young fruit.

[0034] like Figure 6 As shown, the target area for young peach fruit consists of multiple key points, namely... , Based on the shape of the young peach fruit, it can be inferred that the direction of its fruit axis is consistent with the major axis of an ellipse, and the length of the fruit axis is consistent with the longest distance in the young peach fruit. To maintain consistent detection direction, this invention defines the straight line containing the two farthest points in the detected young peach fruit region as (…). This involves calculating the straight-line distance between any two points on the outer contour of the young peach fruit to determine the two farthest points; the angle formed by this distance and the counter-clockwise direction of the x-axis (horizontal to the right) represents the growth posture angle of the young peach fruit. Point A is located above or horizontally to the left of point B. The angle of the target area is calculated by predicting the outer contour coordinates of the peach seedling target region, using the following formula: (3) (4) Among them, through calculation The distance is used to obtain the coordinates of A and B; the angle of line AB is calculated using inverse trigonometric arctan, representing the growth angle of the young peach fruit. .

[0035] Experimental results: Taking young peaches in orchards as the research object, in order to test the accuracy of young peach identification in a real-world environment, the orchard young fruit detection and growth angle estimation network model was deployed on the JETSON AGX ORIN CLB development kit, using the Ubuntu 20.04.6 LTS operating system, Python 3.8 programming language, and PyTorch-1.14 environment to study young peach images. The recognition results on 300 test images were analyzed. These 300 test images covered various scenarios and included 459 young peach targets. The network of this invention accurately identified 403 young peach targets. The verification results show that the model achieved good results in identifying the growth angle of young peaches, meeting the requirements for real-time detection. It should be noted that the recognition module of this invention is mainly applicable to fruits growing downwards.

[0036] When the model of this invention is deployed in the JETSON AGX ORIN CLB development kit, it takes an average of 31ms to detect a 3024×4032 pixel image, which means that it can process 32.15 frames of images per second, realizing real-time detection of young peaches; this shows that the model of this invention can effectively realize the real-time detection of young peaches in natural scenes on JETSON AGX ORIN CLB.

[0037] Based on the above-described preferred embodiments of the present invention, and through the foregoing description, those skilled in the art can make various changes and modifications without departing from the inventive concept. The technical scope of this invention is not limited to the contents of the specification, but must be determined according to the scope of the claims.

Claims

1. A method for detecting young fruit in an orchard and predicting its growth angle, characterized in that, Includes the following steps: Step 1: Collect image data of young fruit and preprocess the images; Step 2: Feed some of the labeled data into the supervised network and obtain pseudo-label data through the confidence threshold; Step 3: The backbone of the YOLOv8n-seg network adopts the MobileNetv3 structure. It uses the Conv_BN_HSwish layer to reduce the dimensionality of the image and improves the ten Bottleneck modules. By optimizing the convolutional layers, the network parameters are reduced while the operation speed is improved, making the network more focused on useful channel information and optimizing the weights of each channel. Improvements to the Bottleneck module include: in the first Bottleneck module, features are first extracted using DW convolution, then BN normalization, h-swish activation function, and SE attention mechanism are introduced; finally, dimensionality reduction is performed using 1×1 convolutional layers and BN normalization. Improvements to the Bottleneck module also include: In the second to tenth Bottleneck modules, 1×1 convolution, BN normalization and h-swish activation function are first used to expand the dimension, then DW convolution and BN normalization are used to extract features, and SE attention mechanism is introduced in combination with h-swish activation function. Then 1×1 convolution layer and BN are used to reduce the dimension of the output, and then residual connection is performed with the input features. Step 4: Neck uses the CARAFE upsampling feature extraction module to enhance the global feature extraction capability of the image; Step 5: Introduce the vector angle between the ground truth bounding box and the predicted bounding box to optimize the bounding box loss; Step 6: Train the young fruit detection model using labeled data and pseudo-labeled data, and estimate the growth angle using young fruit images; Growth angle estimation includes: The coordinate information of the target region of the young fruit was obtained by improving the YOLOv8n-seg network model; Extract the outline coordinates of the young fruit and calculate the straight-line distance between any two points on the outline coordinates. Take the two coordinates with the longest straight-line distance as the coordinates of the two ends of the fruit axis and calculate the angle between the straight line of the fruit axis and the horizontal line to obtain the growth angle of the young fruit. The formula for the growth angle of young fruit is: （4） in,( x i , y i )and( x j , y j ) represents the coordinates of the two ends of the fruit axis.

2. The method for detecting young fruit in orchards and predicting growth angle according to claim 1, characterized in that, Preprocessing includes: Contrast enhancement, brightness enhancement, shearing, flipping, rotation, adding Gaussian noise, and annotation.

3. The method for detecting young fruit in orchards and predicting growth angle according to claim 1, characterized in that, The supervised networks include: Deeplab, unet, Mask-rcnn, YOLOv5-seg, YOLOv7-seg, and YOLOv8n-seg.

4. The method for detecting young fruit in orchards and predicting growth angle according to claim 1, characterized in that, Optimizing the bounding box loss includes: distance loss and angle loss.

5. An orchard young fruit detection and growth angle prediction system, characterized in that, include: Memory is used to store instructions that can be executed by the processor; A processor for executing instructions to implement the orchard young fruit detection and growth angle prediction method as described in any one of claims 1-4.

6. A computer-readable medium storing computer program code, characterized in that, The computer program code, when executed by a processor, implements the orchard young fruit detection and growth angle prediction method as described in any one of claims 1-4.

Images