Method for rapidly detecting pears in natural environment based on lightweight Transform architecture

Abstract

The invention discloses a quick pear detection method in a natural environment based on a lightweight Transform architecture. The quick pear detection method comprises the following steps: A, acquiring pear images in different environments; b, making a pear fruit data set; c, constructing a pome detection model; and D, inputting the to-be-identified image into the pome detection model to carry out pome rapid detection. The defects in the prior art can be overcome, the complexity of a detection model is reduced, and the detection speed and precision are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of target image detection, and in particular to a method for rapid detection of pear fruits in a natural environment based on a lightweight Transformer architecture. Background Art

[0002] The pear industry has a low level of mechanization, and the picking process requires a lot of labor, accounting for 35%-45% of the total labor input, and the picking costs account for 50%-70% of all links. The overall process of intelligent and mechanized picking can be roughly divided into two parts: one is to achieve rapid identification of pears, and the other is to accurately and non-destructively grasp the identified fruits. However, due to the influence of lighting changes, occlusion of branches and leaves, overlapping fruits, and changes in distance environment, the model in the actual picking environment is prone to problems such as difficulty in identification and slow detection speed. Therefore, how to quickly and accurately identify pears has become the primary problem of automated picking.

[0003] The YOLO model has certain advantages in both accuracy and speed, but it usually requires non-maximum suppression (NMS) for post-processing, and introduces hyperparameters that make the accuracy and speed of the model unstable. In addition, a reasonable NMS threshold needs to be selected, which hinders the development of real-time object detection.

[0004] RT-DETR performs slightly worse than YOLO in small object detection, but is superior to current real-time detectors of similar size in terms of speed and accuracy. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a method for rapid detection of pear fruits in a natural environment based on a lightweight Transformer architecture, which can solve the shortcomings of the existing technology, reduce the complexity of the detection model, and improve the detection speed and accuracy.

[0006] In order to solve the above technical problems, the technical solutions adopted by the present invention are as follows.

[0007] A method for rapid detection of pear fruit in a natural environment based on a lightweight Transformer architecture comprises the following steps:

[0008] A. Collect pear fruit images in different environments;

[0009] B. Create a pear fruit dataset;

[0010] C. Construct a pear fruit detection model;

[0011] D. Input the image to be identified into the pear fruit detection model for rapid pear fruit detection.

[0012] Preferably, in step A, pear fruit images are collected during the ripening period and the picking period, the image collection time is 7:00-12:00 and 13:00-19:00 every day, and the image shooting distance is 100-500 mm and 700-1350 mm.

[0013] Preferably, in step B, labelImg is first used to annotate several pear fruit images in different scenes, the annotated images are trained using the YOLOv7 model to obtain the optimal weight file, and then the remaining pear fruit images are semi-automatically annotated, and finally labelImg is used to manually adjust the annotation results.

[0014] Preferably, the pear fruit detection model consists of a lightweight FasterNet backbone network, a hybrid encoder, and a transformer decoder with an auxiliary prediction head.

[0015] Preferably, the lightweight FasterNet backbone network includes four hierarchical stages, and each stage of the FasterNet Block is connected to an embedding layer or a merging layer, which are used for spatial downsampling and channel number expansion respectively; the step size of the embedding layer is 4, and the size of the convolution kernel is Conv4×4; the step size of the merging layer is 2, and the size of the convolution kernel is Conv2×2; each directional residual block is composed of a front-end PConv layer and two back-end PWConv layers, which are used to make full use of information from all channels, wherein a shortcut link consisting of a batch normalization and a rectified linear unit activation function is placed between the two PWConv layers, which is used to strengthen the input features of the image, thereby improving the performance and training speed of the model.

[0016] Preferably, the hybrid encoder consists of an intra-scale feature interaction module and a cross-scale feature fusion module, and is used to convert multi-scale features into an image feature sequence.

[0017] Preferably, in the intra-scale feature interaction module, the HiLo attention mechanism is used to divide the multi-head self-attention layer into two components, one component is used to encode the high-frequency attention branch Hi-Fi with a local self-attention mechanism and a high-resolution feature map, and the other component is used to encode the low-frequency attention branch Lo-Fi through downsampling features; the output of each HiLo attention mechanism is the concatenation of the high-frequency attention branch and the low-frequency attention branch, that is, HiLo = [Hi-Fi (X); Lo-Fi (X)]; the scaled dot product attention in the improved space reduction strategy is used to adapt the two high and low frequency attention branches, one path encodes high-frequency interactions through scaled dot product attention and relatively high-resolution feature maps, and the other path encodes low-frequency interactions through dot product attention and downsampling features, so as to improve information extraction efficiency.

[0018] Preferably, the cross-scale feature fusion module includes a lightweight convolution GSConv, which inputs a downsampled ordinary convolution, then uses DWConv deep convolution to splice the output results of the two convolutions, and finally performs a data allocation shuffle operation to connect the corresponding number of channels of the previous two convolutions in series; it is used to overcome the problem of partial loss of semantic information due to spatial compression and channel expansion of feature maps.

[0019] As a preferred method, the loss function and the small target detection mechanism are combined to perform perceptual query, and a certain number of image features are selected from the encoder output sequence as the initial object query of the decoder; the loss function after the loss function and the small target detection mechanism are combined is:

[0020] Shape_NWD=(1-IoU)(1-0.5NWD)+IoU(1-0.5ShapeIoU), where,

[0021] Shape-IoU=1-IoU+distance shape +0.5×Ω shape ,

[0022]

[0023]

[0024]

[0025] IoU is the actual intersection-over-union ratio, ww and hh represent the weight coefficients in the horizontal and vertical directions respectively, and their values ​​are related to the shape of the GT box.

[0026] Preferably, in step D, the rapid detection of pear fruit comprises the following steps:

[0027] The fruit image is input into the lightweight FasterNet backbone network, and the features of the last three stages of the backbone network are input into the hybrid encoder; the hybrid encoder converts multi-scale features into image feature sequences through the intra-scale feature interaction module and the cross-scale feature fusion module; the Shape_NWD small target detection mechanism is used to jointly perceive the query, and a certain number of image features are selected from the encoder output sequence as the initial object query of the decoder; the decoder with an auxiliary prediction head iteratively optimizes the object query to generate bounding boxes and confidence scores.

[0028] The beneficial effects brought about by adopting the above technical scheme are as follows: based on RT-DETR, the present invention replaces the backbone of the original model with a lightweight FasterNet network, and secondly uses HiLo, an efficient attention mechanism for extracting high- and low-frequency information, to improve the intra-scale feature interaction module, and introduces a GSConv convolution and a cross-stage partial network module VoVGSCSP in the cross-scale feature fusion module, and finally replaces the loss function in the original model with Shape_NWD, thereby improving the detection performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a flow chart of the steps of the present invention.

[0030] Figure 2 It is a structural block diagram of the network model of the present invention.

[0031] Figure 3 It is a schematic diagram of Shape-IoU parameters of the present invention.

[0032] Figure 4 It is the detection result of the fruit target of New Pear No. 7 under different environments using the detection method of the present invention. DETAILED DESCRIPTION

[0033] Reference Figure 1 , the image data collection location of this embodiment is the modern pear orchard demonstration base of Hebei Wokang Agricultural Technology Co., Ltd., the pear variety collected is the new pear No. 7, and the collection time is July-August 2024. Data collection is designed with two modes: close-up shooting (100-500mm) and long-range shooting (700-1350mm). In order to make the time distribution of the collected data set closer to the actual picking time of fruit farmers, the shooting time of this data set is selected in two time periods: 7:00-12:00 and 13:00-19:00. A total of 7468 images of the new pear No. 7 in different environments and scenes were collected, including 4001 close-up images and 3467 long-range images.

[0034] In order to improve the labeling efficiency, YOLO semi-automatic labeling was adopted. First, labelImg was used to label 2000 new pear No. 7 fruit images in different scenes. The YOLOv7 model was applied to train the labeled images to obtain the optimal weight file, and then the remaining new pear No. 7 fruit images were semi-automatically labeled. Finally, labelImg was used to manually adjust the labeling results to prevent mislabeling and missing labels.

[0035] After manual and semi-automatic annotation, a total of 7468 XML files of New Pear No. 7 were obtained, and they were randomly divided into training set, validation set and test set in a ratio of 8:1:1. The training set contained 5974 images, and the validation set and test set each contained 747 images.

[0036] Reference Figure 2 ,The pear fruit detection model consists of a lightweight FasterNet backbone network, a hybrid encoder and a Transformer decoder with an auxiliary prediction head.

[0037] Based on RT-DETR, we improved a fast pear fruit detection model in natural environment. First, the backbone of the original model was replaced with a lightweight FasterNet network. Secondly, the intra-scale feature interaction module was improved using HiLo, an efficient attention mechanism for extracting high- and low-frequency information. A GSConv convolution and a cross-stage partial network module VoVGSCSP were introduced into the cross-scale feature fusion module. Finally, the loss function in the original model was replaced with Shape-NWD, which improved the detection performance of the model.

[0038] The FasterNet backbone network has four hierarchical stages. Each stage of the FasterNet Block is connected to an embedding layer (with a step size of 4, Conv4×4) or a merging layer (with a step size of 2, Conv2×2), which are used for spatial downsampling and channel number expansion respectively. In order to fully and efficiently utilize the information from all channels, each directional residual block consists of a front-end PConv layer and two back-end PWConv layers. A batch normalization (BatchNormalization, BN) and a rectified linear unit (Rectified Linear Unit, ReLU) activation function are placed between the two PWConv layers to strengthen the input features of the new pear No. 7 fruit image, thereby improving the performance and training speed of the model.

[0039] The HiLo attention mechanism divides the MSA layer into two components: one is used to encode the high-frequency attention branch Hi-Fi with local self-attention mechanism and high-resolution feature map, and the other is used to encode the low-frequency attention branch Lo-Fi through global encoding of downsampled features. The output of each HiLo attention mechanism is the concatenation of the high-frequency attention branch and the low-frequency attention branch, that is, HiLo is HiLo = [Hi-Fi (X); Lo-Fi (X)].

[0040] The scaled dot-product attention in the space reduction strategy is improved to adapt the two high- and low-frequency attention branches. One path encodes high-frequency interactions through scaled dot-product attention and relatively high-resolution feature maps, while the other path encodes low-frequency interactions through dot-product attention and down-sampled features, thereby effectively improving the information extraction efficiency, which is more effective than the standard MSA.

[0041] In order to make the pear picking robot further meet the requirements of real-time detection, we introduced lightweight convolution GSConv in the cross-scale feature fusion module (CCFM) to further reduce the complexity of the model while maintaining accuracy and solve the speed problem of prediction calculation in convolutional neural networks. First, GSConv inputs a downsampled ordinary convolution, then uses DWConv deep convolution to splice the output results of the two Convs, and finally performs a data distribution shuffle operation to connect the corresponding channels of the previous two convolutions in series. Therefore, when the spatial information of the input image is gradually transferred to the channel, the GSConv convolution avoids the phenomenon that each feature map is compressed in space (width and height) and the channel is expanded, resulting in partial loss of semantic information.

[0042] In the RT-DETR algorithm, the prediction box regression loss function uses GIoU. When two prediction boxes have the same height and width and are on the same horizontal plane, GIoU degenerates into an IOU loss function, which leads to slow convergence and inaccurate regression. To address the above problems, we use the Shape_NWD loss function to replace the GIoU loss function used by RT-DETR. This method can calculate the loss by focusing on the shape and scale of the bounding box itself, making the bounding box regression more accurate. Figure 3 Schematic diagram of Shape-IoU parameters.

[0043] The allocation ratio strategy of 50% is selected to maximize the detection evaluation index of this model, and the Shape_NWD loss function is:

[0044] Shape_NWD=(1-IoU)(1-0.5NWD)+IoU(1-0.5ShapeIoU),

[0045] Among them, Shape-IoU = 1-IoU + distance shape +0.5×Ω shape ,

[0046]

[0047]

[0048]

[0049] IoU is the actual intersection-over-union ratio, ww and hh represent the weight coefficients in the horizontal and vertical directions respectively, and their values ​​are related to the shape of the GT box.

[0050] Compared with traditional loss functions, Shape_NWD better handles the impact of shape and scale factors on regression results in bounding box regression, especially in small target detection tasks. This strategy can not only improve target detection accuracy but also ensure model detection speed.

[0051] The specific detection process is as follows: the fruit image is input into the lightweight FasterNet backbone network, and the features of the last three stages of the backbone network are input into the hybrid encoder; the hybrid encoder converts multi-scale features into image feature sequences through the intra-scale feature interaction module and the cross-scale feature fusion module; the Shape_NWD small target detection mechanism is used to jointly perform perceptual queries, and a certain number of image features are selected from the encoder output sequence as the initial object query of the decoder; the decoder with an auxiliary prediction head iteratively optimizes the object query to generate bounding boxes and confidence scores.

[0052] The RT-DETR was compared with the YOLOv5m, YOLOv7, YOLOv8m and YOLOv10m models. The results showed that RT-DETR can achieve a good balance between model lightweight and recognition accuracy compared with other models, and can realize the rapid detection of the new pear No. 7 fruit.

[0053] The precision, recall and average precision of the model of the present invention reached 93.7%, 91.9% and 98% respectively, and compared with the original model, the number of parameters, the amount of calculation and the model size were reduced by 48.47%, 56.2% and 48.31% respectively. The study showed that the model provided technical support for the detection of new pear No. 7 fruit and the deployment and application of the model in complex environments. The results of the ablation test are shown in the table below.

[0054]

[0055]

[0056] Figure 4 The results of the original model and the improved model on the detection of the new pear No. 7 fruit target under different scene conditions are shown. Both the RT-DETR model and the improved model can accurately identify the new pear No. 7 fruit target in the image under the close view of the front light, but under the conditions of branch and leaf occlusion, close view of the back light, fruit overlap, long view of the front light, and long view of the back light, various degrees of missed detection occur, such as Figure 4 As shown in the thick solid box, the improved model can detect the missed fruits; and the improved model has a higher confidence in different environments, such as Figure 4The confidence of the fruit detected by the improved model is 17% higher than that of RT-DETR for the fruit of the new pear No. 7 that is blocked by the fruit shown in column b. The test results show that in the complex environment of occlusion by branches and leaves, overlapping fruits, and reverse and forward lighting, the improved model shows better detection effect and higher confidence, meeting the requirements of the new pear No. 7 fruit picking robot for recognition accuracy.

[0057] In the description of the present invention, it should be understood that the terms "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside" and "outside" etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0058] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A method for rapid detection of pear fruit in natural environment based on lightweight Transformer architecture, characterized in that The following steps are involved: A. Collect pear fruit images in different environments; B. Create a pear fruit dataset; C. Construct a pear fruit detection model; D. Input the image to be identified into the pear fruit detection model for rapid pear fruit detection.

2. The method for rapid detection of pear fruit in natural environment based on lightweight Transformer architecture according to claim 1 is characterized in that: In step A, pear fruit images are collected during the ripening period and the picking period, respectively. The image collection time is 7:00-12:00 and 13:00-19:00 every day, and the image shooting distance is 100-500 mm and 700-1350 mm.

3. The method for rapid detection of pear fruit in natural environment based on lightweight Transformer architecture according to claim 1 is characterized in that: In step B, labelImg is first used to annotate several pear fruit images in different scenes. The annotated images are trained using the YOLOv7 model to obtain the optimal weight file, and then the remaining pear fruit images are semi-automatically annotated. Finally, labelImg is used to manually adjust the annotation results.

4. The method for rapid detection of pear fruit in natural environment based on lightweight Transformer architecture according to claim 1 is characterized in that: The pear fruit detection model consists of a lightweight FasterNet backbone network, a hybrid encoder, and a Transformer decoder with an auxiliary prediction head.

5. The method for rapid detection of pear fruit in natural environment based on lightweight Transformer architecture according to claim 4 is characterized in that: The lightweight FasterNet backbone network includes four hierarchical stages. The FasterNetBlock in each stage is connected to an embedding layer or a merging layer, which are used for spatial downsampling and channel number expansion respectively; the step size of the embedding layer is 4, and the size of the convolution kernel is Conv4×4; the step size of the merging layer is 2, and the size of the convolution kernel is Conv2×2; each directional residual block consists of a front-end PConv layer and two back-end PWConv layers, which are used to make full use of information from all channels, and a shortcut link consisting of batch normalization and rectified linear unit activation function is placed between the two PWConv layers to strengthen the input features of the image, thereby improving the performance and training speed of the model.

6. The method for rapid detection of pear fruit in natural environment based on lightweight Transformer architecture according to claim 5 is characterized in that: The hybrid encoder consists of an intra-scale feature interaction module and a cross-scale feature fusion module, which is used to convert multi-scale features into image feature sequences.

7. The method for rapid detection of pear fruit in natural environment based on lightweight Transformer architecture according to claim 6 is characterized in that: In the intra-scale feature interaction module, the HiLo attention mechanism is used to divide the multi-head self-attention layer into two components, one component is used to encode the high-frequency attention branch Hi-Fi with a local self-attention mechanism and a high-resolution feature map, and the other component is used to encode the low-frequency attention branch Lo-Fi through down-sampling features; the output of each HiLo attention mechanism is the concatenation of the high-frequency attention branch and the low-frequency attention branch, that is, HiLo = [Hi-Fi (X); Lo-Fi (X)]; the scaled dot product attention in the improved space reduction strategy is used to adapt the two high and low frequency attention branches, one path encodes the high-frequency interaction through the scaled dot product attention and the relatively high-resolution feature map, and the other path encodes the low-frequency interaction through the dot product attention and the down-sampling features, so as to improve the efficiency of information extraction.

8. The method for rapid detection of pear fruit in natural environment based on lightweight Transformer architecture according to claim 7 is characterized in that: The cross-scale feature fusion module includes a lightweight convolution GSConv, which inputs a downsampled ordinary convolution, then uses DWConv deep convolution to splice the output results of the two convolutions, and finally performs a data distribution shuffle operation to connect the corresponding number of channels of the previous two convolutions in series; it is used to overcome the problem of partial loss of semantic information due to spatial compression and channel expansion of feature maps.

9. The method for rapid detection of pear fruit in natural environment based on lightweight Transformer architecture according to claim 8, characterized in that: The loss function and the small target detection mechanism are combined to form a perceptual query, and a certain number of image features are selected from the encoder output sequence as the initial object query of the decoder; the loss function after the loss function and the small target detection mechanism are combined is: Shape_NWD=(1-IoU)(1-0.5NWD)+IoU(1-0.5ShapeIoU), in, Shape-IoU=1-IoU+distance shape +0.5×Ω shape , IoU is the actual intersection-over-union ratio, ww and hh represent the weight coefficients in the horizontal and vertical directions respectively, and their values ​​are related to the shape of the GT box.

10. The method for rapid detection of pear fruit in natural environment based on lightweight Transformer architecture according to claim 9, characterized in that: In step D, the rapid detection of pear fruit comprises the following steps: The fruit image is input into the lightweight FasterNet backbone network, and the features of the last three stages of the backbone network are input into the hybrid encoder; the hybrid encoder converts multi-scale features into image feature sequences through the intra-scale feature interaction module and the cross-scale feature fusion module; the Shape_NWD small target detection mechanism is used to jointly perceive the query, and a certain number of image features are selected from the encoder output sequence as the initial object query of the decoder; The decoder with an auxiliary prediction head iteratively refines the object query, producing bounding boxes and confidence scores.

Images