A lightweight real-time apple detection method for harvesting robots

Abstract

This invention discloses a lightweight real-time apple detection method for harvesting robots. This method further improves the YOLOv4 algorithm, introducing MobileNetv3-large and depthwise separable convolutions to lightweight the model's network structure specifically for apple detection. A coordinate attention mechanism is introduced into the feature fusion network to mitigate the accuracy loss caused by model simplification, enhancing the model's ability to detect overlapping and occluded apples. Furthermore, a learning strategy combining cross-domain and intra-domain transfer learning is used during model training to improve the overall generalization ability of the model. The method disclosed in this invention is easier to deploy on embedded platforms, offering high response speed while maintaining accuracy. It can efficiently and in real-time detect apples in complex orchard environments, meeting the detection requirements of harvesting robots.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to a real-time apple detection method based on an improved YOLOv4. Background Technology

[0002] Apples are one of the largest fruit-producing crops in my country, with apple orchards accounting for approximately 18% of the country's total orchards and an annual output of about 41.39 million tons. However, due to the complex environment of orchards, apple harvesting is still mainly done manually, resulting in high costs and low efficiency. Therefore, researching apple-harvesting robots to replace manual labor for automated harvesting is of great significance. Harvesting robots mainly consist of a vision system and a robotic arm system. The robotic arm system is guided by the vision system to complete the fruit harvesting; therefore, rapid and accurate identification and positioning of the fruit is key to achieving automated harvesting.

[0003] In recent years, deep learning technology has continued to develop, and apple detection algorithms based on convolutional neural networks have been widely used due to their strong robustness, adaptability, and high accuracy. Among these, algorithms applied to apple detection tasks are mainly divided into two categories: one is region-propagation-based methods, with notable algorithms including R-CNN, Fast R-CNN, and Faster R-CNN. The core idea is to first obtain proposed regions and then classify within the current region, known as a two-stage object detection method. The other category is region-propagation-free methods, with notable algorithms including SSD and YOLO. The core idea is to use a single convolutional network to directly predict the location and attributes of the target based on the entire image, also known as a one-stage object detection method. These methods generally first acquire image data of the target to be detected, perform preprocessing and annotation to construct a dataset, then set model hyperparameters, train the network model using the training set, and select the best model as the final apple detection model.

[0004] Existing mainstream apple detection methods based on convolutional neural networks still have problems in different applications due to their different focuses. Two-stage algorithms prioritize detection accuracy, but the complex network structure leads to poor real-time performance. One-stage algorithms prioritize speed, but when faced with apple detection tasks in unstructured natural environments, they suffer from high false positive rates and serious missed detections due to fruit overlap, foliage occlusion, and changes in lighting. In addition, harvesting robots are mostly equipped with embedded devices to deploy recognition tasks, resulting in limited computing resources. Therefore, the algorithms need to achieve high real-time detection speed while being limited in computational load and parameter count. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art, solve the problems of high false detection rate and serious missed detection of apples in unstructured natural environments due to overlapping, shading and light effects, as well as the poor real-time performance of complex networks when deployed on embedded platforms, and provide a lightweight real-time apple detection method for picking robots.

[0006] The technical solution adopted by the present invention to achieve the above objectives is: a lightweight real-time apple detection method for harvesting robots, the method comprising the following steps:

[0007] (1) Build an Apple real-time detection network based on YOLOv4;

[0008] (2) Data preprocessing and dataset construction;

[0009] (3) Use the data in the dataset to train the model, and adjust and optimize the model parameters;

[0010] (3) Use an optimized network model to identify the image to be detected.

[0011] Step (1) includes the following steps:

[0012] (1-1) Introducing MobileNetv3-large and depthwise separable convolution to improve the lightweight network structure;

[0013] (1-2) Introduce coordinate attention mechanism into feature fusion network.

[0014] Step (1-1) includes the following steps:

[0015] (1-1-1) Replace the CSPDarkNet53 in the original feature extraction network based on YOLOv4 with the first 17 feature layers of MobileNetv3-large, introduce the spatial pyramid module SPP in the last layer of the network, and extract the feature maps extracted by the MobileNetv3-large feature layers 8 and 14 and the SPP module as input to the subsequent feature fusion network.

[0016] (1-1-2) Replace the ordinary convolution in the feature fusion network with depthwise separable convolution.

[0017] The construction of the coordinate attention mechanism module in step (1-2) includes the following steps:

[0018] (1-2-1) A spatial attention mechanism module is introduced after the Concat layer in the bottom-up part of the PAN structure in the feature fusion network. Two spatial pooling kernels are used to perform the encoding operations shown in equations (1)-(2) along the horizontal and vertical coordinates respectively to obtain a C×H×1 feature map. Feature maps of C×1×W

[0019]

[0020]

[0021] in, This represents the feature of the c-th channel at height h after encoding. Let x be the feature of the c-th channel at width w after encoding. c (h,i) represents the input feature of the c-th channel at height h and width i, x c (j,w) represents the input feature of the c-th channel at height j and width w, where h is the height of the input feature map and w is the width of the input feature map;

[0022] (1-2-2) Perform the operation shown in equation (3) on the two feature maps generated in (1-2-1) to form a C×1×(H+W) feature map. Use 1×1 convolution to compress its channel dimension from C dimension to C / r dimension with a shrinkage rate r, and activate it using a nonlinear function to obtain the intermediate attention map f. Then, decompose the obtained result f along the spatial dimension into a C / r×H×1 horizontal attention tensor f. h The vertical attention tensor f of C / r×1×W w ;

[0023] f=δ(F1([z h ,z w ])) (3)

[0024] Where [.,.] represents the splicing operation along the spatial dimension, F1 is the dimensionality reduction operation along the channel dimension using 1×1 convolution, δ is the nonlinear activation function, f is the intermediate attention graph containing horizontal and vertical attention information, and z h This is a horizontal attention diagram before splicing, z w This is a vertical view before splicing.

[0025] (1-2-3) The two spatial tensors f generated by (1-2-2) h and f w Perform the operation shown in equation (4-5), using two sets of 1×1 convolutions to increase the channel dimension from C / r to C dimensions, and then using the Sigmoid function for non-linear activation to obtain the coordinate attention module g that embeds the feature information. h and g w ;

[0026] g h =σ(F h (f h(4)

[0027] g w =σ(F w (f w (5)

[0028] Among them, f h For the horizontal attention tensor, F h For the dimensionality increase operation of the horizontal attention graph, f w For the vertical attention tensor, F w This is an upscaling operation for the vertical attention map, where σ is the Sigmoid activation function.

[0029] (1-2-4) Using the attention module g from (1-2-3) h and g w By performing weighting operations on the input feature map and applying coordinate attention, the structure of the real-time apple detection network model is obtained.

[0030] Step (2) includes the following steps:

[0031] (2-1) Construct the dataset;

[0032] (2-2) Prior box extraction;

[0033] (2-3) Data augmentation.

[0034] The dataset in step (2-1) includes two datasets with different attributes: a single apple dataset without background and an apple dataset in natural environment, which are further divided into training set, validation set and test set.

[0035] In step (2-2), the K-means algorithm is used to cluster the bounding box sizes in the label data of the natural environment apple dataset described in step (2-1), and the clustering results are used as model prior boxes.

[0036] Step (2-3) performs data augmentation on the backgroundless single apple dataset described in step (2-1), including adjusting the brightness of the image and scaling the image based on the prior bounding box size obtained in step (2-2).

[0037] Step (3) employs a learning strategy that combines cross-domain transfer and intra-domain transfer for training, including the following steps:

[0038] (3-3-1) Cross-domain transfer learning: The model is initialized using a general object detection dataset, and the network parameters are adjusted by training and validation data of a single apple dataset without background, in order to learn apple features;

[0039] (3-3-2) Intradomain transfer learning: Using the model trained in (2-3-1), the network parameters are further trained on the training and validation sets of apples in the natural environment to learn the apple features in the unstructured natural environment.

[0040] (3-3-3) In the above process, update and iterate the training network weights and set hyperparameters; use the model network weights and hyperparameters with the lowest loss on the validation set as the final optimized network.

[0041] Step (4) includes the following steps:

[0042] (4-1) Read the image to be detected; the image to be detected is a real-time image or a test set image;

[0043] (4-2) Input the image to be detected into the trained real-time apple detection network for recognition and localization;

[0044] (4-3) Repeat steps (4-1) to (4-2) until the detection of images in the test set is completed.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] (1) Using the one-stage detection algorithm YOLOv4 as the basic framework, and combining MobileNetv3-large and depthwise separable convolution to improve the network lightweighting, reducing the computational cost and parameter count of the model;

[0047] (2) Introduce the coordinate attention mechanism into the feature fusion network to improve the network’s recognition effect on dense targets and improve the accuracy loss caused by branch and leaf occlusion and fruit overlap in apple detection.

[0048] (3) For the apple detection task in the natural environment, a learning strategy that combines cross-domain transfer and intra-domain transfer is adopted for training, which effectively improves the generalization ability of the model and improves the detection accuracy of the model.

[0049] (4) It is easy to deploy on an embedded platform and has a high detection speed while ensuring accuracy, which can meet the detection task requirements of the picking robot.

[0050] (5) This invention proposes a lightweight real-time apple detection method for picking robots. This method further improves the YOLOv4 algorithm. The idea is to lighten the feature extraction network and feature fusion network of the model for apple detection tasks, and use the attention mechanism to alleviate the accuracy loss caused by model simplification, enhance the model's ability to detect overlapping and occluded apples, and introduce a learning strategy that combines cross-domain transfer and intra-domain transfer during the model training stage to improve the overall generalization ability of the model, so that the model can efficiently and in real-time detect apples in complex orchard environments when deployed on embedded platforms. Attached Figure Description

[0051] Figure 1 This is a flowchart of the present invention;

[0052] Figure 2 This is a schematic diagram of the Space Pyramid Module (SPP) structure.

[0053] Figure 3 A schematic diagram of the PAN network structure for applying the coordinate attention mechanism;

[0054] Figure 4 This is a schematic diagram of the coordinate attention mechanism structure;

[0055] Figure 5 A schematic diagram of the network structure of the Apple real-time detection model;

[0056] Figure 6 This is a diagram showing the model's performance in detecting densely packed apples at close range in a natural environment.

[0057] Figure 7 This is a diagram showing the model's performance in detecting densely packed apples at medium distances in a natural environment.

[0058] Figure 8 This is a diagram showing the model's performance in detecting densely packed apples at a long distance in a natural environment.

[0059] Here, concat means concatenation, conv represents a convolutional layer, conv*5 represents 5 convolutional layers, upsampling represents upsampling operation, downsampling represents downsampling operation, ca represents the spatial attention mechanism module, and yolo head represents the model prediction layer. Detailed Implementation

[0060] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0062] This invention uses apples from the natural environment as the detection object. The training platform configuration is as follows: CPU: Intel(R) Core(TM) i7-8700 3.20GHz, RAM: 16GB; GPU: NVIDIA TITAN V, VRAM: 12GB; operating environment: Windows 10; Python version: 3.6. The method of this invention is implemented on the deep learning framework PyTorch version 1.2.0, CUDA version: 10.0, and cuDNN version: 7.4.1.

[0063] like Figure 1 The lightweight real-time apple detection method for harvesting robots shown comprises four parts:

[0064] (1) Build an Apple real-time detection network based on YOLOv4;

[0065] (2) Data preparation;

[0066] (3) Model training;

[0067] (4) Recognize the image to be detected.

[0068] The real-time apple detection network of this invention is an improvement on the one-stage target detection network YOLOv4. The specific network structure improvement includes the following steps:

[0069] (1-1) Lightweight improvement of network structure

[0070] The apple detection task involved in this invention has high real-time requirements. Considering that apple detection algorithms for picking robots are mostly deployed on embedded devices with limited computing resources, lightweight improvements to the model can reduce network computational complexity and improve the detection speed of the model on low-computing-power platforms.

[0071] The original feature extraction network CSPDarkNet53 was replaced with the first 17 feature layers of the lightweight neural network MobileNetv3-large, as shown in Table 1, which significantly reduced the computational cost of the model. A feature extraction method was introduced in the last layer of the network. Figure 2 The Spatial Pyramid Module (SPP) shown uses four max pooling modules to increase the receptive field and extracts the feature maps from feature layers 8 and 14 and the SPP module as input to the subsequent feature fusion network. The ordinary convolution in the feature fusion network is replaced with depthwise separable convolution. Depthwise separable convolution decomposes the convolution process into channel-wise convolution and pointwise convolution, further compressing the model and improving the model's computational efficiency.

[0072] Table 1 MobileNetv3-large

[0073]

[0074]

[0075] (1-2) Introducing coordinate attention mechanism into feature fusion network

[0076] To compensate for the accuracy loss caused by network simplification, this invention introduces a coordinate attention module into the feature fusion network. This module is a lightweight attention mechanism module for computer vision proposed by Hou et al. in 2021. It can be easily inserted between two convolutional layers to enhance target attention and improve the detection of dense targets and the ability to resist background interference.

[0077] like Figure 3 As shown, in the feature fusion network (which includes two parts, SPP and PAN), a coordinate attention mechanism module (CA) is introduced after the Concat layer in the bottom-up part of the PAN structure. The schematic diagram of the CA module structure is shown below. Figure 4 As shown, two spatial pooling modules are used to perform the encoding operations shown in equations (1)-(2) along the horizontal and vertical coordinate directions respectively to obtain a C×H×1 feature map. and the feature map of C×1×W

[0078]

[0079]

[0080] in, This represents the feature of the c-th channel at height h after encoding. Let x be the feature of the c-th channel at width w after encoding. c (h,i) represents the input feature of the c-th channel at height h and width i, xc (j,w) represents the input feature of the c-th channel at height j and width w, where h is the height of the input feature map and w is the width of the input feature map.

[0081] The two generated feature maps are processed as shown in Equation (3) to form a C×1×(H+W) feature map. A 1×1 convolution is then used to compress its channel dimension from C to C / r dimensions with a shrinkage rate r. ReLU is then used for non-linear activation to obtain the intermediate attention map f. The obtained result f is then decomposed along the spatial dimension into a C / r×H×1 horizontal attention tensor f. h The vertical attention tensor f of C / r×1×W w ;

[0082] f=δ(F1([z h ,z w ])) (3)

[0083] Where [.,.] represents the splicing operation along the spatial dimension, F1 is the dimensionality reduction operation along the channel dimension using 1×1 convolution, δ is the nonlinear activation function, f is the intermediate attention graph containing horizontal and vertical attention information, and z h This is a horizontal attention diagram before splicing, z w This is a vertical view before splicing.

[0084] (1-2-3) The two spatial tensors f generated by (1-2-2) h and f w Perform the operations shown in equations (4)-(5), using two sets of 1×1 convolutions to increase the channel dimension from C / r to C, and using the Sigmoid function for non-linear activation to obtain the coordinate attention module g that embeds the feature information. h and g w ;

[0085] g h =σ(F h (f h (4)

[0086] g w =σ(F w (f w (5)

[0087] Among them, f h For the horizontal attention tensor, F h For the dimensionality increase operation of the horizontal attention graph, f w For the vertical attention tensor, F w This is an upscaling operation for the vertical attention map, where σ is the Sigmoid activation function.

[0088] (1-2-4) Using the attention module g from (1-2-3) h and g w Weighting operations are performed on the input feature map to apply coordinate attention.

[0089] The network structure of the YOLOv4-based real-time apple detection model after the introduction of network lightweighting and coordinate attention mechanisms is as follows: Figure 5 As shown.

[0090] The second part, data preparation, includes the following steps:

[0091] (2-1) Dataset Acquisition

[0092] The dataset used in this experiment is divided into two parts: a background-free single apple dataset and a natural environment apple dataset. The background-free single apple dataset comes from the open-source Fruit-360 dataset, which contains 120 different fruits and vegetables. Each image is taken from a laboratory environment, and the background other than the target is removed after acquisition, containing only the apple's own features. Images of six varieties of apples, namely Braeburn, Crimson Snow, Pink Lady, Red 1, Red 2, and Red 3, are extracted as the background-free single apple dataset. The natural environment apple dataset uses a self-built dataset. The images are sourced from the Internet and retrieved using keywords such as apple, apple tree, and natural environment apple. It is divided into training set, validation set, and test set in an 8:1:1 ratio.

[0093] (2-2) Prior Box Extraction

[0094] The dataset was manually labeled using the LabelImg image annotation tool. During the annotation process, targets occluded by more than 80% were ignored. The resulting PASCAL VOC format XML files were used as label files. The resolution of the images in the natural environment apple dataset was adjusted to 416×416 for network input. The K-means algorithm was used to cluster the bounding boxes in the labels, with a cluster center of 9. The clustering results were used as the network's prior bounding boxes, which were (15,21), (28,35), (40,53), (52,73), (67,97), (83,62), (91,116), (113,159), and (167,214).

[0095] (2-3) Data Augmentation

[0096] To improve the model's detection performance, data augmentation was performed on the backgroundless single apple dataset. The images were transformed to the HSI space, and the data in the I channel was randomly adjusted. Based on the prior bounding box size obtained in (2-2), the target scale range was determined, and the images were randomly scaled proportionally. These operations expanded the backgroundless single apple dataset to four times its original size.

[0097] The adjustment parameters need to be determined based on the specific task and data. Taking this invention as an example, the random adjustment parameters for I-channel data are 0.8-4.0; the random adjustment parameters for scale are 0.15-2.0.

[0098] The third part, model training, includes the following steps:

[0099] (3-1) Loss Function The loss function of the model in this invention is shown in Equation (6), which consists of three parts: confidence loss, category loss and bounding box loss.

[0100]

[0101] Among them, K 2 M represents the number of grid cells in the current feature map, and M represents the number of anchor boxes located on each grid cell. To determine whether an object exists in the j-th anchor frame of the i-th grid, set C to 1 if it exists and 0 if it does not. i , Let λ be the confidence score between the predicted bounding box and the ground truth bounding box on the i-th grid. noobj This is a confidence error weighting factor to address the imbalanced sample problem. To determine whether an object exists in the j-th anchor frame of the i-th grid, p is set to 1 if it does not exist and 0 if it does. i (c) Let be the predicted probability and the true probability of the target class c on the i-th grid, IOU be the intersection-union ratio between the predicted and true bounding boxes, d be the Euclidean distance between the center points of the predicted and true bounding boxes, s be the diagonal length of the minimum closure rectangle between the predicted and true bounding boxes, and w be the predicted probability and the true probability of the target class c on the i-th grid. gt h gt ...

[0102] (3-2) Training Strategy

[0103] This invention employs a learning strategy that combines cross-domain and intra-domain transfer learning for training. The first stage involves cross-domain transfer learning, initializing the network parameters using pre-trained weights from the MobileNetv3-large network trained on the general object detection dataset Pascal-VOC, and then fine-tuning the network using a single, background-free apple dataset to learn apple features. The second stage involves intra-domain transfer learning, further fine-tuning the model trained in the first stage on a natural environment apple dataset to learn apple features in unstructured natural environments. General object detection datasets refer to large open-source datasets currently available online, including Pascal-VOC, COCO, and ImageNet.

[0104] (3-3) Training fine-tuning

[0105] The fine-tuning training mentioned in (3-2) is described in detail. The training process consists of two steps. First, the weights of the model's feature extraction network are frozen, and hyperparameters are set for initial training. Then, the weights of the model's feature extraction network are unfrozen, allowing them to update and iterate normally, and hyperparameters are set to complete the training. Tensorboard is used to record data during training. The training set loss is recorded for each iteration, and the validation set loss is recorded for each training epoch, and the model weights are saved. The model with the lowest validation set loss during the subsequent unfrozen training is taken as the training result.

[0106] Hyperparameters need to be set according to the specific task and hardware configuration. Taking this invention as an example, the Adam optimizer is used during training with default parameters and a learning rate decay of 0.9. During the frozen training phase, the batch size is 64, and the initial learning rate is 1×10⁻⁶. -3 The training rounds are 50; during the unfreezing training phase, the batch size is 16, and the initial learning rate is 1×10. -4 The training sessions consist of 50 rounds.

[0107] Part Four: Recognition of the Image to be Detected

[0108] Prepare apple images from the existing dataset test set or other sources, and input the images to be detected sequentially into the real-time apple detection network trained in the third part for recognition and localization. A simplified feature extraction network is used to extract features, and a feature fusion network with coordinate attention mechanism is used to combine feature information at different scales to enhance the detection of dense and occluded targets. After filtering with a confidence threshold of 0.5, the detection results are obtained. The model is tested on the test set of the natural environment apple dataset mentioned in this invention, and the detection results are as follows: Figure 6-8 As shown in Table 2, the overall performance of the present invention is efficient and real-time for detecting apples in complex orchard environments.

[0109] Table 2 Overall Model Performance

[0110]

[0111] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A lightweight real-time apple detection method for harvesting robots, characterized in that: The method includes the following steps: (1) Build an Apple real-time detection network based on YOLOv4; step (1) includes the following steps: (1-1) Introducing MobileNetv3-large and depthwise separable convolutions to improve the network structure with lightweight design; step (1-1) includes the following steps: (1-1-1) Replace the CSPDarkNet53 in the original feature extraction network based on YOLOv4 with the first 17 feature layers of MobileNetv3-large, introduce the spatial pyramid module SPP in the last layer of the network, and extract the feature maps extracted by the MobileNetv3-large feature layers 8 and 14 and the SPP module as input to the subsequent feature fusion network. (1-1-2) Replace the ordinary convolutions in the feature fusion network with depthwise separable convolutions; (1-2) Introduce coordinate attention mechanism into feature fusion network; (2) Data preprocessing and dataset construction; (3) Use the data in the dataset to train the model, which is used to adjust and optimize the model parameters; step (3) adopts a learning strategy that combines cross-domain transfer and intra-domain transfer for training, including the following steps: (3-3-1) Cross-domain transfer learning: The model is initialized using a general object detection dataset, and the network parameters are adjusted by training and validation data using a single apple dataset without background, in order to learn apple features; (3-3-2) Intradomain transfer learning: Using the model trained in (2-3-1), the network parameters are further trained on the training and validation sets of apples in the natural environment to learn the apple features in the unstructured natural environment. (3-3-3) In the above process, update and iterate the training network weights and set hyperparameters; use the model network weights and hyperparameters with the lowest loss on the validation set as the final optimized network; (4) Use an optimized network model to identify the image to be detected.

2. The lightweight real-time apple detection method for harvesting robots according to claim 1, characterized in that: The construction of the coordinate attention mechanism module in step (1-2) includes the following steps: (1-2-1) A spatial attention mechanism module is introduced after the Concat layer in the bottom-up part of the PAN structure in the feature fusion network. Two spatial pooling kernels are used to perform the encoding operations shown in equations (1)-(2) along the horizontal and vertical coordinates respectively to obtain a C×H×1 feature map. and the feature map of C×1×W in, This represents the feature of the c-th channel at height h after encoding. Let x be the feature of the c-th channel at width w after encoding. c (h,i) represents the input feature of the c-th channel at height h and width i, x c (j,w) represents the input feature of the c-th channel at height j and width w, where h is the height of the input feature map and w is the width of the input feature map; (1-2-2) Perform the operation shown in Equation (3) on the two feature maps generated in (1-2-1) to form a C×1×(H+W) feature map. Use 1×1 convolution to compress its channel dimension from C dimension to C / r dimension with a shrinkage rate r, and use a nonlinear function to activate it to obtain the intermediate attention map f. Then, decompose the obtained result f along the spatial dimension into a C / r×H×1 horizontal attention tensor f. h The vertical attention tensor f of C / r×1×W w ; f=δ(F1([z h ,z w ])) (3) Where [.,.] represents the splicing operation along the spatial dimension, F1 is the dimensionality reduction operation along the channel dimension using 1×1 convolution, δ is the nonlinear activation function, f is the intermediate attention graph containing horizontal and vertical attention information, and z h This is a horizontal attention diagram before splicing, z w This is a diagram showing the vertical alignment before splicing. (1-2-3) The two spatial tensors f generated by (1-2-2) h and f w Perform the operation shown in equation (4-5), using two sets of 1×1 convolutions to increase the channel dimension from C / r to C dimensions, and then using the Sigmoid function for non-linear activation to obtain the coordinate attention module g that embeds the feature information. h and g w ; g h =σ(F h (f h )) (4) g w =σ(F w (f w )) (5) Among them, f h For the horizontal attention tensor, F h For the dimensionality increase operation of the horizontal attention graph, f w For the vertical attention tensor, F w This is an upscaling operation for the vertical attention map, where σ is the Sigmoid activation function; (1-2-4) Using the attention module g from (1-2-3) h and g w By performing weighting operations on the input feature map and applying coordinate attention, the structure of the real-time apple detection network model is obtained.

3. The lightweight real-time apple detection method for harvesting robots according to claim 1, characterized in that: Step (2) includes the following steps: (2-1) Construct the dataset; (2-2) Prior box extraction; (2-3) Data augmentation.

4. A lightweight real-time apple detection method for harvesting robots according to claim 3, characterized in that: The dataset in step (2-1) includes two datasets with different attributes: a single apple dataset without background and an apple dataset in natural environment, which are further divided into training set, validation set and test set.

5. A lightweight real-time apple detection method for harvesting robots according to claim 3, characterized in that: In step (2-2), the K-means algorithm is used to cluster the bounding box sizes in the label data of the natural environment apple dataset described in step (2-1), and the clustering results are used as model prior boxes.

6. A lightweight real-time apple detection method for harvesting robots according to claim 3, characterized in that: Step (2-3) performs data augmentation on the backgroundless single apple dataset described in step (2-1), including adjusting the brightness of the image and scaling the image based on the prior bounding box size obtained in step (2-2).

7. A lightweight real-time apple detection method for harvesting robots according to claim 1, characterized in that: Step (4) includes the following steps: (4-1) Read the image to be detected; the image to be detected is a real-time image or a test set image; (4-2) Input the image to be detected into the trained real-time apple detection network for recognition and localization; (4-3) Repeat steps (4-1) to (4-2) until the detection of images in the test set is completed.

Images