A dense small target detection method based on improved YOLOv5

Abstract

The present invention discloses a method for detecting dense small targets based on an improved YOLOv5 algorithm, which further improves the YOLOv5 algorithm. The idea is: (1) adding a coordinated attention mechanism (CA) to the YOLOv5 backbone extraction network, embedding the position information into the channel attention, so that the mobile network can obtain information from a larger area and avoid introducing large overhead; (2) in the feature fusion network of YOLOv5, BiFPN is used instead of PANet, and weights are introduced to better balance feature information of different scales; (3) for dense, mutually occluded small targets, Varifocal Loss is used to train dense targets, so that the network model can accurately identify targets with large-area clusters and overlaps. It also has good robustness to the current color changes of the object itself, complex natural environment conditions, etc.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection, and in particular to a dense small target detection method based on improved YOLOv5. Background Art

[0002] Object detection is a research hotspot in the field of machine vision and artificial intelligence, and is also the core technology of applications such as face recognition, object classification, and automatic sorting. A large number of researchers have conducted a lot of research on object detection and proposed some solutions. The early methods mainly extracted features from images, including color features, texture features, shape features, and spatial relationship features of images. The color histogram feature matching methods mainly include histogram intersection method and reference color table method. Since color cannot measure the direction and size of the image, it cannot extract the local features of the image well. The commonly used methods for texture feature extraction are gray level co-occurrence matrix method and semivariogram. The commonly used models are random field model and fractal model. Texture is a concept in the region, so it will lead to excessive regionalization and ignore the global features. The methods based on shape feature extraction mainly include boundary feature method and geometric parameter method, but the recognition effect of deformed targets is not good. Some researchers have proposed automatic fruit recognition technology based on machine vision, and image acquisition is carried out according to the principle of machine vision. After preprocessing the fruit image by smoothing and sharpening, the fruit color sample value is calculated in the RGB color space, and the image is segmented according to the sample value. Finally, the segmentation result is used for feature extraction. The main problem of the traditional method based on image features is that the method is not scalable, and different features are often required for different targets.

[0003] In recent years, the rise of machine learning, especially deep learning, has brought about a breakthrough in the field of computer vision. Some researchers have proposed a fruit recognition method based on convolutional neural networks. This type of method usually first obtains RGB images of fruits, preprocesses and annotates them, builds a data set, builds a convolutional neural network, sets the parameters of the network model, and then puts the training set into the convolutional neural network for training. Finally, a fruit recognition model can be obtained. Due to the stronger applicability of deep learning technology, it has been promoted and applied in many target detection occasions in recent years.

[0004] In general, the currently widely used deep learning target detection methods can achieve good detection results for targets with large areas and volumes and not severely occluded. However, for some small, dense and severely occluded targets, such as leaves, fruits, flowers on trees, wild animals photographed from high altitudes, etc., accurate and automatic detection is still a challenge. At the same time, for target detection under these outdoor natural conditions, it is also necessary to overcome the influence of various environmental factors such as lighting, rain and fog. Although there are some methods, such as object detection methods based on convolutional neural networks, which focus on the detection of small targets, there are still shortcomings. First, when facing a large number of dense and overlapping targets, accurate recognition cannot be achieved, and the misidentification rate is high; another shortcoming is that these methods pay too much attention to the recognition accuracy of small targets when identifying targets, and do not consider the model size and detection speed of convolutional neural networks, resulting in the final generated detection model being difficult to deploy and use in mobile devices.

[0005] In real application environments, the characteristics and application scenarios of specific targets also need to be considered. For example, in the detection application of fruits on orchard trees, the individual and color characteristics of the fruits show differences with the growth cycle. Even the same variety of fruit trees have different characteristics, postures, and degrees of occlusion. Different varieties of fruit trees have different characteristics. In addition, during the growth of fruits, complex environmental factors such as light intensity, fertilization methods, and water irrigation will affect the final recognition of fruits on the trees. Therefore, the target detection algorithm also needs to consider the changes in the size, color, and surrounding environmental conditions of the target object. Summary of the invention

[0006] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a method for detecting dense small targets based on an improved YOLOv5 algorithm. The method further improves the YOLOv5 algorithm. The idea is: (1) to add a coordinated attention mechanism (CA) to the YOLOv5 backbone extraction network, by embedding the position information into the channel attention, so that the mobile network can obtain information from a larger area and avoid introducing large overhead; (2) in the feature fusion network of YOLOv5, BiFPN is used instead of PANet, and weights are introduced to better balance feature information of different scales; (3) for dense, mutually occluded small targets, Varifocal Loss is used to train dense targets, so that the network model can accurately identify targets with large-area clusters and overlaps. It also has good robustness to the current color changes of the object itself, complex natural environmental conditions, etc.

[0007] The technical solution for achieving the purpose of the present invention is:

[0008] A dense small target detection method based on an improved YOLOv5 algorithm includes the following steps:

[0009] S1. Image acquisition: The user uses an image acquisition device to acquire images of the target object, names the acquired images according to the format of the Pascal VOC dataset, and creates three folders named Annotations, ImageSets, and JPEGImages;

[0010] S2. Image preprocessing:

[0011] S2-1, image labeling: in the image collected in step S1, use the image labeling tool LabelImg to label the target in the image, marking the location and category name of the target;

[0012] S2-2, image augmentation: If the image collected by the user in step S1 cannot meet the requirement of 2000 images required for identifying a single category target, the Augmentor image data augmentation library is used to augment the image. The user selects the image save path and the tag information XML file path, and formulates the output path of the augmented image and XML file, selects the required image enhancer such as brightness, cropping, Gaussian noise, etc., selects the augmentation quantity and augmentation method (sequential, combined, random, etc.) to amplify the image to meet the recognition requirements;

[0013] S2-3, divide the data set: divide the amplified images and labeled files into training set, test set, validation set and training validation set. The training set, test set and validation set account for 50%, 25% and 25% respectively. The training validation set is the sum of the training set and validation set, accounting for 75%;

[0014] S3. Set network model parameters: In the YOLOv5 network model configuration file yaml, set the size of the convolutional neural network input image, the number of recognition types, and the number of iterations according to the size of the computer memory and video memory, the recognition effect and training speed required by the user; and the user needs to use a graphics card type that supports CUDA acceleration;

[0015] S3-1. When the image size of the input network is selected as 608*608 (regardless of the original image size), the batch parameter is 8, the number of iterations epcoh is 300, and the number of detected objects is 2, the user uses a single GPU to train the model, which requires at least 6GB;

[0016] S3-2, when the image size of the input network is selected as 640*640 (regardless of the original image size), the batch parameter is 8, the number of iterations epcoh is 300, and the number of detected objects is 2, the user uses a single GPU to train the model, which requires at least 8GB;

[0017] S4. Improve the original YOLOv5 network structure to obtain an improved YOLOv5 network structure. The improvement process is as follows:

[0018] S4-1: In the original YOLOv5 network structure, a CA coordinated attention mechanism is added after the 3rd, 6th and 9th layers, and two 1D global pooling operations are used to aggregate the input features along the vertical and horizontal directions into two separate position perceptions; then the two feature maps with embedded specific direction information are encoded into two attention maps, each of which captures the long-range dependencies of the input feature map along a spatial direction, and the position information is saved in the generated attention map; both attention maps are applied to the input feature map by multiplication to emphasize the representation of the attention area;

[0019] S4-2: In the original YOLOv5 network structure, BiFPN is used to enhance feature extraction. Based on the simplified version of PANet, if the input and output nodes are in the same layer, an extra edge is added to fuse more features without increasing the cost; P5_in is upsampled, and after upsampling, it is stacked with P4_in by Concat_bifpn to obtain P4_td; then P4_td is upsampled, and after upsampling, it is stacked with P3_in by Concat_bifpn to obtain P3_out; then P3_out is downsampled, and after downsampling, it is stacked with P4 by Concat_bifpn to obtain P4_out; then P4_out is downsampled, and after downsampling, it is stacked with P5_in by Concat_bifpn to obtain P3_out;

[0020] S4-3: Use Varifocal Loss to calculate the loss function to solve the class imbalance problem, where p is the predicted Iou-aware classification score (IACS), q is the target score, for positive samples in training, q is set to the IoU (gt IoU) between the generated bbox and gt box, and for negative samples in training, the training target q for all categories is 0, so the training focuses on candidate detection samples with higher IACS; α and γ are hyperparameters, α is an adjustable scaling factor used to balance the loss between positive and negative examples, 0≤α≤1, so the training avoids focusing on negative examples, the loss function can judge and weigh between difficult samples and easy samples, and ultimately reduce the loss contribution of easy samples, because 0≤p≤1, then γ is set to be greater than 1;

[0021] S5. Train the network model: Set the parameters for the improved YOLOv5 network configuration files train.py and yolov5.yaml, put the yaml file with the set parameters and the improved YOLOv5 network structure into the computer with the configured environment, and use the labeled images in the training set and the validation set for training. During the training process, put the images divided in the test set into the computer for testing to obtain the effect of each stage of training, and set the process monitoring tensorboard --logdir runs / train parameter to observe the mAP value of the training in real time. After the training, save the trained network model weight pt;

[0022] S6. Use the trained network model weights for recognition: prepare the image to be detected on the computer, change the configuration file yaml, the trained weights and the image path to be detected in detect.py, execute detect.py, load the network model configuration file yaml and the network model weights after running, and finally save the image of the target detection result on the computer.

[0023] The present invention provides a dense small target detection method based on the improved YOLOv5 algorithm. The method improves the YOLOv5 network model structure, adds a coordinated attention mechanism and strengthens feature extraction, so as to better identify individual small target objects; and uses the Varifocal Loss function to accurately identify large-area clustered, densely overlapped target objects; compared with the prior art, the present invention has the following advantages:

[0024] (1) When identifying fruits on trees, an improved YOLOv5 network structure was used to train the image dataset. The trained model can more accurately identify dense small targets and can accurately identify target objects with large-area clustered overlaps.

[0025] (2) When performing target detection, an improved YOLOv5 network structure is used to train the data set. The trained detection model is small in size and can adapt to various embedded devices.

[0026] (3) It can be used in outdoor natural environments and has the characteristics of high recognition accuracy and fast speed, which can meet the needs of real-time recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a flowchart of a dense small target detection method based on an improved YOLOv5 algorithm of the present invention;

[0028] Figure 2 Coordinate attention mechanism diagram for CA;

[0029] Figure 3Enhanced feature extraction network structure diagram for BiFPN;

[0030] Figure 4 This is the recognition effect of the improved YOLOv5 network model on mature citrus;

[0031] Figure 5 This is the recognition effect of the improved YOLOv5 network model on mature Nanfeng mandarin oranges;

[0032] Figure 6 This is the improved YOLOv5 network model's effect on identifying citrus fruits in the growing period;

[0033] Figure 7 This is the recognition effect of the improved YOLOv5 network model on Nanfeng mandarin oranges in the growing period;

[0034] Figure 8 This is a diagram showing the recognition effect of the improved YOLOv5 network model on large-area clustered citrus fruits. DETAILED DESCRIPTION

[0035] The present invention is further described below with reference to the accompanying drawings and embodiments, but the present invention is not limited thereto.

[0036] Example:

[0037] This embodiment takes citrus and Nanfeng mandarin oranges as examples to identify the fruits on the citrus trees in the orchard.

[0038] A dense small target detection method based on the improved YOLOv5 algorithm, such as Figure 1 As shown, the following steps are included:

[0039] S1. Image acquisition: The user uses a digital camera or other image acquisition device to collect images of citrus trees with fruits, and names the collected images according to the format of the Pascal VOC dataset, and creates three folders named Annotations, ImageSets, and JPEGImages;

[0040] S2. Image preprocessing:

[0041] S2-1, image labeling: In the image collected in step S1, use the image labeling tool LabelImg to label the citrus fruits in the image, marking the location and category name of the citrus fruits. This embodiment selects two varieties (categories) of citrus and Nanfeng mandarin oranges, so these two varieties are used as examples for explanation below, and it is not limited to these two varieties.

[0042] (1) When selecting citrus, the label can be named orange; when selecting Nanfeng mandarin orange, the label can be named sweet_orange;

[0043] (2) When selecting densely clustered and overlapping citrus fruits, select them one by one and manually select the citrus fruits accurately;

[0044] (3) When the occlusion of the citrus fruit exceeds 95%, the current target is discarded;

[0045] (4) If the pixel area of ​​the selected fruit target is less than 8*8, the current target is discarded;

[0046] S2-2, image amplification: If the images collected by the user in step S1 cannot meet the requirement of 2000 images required for identifying a single variety of citrus, the user can use the Augmentor image data enhancement library to amplify the image; the user selects the image save path and the tag information XML file path, and formulates the output path of the amplified image and XML file, selects the required image enhancer (such as brightness, cropping, Gaussian noise, etc.), selects the amplification quantity and amplification method (such as sequential, combined, random, etc.) to amplify the image to meet the required identification requirements.

[0047] S2-3, divide the data set: divide the amplified images and labeled files into training set, test set, validation set and training validation set. The training set, test set and validation set account for 50%, 25% and 25% respectively. The training validation set is the sum of the training set and validation set, accounting for 75%;

[0048] S3. Set network model parameters: In the configuration file yaml of the YOLOv5 network model, set the size of the convolutional neural network input image, the number of recognition types, the number of iterations, etc. according to the size of the computer memory and video memory, the recognition effect and training speed required by the user; and the user needs to use a graphics card type that supports cuda acceleration;

[0049] S3-1. When the image size of the input network is selected as 608*608 (regardless of the original image size), the batch parameter is 8, the number of iterations epcoh is 300, and the number of detected objects is 2, the user uses a single GPU to train the model, which requires at least 6GB;

[0050] S3-2, when the image size of the input network is selected as 640*640 (regardless of the original image size), the batch parameter is 8, the number of iterations epcoh is 300, and the number of detected objects is 2, the user uses a single GPU to train the model, which requires at least 8GB;

[0051] S4. Improve the original YOLOv5 network structure to obtain an improved YOLOv5 network structure. The improvement process is as follows:

[0052] S4-1. In the original YOLOv5 network structure, a CA coordinated attention mechanism is added after the 3rd, 6th and 9th layers. The CA coordinated attention mechanism is shown in the figure below. Figure 2 As shown in the figure, two 1D global pooling operations are used to aggregate the input features along the vertical and horizontal directions into two separate position-aware features. These two feature maps with embedded specific direction information are then encoded into two attention maps, each of which captures the long-range dependencies of the input feature map along a spatial direction. The position information can therefore be preserved in the generated attention map. Both attention maps are then applied to the input feature map by multiplication to emphasize the representation of the attention area.

[0053] S4-2. In the original YOLOv5 network structure, BiFPN is used to enhance feature extraction. The BiFPN enhanced feature extraction network structure is as follows Figure 3 As shown in the figure, based on the simplified version of PANet, if the input and output nodes are of the same layer, an extra edge is added to fuse more features without increasing the cost, and P5_in is upsampled, and after upsampling, it is stacked with P4_in by Concat_bifpn to obtain P4_td; then P4_td is upsampled, and after upsampling, it is stacked with P3_in by Concat_bifpn to obtain P3_out; then P3_out is downsampled, and after downsampling, it is stacked with P4 by Concat_bifpn to obtain P4_out; then P4_out is downsampled, and after downsampling, it is stacked with P5_in by Concat_bifpn to obtain P3_out.

[0054] S4-3. Use Varifocal Loss to calculate the loss function to solve the class imbalance problem, where p is the predicted Iou-aware classification score (IACS), q is the target score, and for positive samples in training, q is set to the IoU (gt IoU) between the generated bbox and gt box, while for negative samples in training, the training target q for all categories is 0. Then the training focuses on candidate detection samples with higher IACS; α and γ are hyperparameters, α is an adjustable scaling factor used to balance the loss between positive and negative examples, 0≤α≤1, so the training avoids focusing on negative examples. The loss function can judge and weigh between difficult samples and easy samples, and ultimately reduce the loss contribution of easy samples, because 0≤p≤1, then γ is set to be greater than 1;

[0055]

[0056] S5. Train the network model: Set the parameters for the improved YOLOv5 network configuration files train.py and yolov5.yaml, put the yaml files with set parameters and the improved YOLOv5 network structure into the computer with the configured environment, and use the labeled images in the training set and the validation set for training. During the training process, put the images divided in the test set into the computer for testing to obtain the effect of each stage of training, and set the process monitoring tensorboard --logdir runs / train parameter to observe the mAP value of the training in real time. After the training, save the trained network model weights pt.

[0057] S6. Use the trained network model weights for identification: prepare the fruit image to be detected on the computer, change the configuration file yaml, the trained weights and the image path to be detected in detect.py, execute detect.py, load the network model configuration file yaml and the network model weights after running, and finally save the image of the fruit identification result on the computer.

[0058] The above scheme is used to identify the fruits of citrus and Nanfeng mandarin orange trees at different stages. The recognition results are as follows: Figure 4 , Figure 5 , Figure 6 , Figure 7 and Figure 8 As shown, it can be seen from the recognition result diagram that when identifying fruits on trees, the method of the present invention is used to train two types of fruit data sets, and the trained model can more accurately identify individual smaller fruit targets, and can accurately identify large-area clustered and overlapping fruit targets.

Claims

1. A dense small target detection method based on an improved YOLOv5 algorithm, characterized in that: The steps include: S1. Image acquisition: The user uses an image acquisition device to acquire images of the target object, names the acquired images according to the format of the Pascal VOC dataset, and creates three folders named Annotations, ImageSets, and JPEGImages; S2. Image preprocessing: S2-1, image labeling: in the image collected in step S1, use the image labeling tool LabelImg to label the target in the image, marking the location and category name of the target; S2-2, image augmentation: If the image collected by the user in step S1 cannot meet the requirement of 2000 images required for identifying a single category target, the Augmentor image data augmentation library is used to augment the image. The user selects the image save path and the tag information XML file path, and specifies the output path of the augmented image and XML file, selects the required image enhancer, and selects the augmentation quantity and amplification method to amplify the image to meet the recognition requirements; S2-3, divide the data set: divide the amplified images and labeled files into training set, test set, validation set and training validation set. The training set, test set and validation set account for 50%, 25% and 25% respectively. The training validation set is the sum of the training set and validation set, accounting for 75%; S3. Set network model parameters: In the YOLOv5 network model configuration file yaml, set the size of the convolutional neural network input image, the number of recognition types, and the number of iterations according to the size of the computer memory and video memory, the recognition effect and training speed required by the user; and the user needs to use a graphics card type that supports CUDA acceleration; S3-1. When the image size of the input network is selected as 608*608, the batch parameter is 8, the number of iterations epcoh is 300, and the number of detected objects is 2, the user uses a single GPU to train the model, which requires at least 6GB; S3-2, when the image size of the input network is selected as 640*640, the batch parameter is 8, the number of iterations epcoh is 300, and the number of detected objects is 2, the user uses a single GPU to train the model, which requires at least 8GB; S4. Improve the original YOLOv5 network structure to obtain an improved YOLOv5 network structure. The improvement process is as follows: S4-1: In the original YOLOv5 network structure, a CA coordinated attention mechanism is added after the 3rd, 6th and 9th layers, and two 1D global pooling operations are used to aggregate the input features in the vertical and horizontal directions into two separate position perceptions; The two feature maps with embedded specific direction information are then encoded into two attention maps respectively. Each attention map captures the long-range dependencies of the input feature map along a spatial direction, and the position information is saved in the generated attention map. Apply both attention maps to the input feature map by multiplication to emphasize the representation of the attention area; S4-2: In the original YOLOv5 network structure, BiFPN is used to enhance feature extraction. Based on the simplified version of PANet, if the input and output nodes are in the same layer, an extra edge is added to integrate more features without increasing the cost; P5_in is upsampled, and after upsampling, it is stacked with P4_in by Concat_bifpn to obtain P4_td; then P4_td is upsampled, and after upsampling, it is stacked with P3_in by Concat_bifpn to obtain P3_out; then P3_out is downsampled, and after downsampling, it is stacked with P4 by Concat_bifpn to obtain P4_out; then P4_out is downsampled, and after downsampling, it is stacked with P5_in to obtain P3_out; S4-3: Use Varifocal Loss to calculate the loss function to solve the class imbalance problem, where p is the predicted Iou-aware classification score IACS, q is the target score, for positive samples in training, q is set to the IoU (gt IoU) between the generated bbox and gtbox, and for negative samples in training, the training target q for all categories is 0, so the training focuses on candidate detection samples with higher IACS; α and γ are hyperparameters, α is an adjustable scaling factor used to balance the loss between positive and negative examples, 0≤α≤1, so the training avoids focusing on negative examples, the loss function can judge and weigh between difficult samples and easy samples, and ultimately reduce the loss contribution of easy samples, because 0≤p≤1, then γ is set to be greater than 1: ； S5. Train the network model: Set the parameters for the improved YOLOv5 network configuration files train.py and yolov5.yaml, put the yaml files with set parameters and the improved YOLOv5 network structure into the computer with the configured environment, and use the labeled images in the training set and the validation set for training. During the training process, put the images divided in the test set into the computer for testing to obtain the effect of each stage of training, and set the process monitoring tensorboard--logdirruns / train parameter to observe the mAP value of the training in real time. After the training, save the trained network model weight pt; S6. Use the trained network model weights for recognition: prepare the image to be detected on the computer, change the configuration file yaml, the trained weights and the image path to be detected in detect.py, execute detect.py, load the network model configuration file yaml and the network model weights after running, and finally save the image of the target detection result on the computer.

2. A dense small target detection method based on an improved YOLOv5 algorithm according to claim 1, characterized in that: In S2-2, the enhancer includes a brightness enhancer, a cropping enhancer, and a Gaussian noise enhancer; the amplification method includes sequential, combined, and random amplification methods.

Images