Small target detection method, device, readable storage medium and electronic device

Abstract

The present disclosure relates to a small target detection method, device, readable storage medium and electronic device, the method comprising: inputting an image to be detected into a pre-trained small target detection model; wherein, in the small target detection model, paired inverse sub-pixel convolution operations and sub-pixel convolution operations are used to encode and decode the image information to be detected respectively; the small target detection model is used to extract features from the image to be detected, and output the target category and position in the image to be detected. The above scheme is used to solve the technical problem existing in the prior art that the feature fusion method of the traditional FPN network does not consider the correlation between the downsampling of the backbone network and the upsampling of the neck network, thereby resulting in redundant operations and information loss. In addition, the interpolation algorithm used in the FPN method not only does not bring additional information, but increases the amount of calculation.

Description

Technical Field

[0001] The present disclosure relates to the field of target detection, and in particular, to a small target detection method, device, readable storage medium, and electronic device. Background Art

[0002] With the rapid development of deep convolutional networks and GPU computing power, object detection, as the foundation of many computer vision tasks, has been widely applied and researched in fields such as healthcare, transportation, and security. Currently, some excellent object detection algorithms have achieved impressive results on common datasets. However, most current object detection algorithms target medium-to-large objects in natural scenes. Small objects, however, have a small pixel count, small coverage area, and contain little information. Therefore, detecting small objects remains a significant challenge.

[0003] To enhance small target detection performance, a common approach is multi-scale feature fusion, the most typical of which is the FPN (Feature Pyramid Network) network. Traditional FPN networks first compress feature maps on the channel level and then use interpolation algorithms to match spatial resolution during feature fusion. However, the feature fusion methods of traditional FPN networks fail to consider the correlation between downsampling in the backbone network and upsampling in the neck network, resulting in redundant operations and information loss. Furthermore, the interpolation algorithms used in FPN methods not only fail to provide additional information, but actually increase the computational effort. Summary of the Invention

[0004] The present disclosure provides a small target detection method, apparatus, readable storage medium, and electronic device to address the existing technical issues of FPN-based networks: The feature fusion method fails to consider the correlation between the downsampling of the backbone network and the upsampling of the neck network, resulting in redundant operations and information loss. Furthermore, the interpolation algorithm used in FPN-based methods not only fails to provide additional information but also increases the computational complexity.

[0005] In order to achieve the above objectives, the present disclosure provides a small target detection method in a first aspect, the method comprising:

[0006] Inputting the image to be detected into a pre-trained small target detection model; wherein, in the small target detection model, paired inverse sub-pixel convolution operations and sub-pixel convolution operations are used to encode and decode the image information to be detected respectively;

[0007] The small target detection model is used to extract features from the image to be detected, and the target category and position in the image to be detected are output.

[0008] Optionally, the method for constructing the small target detection model includes:

[0009] The small target detection model is constructed based on the YOLOv5s model, and all downsampling convolution layers in the target detection layer and subsequent detection layers in the backbone network of the YOLOv5s model are replaced with the inverse sub-pixel convolution operation, and all upsampling layers in the neck network of the YOLOv5s model are replaced with the sub-pixel convolution operation, so that the inverse sub-pixel convolution operation and the sub-pixel convolution operation exist in pairs, thereby obtaining an improved YOLOv5s model;

[0010] The improved YOLOv5s model is trained using the training image set to obtain the small target detection model.

[0011] Optionally, the target detection layer is the C4 detection layer in the backbone network.

[0012] Optionally, the step of training the improved YOLOv5s model using the training image set to obtain the small target detection model includes:

[0013] Dividing the pre-processed image samples and labels in the training image set into a training set and a validation set;

[0014] Optimize the parameters of the improved YOLOv5s model using the training set;

[0015] The validation set is used to perform parameter selection, and a set of parameters with the highest average accuracy is selected as the optimization result to obtain the small target detection model.

[0016] Optionally, in the process of training the improved YOLOv5s model using the training image set, the method further includes:

[0017] One or more data augmentation methods selected from image cropping, image flipping, image scaling, and histogram equalization are randomly used to increase the number of image samples.

[0018] Optionally, performing feature extraction on the image to be detected by the small target detection model and outputting the target category and position in the image to be detected includes:

[0019] Outputting a feature detection frame in the image to be detected by the small target detection model;

[0020] Calculate the GIoU value of the overlapping parts between adjacent feature detection frames;

[0021] If the adjacent feature detection frames are of the same category and the GIoU value is greater than or equal to a threshold, the adjacent feature detection frames are merged to obtain the target category and position of the image to be detected.

[0022] A second aspect of the present disclosure provides a small target detection device, comprising:

[0023] An input module, configured to input an image to be detected into a pre-trained small target detection model; wherein, in the small target detection model, paired inverse sub-pixel convolution operations and sub-pixel convolution operations are used to encode and decode the image information to be detected, respectively;

[0024] The feature extraction module is used to extract features of the image to be detected through the small target detection model and output the target category and position in the image to be detected.

[0025] A third aspect of the present disclosure provides a non-transitory computer-readable storage medium having a computer program stored thereon, which implements the steps of the method described in the first aspect when the program is executed by a processor.

[0026] A fourth aspect of the present disclosure provides an electronic device, including:

[0027] a memory having a computer program stored thereon;

[0028] A processor is used to execute the computer program in the memory to implement the steps of the method of the first aspect.

[0029] In the solution of the embodiment of the present disclosure, based on a pre-trained small target detection model, paired inverse sub-pixel convolution and sub-pixel convolution operations are utilized to avoid the negative impact of downsampling convolution and upsampling operations in traditional models on small targets, thereby alleviating the technical problem that the feature fusion method of traditional FPN-type networks does not consider the correlation between the downsampling of the backbone network and the upsampling of the neck network, resulting in redundant operations and information loss. By utilizing paired inverse sub-pixel convolution and sub-pixel convolution operations, the extracted feature information can be effectively retained and the detection performance of small targets can be improved.

[0030] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the following detailed description, they are used to explain the present disclosure but do not constitute a limitation of the present disclosure. In the accompanying drawings:

[0032] Figure 1 is a flowchart of a small target detection method according to an exemplary embodiment;

[0033] Figure 2 This is a schematic diagram of the existing YOLOv5s network structure;

[0034] Figure 3 is a schematic structural diagram of an improved YOLOv5s network according to an exemplary embodiment;

[0035] Figure 4 is a block diagram of a small target detection device according to an exemplary embodiment;

[0036] Figure 5 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0037] The following describes the specific embodiments of the present disclosure in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to illustrate and explain the present disclosure and are not intended to limit the present disclosure.

[0038] The present disclosure discloses a small target detection method, which includes the following steps.

[0039] Step 101: Input the image to be detected into a pre-trained small target detection model; wherein, in the small target detection model, paired inverse sub-pixel convolution operations and sub-pixel convolution operations are used to encode and decode the image information to be detected, respectively.

[0040] Step 102: extract features from the image to be detected using the small target detection model, and output the target category and position in the image to be detected.

[0041] In the embodiment of the present disclosure, for the image to be detected, the process of converting spatial information into channel information is called encoding, which is characterized by reduced spatial resolution and increased channel dimension; the process of converting channel information into spatial information is called decoding, which is characterized by reduced channel dimension and increased spatial resolution. The decoding and encoding operations are used in pairs to reduce the difficulty of network decoding and are more conducive to mining spatial positioning features. In the embodiment of the present disclosure, the inverse sub-pixel convolution operation and the sub-pixel convolution operation are used in combination in the target detection task, which can avoid the negative impact of downsampling convolution and upsampling operations on small targets, effectively retain the extracted feature information, and thus improve the detection performance of small targets.

[0042] Next, the method for constructing a small target detection model in the embodiment of the present disclosure is described. It should be noted that the construction method in the embodiment of the present disclosure is applicable to various neural network models. In the embodiment of the present disclosure, the yolov5s network is used as an example for description.

[0043] like Figure 2 and Figure 3 As shown, Figure 2 This is a schematic diagram of the existing YOLOv5s network structure; Figure 3This is a schematic diagram of the structure of an improved YOLOv5s network according to an exemplary embodiment; in the encoding process of the YOLOv5s network (version 5), all downsampling convolution layers in the target detection layer and subsequent detection layers are replaced with inverse sub-pixel convolution operations, and all upsampling layers in the Neck decoding process are replaced with sub-pixel convolution operations to construct an improved YOLOv5s detection model for small targets. In the embodiment of the present disclosure, the inverse sub-pixel convolution operation and the sub-pixel convolution operation are used in pairs in the overall structure. Figure 3 It can be seen that the target detection layer is C4, C4 is the C4 detection layer in the backbone, and the inverse sub-pixel convolution operation and sub-pixel convolution operation used in pairs are Desubpixel-1 and SubpixelConv-1, Desubpixel-2 and SubpixelConv-2 respectively.

[0044] In one possible implementation, during the encoding process, the convolution operation with a kernel size of 3*3 and a step size of 2 in the C4 and subsequent detection layers can be replaced with an inverse sub-pixel convolution operation, so that the length and width of the image are reduced to 1 / 2 of the original, and the number of channels is adjusted to twice the original. Since the downsampling convolution operation will blur the information, and the inverse sub-pixel convolution will not cause information loss, the use of the inverse sub-pixel convolution operation can alleviate the problem of small target information loss caused by downsampling. Among them, the number of channels refers to the channels in the image. For example, the original image (such as a photo taken by a mobile phone) has three RGB channels, but after multiple convolution operations, the number of channels will change.

[0045] During the decoding process, the upsampling layer is replaced with sub-pixel convolution, so that the length and width of the image are expanded to twice the original, and the number of channels is reduced to 1 / 2 of the original, thereby obtaining a higher resolution image.

[0046] After building an improved YOLOv5s detection model for small targets, the original image samples are preprocessed and divided into training sets and test sets. The training set is used for parameter optimization. The parameters here include all parameters in the neural network. The training process randomly uses data augmentation, and then the validation set is used for parameter selection. The set of parameters with the highest average accuracy is selected as the optimization result, thus obtaining the optimized small target detection model.

[0047] In one possible implementation, when training a model, appropriate original image samples can be selected for training based on actual needs. In the disclosed embodiments, the 2017 version of the COCO dataset is used as an example. The 2017 version of the COCO dataset contains 118,287 training images and 5,000 validation images, with a total of 80 categories.

[0048] Then, the backbone network of YOLOv5s (i.e. backbone network, Figure 2 and Figure 3 BackBone in

[15] is pre-trained on the COCO dataset, using cross entropy loss as the loss function and updating the network weights through back propagation.

[0049] Next, part of the weights of the trained network are used as the weights of the improved YOLOv5s backbone network, and the above dataset is used for parameter optimization and parameter selection.

[0050] In the disclosed embodiments, one or more data augmentation techniques, including image cropping, image flipping, image scaling, and histogram equalization, can be randomly applied during training. This process not only increases the amount of training data but also enhances the randomness of the data, facilitating the development of a small object detection model with enhanced generalization performance.

[0051] In the disclosed embodiment, cross entropy can be used to calculate the classification loss, mean square error can be used to calculate the position loss, and cross entropy can be used to calculate the confidence loss to guide parameter optimization. During the training process, stochastic gradient descent is also used to optimize the loss function, with an initial learning rate of 0.001, a batch_size of 64, and a maximum number of iterations of 300. It should be noted that the above data are only examples and do not limit this solution.

[0052] In the disclosed embodiment, after the small target detection model is constructed, when performing a target detection task, the image to be detected is input into the trained small target detection model for feature extraction.

[0053] In the embodiment of the present disclosure, during the target detection process, the small target detection model outputs the feature detection box [x, y, w, h, probability] in the image to be detected, where (x, y) is the coordinate of the upper left corner of the detection box, w is the width of the detection box along the X-axis, h is the height of the detection box along the Y-axis, and probability is the classification probability.

[0054] Next, a non-maximum suppression operation is performed on the predicted target. The Generalized Intersection over Union (GIoU) ​​value of the overlapping portions of adjacent feature detection frames is calculated. If adjacent feature detection frames belong to the same class and the GIoU value is greater than a threshold, the adjacent detection frames are merged to obtain the target category and position in the image to be detected. Whether adjacent detection frames belong to the same class can be determined by the classification subnetwork; the threshold can be set in the range [0, 2], for example, 0.7 or 1.1, and those skilled in the art can adjust it according to actual needs.

[0055] It should be noted that the predicted targets in the embodiments of the present disclosure may be small targets to be detected, as well as medium and large targets, and the present invention does not impose any limitation on this.

[0056] The following is a comparison of the small object detection model in the embodiment of the present disclosure and YOLOv5s through a set of experimental results. The present disclosure uses the COCO dataset to conduct experimental verification in the YOLOv5s framework. The experimental results are shown in the following table.

[0057] model size mAP <![CDATA[AP 0.5 ]]> <![CDATA[AP 0.75 ]]> <![CDATA[AP S ]]> <![CDATA[AP M ]]> <![CDATA[AP L ]]> params FLOPs YOLOv5s 640 0.368 0.555 0.402 0.209 0.423 0.470 7.3 17.0 This program 640 0.376 0.558 0.410 0.216 0.424 0.492 7.0 17.2

[0058] Where size represents the image resolution, params represents the number of parameters (millions), FLOPs represents the number of floating-point operations (billions), and precision P represents the proportion of true positives (TP) among the predicted positive examples.

[0059]

[0060] AP C Represents the precision P of each instance belonging to a certain category C j The ratio of the sum of Nc to the total number of instances Nc of category C. The mean average precision (mean AP) is the average value of AP, which is used to measure the training effect of the model on all categories.

[0061]

[0062]

[0063] Mean AP@0.5 represents the mean AP when the Intersection over Union (IOU) is 0.5; mean AP@0.5:0.95 represents the mean AP when the IOU ranges from 0.5 to 0.95 with an interval of 0.05. This evaluation indicator can better reflect the accuracy of the model than AP@0.5. P and R are both counted at an IOU threshold of 0.5, and mAP@0.5 is recorded as AP. 0.5 , mAP@0.75 is recorded as AP 0.75 , mAP@0.5:0.95 is recorded as mAP. AP S 、AP M 、AP L They represent the mean average precision of small targets, medium targets and large targets under 0.5 intersection-over-union ratio.

[0064] Based on the same inventive concept, Figure 4As shown, the embodiment of the present disclosure also provides a small target detection device 400, including an input module 401, which is used to input the image to be detected into a pre-trained small target detection model; wherein, in the small target detection model, paired inverse sub-pixel convolution operations and sub-pixel convolution operations are used to encode and decode the image information to be detected respectively; a feature extraction module 402 is used to extract features of the image to be detected through the small target detection model, and output the target category and position in the image to be detected.

[0065] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0066] Figure 5 FIG. 5 is a block diagram of an electronic device 500 according to an exemplary embodiment. Figure 5 As shown, the electronic device 500 may include: a processor 501 , a memory 502 , and may further include one or more of a multimedia component 503 , an input / output (I / O) interface 504 , and a communication component 505 .

[0067] The processor 501 is used to control the overall operation of the electronic device 500 to complete all or part of the steps in the above-mentioned small target detection method. The memory 502 is used to store various types of data to support the operation of the electronic device 500. For example, these data may include instructions for any application or method operating on the electronic device 500, as well as application-related data, such as contact information, sent and received messages, pictures, audio, video, etc. The memory 502 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk. The multimedia component 503 may include a screen and an audio component. The screen may be, for example, a touch screen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signals may be further stored in the memory 502 or transmitted via the communication component 505. The audio component also includes at least one speaker for outputting audio signals. The I / O interface 504 provides an interface between the processor 501 and other interface modules. The above-mentioned other interface modules may be a keyboard, a mouse, buttons, etc. These buttons may be virtual buttons or physical buttons. The communication component 505 is used for wired or wireless communication between the electronic device 500 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, 4G, NB-IOT, eMTC, or other 5G, etc., or a combination of one or more thereof, is not limited here. Therefore, the corresponding communication component 505 may include: a Wi-Fi module, a Bluetooth module, an NFC module, etc.

[0068] In an exemplary embodiment, the electronic device 500 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above-mentioned small target detection method.

[0069] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided. When executed by a processor, the program instructions implement the steps of the above-described small target detection method. For example, the computer-readable storage medium may be the aforementioned memory 502 including the program instructions. The program instructions may be executed by the processor 501 of the electronic device 500 to implement the above-described small target detection method.

[0070] In another exemplary embodiment, a computer program product is further provided. The computer program product includes a computer program that can be executed by a programmable device, and the computer program has a code portion for performing the above-mentioned small target detection method when executed by the programmable device.

[0071] The preferred embodiments of the present disclosure are described in detail above in conjunction with the accompanying drawings. However, the present disclosure is not limited to the specific details of the above embodiments. Within the technical concept of the present disclosure, various simple modifications can be made to the technical solutions of the present disclosure, and these simple modifications all fall within the scope of protection of the present disclosure.

[0072] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any appropriate manner without contradiction. In order to avoid unnecessary repetition, the present disclosure will not further describe various possible combinations.

[0073] In addition, the various embodiments of the present disclosure may be arbitrarily combined, and as long as they do not violate the concept of the present disclosure, they should also be regarded as the contents disclosed by the present disclosure.

Claims

1. A small target detection method, characterized in that: The method comprises: A small target detection model is constructed based on the YOLOv5s model, and all downsampling convolution layers in the target detection layer and subsequent detection layers in the backbone network of the YOLOv5s model are replaced with inverse sub-pixel convolution operations, and all upsampling layers in the neck network of the YOLOv5s model are replaced with sub-pixel convolution operations, so that the inverse sub-pixel convolution operations and the sub-pixel convolution operations exist in pairs, thereby obtaining an improved YOLOv5s model; Using the training image set to train the improved YOLOv5s model to obtain the small target detection model; Inputting the image to be detected into a pre-trained small target detection model; wherein, in the small target detection model, paired inverse sub-pixel convolution operations and sub-pixel convolution operations are used to encode and decode the image information to be detected respectively; Performing feature extraction on the image to be detected using the small target detection model, and outputting the target category and position in the image to be detected; Among them, the target detection layer is the C4 detection layer in the backbone network.

2. The method according to claim 1, wherein The improved YOLOv5s model is trained using the training image set to obtain the small target detection model, including: Dividing the pre-processed image samples and labels in the training image set into a training set and a validation set; Optimize the parameters of the improved YOLOv5s model using the training set; The validation set is used to perform parameter selection, and a set of parameters with the highest average accuracy is selected as the optimization result to obtain the small target detection model.

3. The method according to claim 2, wherein In the process of training the improved YOLOv5s model using the training image set, the method further includes: One or more data augmentation methods selected from image cropping, image flipping, image scaling, and histogram equalization are randomly used to increase the number of image samples.

4. The method according to claim 1, wherein Performing feature extraction on the image to be detected by the small target detection model and outputting the target category and position in the image to be detected includes: Outputting a feature detection frame in the image to be detected by the small target detection model; Calculate the GIoU value of the overlapping parts between adjacent feature detection frames; If the adjacent feature detection frames are of the same category and the GIoU value is greater than or equal to a threshold, the adjacent feature detection frames are merged to obtain the target category and position of the image to be detected.

5. A small target detection device, characterized in that: include: An input module, configured to input an image to be detected into a pre-trained small target detection model; wherein, in the small target detection model, paired inverse sub-pixel convolution operations and sub-pixel convolution operations are used to encode and decode the image information to be detected, respectively; A feature extraction module is used to extract features from the image to be detected using the small target detection model, and output the target category and position in the image to be detected; The small target detection model is constructed based on the YOLOv5s model. The target detection layer and all downsampling convolution layers in the subsequent detection layers in the backbone network of the YOLOv5s model are replaced with the inverse sub-pixel convolution operation. The upsampling layers in the neck network of the YOLOv5s model are replaced with the sub-pixel convolution operation, so that the inverse sub-pixel convolution operation and the sub-pixel convolution operation exist in pairs, thereby obtaining an improved YOLOv5s model. Using the training image set to train the improved YOLOv5s model to obtain the small target detection model; The target detection layer is the C4 detection layer in the backbone network.

6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

7. An electronic device, characterized in that: include: a memory having a computer program stored thereon; A processor, configured to execute the computer program in the memory to implement the steps of the method according to any one of claims 1 to 4.

Images