A small target detection method based on multi-receptive field and deep features
By adopting a multi-receptive field and depth feature method in a weak object detection system, using convolutional neural network and expanded convolution block to extract depth features and context information, the problem of insufficient detection performance of existing systems in complex scenarios is solved, and the implementation of the realization of the realization of the existing system is achieved. Accurate detection and identification of weak targets.
Patent Information
- Application Number
- CN202210630725.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-06
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2042-06-06
AI Technical Summary
Existing weak target detection systems have problems with insufficient detection performance when dealing with complex scenarios and fuzzy targets, especially in drone and aerial photography scenarios, which are difficult to adapt to the detection and identification of weak targets.
Using a detection method based on multi-receptive fields and depth features, the depth features and context information of sequence images are extracted through convolutional neural networks and expanded convolution blocks, and the feature representation of weak targets is enhanced, thereby improving the adaptability and robustness of the system. This method includes multi-receptive field feature extraction, deep feature extraction, multi-receptive field feature aggregation and multi-scale fusion, combining an improved path aggregation network and a dual-channel prediction head to achieve accurate regression and judgment of weak target box positions and categories.
It realizes accurate detection and correct classification of weak targets in images or videos, improves the robustness and adaptability of the detection system, can effectively identify weak targets below 7×7 in complex scenarios, and enhances the detection of small targets ability.
Smart Images

Figure CN115115973B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of computer vision application, relates to target detection, and in particular to a small target detection method based on multi-receptive fields and depth features. Background Art
[0002] As the basis of advanced visual tasks such as scene understanding, target tracking, and intelligent decision-making, target detection plays an indispensable and important role in the fields of drones, autonomous driving, and smart cities. Among them, weak target detection has always been a research hotspot and difficulty in this direction. Long-distance detection scenarios such as drone reconnaissance and border warning urgently need a weak target detection system with good performance. However, there are several difficulties in weak target detection. First, the resolution of weak targets is too low to lack detailed information, which will cause the feature extraction backbone network containing multiple pooling layers to be unable to extract effective features. Second, several weak targets with high similarity are difficult to distinguish due to the lack of detailed information. Third, due to changes in the camera position, the scales of weak targets vary greatly. This makes the general detection system unable to adapt to weak target scenes.
[0003] At present, small and weak target detection can be roughly divided into traditional methods and deep learning-based methods. Traditional small and weak target detection methods usually include two stages, the first stage is background suppression, and the second stage is target detection. Filter-based methods mainly predict the image background by constructing filters to achieve background suppression. The detection stage can be divided into two categories: detect before track (DBT) and track before detect (TBD). DBT can be further divided into filter-based methods, methods based on the human visual system (HVS), and methods based on image data structures. Unlike DBT, TBD processes sequence images, first tracks small and weak targets in sequence images, obtains the possible trajectory of the target, and then detects the target by determining the trajectory. Although traditional small and weak target detection algorithms have good detection effects, they rely more on prior knowledge and require artificial rules to be set. However, in complex environments with changing backgrounds, setting rules that take all situations into account is a recognized problem in the industry, resulting in poor generalization of traditional algorithms. In recent years, with the rapid improvement of hardware computing power and the continuous improvement of deep learning frameworks, target detection methods based on deep convolutional neural networks (DCNNs) have gradually replaced traditional manual feature extraction methods and become mainstream. Compared with traditional methods, methods based on DCNNs have powerful feature extraction and nonlinear fitting capabilities. Depending on the framework, target detection algorithms based on DCNNs can be divided into one-stage detectors and two-stage detectors. They have their own advantages and are suitable for different scenarios. Although many DCNN detectors have achieved good results on natural scenery datasets, it is still difficult to adapt to aerial photography, remote sensing and other scenes dominated by weak targets.
[0004] Although existing methods have improved the detection performance of small targets to a certain extent, it is still a challenge to deal with complex scenes with severe interference and blurred targets. How to mine contextual information from images that is conducive to improving the feature representation of small target areas is an urgent problem to be solved. Therefore, at present, drones, unmanned vehicles and other fields urgently need a system that can robustly detect small targets. Summary of the invention
[0005] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a small target detection method based on multi-receptive fields and deep features, which utilizes convolutional neural networks and dilated convolution blocks to extract deep features and contextual information of sequence images, enhance the sparse features of small targets, and improve the adaptability and robustness of the system for small targets.
[0006] In order to achieve the above object, the technical solution adopted by the present invention is:
[0007] A small target detection method based on multiple receptive fields and depth features, the method comprising the following steps:
[0008] S1: Serialize the raw video acquired by the airborne or vehicle-mounted camera;
[0009] S2: Use a multi-receptive field feature extraction algorithm to extract features from each input frame image I, and obtain mixed receptive field features without increasing the number of parameters;
[0010] S3: Send the mixed receptive field features obtained in S2 into the deep convolutional neural network to extract deep features;
[0011] S4: Use the multi-receptive field feature aggregation algorithm to extract and integrate context information of different ranges from the deep features in S3, and output a set of multi-receptive field aggregation features f1 * ;
[0012] S5: f1 obtained in S4 * It is sent to the improved path aggregation network for multi-scale fusion and outputs a set of deep fusion features f u ;
[0013] S6: Deep fusion feature f obtained in S5 u Perform regression of the position of the weak target box and determination of the category.
[0014] Furthermore, the multi-receptive field feature extraction algorithm in step S2 includes:
[0015] S21: Use four parallel dilated convolution blocks DC1, DC2, DC3, and DC4 with different dilation coefficients to extract multi-level features from the initial input image respectively;
[0016] S22: merge the four groups of features obtained in S21 into one group of features;
[0017] S23: Send a set of features obtained in S22 into a module consisting of a 3×3 convolutional layer, a BN layer, and a SiLu layer cascade, and output the mixed receptive field features.
[0018] Furthermore, the dilated convolution block DC in step S21 n It consists of a 1×1 convolutional layer with a stride of 1, a 3×3 dilated convolutional layer with a dilation factor of n and a stride of 1, a BN layer, and a SiLu layer in cascade.
[0019] Furthermore, the deep feature extraction in S3 includes the following steps:
[0020] S31: Send the mixed receptive field features output by S2 to a CBS module and output a 2x downsampled feature map p2;
[0021] S32: Send p2 obtained in S31 to a C3 module and output p'2;
[0022] S33: Send the p'2 obtained in S32 to a CBS module and output a 4-fold downsampled feature map p4;
[0023] S34: Send p4 obtained in S33 to a C6 module and output p'4;
[0024] S35: Send p'4 obtained in S34 to a CBS module and output a feature map p8 downsampled by 8 times;
[0025] S36: Send p8 obtained in S35 to a C9 module and output p'8;
[0026] S37: Send p'8 obtained in S36 to a CBS module and output a 16-fold downsampled feature map p 16 ;
[0027] S38: The p obtained in S37 is 16 Feed into a C12 module, output p' 16 .
[0028] Furthermore, the CBS modules in S31, S33, S35, and S37 are composed of a 3×3 convolution with a stride of 2, a BN layer, and a SiLu layer cascaded; the C3, C6, C9, and C12 modules in S32, S34, S36, and S38 are composed of 3, 6, 9, and 12 C modules cascaded respectively; the C module is composed of a 3×3 convolution layer, a BN layer, and a SiLu layer cascaded.
[0029] Furthermore, the multi-receptive field feature aggregation algorithm in S4 includes:
[0030] S41: The deep features obtained in S38 are sequentially passed through four cascaded dilated convolution blocks DC1, DC2, DC3, and DC4 with different dilation ratios;
[0031] S42: concatenate the four sets of features output by each dilated convolution block in S41 into one set of features;
[0032] S43: Use a 1×1 convolutional layer to adjust the weights of each channel of the output feature in S42.
[0033] Furthermore, the multi-scale fusion step in S5 includes:
[0034] S51: Multi-receptive field aggregation feature f1 obtained in S4 * Perform upsampling operation and output p"8;
[0035] S52: concatenate the p'8 obtained in S36 and the p"8 obtained in S51 and send them to a C3 module to output a set of mixed-scale features;
[0036] S53: Use the multi-receptive field feature aggregation algorithm to extract and integrate the context information of the mixed-scale features obtained in S52, and output a set of multi-receptive field aggregation features f2 * ;
[0037] S54: f2 obtained from S53 * Perform upsampling operation and output p"4;
[0038] S55: concatenate the p'4 obtained in S34 and the p"4 obtained in S54 and send them to a C3 module to output a set of deep fusion features f u .
[0039] Further, the up-sampling operation in S51 (same as S54) includes:
[0040] S511: f1 * (S51) or f2 * (S54) is fed into a module composed of a cascade of a 1×1 convolutional layer, a BN layer, and a SiLu layer, and outputs a set of integrated feature maps;
[0041] S512: Use bilinear interpolation to upsample the integrated feature map obtained in S511 to double its resolution.
[0042] Furthermore, the target category determination and box position regression in S6 include:
[0043] S61: The deep fusion feature f obtained in S5 is obtained through the fully connected layer u Mapped to a new feature space, and then passed through a 1×1 convolution layer to output a three-dimensional tensor with 5 channels. The first four channels regress the offsets of the x and y coordinates of the center point of the weak target from the true value, and the offsets of the width and height from the true value;
[0044] S62: The deep fusion feature f obtained in S5 is obtained through the fully connected layer u It is mapped to another new feature space and then passes through a 1×1 convolution layer to output a three-dimensional tensor representing the probability of the category to which the target belongs. The number of channels is the total number of target categories in the database.
[0045] All parameters of the detection model of the present invention are obtained through end-to-end training.
[0046] Compared with the prior art, the present invention can accurately detect and correctly classify small targets in images or videos, providing support for subsequent research in the field of small target detection and recognition. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] In order to make the purpose, technical solution and actual measurement performance of the present invention clearer, the present invention provides the following drawings for illustration:
[0048] Figure 1 Schematic diagram of the overall framework of the small target detection method based on multiple receptive fields and deep features.
[0049] Figure 2 Flowchart of the multi-receptive field feature extraction algorithm.
[0050] Figure 3 Schematic diagram of the dilated convolution block.
[0051] Figure 4 Flowchart of the multi-receptive field feature aggregation algorithm.
[0052] Figure 5 Schematic diagram of dual-channel prediction head.
[0053] Figure 6 This is a diagram showing the actual measured effect of the method of the present invention. DETAILED DESCRIPTION
[0054] The embodiments of the present invention are described in detail below with reference to the accompanying drawings and examples.
[0055] In the present invention, the multi-receptive field feature extraction algorithm uses parallel dilated convolution blocks with different dilation ratios to extract shallow features from each frame of the input sequence;
[0056] The deep convolutional neural network extracts deep features from the shallow features of the input;
[0057] The deep convolutional neural network here does not use pooling operations to avoid the loss of small target information, and a multi-receptive field feature aggregation module is added to the tail of the deep convolutional network to obtain contextual information in different ranges;
[0058] The improved path aggregation network fuses deep features of different scales;
[0059] The improvement of the path aggregation network in the present invention is to add a multi-receptive field feature aggregation module at the front end of each path branch to integrate features of different scales;
[0060] The dual-channel prediction head separates the tasks of target category determination and box position regression, using two convolutional channels to predict the target category and regress the box position coordinates respectively, making the system more robust.
[0061] Figure 1 The overall flow chart of the small target detection method based on multiple receptive fields and depth features of the present invention is as shown in the figure. The method of the present invention specifically includes the following steps:
[0062] Step S1: First, obtain the video of the small target scene through the camera carried by the carrier such as the drone or satellite, and serialize it to obtain an image sequence.
[0063] Step S2: Use the multi-receptive field feature extraction module to extract shallow features of different receptive fields from the serialized single-frame image, i.e., each frame image I, to obtain mixed receptive field features without increasing the number of parameters. The specific process is as follows: Figure 2 As shown:
[0064] S21: Use four parallel expansion convolution blocks DC1, DC2, DC3, and DC4 with expansion coefficients of 1, 2, 3, and 4 respectively to extract multi-level shallow features from the initial input image and obtain four groups of shallow features. For example, refer to Figure 3 , dilated convolution block DC n It consists of a 1×1 convolutional layer with a stride of 1, a 3×3 dilated convolutional layer with a dilation factor of n and a stride of 1, a BN layer, and a SiLu layer cascaded, where n is equal to 1, 2, 3, or 4.
[0065] S22: Merge these four groups of shallow features into one group of features by channel.
[0066] S23: This set of features is sent to a module consisting of a 3×3 convolutional layer, a BN layer, and a SiLu layer cascade for integration, and the mixed receptive field features are output.
[0067] The multi-receptive field feature extraction algorithm captures the multi-level features of the initial input image without reducing the resolution, and transmits multi-level features of different receptive fields to the backbone network, which improves the accuracy of small target detection to a certain extent.
[0068] Step S3: Use a deep convolutional neural network without a pooling layer to perform deep feature extraction on the obtained mixed receptive field features. The specific process is as follows: Figure 1 As shown, the process includes:
[0069] S31: Send the mixed receptive field features output by S2 to a CBS module and output a 2x downsampled feature map p2;
[0070] S32: Send p2 obtained in S31 to a C3 module and output p'2;
[0071] S33: Send the p'2 obtained in S32 to a CBS module and output a 4-fold downsampled feature map p4;
[0072] S34: Send p4 obtained in S33 to a C6 module and output p'4;
[0073] S35: Send p'4 obtained in S34 to a CBS module and output a feature map p8 downsampled by 8 times;
[0074] S36: Send p8 obtained in S35 to a C9 module and output p'8;
[0075] S37: Send p'8 obtained in S36 to a CBS module and output a 16-fold downsampled feature map p 16 ;
[0076] S38: The p obtained in S37 is 16 Feed into a C12 module, output p' 16 .
[0077] Among them, the CBS module is composed of a 3×3 convolution with a stride of 2, a BN layer and a SiLu layer cascaded; the C3 module, C6 module, C9 module, and C12 module are composed of 3, 6, 9, and 12 C modules cascaded respectively; the C module is composed of a 3×3 convolution layer, a BN layer and a SiLu layer cascaded.
[0078] Step S4: Use the multi-receptive field feature aggregation algorithm to extract and integrate context information of different ranges from the deep features output by the deep convolutional neural network, and output a set of multi-receptive field aggregation features f1 * The specific process is as follows Figure 4 As shown:
[0079] S41: The deep features obtained in S38 are sequentially passed through four cascaded dilated convolution blocks DC1, DC2, DC3, and DC4 with different dilation ratios. The process of the dilated convolution block is as follows: Figure 3 shown.
[0080] S42: Concatenate the four sets of features output by the four dilated convolution blocks into one set of features.
[0081] S43: Use a 1×1 convolutional layer to adjust the weights of each channel of this set of features to balance the proportion of feature maps with different receptive fields.
[0082] Step S5: Aggregate the multi-receptive field feature f1 * It is sent to the improved path aggregation network for multi-scale fusion and outputs a set of deep fusion features f u The specific process is as follows Figure 1 As shown, specifically including:
[0083] S51: Aggregate features f1 for multiple receptive fields * Perform upsampling operation and output p"8. For example, f1 *It is sent into a module consisting of a 1×1 convolutional layer, a BN layer, and a SiLu layer cascade, outputting a set of integrated feature maps; then the obtained integrated feature maps are upsampled using bilinear interpolation to double their resolution.
[0084] S52: concatenate the p'8 obtained in S36 and the p"8 obtained in S51 and send them to a C3 module to output a set of mixed-scale features;
[0085] S53: Use the multi-receptive field feature aggregation algorithm to extract and integrate the context information of the mixed-scale features obtained in S52, and output a set of multi-receptive field aggregation features f2 * ;
[0086] S54: f2 obtained from S53 * Perform upsampling operation and output p"4. The method adopts the upsampling method of S51, and the input is f2 * .
[0087] S55: concatenate the p'4 obtained in S34 and the p"4 obtained in S54 and send them to a C3 module to output a set of deep fusion features f u .
[0088] Step S6: The deep fusion feature output by the path aggregation network u The dual-channel prediction head is used to determine the target category and regress the box position. The specific process is as follows: Figure 5 As shown:
[0089] S61: Deep fusion feature f is integrated through the fully connected layer u Mapped to a new feature space, and then passed through a 1×1 convolution layer to output a three-dimensional tensor with 5 channels. The first four channels regress the offsets of the x and y coordinates of the center point of the weak target from the true value, and the offsets of the width and height from the true value;
[0090] S62: Deep fusion feature f is integrated through the fully connected layer u It is mapped to another new feature space and then passes through a 1×1 convolution layer to output a three-dimensional tensor representing the probability of the category to which the target belongs. The number of channels is the total number of target categories in the database.
[0091] When the model is in the training stage, in step S6, binary cross entropy is used to calculate the loss of target category and confidence, and CIOU is used to calculate the error of target position regression.
[0092] Finally, the small target detection method based on multi-receptive field and deep features can detect small targets below 7×7 in actual combat and classify them correctly. This is the progress of the present invention compared with the current advanced methods. The detection results are as follows: Figure 6shown.
[0093] The platform for this experiment is a computer equipped with a 12-core Ryzen processor, 128G RAM, RTX 3090 GPU, pytorch1.8, and CUDA 11.2. During the training phase, the batch size was set to 16, the optimizer used SGD, the training set used VisDrone2021, the initial learning rate was set to 0.0001, and the cosine decay strategy was used for training for 280 rounds. The data enhancement techniques used in this method during the training phase include mosaic, flipping, HSV color transformation, blurring, and histogram equalization. During the testing phase, the confidence threshold of the detector was set to 0.25, and the IOU threshold of non-maximum suppression was set to 0.45. No test-time enhancement technology was used. Figure 6 As shown in the figure, this method not only does not lose medium and large targets such as vehicles, but can also detect tiny pedestrians in drone aerial images. The actual test results illustrate the advancement and robustness of the small target detection method based on multi-receptive field and deep features.
[0094] The above preferred embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the present invention. Although the present invention has been described in detail through the above preferred embodiments, those skilled in the art should understand that various changes can be made in form and details without departing from the scope defined by the claims of the present invention.
Claims
1. A small target detection method based on multiple receptive fields and depth features, characterized in that: The following steps are involved: S1: Acquire sequence images through an airborne or vehicle-mounted camera; S2: extracting features from each input frame image I using a multi-receptive field feature extraction algorithm to obtain mixed receptive field features without increasing the number of parameters; wherein the multi-receptive field feature extraction algorithm includes: S21: Use four parallel dilated convolution blocks DC1, DC2, DC3, and DC4 with different dilation coefficients to extract multi-level features from the initial input image respectively; S22: merge the four groups of features obtained in S21 into one group of features; S23: Send a set of features obtained in S22 into a module consisting of a 3×3 convolutional layer, a BN layer, and a SiLu layer cascade, and output the mixed receptive field features; S3: Send the mixed receptive field features obtained in S2 into the deep convolutional neural network for deep feature extraction to obtain feature maps p'2, p'4, p'8, and p1'6; S4: Use the multi-receptive field feature aggregation algorithm to extract and integrate context information of different ranges from the deep features obtained in S3, and output a set of multi-receptive field aggregation features f1 * ; Wherein, the multi-receptive field feature aggregation algorithm includes: S41: The feature map p1'6 obtained by the deep convolutional neural network is sequentially passed through four cascaded dilated convolution blocks with different dilation ratios; S42: concatenate the four sets of features output by each dilated convolution block in S41 into one set of features; S43: Use a 1×1 convolutional layer to adjust the weights of each channel of the output feature in S42; S5: Aggregate the multi-receptive field feature f1 obtained in S4 * The feature maps p'4 and p'8 obtained from the deep convolutional neural network are sent to the improved path aggregation network for multi-scale fusion, and a set of deep fusion features f is output. u ; Wherein, the multi-scale fusion includes: S51: Multi-receptive field aggregation feature f1 obtained in S4 * Perform upsampling operation and output p'8'; S52: The feature map p'8 obtained by the deep convolutional neural network is concatenated with p'8' obtained by S51 and sent to a C3 module to output a set of mixed-scale features; S53: Use the multi-receptive field feature aggregation algorithm to extract and integrate the context information of the mixed-scale features obtained in S52, and output a set of multi-receptive field aggregation features S54: What S53 got Perform upsampling operation and output p'4'; S55: The feature map p'4 obtained by the deep convolutional neural network is concatenated with p'4' obtained by S54 and sent to a C3 module to output a set of deep fusion features f u ; S6: Deep fusion feature f obtained in S5 u Perform regression of the position of the weak target box and determination of the category.
2. According to claim 1, the method for detecting small and weak targets based on multiple receptive fields and depth features is characterized in that: In step S21, the dilated convolution block DC n It consists of a 1×1 convolutional layer with a stride of 1, a 3×3 dilated convolutional layer with a dilation factor of n and a stride of 1, a BN layer, and a SiLu layer cascaded, where n is equal to 1, 2, 3, or 4.
3. According to claim 2, the method for detecting small and weak targets based on multiple receptive fields and depth features is characterized in that: In step S3, the process of deep feature extraction includes: S31: Send the mixed receptive field features output by S2 to a CBS module and output a 2x downsampled feature map p2; S32: Send p2 obtained in S31 to a C3 module and output p'2; S33: Send the p'2 obtained in S32 to a CBS module and output a 4-fold downsampled feature map p4; S34: Send p4 obtained in S33 to a C6 module and output p'4; S35: Send p'4 obtained in S34 to a CBS module and output a feature map p8 downsampled by 8 times; S36: Send p8 obtained in S35 to a C9 module and output p'8; S37: Send p'8 obtained in S36 to a CBS module and output a 16-fold downsampled feature map p 16 ; S38: The p obtained in S37 is 16 Feed into a C12 module and output p1'6.
4. According to claim 3, the method for detecting small and weak targets based on multiple receptive fields and depth features is characterized in that: The CBS module is composed of a 3×3 convolution with a step size of 2, a BN layer and a SiLu layer cascaded; the C3 module, C6 module, C9 module, and C12 module are respectively composed of 3, 6, 9, and 12 C modules cascaded; the C module is composed of a 3×3 convolution layer, a BN layer and a SiLu layer cascaded.
5. According to claim 4, the method for detecting small and weak targets based on multiple receptive fields and depth features is characterized in that: The up-sampling operation in step S51 or S54 includes: S511: f1 * or It is fed into a module consisting of a 1×1 convolutional layer, a BN layer, and a SiLu layer cascade, outputting a set of integrated feature maps; S512: Use bilinear interpolation to upsample the integrated feature map obtained in S511 to double its resolution.
6. According to claim 4, the method for detecting small and weak targets based on multiple receptive fields and depth features is characterized in that: In step S6, the regression of the target frame position and the determination of the category include S61: The deep fusion feature f obtained in S5 is combined through the fully connected layer u Mapped to a new feature space, and then passed through a 1×1 convolution layer to output a three-dimensional tensor with 5 channels. The first four channels regress the offsets of the x and y coordinates of the center point of the weak target from the true value, and the offsets of the width and height from the true value; S62: The deep fusion feature f obtained in S5 is obtained through the fully connected layer u It is mapped to another new feature space and then passes through a 1×1 convolution layer to output a three-dimensional tensor representing the probability of the category to which the target belongs. The number of channels is the total number of target categories in the database.
7. According to claim 1, the method for detecting small and weak targets based on multiple receptive fields and depth features is characterized in that: All parameters of the detection model are obtained through end-to-end training.
Citation Information
Patent Citations
Small target detection method based on unmanned aerial vehicle image
CN112418117A
Multilevel feature pyramid target detection method based on depth separable convolution and fusion PAN
CN113111736A
Apparatus and method for detecting scene text in an image
US20190130204A1
