Multi-mode air dim and small target detection method for noise suppression

Through asymmetric context noise suppression network and multimodal image fusion technology, noise interference and environmental complexity problems in air small object detection are solved, detection accuracy and robustness are improved, and effective extraction and detection of small object features are achieved. .

CN118918314APending Publication Date: 2024-11-08CHINESE PEOPLES LIBERATION ARMY ARMY ARTILLERY & AIR DEFENSE ACAD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410983071.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-22
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

The prior art is difficult to accurately and quickly detect small air targets, especially in noise interference and complex environments, resulting in low detection accuracy and difficulty in feature extraction.

Method used

The infrared image is noise suppressed by asymmetric context noise suppression network, and features are fused with visible light images. Through the accumulation and fusion of multimodal images, combined with the air weak target detection network, small target features are extracted and predicted.

Benefits of technology

It significantly improves the accuracy and robustness of small-object detection, reduces noise interference, enhances the extraction ability of small-object features, and improves the detection performance of the system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918314A_ABST
    Figure CN118918314A_ABST
Patent Text Reader

Abstract

The invention discloses a noise-suppressed multi-mode air dim and small target detection method, which comprises the following steps: acquiring a visible light image data set and an infrared image data set, and inputting the infrared image data set into an asymmetric context noise suppression network for noise suppression processing to obtain a noise-suppressed infrared image data set; performing feature extraction on the visible light image data set and the infrared image data set after noise suppression, and fusing the extracted visible light features and infrared features to obtain a fused image data set; inputting the fused image data set into an air weak and small target detection network to obtain an air weak and small target prediction result; according to the invention, the weak and small target in the air can be detected more accurately and quickly.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image processing and target detection, and in particular to a noise-suppressed multi-modal aerial dim small target detection method. Background Art

[0002] Compared with regular-sized object detection, small object detection usually lacks sufficient appearance information, making it difficult to distinguish small objects from the background or similar objects.

[0003] Driven by deep learning, although target detection algorithms have made significant breakthroughs, the detection of small targets is still unsatisfactory. Whether based on absolute or relative scale, small targets have low resolution relative to large / medium-scale targets. Low-resolution small targets have little visual information, making it difficult to extract discriminative features, and are easily disturbed by environmental factors, which makes it difficult for the detection model to accurately locate and identify small targets. In addition, real scenes are complex, and there are usually problems such as drastic changes in lighting, target occlusion, densely connected targets, changes in target scale, and noise interference. These factors have a more drastic impact on the features of small targets, further increasing the difficulty of detecting small targets, especially small targets in the air, which will be blocked by clouds, etc., and in dim light conditions such as rainy days, the color of small targets will be close to the background and difficult to distinguish.

[0004] At present, there are generally the following ideas for studying small target detection at home and abroad:

[0005] (1) Data augmentation

[0006] Data augmentation is the simplest and most effective method to improve the performance of small target detection. Different data augmentation strategies can expand the scale of the training data set and enrich the diversity of the data set, thereby enhancing the robustness and generalization ability of the detection model. In relatively early studies, the accuracy of handwriting recognition was significantly improved by using data augmentation methods such as distortion, rotation and scaling. Later, strategies such as elastic deformation, random cropping and translation were derived from data augmentation. At present, these data augmentation strategies have been widely used in target detection. Focusing on the field of small target detection, small targets face many challenges such as low resolution, few extractable features, lack of samples and uneven distribution. The importance of data augmentation has become increasingly prominent. In recent years, some data augmentation methods suitable for small targets have emerged. In the processing of data, a scale matching strategy is proposed to crop according to different target sizes to narrow the gap between targets of different sizes, thereby avoiding the situation where small target information is easily lost in conventional scaling operations. In order to solve the problems of small targets covering a small area, lack of diversity in their locations, and the intersection-over-union ratio between the detection box and the true value box being far less than the expected threshold, a copy enhancement method is proposed. By copying and pasting small targets in the image multiple times, the number of training samples of small targets is increased, thereby improving the detection performance of small targets. On this basis, an adaptive resampling strategy is proposed in RRNet for data enhancement. This strategy copies the target image with context information based on the pre-trained semantic segmentation network to solve the background mismatch and scale mismatch problems that may occur in the simple copying process, thereby achieving a better data enhancement effect. Starting from the problems of small number of small targets and little information contained in themselves, the images are scaled and spliced ​​during the training process, and the large-sized targets in the data set are converted to medium-sized targets, and the medium-sized targets are converted to small-sized targets. While improving the number and quality of medium / small-sized targets, the computational cost is also taken into account. In addition to designing corresponding data enhancement strategies for the characteristics of small targets, it goes beyond the target characteristic limitations and proposes a data enhancement strategy that selects the best data enhancement strategy through adaptive learning methods such as reinforcement learning, which has achieved certain performance improvements in small target detection.

[0007] (2) Multi-scale learning

[0008] Compared with regular targets, small targets have fewer available pixels, making it difficult to extract good features. Moreover, as the number of network layers increases, the feature information and position information of small targets are gradually lost, making them difficult to be detected by the network. These characteristics lead to the need for both deep semantic information and shallow representation information for small targets. Multi-scale learning combines these two, which is an effective strategy to improve the performance of small target detection.

[0009] (3) Contextual Learning

[0010] In the real world, there is usually a coexistence relationship between "target and scene" and "target and target". By utilizing this relationship, it will help improve the detection performance of small targets. Before deep learning, studies have shown that the performance of target detection can be improved by properly modeling the context, especially for small targets with unclear appearance features. With the widespread application of deep neural networks, some studies have also tried to integrate the context around the target into deep neural networks and have achieved certain results.

[0011] (4) Generative Adversarial Learning

[0012] The generative adversarial learning method aims to achieve the same detection performance as larger targets by mapping the features of low-resolution small targets into features equivalent to those of high-resolution targets. Although the methods mentioned above, such as data enhancement, feature fusion, and contextual learning, can effectively improve the detection performance of small targets, the performance gains brought by these methods are often limited by the computational cost. To address the problem of low resolution of small targets, an end-to-end joint training method for super-resolution and detection models is proposed, which improves the detection performance of low-resolution targets to a certain extent. However, this method has high requirements for the training dataset and does not improve the detection performance of small targets sufficiently.

[0013] (5) Anchor-free mechanism

[0014] The anchor box mechanism plays an important role in object detection. Many advanced object detection methods are designed based on the anchor box mechanism, but the anchor box design is extremely unfriendly to the detection of small objects. The existing anchor box design is difficult to balance the contradiction between the recall rate of small objects and the computational cost. Moreover, this method leads to an extreme imbalance between the positive samples of small objects and the positive samples of large objects, making the model more focused on the detection performance of large objects, thereby ignoring the detection of small objects. In extreme cases, if the designed anchor box is much larger than the small object, there will be no positive samples for the small object. The lack of positive samples for small objects will make the algorithm only learn the detection model suitable for larger objects. In addition, the use of anchor boxes introduces a large number of hyperparameters, such as the number, aspect ratio and size of anchor boxes, which makes it difficult to train the network and not easy to improve the detection performance of small objects.

[0015] In the related art, the patent application document with publication number CN116958782A proposes to realize feature extraction of infrared light and visible light modal data through a dual-stream backbone network model, and to realize the complementarity of the two modal features through cross-modal multi-level inter-level feature fusion, and to realize the positioning and classification of small targets using a deep learning-based network; however, this solution ignores the impact of noise on the detection of small targets in the air, and may not be applicable in some scenarios, such as when the image collected by the camera is not accurate enough and contains interference noise.

[0016] The patent application document with publication number CN116416503A proposes to design feature extraction networks separately for different modalities, so that the subtle features of different models can be better extracted, and a modal feature fusion network is designed to perform feature-level fusion, increase the number of effective features, and effectively improve the detection and recognition performance; however, this solution also ignores the impact of noise on the detection of small aerial targets, and designs feature extraction networks separately for different modalities, and the network structure is complex. Summary of the invention

[0017] The technical problem to be solved by the present invention is how to detect small aerial targets more accurately and quickly.

[0018] The present invention solves the above technical problems by the following technical means:

[0019] The present invention proposes a noise-suppressed multi-modal aerial small target detection method, the method comprising:

[0020] Acquire a visible light image dataset and an infrared image dataset, and input the infrared image dataset into an asymmetric context noise suppression network for noise suppression processing to obtain a noise-suppressed infrared image dataset, wherein the asymmetric context noise suppression network includes a low-level feature extraction branch and a high-level feature extraction branch, a low-dimensional feature image of the infrared image dataset is used as input of the low-level feature extraction branch, and a high-dimensional feature image of the infrared image dataset is used as input of the high-level feature extraction branch, and an output of the low-level feature extraction branch and an accumulated fusion result of the high-dimensional feature image are accumulated and fused with an output of the high-level feature extraction branch and an accumulated fusion result of the low-dimensional feature image to obtain the noise-suppressed infrared image dataset;

[0021] Extracting features from the visible light image data set and the infrared image data set after noise suppression respectively, and fusing the extracted visible light features and infrared features to obtain a fused image data set;

[0022] The fused image data set is input into an aerial small target detection network to obtain an aerial small target prediction result.

[0023] Furthermore, the low-level feature extraction branch includes a convolution PW-Conv_1 and a convolution PW-Conv_2 connected in series, and the convolution PW-Conv_2 is followed by an activation function Sigmod_1;

[0024] The high-level feature extraction branch includes a GlobalAvgPool2D function, a fully connected layer Fully-Connected_1 and a fully connected layer Fully-Connected_2 connected in series, and the fully connected layer Fully-Connected_2 is followed by an activation function Sigmod_2.

[0025] Furthermore, the extracting features from the visible light image data set and the infrared image data set after noise suppression respectively, and fusing the extracted visible light features and infrared features to obtain a fused image, comprises:

[0026] Inputting the visible light image data set and the infrared image data set after noise suppression into an image fusion generation network respectively, wherein the image fusion generation network comprises an image feature extraction module and a feature fusion reconstruction module connected in sequence;

[0027] Using the image feature extraction module to perform feature extraction on the visible light image data set and the infrared image data set after noise suppression, respectively, to obtain the visible light feature and the infrared feature;

[0028] The feature fusion and reconstruction module is used to perform channel superposition and fusion on the visible light feature and the infrared feature to obtain the fused image.

[0029] Furthermore, the image feature extraction module includes a visible light feature extraction branch and an infrared feature extraction branch, wherein:

[0030] The visible light feature extraction branch includes a convolution blocks connected in series, and the i-th convolution block in the visible light feature extraction branch is jump-connected with the next ai convolution blocks, i=1, 2, ..., a;

[0031] The infrared feature extraction branch includes b convolution blocks connected in series, and in the visible light feature extraction branch, the jth convolution block is jump-connected with the next bj convolution blocks, where j=1, 2, ..., b.

[0032] Furthermore, the feature fusion reconstruction module includes a feature fusion network and c convolution blocks connected in series, the outputs of the visible light feature extraction branch and the infrared feature extraction branch are both connected to the input of the feature fusion network, and the output of the feature fusion network is connected to the first convolution block among the c convolution blocks connected in series.

[0033] Furthermore, the small target detection network includes a multi-dimensional feature extraction module, a multi-dimensional feature fusion module and a result prediction module, wherein:

[0034] The multidimensional feature extraction module is used to extract the multidimensional features of the fused image data set;

[0035] The multidimensional feature fusion module is used to fuse the multidimensional features to generate a multidimensional prediction feature map;

[0036] The result prediction module is used to separately process and then fuse the multi-dimensional prediction feature graph to obtain a prediction result of a small and weak target in the air.

[0037] Further, the multidimensional feature extraction module includes d Dark blocks connected in series, wherein the outputs of the d-th Dark block, the d-1-th Dark block and the d-2-th Dark block are all connected to the multidimensional feature fusion module;

[0038] The d-2th Dark block, the d-1th Dark block and the dth Dark block output the one-dimensional features, two-dimensional features and three-dimensional features related to the weak targets in the air respectively.

[0039] Furthermore, the multi-dimensional feature fusion module includes a one-dimensional feature fusion layer, a two-dimensional feature fusion layer and a three-dimensional feature fusion layer, wherein:

[0040] The three-dimensional feature fusion layer includes a residual convolution block Lateral_conv0 and a deconvolution unsample_1 connected in sequence, the three-dimensional feature output of the d-th Dark block is used as the input of the residual convolution block Lateral_conv0, and the output of the deconvolution Unsample_1 and the two-dimensional feature output of the d-1-th Dark block are both used as the input of the two-dimensional feature fusion layer;

[0041] The two-dimensional feature fusion layer includes a fusion operation Cat_1, a residual convolution block C3_p4, Reduce_conv1 and a deconvolution Unsample_2 connected in sequence. The output of the deconvolution Unsample_1 and the two-dimensional feature output by the d-1th Dark block are fused through the fusion operation Cat_1 to obtain an enhanced two-dimensional feature. The enhanced two-dimensional feature is used as the input of C3_p4. The output of the deconvolution Unsample_2 and the one-dimensional feature output by the d-2th Dark block are used as the input of the one-dimensional feature fusion layer.

[0042] The one-dimensional feature fusion layer includes a fusion operation Cat_2 and a residual convolution block C3_p3 connected in sequence, the output of the deconvolution Unsample_2 is fused with the one-dimensional feature output by the d-2th Dark block through the fusion operation Cat_2 to obtain an enhanced one-dimensional feature, and the enhanced one-dimensional feature is outputted as a one-dimensional prediction feature map after the residual convolution block C3_p3;

[0043] The output of the residual convolution block C3_p3 is connected to the convolution Bu_conv2 and the result prediction module respectively, and the output of Reduce_conv1 and the output of the convolution Bu_conv2 are fused through the fusion operation Cat_3 and then output a two-dimensional prediction feature map through the residual convolution block C3_n3;

[0044] The output of the residual convolution block C3_n3 is connected to the convolution Bu_conv1 and the result prediction module respectively, the output of the convolution Bu_conv1 is connected to the residual convolution block C3_n4 through the fusion operation Cat_4, and the residual convolution block C3_n4 outputs a three-dimensional prediction feature map to the result prediction module.

[0045] Furthermore, before extracting features from the visible light image data set and the infrared image data set after noise suppression, respectively, and fusing the extracted visible light features and infrared features to obtain a fused image data set, the method further includes:

[0046] The image fusion generation network is pre-trained to obtain the trained image fusion generation network for generating a fused image data set, wherein the loss function LOSS used in the training is:

[0047] LOSS=λL SSIM +L tv

[0048] Where λ is the equilibrium parameter, L SSIM represents the image structure similarity loss function, L tv Represents the image noise loss function.

[0049] Furthermore, the loss function L SSIM The formula is:

[0050]

[0051] In the formula, I A represents the visible light image, I B Indicates infrared image, I F represents the fused image, W represents the sliding window, N represents the total number of sliding windows, Score(I A ,I B ,I F |W) means retaining the information of the visible light image or infrared image that has a higher similarity with the fused image in window W;

[0052] The loss function L tv The formula is:

[0053] Ltv =∑ i,j ||R(i,j+1)-R(i,j)|| 2 +||R(i+1,j)-R(i,j)|| 2

[0054] Where R represents the difference between the visible light image and the fused image, R(i,j+1) represents the difference between the pixel value of the i-th row and j+1 column of the visible light image and the pixel value of the i-th row and j+1 column of the fused image, R(i,j) represents the difference between the pixel value of the i-th row and j column of the visible light image and the pixel value of the i-th row and j column of the fused image, R(i+1,j) represents the difference between the pixel value of the i+1-th row and j column of the visible light image and the pixel value of the i+1-th row and j column of the fused image, || || 2 represents the Euclidean distance.

[0055] The advantages of the present invention are:

[0056] (1) The present invention simultaneously obtains a visible light image dataset and an infrared image dataset of small and weak targets in the air, and performs noise suppression processing on the infrared image by using an asymmetric context noise suppression network, thereby reducing the interference of noise in the infrared image on small targets, so as to more accurately obtain the semantic information of the target, and solve the problem that the real scene is complicated and usually there are factors such as drastic changes in illumination, target occlusion, dense connection of targets, and change in target scale that cause interference to small target images; then, feature extraction is performed on the visible light image dataset and the infrared image dataset after noise suppression, and the extracted visible light features and infrared features are fused. Through the cumulative fusion of multimodal images, the characteristics of visible light and infrared images are fully utilized, and the fusion method is used to highlight small and weak targets, and the extraction of small target image features is strengthened, so as to solve the current problems of small target detection with little visual information, difficulty in extracting discriminative features, and susceptibility to interference from environmental factors, and significantly improve the small target detection accuracy.

[0057] (2) When detecting small targets, the image fusion generation network established by the present invention reduces the uncertainty and redundancy of the output on the basis of maximally merging relevant information. The advantages of image fusion are obvious. It can expand the temporal and spatial information contained in the image, reduce uncertainty, increase reliability, and improve the robustness of the system.

[0058] (3) The aerial small target detection network established by the present invention uses a decoupling head to separately process three different feature maps (one-dimensional, two-dimensional, and three-dimensional features of aerial small targets) and then fuse them to obtain the final predicted image of aerial small targets, thereby further improving the small target detection performance.

[0059] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is a flowchart of a noise-suppressed multi-modal aerial small target detection method proposed in one embodiment of the present invention;

[0061] Figure 2 is a structural diagram of an asymmetric context noise suppression network in one embodiment of the present invention;

[0062] Figure 3 is a structural diagram of an image fusion generation network in one embodiment of the present invention;

[0063] Figure 4 It is a structural diagram of an aerial small target detection network in one embodiment of the present invention. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in combination with the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0065] like Figure 1 As shown, an embodiment of the present invention provides a noise-suppressed multi-modal aerial small target detection method, the method comprising the following steps:

[0066] S10, obtaining a visible light image dataset and an infrared image dataset, and inputting the infrared image dataset into an asymmetric context noise suppression network for noise suppression processing to obtain a noise-suppressed infrared image dataset;

[0067] Among them, the asymmetric contextual noise suppression network includes a low-level feature extraction branch and a high-level feature extraction branch, the low-dimensional feature image of the infrared image data set is used as the input of the low-level feature extraction branch, and the high-dimensional feature image of the infrared image data set is used as the input of the high-level feature extraction branch. The output of the low-level feature extraction branch and the cumulative fusion result of the high-dimensional feature image are cumulatively fused with the output of the high-level feature extraction branch and the low-dimensional feature image to obtain the infrared image data set after noise suppression.

[0068] It should be noted that for weak targets in the air, visible light images are easily affected by the environment, such as dim light and complex background, which makes the target not prominent. Although infrared images can highlight the target, the complex background may also cause the weak target to be submerged in the clutter. Therefore, infrared images can effectively resist clutter interference after being processed by the noise suppression network, and are easier to highlight the target after being fused with the visible light image. The subsequent fused image is easier to detect by the detection network.

[0069] S20, performing feature extraction on the visible light image data set and the infrared image data set after noise suppression respectively, and fusing the extracted visible light features and infrared features to obtain a fused image data set.

[0070] It should be noted that this embodiment makes full use of the characteristics of visible light and infrared images, uses a fusion method to highlight weak targets, strengthens the extraction of small target image features, solves the current problems of small target detection with little visual information, difficulty in extracting discriminative features, and being easily disturbed by environmental factors, and significantly improves the accuracy of small target detection

[0071] S30, inputting the fused image data set into an aerial small target detection network to obtain an aerial small target prediction result.

[0072] As a further preferred technical solution, Figure 2 As shown, the low-level feature extraction branch includes a series of convolutions PW-Conv_1 and PW-Conv_2, and the convolution PW-Conv_2 is followed by an activation function Sigmod_1;

[0073] The high-level feature extraction branch includes a GlobalAvgPool2D function, a fully connected layer Fully-Connected_1 and a fully connected layer Fully-Connected_2 connected in series, and the fully connected layer Fully-Connected_2 is followed by an activation function Sigmod_2.

[0074] Specifically, this embodiment obtains video data of a small target in the air and processes it into a small target visible light image data set S = {s 1 ,s 2 ,…,s n ,…,s N}, where s n represents the nth visible light image of a small target in the air, n = 1, 2, ..., N, where N is the total number of visible light images of small targets in the air;

[0075] Obtain infrared image data set of small and weak targets in the air T = {t 1 ,t 2 ,…,tn ,…,t N}, where t n It represents the nth infrared image of a small and weak target in the air, n = 1, 2, ..., N, and N is the total number of infrared images of small and weak targets in the air.

[0076] Through convolution of different scales, low-dimensional feature extraction and high-dimensional feature extraction are performed on the infrared image dataset T of small and weak targets in the air to obtain the low-dimensional feature image dataset TL = {tl 1 ,tl 2 ,…,tl n ,…,tl N} and high-dimensional feature image dataset TH = {th 1 ,th 2 ,…,th n ,…,th N}, by performing high-dimensional feature extraction and low-dimensional feature extraction on infrared images, the semantic information of the target can be obtained more accurately.

[0077] Next, the nth low-dimensional feature image tl n After inputting the low-level feature extraction branch for processing, the low-dimensional infrared image feature of the weak target in the air is obtained. n , the nth high-dimensional feature image th n After inputting the advanced feature extraction branch for processing, the high-dimensional infrared image feature rth of the small target in the air is obtained. n , tl n and rth n After cumulative fusion, add rtl n and n The result of cumulative fusion is the infrared image Dnt after noise suppression. n Finally, the infrared image data set D = {Dnt 1 ,Dnt 2 ,…,Dnt n ,…,Dnt N}.

[0078] It should be noted that the asymmetric contextual noise suppression network constructed in this embodiment includes two branches, the first branch is a low-level feature extraction branch composed of a PointWise convolution in series and a Sigmod activation function, and the second branch is a high-level feature extraction branch composed of a GlobalAvgPool2D function, a fully connected layer and a Sigmod activation function in series; by using the low-level feature extraction branch and the high-level feature extraction branch to exchange the semantic scale context multiple times, richer semantic information and spatial detail encoding can be obtained, so as to more accurately obtain the semantic information of the target.

[0079] It should be noted that for infrared images of small aerial targets, once the background is complex, they are easily submerged in the complex background of low signal-to-clutter ratio, so noise suppression is needed to highlight them. The purpose of using low-level features and high-level features is to solve the problem of scale mismatch between subsequent infrared small targets and visible light small targets, and at the same time, the visual details of smaller scales can be encoded to a deeper level. The mutual modulation between high-level features and low-level features is achieved in an asymmetric way to maintain and highlight the infrared small targets in the high-level features.

[0080] As a further preferred technical solution, the step S20: extracting features from the visible light image data set and the infrared image data set after noise suppression, respectively, and fusing the extracted visible light features and infrared features to obtain a fused image, comprises the following steps:

[0081] Inputting the visible light image data set and the infrared image data set after noise suppression into an image fusion generation network respectively, wherein the image fusion generation network comprises an image feature extraction module and a feature fusion reconstruction module connected in sequence;

[0082] Using the image feature extraction module to perform feature extraction on the visible light image data set and the infrared image data set after noise suppression, respectively, to obtain the visible light feature and the infrared feature;

[0083] The feature fusion and reconstruction module is used to perform channel superposition and fusion on the visible light feature and the infrared feature to obtain the fused image.

[0084] As a further preferred technical solution, Figure 3 As shown, the image feature extraction module includes a visible light feature extraction branch and an infrared feature extraction branch, wherein:

[0085] The visible light feature extraction branch includes a convolution blocks connected in series, and the i-th convolution block in the visible light feature extraction branch is jump-connected with the next ai convolution blocks, i=1, 2, ..., a;

[0086] The infrared feature extraction branch includes b convolution blocks connected in series, and in the visible light feature extraction branch, the jth convolution block is jump-connected with the next bj convolution blocks, where j=1, 2, ..., b.

[0087] The advantage of the visible light feature extraction branch and the infrared feature extraction branch designed in this embodiment is that the two share the same weights to extract the same type of depth features, which effectively extracts features while reducing the computational complexity of the model, making the model lighter.

[0088] As a further preferred technical solution, the feature fusion reconstruction module includes a feature fusion network and c sequentially connected convolution blocks, the outputs of the visible light feature extraction branch and the infrared feature extraction branch are connected to the input of the feature fusion network, and the output of the feature fusion network is connected to the first convolution block of the c sequentially connected convolution blocks. Among them, the feature fusion network directly performs channel superposition through convolution, the purpose is to fuse the visible light and infrared image features, and the c sequentially connected convolution blocks are used to reconstruct the fusion result from the fused features.

[0089] Specifically, the image feature extraction module is divided into two branches, wherein the visible light feature extraction branch is composed of a convolution blocks connected in series, where a is 4, and the first convolution block is jump-connected to the next a-1 convolution blocks; the infrared feature extraction branch is composed of b convolution blocks connected in series, where b is 4, and the first convolution block is jump-connected to the next b-1 convolution blocks.

[0090] The nth visible light image s n After inputting the visible light feature extraction branch for processing, the visible light feature Is corresponding to the nth visible light image is obtained. n , the nth infrared image after noise suppression Dnt n After inputting the infrared feature extraction branch for processing, the infrared feature It corresponding to the nth infrared image after noise suppression is obtained. n .

[0091] The visible light feature Is corresponding to the nth visible light image n The infrared feature It corresponding to the nth noise-suppressed infrared image n After being input into the feature fusion reconstruction module, the channel superposition is first directly performed through the feature fusion network to obtain the fusion image of the weak target in the air, and then the reconstructed fusion image ST is output after c convolutions. n .

[0092] The image fusion generation network established in this embodiment reduces the uncertainty and redundancy of the output on the basis of maximizing the merging of relevant information when detecting small targets. The advantages of image fusion are obvious. It can expand the temporal and spatial information contained in the image, reduce uncertainty, increase reliability, and improve the robustness of the system.

[0093] It should be noted that, in this embodiment, the noise in the infrared image is suppressed by an asymmetric context noise suppression network before the infrared image and the visible light image are fused; secondly, a dual-stream backbone network is used for image fusion in file 1, and the features of each layer of the visible light and infrared images are fused with each other to finally generate a fused image, while the present application scheme does not fuse the features of each other before the two types of images are fused, but instead fully extracts the features of the visible light and infrared images through residual connections before fusing them, and then performs convolution operations on the fusion to reconstruct the fusion result from the fused features for subsequent detection.

[0094] As a further preferred technical solution, before fusing the extracted visible light features and infrared features, the present embodiment needs to pre-train the image fusion generation network. After the training is completed, the image fusion generation network can be used to output the optimal fused image. The nth aerial dim target fusion image ST n The optimal aerial dim target fusion image ST′ is obtained through the loss function LOSS training optimization. n , and so on, we get the optimal aerial weak target fusion feature data set ST = {ST′ 1 ,ST′ 2 ,…,ST′ n ,…,ST′ N}, N is the total number of optimal aerial dim target fusion images.

[0095] Among them, the loss function LOSS used is:

[0096] LOSS=λL SSIM +L tv

[0097] Where λ is the equilibrium parameter, L SSIM represents the image structure similarity loss function, L tv Represents the image noise loss function, tv stands for Total Variation, which is an indicator to measure image noise.

[0098] As a further preferred technical solution, construct the loss function L SSIM as follows:

[0099]

[0100] In the formula, x and I A Represents a visible light image, y and I B Indicates infrared image, I F represents the fused image, σ x and σ y They represent the standard deviation of the visible light image pixel value and the standard deviation of the infrared image pixel value, σ xyis the covariance of the visible light image and the infrared image, C represents a constant, W represents a sliding window with a size of m×n, and P i represents the pixel value of the sliding window area, and N represents the total number of sliding windows; SSIM M (x, y|W) represents the formula for calculating the structural similarity in the sliding window W, E(I|W) represents the average value of the pixels in the sliding window W, Score(I A ,I B ,I F |W) indicates retaining the information of the visible light image or infrared image that has a higher similarity with the fused image in window W, that is, whichever of the visible light image and the infrared image is more similar to the fused image in window W, the image information with higher similarity is retained.

[0101] This embodiment can adaptively retain detail information and learn deep features by designing an image structure similarity loss function.

[0102] The loss function L constructed using the neighbor method tv as follows:

[0103] R(i,j)=I A (i,j)-I F (i,j)

[0104] L tv =∑ i,j ||R(i,j+1)-R(i,j)|| 2 +||R(i+1,j)-R(i,j)|| 2

[0105] Where R represents the difference between the visible light image and the fused image, I A (i, j) represents the pixel value of the i-th row and j-th column of the visible light image. F (i,j) represents the pixel value of the i-th row and j-th column of the fused image, || || 2 represents the Euclidean distance.

[0106] The image noise loss function designed in this embodiment can be used to reduce the impact of noise on the variation between pixels during the training process.

[0107] As a further preferred technical solution, the small target detection network used in step S30 is a pre-built and trained network structure, and the small target detection network includes a multi-dimensional feature extraction module, a multi-dimensional feature fusion module and a result prediction module, wherein:

[0108] The multidimensional feature extraction module is used to extract the multidimensional features of the fused image data set;

[0109] The multidimensional feature fusion module is used to fuse the multidimensional features to generate a multidimensional prediction feature map;

[0110] The result prediction module is used to separately process and then fuse the multi-dimensional prediction feature graph to obtain a prediction result of a small and weak target in the air.

[0111] Specifically, Figure 4 As shown, the multidimensional feature extraction module includes d Dark blocks connected in series, wherein the outputs of the d-th Dark block, the d-1-th Dark block and the d-2-th Dark block are all connected to the multidimensional feature fusion module;

[0112] The d-2th Dark block, the d-1th Dark block and the dth Dark block output the one-dimensional features, two-dimensional features and three-dimensional features related to the weak targets in the air respectively.

[0113] By fusion image ST′ of the nth optimal aerial dim target n Input into the multi-dimensional feature extraction module, after the feature extraction of d-2 Dark blocks, the one-dimensional feature ST-1′ of the small target in the air is obtained n , ST-1′ n After extracting the features of the d-1th Dark block, we get the two-dimensional features of the small target in the air, ST-2′ n , ST-2′ n After extracting the features of the dth Dark block, we get the three-dimensional features of the small aerial target ST-3′ n .

[0114] It should be noted that the feature extraction module is implemented using the darknet53 network, which is composed of d Dark blocks connected in series, where d is 5.

[0115] Specifically, Figure 4 As shown, the multi-dimensional feature fusion module includes a one-dimensional feature fusion layer, a two-dimensional feature fusion layer and a three-dimensional feature fusion layer, wherein:

[0116] The three-dimensional feature fusion layer includes a residual convolution block Lateral_conv0 and a deconvolution unsample_1 connected in sequence, the three-dimensional feature output of the d-th Dark block is used as the input of the residual convolution block Lateral_conv0, and the output of the deconvolution Unsample_1 and the two-dimensional feature output of the d-1-th Dark block are both used as the input of the two-dimensional feature fusion layer;

[0117] The two-dimensional feature fusion layer includes a fusion operation Cat_1, a residual convolution block C3_p4, an upsampling convolution Reduce_conv1 and a deconvolution Unsample_2 connected in sequence. The output of the deconvolution Unsample_1 and the two-dimensional feature output by the d-1th Dark block are fused through the fusion operation Cat_1 to obtain an enhanced two-dimensional feature. The enhanced two-dimensional feature is used as the input of C3_p4. The output of the deconvolution Unsample_2 and the one-dimensional feature output by the d-2th Dark block are used as the input of the one-dimensional feature fusion layer.

[0118] The one-dimensional feature fusion layer includes a fusion operation Cat_2 and a residual convolution block C3_p3 connected in sequence, the output of the deconvolution Unsample_2 is fused with the one-dimensional feature output by the d-2th Dark block through the fusion operation Cat_2 to obtain an enhanced one-dimensional feature, and the enhanced one-dimensional feature is outputted as a one-dimensional prediction feature map after the residual convolution block C3_p3;

[0119] The output of the residual convolution block C3_p3 is connected to the convolution Bu_conv2 and the result prediction module respectively, and the output of the convolution Reduce_conv1 and the output of the convolution Bu_conv2 are fused through the fusion operation Cat_3 and then output a two-dimensional prediction feature map through the residual convolution block C3_n3;

[0120] The output of the residual convolution block C3_n3 is connected to the downsampling convolution Bu_conv1 and the result prediction module respectively, the output of the convolution Bu_conv1 is connected to the residual convolution block C3_n4 through the fusion operation Cat_4, and the residual convolution block C3_n4 outputs a three-dimensional prediction feature map to the result prediction module.

[0121] It should be noted that the multi-dimensional feature fusion module contains a three-layer structure. The first layer is a one-dimensional feature fusion layer, which is composed of residual convolution blocks, the second layer is a two-dimensional feature fusion layer, which is composed of residual convolution blocks and dimensionality reduction convolution blocks, and the third layer is a three-dimensional feature fusion layer, which is composed of residual convolution and dimensionality reduction convolution. For small and weak targets in the air, the use of a multi-dimensional feature fusion module can extract small target features of different dimensions, which is conducive to improving the accuracy of subsequent result prediction.

[0122] ST-3′ n The input feature fusion network is processed by the residual convolution block and then deconvolution to enter the second layer of the feature fusion network and merge with ST-2′ n Fusion enhanced two-dimensional features of small aerial targets RST-2′ n , RST-2′ nAfter being processed by the residual block, it is deconvolved and enters the one-dimensional feature fusion layer and is combined with ST-1′ n Fusion enhanced one-dimensional features of small aerial targets RST-1′ n , RST-1′ n After the residual convolution block is processed, the one-dimensional prediction feature map of the small target in the air is obtained. n ;pan-1 n At the same time, it is convolved and combined with RST-2′ n After fusion, the residual convolution block is used to process the two-dimensional prediction feature map of the small target in the air, pan-2. n ;pan-2 n At the same time, after convolution processing, the residual convolution block is processed to obtain the three-dimensional prediction feature map of the weak target in the air pan-3 n .

[0123] As a further preferred technical solution, the result prediction module is composed of a decoupling head, and the multi-dimensional feature fusion module combines pan-1 n 、pan-2 n and pan-3 n The input is sent to the result prediction module, and the three different feature maps are processed separately and then fused through the decoupling head to obtain the final prediction image of the small target in the air, Finalpan n .

[0124] In this embodiment, the result prediction module is composed of three branches, which contain convolutions and downsampling rates of different scales. The result prediction module uses a decoupling head to separately process three different feature maps (one-dimensional, two-dimensional, and three-dimensional features of small and weak targets in the air) and then fuses them to obtain the final predicted image of small and weak targets in the air. The use of a decoupling head to form a result prediction module not only improves the model detection accuracy but also makes the network converge faster, further improving the small target detection performance.

[0125] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0126] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0127] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A noise-suppressed multi-modal aerial small target detection method, characterized in that: The method comprises: Acquire a visible light image dataset and an infrared image dataset, and input the infrared image dataset into an asymmetric context noise suppression network for noise suppression processing to obtain a noise-suppressed infrared image dataset, wherein the asymmetric context noise suppression network includes a low-level feature extraction branch and a high-level feature extraction branch, a low-dimensional feature image of the infrared image dataset is used as input of the low-level feature extraction branch, and a high-dimensional feature image of the infrared image dataset is used as input of the high-level feature extraction branch, and an output of the low-level feature extraction branch and an accumulated fusion result of the high-dimensional feature image are accumulated and fused with an output of the high-level feature extraction branch and an accumulated fusion result of the low-dimensional feature image to obtain the noise-suppressed infrared image dataset; Extracting features from the visible light image data set and the infrared image data set after noise suppression respectively, and fusing the extracted visible light features and infrared features to obtain a fused image data set; The fused image data set is input into an aerial small target detection network to obtain an aerial small target prediction result.

2. The noise-suppressed multi-modal aerial small target detection method according to claim 1, characterized in that: The low-level feature extraction branch includes a convolution PW-Conv_1 and a convolution PW-Conv_2 connected in series, and the convolution PW-Conv_2 is followed by an activation function Sigmod_1; The high-level feature extraction branch includes a GlobalAvgPool2D function, a fully connected layer Fully-Connected_1 and a fully connected layer Fully-Connected_2 connected in series, and the fully connected layer Fully-Connected_2 is followed by an activation function Sigmod_2.

3. The noise-suppressed multi-modal aerial small target detection method according to claim 1, characterized in that: The extracting features of the visible light image data set and the infrared image data set after noise suppression respectively, and fusing the extracted visible light features and infrared features to obtain a fused image, comprises: Inputting the visible light image data set and the infrared image data set after noise suppression into an image fusion generation network respectively, wherein the image fusion generation network comprises an image feature extraction module and a feature fusion reconstruction module connected in sequence; Using the image feature extraction module to perform feature extraction on the visible light image data set and the infrared image data set after noise suppression, respectively, to obtain the visible light feature and the infrared feature; The feature fusion and reconstruction module is used to perform channel superposition and fusion on the visible light feature and the infrared feature to obtain the fused image.

4. The noise-suppressed multi-modal aerial small target detection method according to claim 3, characterized in that: The image feature extraction module includes a visible light feature extraction branch and an infrared feature extraction branch, wherein: The visible light feature extraction branch includes a convolution blocks connected in series, and the i-th convolution block in the visible light feature extraction branch is jump-connected with the next ai convolution blocks, i=1, 2, ..., a; The infrared feature extraction branch includes b convolution blocks connected in series, and in the visible light feature extraction branch, the jth convolution block is jump-connected with the next bj convolution blocks, where j=1, 2, ..., b.

5. The noise-suppressed multi-modal aerial small target detection method according to claim 4, characterized in that: The feature fusion reconstruction module includes a feature fusion network and c convolution blocks connected in series, the outputs of the visible light feature extraction branch and the infrared feature extraction branch are both connected to the input of the feature fusion network, and the output of the feature fusion network is connected to the first convolution block among the c convolution blocks connected in series.

6. The noise-suppressed multi-modal aerial small target detection method according to claim 1, characterized in that: The small target detection network includes a multi-dimensional feature extraction module, a multi-dimensional feature fusion module and a result prediction module, wherein: The multidimensional feature extraction module is used to extract the multidimensional features of the fused image data set; The multidimensional feature fusion module is used to fuse the multidimensional features to generate a multidimensional prediction feature map; The result prediction module is used to separately process and then fuse the multi-dimensional prediction feature graph to obtain a prediction result of a small and weak target in the air.

7. The noise-suppressed multi-modal aerial small target detection method according to claim 6, characterized in that: The multidimensional feature extraction module includes d Dark blocks connected in series, wherein the outputs of the d-th Dark block, the d-1-th Dark block and the d-2-th Dark block are all connected to the multidimensional feature fusion module; The d-2th Dark block, the d-1th Dark block and the dth Dark block output the one-dimensional features, two-dimensional features and three-dimensional features related to the weak targets in the air respectively.

8. The noise-suppressed multi-modal aerial small target detection method according to claim 7, characterized in that: The multi-dimensional feature fusion module includes a one-dimensional feature fusion layer, a two-dimensional feature fusion layer and a three-dimensional feature fusion layer, wherein: The three-dimensional feature fusion layer includes a residual convolution block Lateral_conv0 and a deconvolution unsample_1 connected in sequence, the three-dimensional feature output of the d-th Dark block is used as the input of the residual convolution block Lateral_conv0, and the output of the deconvolution Unsample_1 and the two-dimensional feature output of the d-1-th Dark block are both used as the input of the two-dimensional feature fusion layer; The two-dimensional feature fusion layer includes a fusion operation Cat_1, a residual convolution block C3_p4, Reduce_conv1 and a deconvolution Unsample_2 connected in sequence. The output of the deconvolution Unsample_1 and the two-dimensional feature output by the d-1th Dark block are fused through the fusion operation Cat_1 to obtain an enhanced two-dimensional feature. The enhanced two-dimensional feature is used as the input of C3_p4. The output of the deconvolution Unsample_2 and the one-dimensional feature output by the d-2th Dark block are used as the input of the one-dimensional feature fusion layer. The one-dimensional feature fusion layer includes a fusion operation Cat_2 and a residual convolution block C3_p3 connected in sequence, the output of the deconvolution Unsample_2 is fused with the one-dimensional feature output by the d-2th Dark block through the fusion operation Cat_2 to obtain an enhanced one-dimensional feature, and the enhanced one-dimensional feature is outputted as a one-dimensional prediction feature map after the residual convolution block C3_p3; The output of the residual convolution block C3_p3 is connected to the convolution Bu_conv2 and the result prediction module respectively, and the output of Reduce_conv1 and the output of the convolution Bu_conv2 are fused through the fusion operation Cat_3 and then output a two-dimensional prediction feature map through the residual convolution block C3_n3; The output of the residual convolution block C3_n3 is connected to the convolution Bu_conv1 and the result prediction module respectively, the output of the convolution Bu_conv1 is connected to the residual convolution block C3_n4 through the fusion operation Cat_4, and the residual convolution block C3_n4 outputs a three-dimensional prediction feature map to the result prediction module.

9. The noise-suppressed multi-modal aerial small target detection method according to claim 3, characterized in that: Before extracting features from the visible light image data set and the infrared image data set after noise suppression, respectively, and fusing the extracted visible light features and infrared features to obtain a fused image data set, the method further includes: The image fusion generation network is pre-trained to obtain the trained image fusion generation network for generating a fused image data set, wherein the loss function LOSS used in the training is: LOSS=λL SSIM +L tv Where λ is the equilibrium parameter, L SSIM represents the image structure similarity loss function, L tv Represents the image noise loss function.

10. The noise-suppressed multi-modal aerial small target detection method according to claim 9, characterized in that: The loss function L SSIM The formula is: In the formula, I A represents the visible light image, I B Indicates infrared image, I F represents the fused image, W represents the sliding window, N represents the total number of sliding windows, Score(I A ,I B ,I F |W) means retaining the information of the visible light image or infrared image that has a higher similarity with the fused image in window W; The loss function L tv The formula is: L tv =∑ i,j ||R(i,j+1)-R(i,j)||2+||R(i+1,j)-R(i,j)||2 Where R represents the difference between the visible light image and the fused image, R(i,j+1) represents the difference between the pixel value of the i-th row and j+1 column of the visible light image and the pixel value of the i-th row and j+1 column of the fused image, R(i,j) represents the difference between the pixel value of the i-th row and j column of the visible light image and the pixel value of the i-th row and j column of the fused image, R(i+1,j) represents the difference between the pixel value of the i+1-th row and j column of the visible light image and the pixel value of the i+1-th row and j column of the fused image, and || ||2 represents the Euclidean distance.

Citation Information

Patent Citations

  • Target identification and tracking method, device and system on complex battlefield

    CN116310900A

  • Infrared image denoising and small target detection method used in severe weather

    CN118154886A