A target detection method based on a complementary perception feature fusion mechanism

Abstract

The application discloses a target detection method based on a complementary perception feature fusion mechanism, proposes a complementary perception feature fusion detection network, extracts multi-level semantic features of optical and SAR modal remote sensing images in a spatial dimension respectively, quantitatively analyzes complementary differences of the two modal features on an aircraft target representation, adaptively weights and fuses complementary features of the two modalities on this basis, realizes accurate detection of an aircraft in a complex scene, proposes a differential feature space perception complementary module, can sensitively capture deep feature differences of optical and SAR modalities in the spatial dimension, enhances rich context information related to the aircraft in the two modal feature maps, proposes a gate generation weighted fusion module, calculates a weight matrix consistent with the input feature dimension, measures the contribution of optical and SAR input features to the detection task, and reweights according to the same to effectively fuse target information in the two modalities, so as to generate an accurate final detection result.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of target detection and identification, and relates to a remote sensing image target detection method, in particular to a target detection method based on a complementary perception feature fusion mechanism. BACKGROUND

[0002] In recent years, aircraft target detection technology based on remote sensing images has attracted more and more attention and plays an increasingly important role in civil fields. For example, aircraft target detection can provide timely and accurate information support for airport traffic management and emergency rescue response applications. With the rapid development of computer vision technology, target detection methods based on deep neural networks (such as Faster R-CNN, YOLO and vision transformer) are widely used in aircraft target detection tasks due to their advantages in adaptively extracting high-level semantic features of input images. Visible light images and synthetic aperture radar (SAR) images are two main data sources for aircraft target detection tasks. Optical remote sensing images have the characteristics of rich texture and color information, intuitive visual effect, and low acquisition cost, but are easily affected by environmental factors such as light changes and shadow interference. SAR imaging has the characteristics of all-weather and all-day use, and is not affected by natural environmental factors such as clouds, rain, snow and smoke, but its images have the limitations of significant noise interference and lack of color and detail texture.

[0003] In recent years, multi-modal data fusion has become a research hotspot in the field of aircraft target detection. This method can overcome the limitations of single modal data and significantly improve the environmental adaptability of aircraft detection networks. Considering the high complementarity of optical images and SAR images in information dimension, fusing optical and SAR image features for joint detection is a feasible way to improve detection accuracy and algorithm robustness. In fact, researchers have proposed various aircraft detection methods based on multi-modal data fusion applications and have made certain achievements in improving detection accuracy and overcoming environmental interference. Among them, feature-level fusion is a widely used and promising processing strategy for building multi-modal target detection models. It can effectively fuse the information in the feature dimension from complementary modalities to obtain excellent target detection results.

[0004] However, current multi-modal detection methods based on feature-level fusion still face the following problems that need to be solved: (1) In the feature extraction stage, existing methods have difficulty in comprehensively considering and effectively utilizing multi-level features such as structure, texture and semantics of complementary modalities and their spatial context information, resulting in insufficient complementary feature extraction; (2) When performing multi-modal feature-level fusion, existing methods lack fine measurement and efficient fusion of the contribution degree of different modal features, limiting the detection performance of the model under complex environmental interference. SUMMARY

[0005] In order to solve the problems of insufficient multi-modal complementary feature extraction and lack of fine measurement of cross-modal feature contribution in the multi-modal aircraft target fusion detection task, the application provides a target detection method based on a complementary perception feature fusion mechanism. The method proposes a complementary perception feature fusion detection network, extracts multi-level semantic features of optical and SAR modal remote sensing images in the spatial dimension, quantitatively analyzes the complementarity difference of the two modal features in the aircraft target representation, and adaptively weights and fuses the complementary features of the two modalities to realize accurate detection of aircraft in complex scenes.

[0006] The purpose of the application is realized by the following technical solutions:

[0007] A target detection method based on a complementary perception feature fusion mechanism comprises the following steps:

[0008] Step one: build a complementary perception feature fusion detection network (Complementarity-aware Feature Fusion Detection Network, CFFDNet), input optical and SAR images of the same scene, realize strict alignment of optical and SAR images in the spatial dimension, extract optical, SAR image features and optical-SAR differential features by using a differential feature spatial perception complementary module (Differential Feature Spatial-aware Complementary, DFSC), build a cascade large-kernel spatial perception (Cascade Large-kernel Spatial Perception, CLSP) structure to process the differential features to generate a corresponding differential spatial attention map, and combine a classic residual structure to extract rich context information and enhance the perception of aircraft target structure by the two modal feature maps;

[0009] Step two: input the extracted optical and SAR features, use a gate-generated weighted fusion module (Gate-generated Weighted Fusion, GWF) to measure the influence of the two modal features on the final aircraft detection result, and re-weight and fuse the effective information therein;

[0010] Step three: pass the weighted and fused features to the neck of the CFFDNet, use a multi-scale feature pyramid structure to further integrate and enhance the feature information at different levels, then input the fused multi-scale features into the detection head of the CFFDNet, and realize fast and accurate detection of aircraft targets in optical and SAR images.

[0011] Compared with the prior art, the present application has the following advantages:

[0012] 1. A differential feature space perception complementary module is proposed, which can sensitively capture the deep feature differences of optical and SAR modalities in the spatial dimension, and enhance the rich context information related to the aircraft in the feature maps of the two modalities.

[0013] 2. A gate generation weighted fusion module is proposed, which calculates a weight matrix consistent with the input feature dimension to measure the contribution of optical and SAR input features to the detection task, and reweights accordingly to effectively fuse the target information under the two modalities, thereby generating an accurate final detection result. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 is a schematic diagram of the architecture of CFFDNet;

[0015] Figure 2 is a specific structure diagram of the main modules of CFFDNet;

[0016] Figure 3 is a comparison diagram of single-modality and multi-modality feature fusion detection results. DETAILED DESCRIPTION

[0017] The technical solutions of the present application will be further described below in conjunction with the drawings, but are not limited thereto, and any modification or equivalent replacement of the technical solutions of the present application without departing from the spirit and scope of the present application shall be covered in the protection scope of the present application.

[0018] The present application provides a target detection method based on a complementary perception feature fusion mechanism, which designs a CFFDNet to capture the deep feature differences of optical and SAR modal remote sensing images in the spatial dimension. First, the DFSC module is used to extract the differential features of optical and SAR images of the same scene, and the learning of aircraft target features in the feature maps of the two modalities is enhanced. Then, the GWF module is used to reweight and fuse the effective information in the two modalities, realize the efficient use of the complementary features between the optical and SAR modalities, and improve the detection accuracy of the aircraft target. The overall architecture is shown in Figure 1 The specific steps include the following:

[0019] Step 1: Construct CFFDNet, taking optical and SAR images of the same scene as input to achieve strict spatial alignment between the optical and SAR images. Utilize the DFSC module (mainly composed of CLSP units and classical residual structure units) to extract features from optical and SAR images, as well as optical-SAR differential features. Simultaneously, construct a CLSP structure to process the differential features to generate corresponding differential spatial attention maps. Combined with the classical residual structure, this enhances the perception of aircraft target structures by extracting rich contextual information from both modal feature maps. The specific steps are as follows:

[0020] Step 11: Construct CFFDNet using a ResNet-50-based dual-branch backbone, neck, and head network, while simultaneously introducing DFSC and GWF modules to extract the optical image X. OPT and SAR image X SAR The differential features of optical and SAR images are weighted and fused to achieve the integration and utilization of complementary features between optical and SAR images. See the detailed structure diagram of the main modules of CFFDNet. Figure 2 As shown.

[0021] Steps 1 and 2: Utilize the dual-branch backbone network of CFFDNet to process the optical image X of the same scene. OPT and SAR image X SAR Multi-level deep feature extraction is performed, and these features are simultaneously input into the DFSC module. For the input optical features and SAR features at the same level, the differential feature X is calculated. D,OPT and X D,SAR Thus inhibiting X OPT and X SAR Shared information in X amplifies the difference between the two. D,OPT and X D,SAR The calculation method is shown in the following formula:

[0022] X D,OPT =X OPT -X SAR

[0023] X D,SAR =X SAR -X OPT

[0024] Step 13: Process the obtained differential features using two cascaded depthwise convolutions to obtain rich semantic and contextual information representations of the aircraft target X within different receptive fields. D,i The calculation process is shown in the following formula:

[0025] X D,i =W i dw (X D), i = 1, 2

[0026] In the formula, X D X represents D,OPT or X D,SAR W i dw (·) denotes a cascaded depthwise convolution operation, W1 dw (·) represents the first depthwise convolution, W2 dw (·) represents the second depthwise convolution, X D,1 X represents the output of the first depthwise convolution layer. D,2 This represents the output calculated after two layers of depthwise convolution.

[0027] Then, two 1×1 convolution kernels were used to process X respectively. D,i Convolution computes and generates differential spatial attention maps V with different receptive fields. D,1 and V D,2 V D,i The calculation method is shown in the following formula;

[0028] V D,i =W 1×1 (X D,i ), i = 1, 2

[0029] In the formula, W 1×1 (·) represents a 1×1 convolution operation.

[0030] Step 14: Apply the obtained difference space attention map V D,1 and V D,2 As input, the two are concatenated along the channel dimension, and then averaged and max-pooled operations are performed to obtain the feature map M. avg and M max M avg and M max The calculation process is as follows:

[0031] M avg =CAP(concat(V) D,1 V D,2 ))

[0032] M max =CMP(concat(V D,1 V D,2 ))

[0033] In the formula, concat(·) represents the fully connected operation, and CAP(·) and CMP(·) represent the average pooling and max pooling operations on the channel dimension, respectively.

[0034] For the obtained M avg and M maxThe two are concatenated along the channel dimension, and a large kernel convolutional layer is applied to realize the information exchange between them. The final spatial mask M is generated by the sigmoid function. This process can be represented by the following formula.

[0035] M = Sigmoid(W) 1×1 (concat(M avg M max )))

[0036] In the formula, Sigmoid(·) represents the Sigmoid activation function.

[0037] The two-level difference spatial attention map V D,i Each value is spatially multiplied and weighted with a spatial selection mask, and then fused using a 1×1 convolution kernel to generate the final differential spatially weighted attention map V. D V D The calculation method is shown in the following formula:

[0038] V D =W 1×1 (M1⊙V D,1 +M2⊙V D,2 )

[0039] In the formula, ⊙ represents weighted multiplication across spaces, and V D V represents D,OPT or V D,SAR .

[0040] Step 15: Utilize the obtained difference space attention map V D,OPT and V D,SAR Compare with the corresponding original input optical image X OPT and SAR image X SAR By performing spatial multiplication and weighting, we obtain the spatially weighted differential features that highlight local characteristics. and This process can be represented as follows:

[0041]

[0042] Step 16: Obtain and As supplementary information, add them to X respectively. SAR and X OPT In order to generate more information features, classical residual structures are then added to the optical and SAR feature branches to improve the stability of the fusion module structure, resulting in a feature-enhanced optical image. and SAR images The specific process can be represented as follows:

[0043]

[0044] In the formula, Res(·) represents the classical residual structure.

[0045] Step 2: Using the extracted optical and SAR features as input, the GWF module (mainly composed of convolution, sigmoid, and weighted summation operations) is used to measure the impact of the two modalities on the final aircraft detection result, and the effective information is re-weighted and combined. The specific steps are as follows:

[0046] Step 21: Apply optical features and SAR features As input, the data is concatenated along the channel dimension, and the resulting combined features are sequentially fed into a 1×1 convolutional kernel and a sigmoid function to obtain the weight matrix. The calculation process is represented by the following formula:

[0047]

[0048] Step 22: Use the obtained G and 1-G as the weight matrices for the optical mode and SAR mode, thereby obtaining the final gated fusion feature X. GWF The feature fusion of optical and SAR images is completed, and the fused feature map X is obtained. GWF The calculation process is shown in the following formula:

[0049]

[0050] Step 3: The weighted fused features are passed to the neck of CFFDNet, where a multi-scale feature pyramid structure is used to further integrate and enhance feature information from different levels. Then, the fused multi-scale features are input into the CFFDNet head. The head contains a classification subnetwork and a regression subnetwork, used to predict the aircraft target's category and accurate location information, respectively. The head employs a lightweight design to improve computational efficiency while ensuring detection accuracy, achieving fast and accurate detection of aircraft targets in optical and SAR images. The specific steps are as follows:

[0051] Step 31: Combine the fused features X obtained in Step 2 GWF A series of convolution and pooling operations are performed to generate feature maps {C1, C2, C3, C4, C5} of different scales, where C i The scale is the same as the original image. The calculation process is shown in the following formula:

[0052] C2 = MaxPool(Conv) 3×3 (X GWF ))

[0053] C i+1=MaxPool(Conv 3×3 (C i ), i∈{2,3,4}

[0054] In the formula, Conv n×n (·) represents an n×n convolution operation, and MaxPool(·) represents a max pooling operation.

[0055] Step 32: Using a top-down approach, the high-level semantic features are upsampled using bilinear interpolation and added to the principal elements of the shallow high-resolution features to generate a multi-scale feature pyramid {P1, P2, P3, P4, P5}. The calculation process is as follows:

[0056] P5 = Conv 1×1 (C5)

[0057] P i =Conv 3×3 (C i )+Upsample(P i+1 ), i∈{4,3,2}

[0058] In the formula, Upsample(·) represents the upsampling operation.

[0059] Each layer of P obtained i The feature map P is obtained by applying the spatial attention module. i To enhance the saliency of the target region, the calculation process can be expressed by the following formula:

[0060]

[0061] in, This indicates element-wise multiplication.

[0062] Step 33: For each layer P obtained i Apply a lightweight classification network to output the category score cls for each anchor point. i At the same time, for each layer P i 'Apply a regression network to predict the offset of the target bounding box (bbox)' i cls i and bbox i The calculation process is as follows:

[0063] cls i =Softmax(Conv 3×3 (ReLU(Conv 1×1 (P i '))))

[0064] bbox i =Conv 3×3(ReLU(Conv 1×1 (P i ')))

[0065] In the formula, ReLU(·) represents the activation function and Softmax(·) represents the normalization function.

[0066] Steps 3 and 4: Summarize the classification and regression results from different levels, and apply non-maximum suppression (NMS) to remove redundant detection boxes to obtain the final detection results.

[0067]

[0068] In the formula, NMS(·) represents the nonmaximum suppression operation, which removes duplicate results.

[0069] Step 4: Validate the algorithm's performance using a publicly available, high-quality dataset. The specific steps are as follows:

[0070] This invention uses the publicly available MAR20 dataset for experiments, dividing it into training and test sets in a 7:3 ratio. All unimodal baseline methods and the proposed method are implemented using MMDetection. For the comparative multimodal detection model, experiments are conducted using open-source code and laboratory replication. During model training, the batch size is set to 4, the training phase lasts 24 epochs, and the SGD optimizer is used to update model parameters. The initial learning rate, momentum, and weight decay are 0.005, 0.9, and 0.0001, respectively.

[0071] Figure 3 The detection performance differences between single-modal and multi-modal feature fusion were compared. Figure 3 (a) shows the detection results for a single optical mode. Figure 3 (b) shows the SAR single-mode detection results. Figure 3 (c) is the fusion detection result of direct summation of optical and SAR modes. Figure 3 (d) is the feature-level fusion detection result proposed in this invention. Figure 3 (e) represents the ground truth of the dataset images. It should be noted that the green, yellow, and red boxes in the figure represent correct detection, missed detection, and false alarm, respectively. Experimental results show that the remote sensing image target detection method based on complementary sensing feature fusion mechanism proposed in this invention can effectively fuse complementary information from optical and SAR modes, and effectively improve the performance of aircraft target detection in complex remote sensing scenarios.

Claims

1. A target detection method based on a complementary perception feature fusion mechanism, characterized in that The method comprises the following steps: Step one: build a complementary perception feature fusion detection network by a double-branch backbone network based on ResNet-50, a neck and a detection head, and meanwhile introduce a differential feature space perception complementary module and a gate generation weighted fusion module to extract the differential features of optical images and SAR images , and perform weighted fusion to realize the complementary feature integration and utilization of optical images and SAR images; Dual-branch backbone network of complementary perception feature fusion detection network for optical image of same scene and SAR image Multi-level deep feature extraction is performed, and the features are simultaneously input into a differential feature space perception complementary module; the differential features are calculated for the input optical features and SAR features of the same level and ; two cascaded deep convolutional processes are used to obtain the obtained differential features to obtain rich aircraft target semantic and context information representation in different receptive fields , and then two 1×1 convolution kernels are used to calculate convolution and generate differential space attention maps with different receptive fields and ; The obtained difference spatial attention map And As input, after concatenating the two in the channel dimension, the average and maximum pooling operations are performed to obtain the feature map And , the obtained And Concatenate along the channel dimension, while applying a large kernel convolution layer to exchange information between the two, and generate the final spatial mask M through the sigmoid function, and the difference spatial attention map of the two levels Respectively multiply the spatial selection mask, and fuse through the 1×1 convolution kernel to generate the final difference spatial weighted attention map ; The obtained difference spatial attention map is used for weighting and respectively, and the optical image and the SAR image corresponding to the original input are multiplied spatially to obtain the difference feature highlighting the local features after spatial weighting and ; the obtained and are added as supplementary information to and respectively to generate more information features, and then a residual structure is added to the optical feature and the SAR feature branch to improve the stability of the fusion module structure, thereby obtaining the feature-enhanced optical image and the SAR image ; Step two: the optical feature and SAR feature As input, the generated combined features are sequentially fed into a 1x1 convolution kernel and a sigmoid function along the channel dimension to obtain a weight matrix ; the obtained G and 1-G are taken as the weight matrices of the optical and SAR modalities, thereby obtaining the final gated fusion feature , completing the feature fusion of the optical and SAR images to obtain the fused feature map ; Step three: the weighted fused features are transmitted to the neck of the complementary perception feature fusion detection network, multi-scale feature pyramid structures are used to further integrate and enhance the feature information of different levels, then the fused multi-scale features are input into the detection head of the complementary perception feature fusion detection network, and rapid and accurate detection of the aircraft target in the optical and SAR images is realized. 2.The target detection method based on the complementary perception feature fusion mechanism according to claim 1, characterized in that The and The calculation method is shown by the following formula: The calculation process is shown in the following formula: wherein represents or , represents a cascaded deep convolution operation, is a first layer deep convolution, is a second layer deep convolution, represents an output of the first layer deep convolution, represents an output of the two-layer deep convolution calculation; The calculation method is shown in the following formula; In the formula, denotes a 1 x 1 convolution operation; and The calculation process is as follows: wherein denotes a full connection operation, and denote average and max pooling operations in the channel dimension, respectively; M is represented by the following formula: In the formula, denotes a sigmoid activation function; The calculation is as follows: wherein denotes a spatially multiplicative weighting, denotes or ; and is represented by: In the formula, represents the residual structure. 3.The target detection method based on the complementary perception feature fusion mechanism according to claim 1, characterized in that The calculation process of the G is represented by the following formula: The calculation process is shown in the following formula: 。 4.The target detection method based on the complementary perception feature fusion mechanism according to claim 1, characterized in that The specific steps of the step three are as follows: Step three one: the fusion features obtained in step two are input into a convolutional neural network to obtain a feature map of a first size A series of convolution and pooling operations are performed to generate feature maps of different scales ; Step three two: adopt the top-down path, up-sample the high-level semantic features through bilinear interpolation, and add them to the main elements of the shallow high-resolution features to generate a multi-scale feature pyramid Each layer obtained Apply the spatial attention module to obtain the feature map , and enhance the saliency of the target region; Step three: apply a lightweight classification network to each feature map to output the class score of each anchor Meanwhile, for each feature map apply a regression network to predict the offset of the bounding box ; Step three four: aggregate the classification and regression results of different levels, and apply non-maximum suppression to remove redundant detection boxes to obtain the final detection result .

5. The target detection method based on the complementary perception feature fusion mechanism according to claim 4, characterized in that The The calculation process is as follows: wherein denotes an n x n convolution operation, denotes a max-pooling operation; The calculation process is as follows: In the formula, denotes an up-sampling operation; The calculation process is represented by the following equation: wherein denotes element-wise multiplication; and The calculation process is as follows: wherein denotes an activation function, denotes a normalization function; The calculation process is as follows: In the formula, denotes a non-maximum suppression operation, which removes duplicate results.

Images