Double-dynamic cross-modal interaction remote sensing image target detection method

Abstract

The invention relates to a double-dynamic cross-modal interaction remote sensing image target detection method, which comprises the following steps of: constructing a double-flow feature extraction network, and extracting features of a visible light image and an infrared SAR (Synthetic Aperture Radar) image; a double-branch dynamic cross-modal interaction module is constructed, and cross-modal interaction of features between two modals of the visible light image and the infrared SAR image is completed; a cross-modal dynamic attention network is constructed on one branch of the double-branch dynamic cross-modal interaction module, and a cross-modal dynamic filtering network is constructed on the other branch of the double-branch dynamic cross-modal interaction module; constructing a double-dynamic cross-modal interaction remote sensing image target detection network and completing training; and performing remote sensing image target detection by using the double-dynamic cross-modal interaction remote sensing image target detection network. According to the method, through a double-dynamic cross-modal interaction mode, the modal heterogeneity and redundancy are relieved at the same time, so that complementary features are better extracted and aggregated, and the detection performance is greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing images and artificial intelligence technology, and in particular to a dual-dynamic cross-modal interactive remote sensing image target detection method. Background Art

[0002] Remote sensing image target detection aims to locate and identify targets from a given remote sensing image, and has been widely used in intelligent monitoring, urban planning, port management, and military reconnaissance. With the rapid development of artificial intelligence, optical remote sensing image target detection has made significant progress, because optical images can provide rich target edge, texture, and color information. However, this single-modal target detection still faces problems such as insufficient sensor information acquisition, such as poor lighting (nighttime) or diverse weather conditions (rain, dust, and fog).

[0003] The rapid development of multi-sensor satellites and drones in recent years has provided a large amount of data for remote sensing image target detection. Different types of sensors can provide rich complementary information for optical images. For example, thermal cameras can present clear object outlines even in low light conditions because of their ability to capture radiant heat. Synthetic aperture radar (SAR) is an active microwave remote sensing imaging radar that has the ability to work in all-day and all-weather conditions, and different types of ground objects present unique scattering characteristics. The same target presents inherent complementary properties under different sensors. Therefore, multimodal remote sensing image target detection can utilize modal complementary information to produce more accurate and reliable detection results. Although the current multimodal remote sensing image target detection methods have achieved certain results, the following problems still exist:

[0004] 1) Modality heterogeneity. Due to the significant differences in the imaging mechanisms of different sensors, multimodal remote sensing images exhibit obvious modality heterogeneity or modality inconsistency, which will lead to semantic and spatial misalignment problems. Specifically, there are inevitable misclassifications and mislocalizations of the same objects in different modalities, which leads to confusion in feature fusion.

[0005] 2) Modal redundancy. Modal redundancy means that multimodal data usually contains a large amount of modal-correlated noise. For example, in a poor light environment, optical images usually have noise. Infrared images, due to the lack of target texture details, tend to present an appearance similar to the surrounding environment and show thermal noise. SAR images have obvious scattering characteristics, speckle noise, and geometric distortion.

[0006] The aforementioned modal heterogeneity and redundancy severely hinder the complementary feature extraction of the two-stream network, resulting in suboptimal multimodal feature aggregation and poor detection performance. Summary of the invention

[0007] In order to solve the technical problems existing in the above-mentioned prior art, the purpose of the present invention is to provide a dual dynamic cross-modal interactive remote sensing image target detection method, which alleviates modal heterogeneity and redundancy through dual dynamic cross-modal interaction, thereby better extracting and aggregating complementary features, greatly improving the detection performance.

[0008] To achieve the above-mentioned object of the invention, the present invention provides a dual-dynamic cross-modal interactive remote sensing image target detection method comprising the following steps:

[0009] Step S1, constructing a dual-stream feature extraction network to extract features of visible light images and infrared SAR images;

[0010] Step S2: construct a dual-branch dynamic cross-modal interaction module to complete the cross-modal interaction of features between the two modalities of visible light image and infrared SAR image;

[0011] Step S3: constructing a cross-modal dynamic attention network on one branch of the dual-branch dynamic cross-modal interaction module, and constructing a cross-modal dynamic filtering network on the other branch;

[0012] Step S4, constructing a dual dynamic cross-modal interactive remote sensing image target detection network and completing training;

[0013] Step S5: Utilize the dual dynamic cross-modal interactive remote sensing image target detection network to perform remote sensing image target detection.

[0014] According to a technical solution of the present invention, the step S1 specifically includes:

[0015] Obtain target detection data of visible light images and infrared SAR images, and construct a dataset that matches visible light images and infrared SAR images;

[0016] Divide the data set into a training set and a test set according to a preset ratio;

[0017] The ResNet50 backbone network is used as the feature extraction network for visible light images and infrared SAR images, and the training set is input into the feature extraction network to form a multi-scale visible light-infrared SAR feature map, and output visible light-infrared SAR image feature pairs at multiple feature extraction levels.

[0018] According to a technical solution of the present invention, step S3 specifically includes:

[0019] A cross-modal dynamic attention network is built on one of the branches of the dual-branch dynamic cross-modal interaction module to calculate the similarity between each pixel of the current modality feature and the feature of the other modality and use it as a coefficient to re-weight the current modality feature to complete the association between multi-modal images.

[0020] A cross-modal dynamic filtering network is constructed on the other branch of the dual-branch dynamic cross-modal interaction module, and input-related spatially dynamically changing filters are jointly generated through multi-modal features.

[0021] According to a technical solution of the present invention, the step S4 specifically includes:

[0022] Constructing a dual-dynamic cross-modal interactive remote sensing image target detection network, and using the training set in step S1 to train the dual-dynamic cross-modal interactive remote sensing image target detection network;

[0023] Based on the stochastic gradient descent optimization algorithm, a learning rate decay strategy is set to update the parameters of the dual dynamic cross-modal interactive remote sensing image target detection network until convergence.

[0024] According to a technical solution of the present invention, the step S5 specifically includes:

[0025] The test set in step S1 is used to perform numerical verification on the trained dual-dynamic cross-modal interactive remote sensing image target detection network;

[0026] The verified dual-dynamic cross-modal interactive remote sensing image target detection network is used to perform remote sensing image target detection.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] The present invention proposes a dual-dynamic cross-modal interactive remote sensing image target detection method, which eliminates modal heterogeneity and enhances modal consistency through a cross-modal dynamic attention network. At the same time, it suppresses complex noise information and eliminates modal redundancy through a cross-modal dynamic filtering network, thereby facilitating the extraction and fusion of multi-modal complementary features, improving the application scenarios of multi-modal remote sensing image target detection, and improving detection performance.

[0029] By calculating the pixel similarity between multimodal features, the most useful information in another modality for extracting the current target feature is integrated into the current modality feature, which enhances the global correlation between multimodal features and contributes to better multimodal feature fusion.

[0030] By fully considering multimodal features and generating input-related spatially different filters at different spatial positions, the problem of static filters sliding evenly over the entire image and treating all pixels equally without considering information differences at different positions is overcome, which is conducive to preserving complementary information and suppressing complex noise information. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0032] Figure 1 A schematic diagram schematically shows a dual dynamic cross-modal interactive remote sensing image target detection network according to one embodiment of the present invention;

[0033] Figure 2 A schematic diagram of a dual dynamic cross-modal interaction module according to an embodiment of the present invention is schematically shown;

[0034] Figure 3 Schematically showing a cross-modal attention diagram according to one embodiment of the present invention;

[0035] Figure 4 A schematic diagram of cross-modal dynamic filtering according to an embodiment of the present invention is schematically shown;

[0036] Figure 5 The flowchart schematically shows a dual-dynamic cross-modal interactive remote sensing image target detection method according to one embodiment of the present invention. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] like Figures 1 to 5 As shown, a dual-dynamic cross-modal interactive remote sensing image target detection method of the present invention comprises the following steps:

[0039] Step S1: construct a dual-stream feature extraction network to extract features of visible light images and infrared SAR images, specifically including:

[0040] Obtain target detection data of visible light images and infrared SAR images, and construct a dataset that matches visible light images and infrared SAR images;

[0041] Divide the data set into a training set and a test set according to a preset ratio;

[0042] like Figure 1As shown in the figure, the ResNet50 backbone network is used as the feature extraction network for visible light images and infrared SAR images, and the training set is input into the feature extraction network to form a multi-scale visible light-infrared SAR feature map, and output visible light-infrared SAR image feature pairs at multiple feature extraction levels.

[0043] Take the DroneVehicle dataset as an example. The DroneVehicle dataset is a large-scale visible light-infrared pairing dataset based on drones, containing 28,439 image pairs and 953,087 vehicle instances. The dataset covers multiple scenes including urban roads, residential areas, parking lots, and other scenes from day to night. The image size is 512×640.

[0044] In the visible light-infrared remote sensing image target detection task, the DroneVehicle dataset is randomly divided into a training set and a test set in a ratio of 7:3. For each data pair in the dataset, such as Figure 1 As shown in the figure, the ResNet50 backbone network is used as the feature extraction network for visible light and infrared images, and the visible light and infrared feature maps of 5 feature extraction levels are output.

[0045] Step S2: construct a dual-branch dynamic cross-modal interaction module to complete the cross-modal interaction of features between the two modalities of visible light image and infrared SAR image;

[0046] like Figure 2 As shown in Figure 2, starting from the second feature extraction level, a dual dynamic cross-modal interaction module is used to perform cross-modal interaction on visible light and infrared image features, output multi-modal features with enhanced complementarity, and input them into the feature extraction network of the next level. Assume that the visible light and infrared image features are represented by and Represents, then the output feature and Obtained by the following formula:

[0047]

[0048]

[0049] X1′,Y1′=CMCA(X1,Y1)

[0050] X2′,Y2′=CMDF(X2,Y2)

[0051]

[0052]

[0053] in represents a standard convolution with a kernel size of 1×1, which is used to convert the number of channels from C to C. Split represents a split operation along the channel dimension, while Cat represents a concatenation operation along the channel dimension.

[0054] Specifically, given the multimodal feature maps X and Y, they are first evenly divided into two sub-feature maps along the channel dimension, represented as and Then X1 and Y1 are input into the Cross-Modal Cross Attention (CMCA) module to alleviate modality heterogeneity, while X2 and Y2 are input into the Cross-Modal Dynamic (CMDF) module to alleviate modality redundancy. Output features and Then concatenate along the channel dimension to generate a new multimodal feature map and A single-layer network is used before the splitting operation and after the concatenation operation, respectively, to effectively enhance the representation learning ability and collaboration ability between the CMCA module and the CMDF module.

[0055] Step S3: constructing a cross-modal dynamic attention network on one branch of the dual-branch dynamic cross-modal interaction module, and constructing a cross-modal dynamic filtering network on the other branch. Step S3 specifically includes:

[0056] A cross-modal dynamic attention network is built on one of the branches of the dual-branch dynamic cross-modal interaction module to calculate the similarity between each pixel of the current modality feature and the feature of the other modality and use it as a coefficient to re-weight the current modality feature to complete the association between multi-modal images.

[0057] Cross attention can integrate global information from other modalities and establish complementarity between multimodal features. Spatially reduced cross attention is used to reduce resource consumption during cross-modal interaction, solving the problem of high computational and memory costs and the original cross attention being unsuitable for pixel-level dense prediction tasks. Figure 3 As shown, input X1 and Y1 sub-feature maps, the output feature map can be obtained by the following formula:

[0058]

[0059] Among them, Reshape is the operation of reshaping the image features of shape C×H×W into a sequence of HW×C, and Reshape′ is the inverse operation of Reshape. SR represents the operation for reducing the spatial dimension of the input feature, which actually represents a depth-separable convolution with a kernel size equal to the stride. represents a depthwise convolution with a kernel size of s×s. Softmax and ReLU represent the functions of softmax and relu activation, respectively. It is also possible to perform multi-head cross attention operations from different representation subspaces, and d represents the number of channels in each attention head.

[0060] The present invention proposes a cross-modal dynamic attention network, which integrates the most useful information for extracting the current target feature in another modality into the current modality feature by calculating the pixel similarity between multimodal features, thereby enhancing the global correlation between multimodal features and facilitating better multimodal feature fusion.

[0061] A cross-modal dynamic filtering network is constructed on the other branch of the dual-branch dynamic cross-modal interaction module, and input-related spatially dynamically changing filters are jointly generated through multi-modal features.

[0062] like Figure 4 The cross-modal dynamic filter network shown in Figure 2 generates spatial variant filters at different positions from cross-modal interactions in an input-dependent manner. Inputting the X2 and Y2 sub-feature maps, the output feature map can be obtained by the following formula

[0063]

[0064] in, represents convolution operation, Avg represents spatial average pooling, and r represents channel compression rate. Taking the X2 feature as an example, the cross-modal dynamic filtering network first uses static filtering to filter the spatial invariant noise information of the entire feature, thereby generating a static feature Then Connect it with modality Y2 to obtain the cross-modal context feature T X .because can be regarded as a static representation of the input and reflects the contextual information between local neighbors, so cross-modal contextual interaction can be easily performed through concatenation operations. X The spatial filter generation module is decoupled into a spatial filter generation module and a channel filter generation module to reduce the number of parameters and limit the computational cost. and denote the generated spatial filter and channel filter respectively. Then the generated filter Slide on X2 to filter out spatial variable noise information and generate dynamic features Finally, along the channel dimension, and Concatenate them and fuse them together to obtain the purified feature X′2 through standard convolution.

[0065] The present invention proposes a cross-modal dynamic filtering network, which generates input-related spatially different filters at different spatial positions by fully considering multimodal features, overcomes the problem that the static filter slides evenly over the entire image, treats all pixels equally without considering the information differences at different positions, and is conducive to preserving complementary information and suppressing complex noise information.

[0066] Step S4: constructing a dual-dynamic cross-modal interactive remote sensing image target detection network and completing training, specifically including:

[0067] Constructing a dual-dynamic cross-modal interactive remote sensing image target detection network, and using the training set in step S1 to train the dual-dynamic cross-modal interactive remote sensing image target detection network;

[0068] Based on the stochastic gradient descent optimization algorithm, a learning rate decay strategy is set to update the parameters of the dual dynamic cross-modal interactive remote sensing image target detection network until convergence.

[0069] The dual-dynamic cross-modal interactive remote sensing image target detection network was trained on the training set of the DroneVehicle dataset. The ResNet50 pre-trained on the ImageNet dataset was used to initialize the two-stream backbone network to accelerate convergence. The stochastic gradient descent method was used for iterative training for 12 rounds, with a batch size of 16, an initial learning rate of 0.005, and a weight decay of 0.0005. The intersection-over-union ratio threshold between target boxes in the post-processing algorithm was set to 0.5 to filter out redundant boxes.

[0070] Step S5, performing remote sensing image target detection using the dual dynamic cross-modal interactive remote sensing image target detection network, specifically comprising:

[0071] The test set in step S1 is used to perform numerical verification on the trained dual-dynamic cross-modal interactive remote sensing image target detection network;

[0072] The verified dual-dynamic cross-modal interactive remote sensing image target detection network is used to perform remote sensing image target detection.

[0073] As shown in Table 1, the mean average precision mAP of the basic target detection network and the dual dynamic cross-modal interactive target detection network on the test set of the DroneVehicle dataset are compared. Compared with the basic detection method, the dual dynamic cross-modal interactive target detection network can achieve 7.5%, 6.9% and 4.9% performance improvements on the three indicators of mAP50, mAP75 and mAP, respectively, and finally achieved a mAP50 detection result of 77.8%. These improved results prove that the dual dynamic cross-modal interactive target detection network proposed in this application can dynamically eliminate modal heterogeneity and redundancy at the same time to enhance the extraction and fusion of multi-modal complementary features, and significantly improve the target detection performance of multiple remote sensing images.

[0074]

[0075]

[0076] Table 1

[0077] A dual dynamic cross-modal interactive remote sensing image target detection method of the present invention constructs a dual-stream feature extraction network to extract features of visible light images and infrared SAR images; constructs a dual-branch dynamic cross-modal interaction module to complete the cross-modal interaction of features between the two modalities of visible light images and infrared SAR images; constructs a cross-modal dynamic attention network on one branch of the dual-branch dynamic cross-modal interaction module, and constructs a cross-modal dynamic filtering network on the other branch; constructs a dual dynamic cross-modal interactive remote sensing image target detection network and completes training; and uses the dual dynamic cross-modal interactive remote sensing image target detection network to perform remote sensing image target detection. The dual dynamic cross-modal interactive remote sensing image target detection network proposed by the present invention is a unified, dynamic, and effective framework that can simultaneously eliminate modal heterogeneity and redundancy, thereby facilitating the extraction and fusion of multi-modal complementary features, improving the application scenarios of multi-modal remote sensing image target detection, and improving detection performance.

[0078] Parts of the present invention that are not described in detail belong to the well-known technology in the art.

[0079] It should be noted that although the embodiments of the present invention described above are illustrative, they are not intended to limit the present invention, and therefore the present invention is not limited to the above specific embodiments. Without departing from the principles of the present invention, any other embodiments obtained by those skilled in the art under the guidance of the present invention are deemed to be within the protection of the present invention.

Claims

1. A dual-dynamic cross-modal interactive remote sensing image target detection method, characterized in that: The following steps are involved: Step S1, constructing a dual-stream feature extraction network to extract features of visible light images and infrared SAR images; Step S2: construct a dual-branch dynamic cross-modal interaction module to complete the cross-modal interaction of features between the two modalities of visible light image and infrared SAR image; Step S3: constructing a cross-modal dynamic attention network on one branch of the dual-branch dynamic cross-modal interaction module, and constructing a cross-modal dynamic filtering network on the other branch; Step S4, constructing a dual dynamic cross-modal interactive remote sensing image target detection network and completing training; Step S5: Utilize the dual dynamic cross-modal interactive remote sensing image target detection network to perform remote sensing image target detection.

2. The dual dynamic cross-modal interactive remote sensing image target detection method according to claim 1, characterized in that: The step S1 specifically includes: Obtain target detection data of visible light images and infrared SAR images, and construct a dataset that matches visible light images and infrared SAR images; Divide the data set into a training set and a test set according to a preset ratio; The ResNet50 backbone network is used as the feature extraction network for visible light images and infrared SAR images, and the training set is input into the feature extraction network to form a multi-scale visible light-infrared SAR feature map, and output visible light-infrared SAR image feature pairs at multiple feature extraction levels.

3. The dual dynamic cross-modal interactive remote sensing image target detection method according to claim 1, characterized in that: The step S3 specifically includes: A cross-modal dynamic attention network is built on one of the branches of the dual-branch dynamic cross-modal interaction module to calculate the similarity between each pixel of the current modality feature and the feature of the other modality and use it as a coefficient to re-weight the current modality feature to complete the association between multi-modal images. A cross-modal dynamic filtering network is constructed on the other branch of the dual-branch dynamic cross-modal interaction module, and input-related spatially dynamically changing filters are jointly generated through multi-modal features.

4. The dual dynamic cross-modal interactive remote sensing image target detection method according to claim 1, characterized in that: The step S4 specifically includes: Constructing a dual-dynamic cross-modal interactive remote sensing image target detection network, and using the training set in step S1 to train the dual-dynamic cross-modal interactive remote sensing image target detection network; Based on the stochastic gradient descent optimization algorithm, a learning rate decay strategy is set to update the parameters of the dual dynamic cross-modal interactive remote sensing image target detection network until convergence.

5. The dual dynamic cross-modal interactive remote sensing image target detection method according to claim 2, characterized in that: The step S5 specifically includes: The test set in step S1 is used to perform numerical verification on the trained dual-dynamic cross-modal interactive remote sensing image target detection network; The verified dual-dynamic cross-modal interactive remote sensing image target detection network is used to perform remote sensing image target detection.

Images