Visible light / SAR-based cross-modal feature fusion target detection method

Abstract

The invention discloses a cross-modal feature fusion target detection method based on visible light / SAR, and solves the problem of low target detection precision of a visible light remote sensing image in a complex environment (cloud, rain and fog) in the prior art, and the method comprises the steps: firstly, obtaining image data, and carrying out the preprocessing operation, and obtaining a visible light-SAR image; secondly, single-mode feature extraction is carried out on the visible light image and the SAR image; thirdly, in a feature fusion stage, performing cross-modal feature fusion based on a cross attention mechanism; then, inputting the fused feature map into a multi-scale detection head, and carrying out target classification and frame regression; the accuracy of visible light remote sensing image target detection is improved.

Description

Technical Field

[0001] The present invention relates to the field of synthetic aperture radar imaging technology, and in particular to a cross-modal feature fusion target detection method based on visible light / SAR. Background Art

[0002] With the development and innovation of remote sensing technology in recent years, a large number of optical remote sensing satellites and SAR satellites have been successfully launched and operated around the world. The acquisition of remote sensing images has become more efficient and faster, and how to achieve accurate remote sensing data interpretation in practical application fields is particularly important.

[0003] At present, most detection methods based on remote sensing information use visible light images, and a small number use SAR images. Although optical images are a common remote sensing image, they are easily affected by external conditions such as clouds, rain, and fog, resulting in unstable image quality and insufficient information expression, making it difficult to effectively support high-precision target detection in complex environments. As an active microwave earth observation system, SAR sensors have all-day and all-weather detection capabilities and strong penetration capabilities. They can obtain high-resolution radar images similar to optical photography under complex conditions. Therefore, SAR images can effectively make up for the monitoring blind spots of optical remote sensing images at night and under complex meteorological conditions, and overcome the limitations of optical images in certain situations. However, SAR images have always had the problems of severe speckle noise and poor visual effects. In contrast, visible light remote sensing images contain rich details and texture information, which can make up for the poor visualization of SAR images. Therefore, if an effective fusion strategy can be designed to make up for the shortcomings of SAR images in visibility and visible light images in quality, it will be possible to more comprehensively and accurately reflect the multi-dimensional characteristics of the ground objects, thereby significantly improving the interpretation accuracy and application effect of remote sensing data.

[0004] Nowadays, most of the research on dual-modal fusion detection is concentrated in the visible light and infrared fields. For example, Zhang Yanning et al. realized multi-scale feature extraction of dual-modal image pairs based on the target detection framework of yolov5, and constructed a dynamic network multi-source feature fusion module to improve the robustness of the model; He Lin et al. constructed a heterogeneous remote sensing target detection framework based on iterative fusion, which coupled the iterative feature extraction module with the feature enhancement module of scale transformation, effectively alleviating the defect of insufficient expression of scene information of single-source image; in the fusion detection of visible light and SAR images, Yang Shuyuan et al. introduced the Faster R-CNN network, performed multi-source fusion in the attention mechanism module, and improved the detection ability of occluded targets; Li Xue et al. obtained detection results by fusing visible light image feature maps and SAR image feature maps of multiple scales, thereby improving the accuracy of target detection.

[0005] Existing feature fusion networks usually only use simple superposition or connection operations to process bimodal features. Especially for the fusion task of visible light images and SAR images, existing methods fail to fully couple the complementary information of the two, which leads to limited fusion detection effect. Summary of the invention

[0006] The present invention solves the problem of low target detection accuracy of visible light remote sensing images in complex environments (clouds, rain, and fog) in the prior art by providing a cross-modal feature fusion target detection method based on visible light / SAR, thereby improving the accuracy of target detection in visible light remote sensing images.

[0007] The present invention provides a cross-modal feature fusion target detection method based on visible light / SAR, the method comprising:

[0008] Obtain a visible light image and a SAR image corresponding to the original image, and preprocess the visible light image and the SAR image to obtain an initial visible light image I opt and the initial SAR image I SAR ;

[0009] The initial visible light image I opt and the initial SAR image I SAR Perform feature extraction to obtain the initial visible light feature F o and the initial SAR feature F S ;

[0010] The initial visible light feature F is analyzed layer by layer using the YOLO v5 network. o Perform single-mode feature extraction to obtain a visible light feature set; use the SAR feature single-mode feature extraction method to extract the initial SAR feature F layer by layer S Perform single-mode feature extraction to obtain the SAR feature set;

[0011] Based on the Transformer cross attention mechanism, cross-modal feature enhancement is performed on the visible light feature set and the SAR feature set to obtain a visible light enhanced feature set and a SAR enhanced feature set;

[0012] Fusing the visible light enhancement features and the SAR enhancement features of the corresponding layers in the visible light enhancement feature set and the SAR enhancement feature set respectively to obtain the fused features corresponding to each layer;

[0013] The fused features corresponding to each layer are input into the multi-scale detection head for target classification and bounding box regression.

[0014] In a possible implementation, the visible light image and the SAR image are preprocessed to obtain an initial visible light image I optand the initial SAR image I SAR ,include:

[0015] The visible light image and the SAR image are cropped using Envi software to obtain an initial visible light image I of the same size. opt and the initial SAR image I SAR .

[0016] In a possible implementation, the initial visible light feature F o Perform single-modal feature extraction, including:

[0017] Set the number of extraction layers to i, and use the single-mode feature extraction function to extract the initial visible light feature F layer by layer. o Perform single-mode feature extraction; the input of the i-th layer is: visible light feature Output: Visible light features i≥2; and the input of the first layer is the initial visible light feature F o , the output is the first visible light feature

[0018] In a possible implementation, the unimodal feature extraction function is expressed as:

[0019]

[0020] in, represents the visible light feature of the i-th layer input; Ψ backbone (·) represents the feature extraction function of YOLO v5; θ opt Represents the network parameters of YOLO v5; Represents the visible light features output by the i-th layer; i represents the number of extraction layers.

[0021] In a possible implementation, the SAR feature single-modal feature extraction method includes:

[0022] The SAR feature features of the current layer input are processed by three standard convolution processing units, batch normalization processing and activation function processing in sequence to obtain the preliminary extracted SAR features.

[0023] The preliminary extraction of SAR features Perform global average pooling operation to obtain global SAR features

[0024] The global SAR features are analyzed by using two convolutional layers and activation function layers connected in sequence. Perform channel feature extraction to obtain global SAR channel features;

[0025] The global SAR features are analyzed by sequentially connecting the depth separation convolution layer and the point-by-point convolution layer. Perform spatial feature extraction to obtain global SAR spatial features;

[0026] The global SAR channel feature and the global SAR spatial feature are fused, and the fused feature is weighted to obtain an enhanced global SAR feature.

[0027] The input SAR features are activated by using two activation function layers connected in sequence, and the activated SAR features are combined with the enhanced global SAR features. The spliced ​​features are then preprocessed using the activation function layer and downsampling layer connected in sequence. The SAR features output by the current layer Take the SAR features output by all layers Constitute the SAR feature set.

[0028] In a possible implementation, the cross-modal feature enhancement of the visible light feature set and the SAR feature set is performed based on the Transformer cross attention mechanism to obtain a visible light enhanced feature set and a SAR enhanced feature set, including:

[0029] Compressing each visible light feature in the visible light feature set by a hybrid pooling operation to obtain a compressed visible light enhanced feature set; and compressing each SAR feature in the SAR feature set by a hybrid pooling operation to obtain a compressed SAR enhanced feature set;

[0030] Flattening the compressed visible light enhancement features and the compressed SAR enhancement features in the compressed visible light enhancement feature set and the compressed SAR enhancement feature set respectively to obtain a flattened visible light enhancement feature set and a flattened SAR enhancement feature set;

[0031] The flattened SAR enhancement feature set and a flattened visible light enhancement feature in the flattened visible light enhancement feature set The features are sequentially input into the cross-modal feature enhancement module for feature coupling to obtain the enhanced SAR modal features of the corresponding layer.

[0032] The flattened visible light enhancement feature in the flattened visible light enhancement feature set and flattening the SAR enhancement features in the flattened SAR enhancement feature set The features are sequentially input into the cross-modal feature enhancement module for feature coupling to obtain the enhanced visible light modal features of the corresponding layers.

[0033] According to the enhanced SAR modal features of each layer and the enhanced visible light modal characteristics Obtain visible light enhancement feature set and SAR enhancement feature set.

[0034] In a possible implementation, the flattening SAR enhancement feature set includes: and a flattened visible light enhancement feature in the flattened visible light enhancement feature set The features are sequentially input into the cross-modal feature enhancement module for feature coupling to obtain the enhanced SAR modal features of the corresponding layer. include:

[0035] Flatten SAR enhancement features Projection to the first matrix and the second matrix and flattening the visible light enhancement features Projection to the third matrix Among them, the first matrix and the first parameter W of the trained cross-modal feature enhancement module V Related, the second matrix The second parameter W of the trained cross-modal feature enhancement module K Related, the third matrix The third parameter of the trained cross-modal feature enhancement module Related;

[0036] According to the second matrix and the third matrix A correlation matrix is ​​obtained by using dot product calculation, and the correlation matrix is ​​normalized to obtain a fusion correlation matrix;

[0037] The fusion correlation matrix and the first matrix Multiply them together to get the cross-modal feature vector of the i-th layer

[0038] For the cross-modal feature vector Perform linear transformation, perform weighted summation with the input SAR features of the i-th layer, and obtain the enhanced SAR modal features of the i-th layer based on the weighted summation result

[0039] In a possible implementation, the enhanced visible light modality feature It is expressed as:

[0040]

[0041] The enhanced SAR modal characteristics It is expressed as:

[0042]

[0043] Among them, CAFE-opt (·) represents the visible light enhancement unit in the cross-modal feature enhancement module; represents the flattened visible light enhancement feature of the i-th layer; represents the flattened SAR enhancement feature of the i-th layer; represents the enhanced visible light modal features of the i-th layer; Π CAFE-SAR (·) represents the SAR enhancement unit in the cross-modal feature enhancement module; represents the enhanced visible light modality features of the i-th layer.

[0044] In a possible implementation, respectively fusing the visible light enhancement features and the SAR enhancement features of corresponding layers in the visible light enhancement feature set and the SAR enhancement feature set to obtain the fused features corresponding to each layer includes:

[0045] Adding each visible light enhancement feature in the visible light enhancement feature set and the visible light features inputted at each layer to obtain a visible light fusion feature set;

[0046] Adding each SAR enhancement feature in the SAR enhancement feature set and the SAR features inputted at each layer to obtain a SAR fusion feature set;

[0047] According to the feature fusion formula, the E visible light fusion feature in the visible light fusion feature set and the SAR fusion feature in the SAR fusion feature set of the same layer are fused to obtain the fusion feature corresponding to each layer.

[0048] In a possible implementation, the feature fusion formula is expressed as:

[0049]

[0050] in, represents the SAR fusion feature of the i-th layer; represents the i-th layer of visible light fusion features; θ f represents the learnable parameters of the feature fusion function; Φ fusion (·) represents the feature fusion function; Represents the fusion features of the i-th layer.

[0051] One or more technical solutions provided in the present invention have at least the following technical effects or advantages:

[0052] The present invention achieves deep coupling of multi-source image features by designing a multi-scale feature extraction network and taking advantage of the Transformer's advantages in bimodal feature fusion. In the feature fusion process, hybrid pooling and cross-attention mechanisms are used to effectively reduce resource consumption, and combined with the YOLOv5 detection network with good generalization ability, the complementary information of global features is fully mined. The ultimate goal is to improve the accuracy of target detection, especially to reduce the missed detection rate and false detection rate of visible light images in complex environments, thereby improving the accuracy and robustness of overall target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A flowchart of the steps of a cross-modal feature fusion target detection method based on visible light / SAR provided in an embodiment of the present invention;

[0054] Figure 2 A flowchart of SAR feature extraction steps provided by an embodiment of the present invention;

[0055] Figure 3 A schematic diagram of a SAR-visible light dual-modal fusion module provided in an embodiment of the present invention;

[0056] Figure 4 The overall framework network diagram provided for the embodiment of the present invention. DETAILED DESCRIPTION

[0057] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] The present invention provides a cross-modal feature fusion target detection method based on visible light / SAR, such as Figure 1 As shown, the method includes the following steps S101 to S106.

[0059] S101, obtaining a visible light image and a SAR image corresponding to the original image, and preprocessing the visible light image and the SAR image to obtain an initial visible light image I opt and the initial SAR image I SAR ;

[0060] Specifically, the visible light image and the SAR image are preprocessed to obtain an initial visible light image I opt and the initial SAR image I SAR ,include:

[0061] Envi software is used to crop the visible light image and SAR image to obtain the initial visible light image I of the same size. opt and the initial SAR image I SAR .

[0062] S102, respectively, the initial visible light image I opt and the initial SAR image I SAR Perform feature extraction to obtain the initial visible light feature F o and the initial SAR feature F S ;

[0063] S103, use the YOLO v5 network to analyze the initial visible light features F layer by layer o Perform single-mode feature extraction to obtain the visible light feature set; use the SAR feature single-mode feature extraction method to extract the initial SAR feature F layer by layer S Perform single-modal feature extraction to obtain the SAR feature set.

[0064] Specifically, the initial visible light feature F is layer by layer o Perform single-modal feature extraction, including:

[0065] Set the number of extraction layers to i, and use the single-mode feature extraction function to extract the initial visible light features F layer by layer. o Perform single-mode feature extraction; the input of the i-th layer is: visible light feature Output: Visible light features i≥2; and the input of the first layer is the initial visible light feature F o , the output is the first visible light feature

[0066]

[0067] Here, the unimodal feature extraction function is expressed as:

[0068]

[0069] in, represents the visible light feature of the i-th layer input; Ψ backbone (·) represents the feature extraction function of YOLO v5; θ opt Represents the network parameters of YOLO v5; Represents the visible light features output by the i-th layer; i represents the number of extraction layers.

[0070] YOLOv5 has good generalization performance and is used as the default feature extraction network.

[0071] For the texture information and morphological features of SAR images, such as Figure 2As shown in Figure 2, a SAR feature extraction module (SAR Feature Extraction, SFE) is designed so that the model can focus more on the key features in the image. For the specific process, refer to Figure 2 To implement.

[0072] Specifically, the SAR feature single-mode feature extraction method includes:

[0073] (1) Input SAR feature F s After passing through the standard convolution processing unit three times in sequence, each time through the standard convolution processing unit, it passes through the CBS (Conv-BN-SiLU, CBS) composed of the convolution layer, batch normalization (Batch Normalization) and activation function to obtain the preliminary extracted SAR features. The convolution operation helps to capture local features of the image from different levels, batch normalization stabilizes training, and the activation function SiLU enhances the nonlinear expression ability of the network.

[0074] (2) Preliminary extraction of SAR features Perform global average pooling (GAP) operation to obtain global SAR features Global pooling extracts global information of the image, which helps to retain overall information, reduce redundancy and highlight key features.

[0075] (3) Use two convolutional layers and activation function layers connected in sequence to analyze the global SAR features. Channel feature extraction is performed to obtain global SAR channel features. Specifically, two 1×1 convolutional layers and ReLU activation functions are used to extract key information between channels and enhance the dependency between channels.

[0076] (4) Use the sequentially connected depth separation convolution layer and point-by-point convolution layer to analyze the global SAR features. Spatial feature extraction is performed to obtain global SAR spatial features. Specifically, 3×3 depthwise separable convolution and 1×1 point-by-point convolution layers are used to extract spatial features, which reduces the computational complexity while maintaining effective spatial feature representation.

[0077] (5) The global SAR channel features and the global SAR spatial features are fused, and the fused features are weighted to obtain the enhanced global SAR features. The fused features are weighted through the Sigmoid function, which strengthens the expression of features in the channel and spatial dimensions and ensures that key information is effectively retained.

[0078] (6) Using two standard convolution processing units CBS connected in sequence to process the input SAR feature F sPerform standard convolution processing and combine the processed SAR features with the enhanced global SAR features The spliced ​​features are sequentially passed through a standard convolution processing unit and a downsampling layer to obtain the SAR features output by the current layer. Take the SAR features output by all layers Constitute the SAR feature set.

[0079] Through these steps, the SFE module can effectively extract, fuse and enhance key information in SAR images, providing richer and more effective feature representation for downstream task target detection.

[0080] Aiming at the unique characteristics of SAR images, the present invention designs a multi-scale feature extraction network, which effectively exploits the advantages of SAR images in complex environments, such as penetration and anti-interference, and makes up for the deficiency that the existing fusion detection model cannot fully utilize the characteristics of SAR images.

[0081] S104, based on the Transformer cross attention mechanism, cross-modal feature enhancement is performed on the visible light feature set and the SAR feature set to obtain a visible light enhanced feature set and a SAR enhanced feature set;

[0082] Specifically, based on the Transformer cross-attention mechanism, cross-modal feature enhancement is performed on the visible light feature set and the SAR feature set to obtain a visible light enhanced feature set and a SAR enhanced feature set, including the following steps S1041 to S1046.

[0083] S1041, compressing each visible light enhancement feature in the visible light enhancement feature set by a hybrid pooling operation to obtain a compressed visible light enhancement feature set; and compressing each SAR enhancement feature in the SAR enhancement feature set by a hybrid pooling operation to obtain a compressed SAR enhancement feature set;

[0084] S1042, flattening the compressed visible light enhancement features and the compressed SAR enhancement features in the compressed visible light enhancement feature set and the compressed SAR enhancement feature set respectively to obtain a flattened visible light enhancement feature set and a flattened SAR enhancement feature set;

[0085] S1043, flattening the SAR enhancement feature set and Flattened Visible Enhanced Features in the Flattened Visible Enhanced Feature Set The features are sequentially input into the cross-modal feature enhancement module for feature coupling to obtain the enhanced SAR modal features of the corresponding layer.

[0086] Here, the flattened SAR enhanced features in the flattened SAR enhanced feature set and Flattened Visible Enhanced Features in the Flattened Visible Enhanced Feature Set The features are sequentially input into the cross-modal feature enhancement module for feature coupling to obtain the enhanced SAR modal features of the corresponding layer. include:

[0087] (1) Flatten the SAR enhancement features Projection to the first matrix and the second matrix And flatten the visible light enhancement features Projection to the third matrix Among them, the first matrix The network parameters W of YOLO v5 with the trained cross-modal feature enhancement module V Correlation, second matrix The second parameter W of the trained cross-modal feature enhancement module K Correlation, third matrix The third parameter of the trained cross-modal feature enhancement module Related;

[0088] (2) According to the second matrix and the third matrix The correlation matrix is ​​obtained by using dot product calculation, and the correlation matrix is ​​normalized to obtain the fusion correlation matrix;

[0089] (3) Fusion correlation matrix and first matrix Multiply them together to get the cross-modal feature vector of the i-th layer

[0090] (4) Cross-modal feature vectors Perform linear transformation, perform weighted summation with the input SAR features of the i-th layer, and obtain the enhanced SAR modal features of the i-th layer based on the weighted summation result

[0091] S1046, based on the enhanced SAR modal features of each layer and enhanced visible light modal features Obtain visible light enhancement feature set and SAR enhancement feature set.

[0092] Here, the visible light modal features are enhanced It is expressed as:

[0093]

[0094] S1044, Flatten visible light enhancement features in the visible light enhancement feature set and the flattened SAR enhancement feature set The features are sequentially input into the cross-modal feature enhancement module for feature coupling to obtain the enhanced visible light modal features of the corresponding layers.

[0095] Here, the enhanced SAR modal characteristics It is expressed as:

[0096]

[0097] Among them, CAFE-opt (·) represents the visible light enhancement unit in the cross-modal feature enhancement module; represents the flattened visible light enhancement feature of the i-th layer; represents the flattened SAR enhancement feature of the i-th layer; represents the enhanced visible light modal features of the i-th layer; Π CAFE-SAR (·) represents the SAR enhancement unit in the cross-modal feature enhancement module; represents the enhanced visible light modality features of the i-th layer.

[0098] S1045, based on the enhanced SAR modal features of each layer and enhanced visible light modal features Obtain visible light enhancement feature set and SAR enhancement feature set.

[0099] For example, Figure 3 As shown in Figure 2, a hybrid pooling operation is first used to enhance the SAR features of both SAR and optical modes. Visible light enhancement features Compress, then compress the SAR enhancement features Compressed visible light enhancement features Flattening is performed to obtain two sets of tokens, namely the flattened visible light enhancement feature set and the flattened SAR enhancement feature set. Then the mapped flattened SAR enhancement feature set is and flattening visible light enhancement features Input to the transformer-based cross-modal feature enhancement (CAFE) module for feature coupling to obtain enhanced SAR modal features and enhanced visible light modal features This not only preserves the original information of the two modalities, but also achieves feature complementarity between the modalities and improves the expressiveness of the target features. The specific process is as follows:

[0100] 1) Feature Compression (FC)

[0101] In order to reduce the subsequent computational cost of the module and reduce the information loss in the feature map, a hybrid pooling operation is used to compress the original features. In the pooling operation, average pooling is used to calculate the average value of all elements in the pooling area to retain the background information in the image; maximum pooling considers the maximum value of the elements in the area to retain the texture features of the object. Taking the above into consideration, the visible light image is taken as an example to enhance the visible light feature. Compression is performed. The adaptive aggregation average pooling and maximum pooling method is adopted, and the expression is as follows:

[0102]

[0103] in, represents the input feature, S represents the scaling factor of the pooling area, and Represent the features after average pooling and maximum pooling respectively. It is the feature of the final output, which is a weighted combination of the two pooling results. λ is a learnable parameter between 0 and 1, allowing the network to be adjusted according to different tasks.

[0104] Similarly, for SAR enhancement features The result of compression is expressed as:

[0105]

[0106] 2) Cross-modal feature enhancement, Figure 3 CAFE(SAR) and CAFE(opt) in.

[0107] The compressed feature map is flattened to obtain a set of tokens, namely the flattened visible light enhancement feature set and the flattened SAR enhancement feature set, and a learnable positional encoding is added to each token. This can convert the spatial features into a sequence form, so that they can be applied to the Transformer model later.

[0108] The following uses the SAR mode as an example for explanation, and the same is true for the visible light mode. The formula is as follows:

[0109] 2.1) Calculate the query (query, Q), key (key, K), and value (value, V).

[0110] Obtained flattened visible light enhancement features and flattened SAR enhancement features As the input of CAFE module, the SAR enhancement features are flattened Projected into two separate matrices, the first matrix and the second matrix Flattening visible light enhancement features is projected onto the third matrix Used to calculate That is used to calculate the attention mechanism:

[0111]

[0112] in, It is a learnable weight matrix used to map tokens to Q, K, V space.

[0113] Here It is obtained based on the trained cross-modal feature enhancement module. When training the cross-modal feature enhancement module, the training data set is based on the task requirements. The present invention uses the Space-Net 6 data set, which contains paired visible light images and SAR images, and the main target is the ships docked near the port. The specific process is as follows:

[0114] First, the dataset is preliminarily integrated to filter out images containing ship targets;

[0115] Then, the images were cropped using Envi software and uniformly adjusted to a fixed size of 512 × 512 pixels to ensure data consistency;

[0116] Secondly, the data set is expanded through data augmentation operations such as rotation and mirroring to increase the robustness of the model;

[0117] Finally, LabelImg software is used to annotate visible light and SAR images to provide high-quality labeled data for subsequent network training.

[0118] 2.2) Construct the correlation matrix.

[0119] First, the second matrix With the third matrix The correlation matrix is ​​constructed through dot product operation, and then the Softmax function is used to normalize the correlation matrix. The fused correlation matrix and the first matrix V SAR Multiply them together to get the cross-modal feature vector Cross-modal feature vector The expression is as follows:

[0120]

[0121] in, Represents the second matrix The square root of the dimension is used to scale to control the value of Softmax; represents the cross-modal feature vector of the i-th layer.

[0122] 2.3) Refine features.

[0123] Get the cross-modal feature vector After that, it is linearly transformed to make it the same as the original feature dimension, and then combined with the input SAR feature of the i-th layer by weighted summation:

[0124]

[0125] Among them, α and β represent the learned weight coefficients.

[0126] Afterwards, the features are further enhanced through the Feed-Forward Network (FFN) and the enhanced features are combined with Add them together to get the final enhanced SAR modal features of the i-th layer:

[0127]

[0128] in, is the final feature vector obtained by FFN refinement, γ represents the learned first weight coefficient, and δ represents the learned second weight coefficient.

[0129] Enhanced features of visible light modality The acquisition process is the same as above and will not be described in detail. Its expression is as follows:

[0130]

[0131] It is worth noting that the CAFE module is a transformer-based method that uses a cross-attention mechanism and only uses queries from the auxiliary modality to calculate the relevance to the main modality. This design significantly reduces the computational complexity and number of parameters, thereby improving the efficiency of cross-modal interaction.

[0132] The present invention leverages the advantages of Transformer in cross-modal tasks, deeply couples the multi-source features of visible light and SAR images, enhances the complementarity between images, helps improve the target detection accuracy of visible light remote sensing images in complex environments, and reduces missed detection rates and false detection rates.

[0133] The present invention introduces optimization measures such as hybrid pooling and cross attention in the early fusion module. Compared with traditional methods, hybrid pooling combines the advantages of different pooling strategies, can better extract multi-level and multi-scale features of images, and enhance the expressiveness of features. The cross attention mechanism optimizes information flow and strengthens feature selection capabilities by guiding the model to share attention weights between different modalities. The introduction of these optimization measures enables the model to achieve efficient cross-modal fusion under low computing resources, significantly improving the scalability and applicability in practical applications, and is particularly suitable for large-scale remote sensing image data processing and real-time target detection tasks.

[0134] The feature fusion method proposed in this invention has good versatility and can be integrated into different backbone networks and used in combination with different target detection frameworks. The versatility of this method enables it to adapt to various remote sensing image target detection tasks, including but not limited to urban monitoring, disaster assessment, military reconnaissance and other fields. By integrating this method into the existing detection framework, the accuracy and efficiency of target detection tasks can be effectively improved, and it has broad application prospects.

[0135] S105, fusing the visible light enhancement features and SAR enhancement features of the corresponding layers in the visible light enhancement feature set and the SAR enhancement feature set, respectively, to obtain the fused features corresponding to each layer;

[0136] Specifically, the visible light enhancement features and SAR enhancement features of corresponding layers in the visible light enhancement feature set and the SAR enhancement feature set are fused respectively to obtain the fused features corresponding to each layer, including S1051 to S1053.

[0137] S1051, adding each visible light enhancement feature in the visible light enhancement feature set and the visible light features inputted from each layer to obtain a visible light fusion feature set;

[0138] S1052, adding each SAR enhancement feature in the SAR enhancement feature set and the SAR features inputted at each layer to obtain a SAR fusion feature set;

[0139] S1053, according to the feature fusion formula, the visible light fusion features in the visible light fusion feature set and the SAR fusion features in the SAR fusion feature set of the same layer are fused to obtain the fusion features corresponding to each layer.

[0140] Here, the feature fusion formula is expressed as:

[0141]

[0142] in, Indicates that the SAR fusion feature is obtained at the i-th layer; represents the i-th layer of visible light fusion features; θf represents the learnable parameters of the feature fusion function; Φ fusion (·) represents the feature fusion function; Represents the fusion features of the i-th layer.

[0143] For example, the enhanced SAR modal features obtained after the CAFE module and enhanced visible light modal features With the original features Perform element-by-element addition to obtain the SAR fusion features and visible light fusion features corresponding to each layer.

[0144] S106, input the fusion features corresponding to each layer into the multi-scale detection head to perform target classification and bounding box regression.

[0145] After completing the cross-modal feature fusion, the obtained fused feature set is input into the "neck" part of the detector for multi-scale feature fusion, and then the fused features are sent to the detection head for subsequent classification and regression operations. This process can be expressed by the following formula:

[0146]

[0147] Among them, D cls , D bbox Represent the classification results and bounding box regression results respectively; represents all fused feature maps from layer 1 to layer L in the cross-modal fusion process; ξ neck and head They represent the multi-scale feature aggregation function and the detection head function, respectively. The detection head ξ neck Responsible for performing classification tasks and bounding box regression tasks, and using parameters θ h .

[0148] Common multi-scale feature aggregation functions include FPN (Feature Pyramid Networks) and PANet (Path Aggregation Network), which are widely used to enhance the semantic expression and positioning capabilities of features. The detection head structure includes the YOLO structure and the FCOS detection head. Based on the strong generalization of YOLOv5, the present invention adopts the default settings in the original paper, including the "neck" and "head" parts of the detector.

[0149] In a specific use embodiment provided by the invention, Figure 4 As shown, the number of extraction layers is set to i=5, and at the third layer, a cross-modal feature enhancement module is used for subsequent feature fusion, target classification, and bounding box regression.

[0150] The present invention makes full use of the advantages of SAR images in active imaging, all-weather and anti-interference, and makes up for the high missed detection rate of target detection in visible light images in complex environments (such as low light, haze, cloudy days, etc.), thereby significantly improving the accuracy of single-modal image target detection. The attention mechanism of the Transformer model is used to deeply fuse images of different modalities. This method fully mines the complementary information in bimodal images by adaptively adjusting the importance of features between different modalities, effectively solving the heterogeneity problem in cross-modal fusion. Feature enhancement strategy based on cross-attention mechanism. This strategy can automatically focus on the most relevant areas between images of different modalities, strengthen the representation of key features through the attention mechanism, and effectively reduce the impact of redundant information on model performance.

[0151] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and the key points of each embodiment are the differences from other embodiments. All or part of the present invention can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, mobile communication terminals, multi-processor systems, microprocessor-based systems, programmable electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc.

[0152] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the present invention.

Claims

1. A cross-modal feature fusion target detection method based on visible light / SAR, characterized in that: include: Obtain a visible light image and a SAR image corresponding to the original image, and preprocess the visible light image and the SAR image to obtain an initial visible light image I opt and the initial SAR image I SAR ; The initial visible light image I opt and the initial SAR image I SAR Perform feature extraction to obtain the initial visible light feature F o and the initial SAR feature F S ; The initial visible light feature F is analyzed layer by layer using the YOLO v5 network. o Perform single-mode feature extraction to obtain a visible light feature set; use the SAR feature single-mode feature extraction method to extract the initial SAR feature F layer by layer S Perform single-mode feature extraction to obtain the SAR feature set; Based on the Transformer cross attention mechanism, cross-modal feature enhancement is performed on the visible light feature set and the SAR feature set to obtain a visible light enhanced feature set and a SAR enhanced feature set; Fusing the visible light enhancement features and the SAR enhancement features of the corresponding layers in the visible light enhancement feature set and the SAR enhancement feature set respectively to obtain the fused features corresponding to each layer; The fused features corresponding to each layer are input into the multi-scale detection head for target classification and bounding box regression.

2. The cross-modal feature fusion target detection method based on visible light / SAR according to claim 1 is characterized in that: The visible light image and the SAR image are preprocessed to obtain an initial visible light image I opt and the initial SAR image I SAR ,include: The visible light image and the SAR image are cropped using Envi software to obtain an initial visible light image I of the same size. opt and the initial SAR image I SAR .

3. The cross-modal feature fusion target detection method based on visible light / SAR according to claim 1 is characterized in that: The initial visible light feature F o Perform single-modal feature extraction, including: Set the number of extraction layers to i, and use the single-mode feature extraction function to extract the initial visible light feature F layer by layer. o Perform single-mode feature extraction; the input of the i-th layer is: visible light feature Output: Visible light features i≥2; and the input of the first layer is the initial visible light feature F o , the output is the first visible light feature 4. The cross-modal feature fusion target detection method based on visible light / SAR according to claim 3 is characterized in that: The unimodal feature extraction function is expressed as: in, represents the visible light feature of the i-th layer input; Ψ backbone (·) represents the feature extraction function of YOLO v5; θ opt Represents the network parameters of YOLO v5; Represents the visible light features output by the i-th layer; i represents the number of extraction layers.

5. The cross-modal feature fusion target detection method based on visible light / SAR according to claim 1 is characterized in that: The SAR feature single-mode feature extraction method comprises: The SAR features of the current layer input are processed by three standard convolution processing units in sequence to obtain the preliminary extracted SAR features. Wherein, each of the standard convolution processing units includes sequentially undergoing a standard convolution processing unit, batch normalization processing, and activation function processing; The preliminary extraction of SAR features Perform global average pooling operation to obtain global SAR features The global SAR features are analyzed by using two convolutional layers and activation function layers connected in sequence. Perform channel feature extraction to obtain global SAR channel features; The global SAR features are analyzed by sequentially connecting the depth separation convolution layer and the point-by-point convolution layer. Perform spatial feature extraction to obtain global SAR spatial features; The global SAR channel feature and the global SAR spatial feature are fused, and the fused feature is weighted to obtain an enhanced global SAR feature. The input SAR features are processed by standard convolution processing units connected in sequence, and the processed SAR features are combined with the enhanced global SAR features. The spliced ​​features are sequentially passed through a standard convolution processing unit and a downsampling layer to obtain the SAR features output by the current layer. Take the SAR features output by all layers Constitute the SAR feature set.

6. The cross-modal feature fusion target detection method based on visible light / SAR according to claim 1, characterized in that: The cross-modal feature enhancement of the visible light feature set and the SAR feature set is performed based on the Transformer cross attention mechanism to obtain a visible light enhanced feature set and a SAR enhanced feature set, including: Compressing each visible light feature in the visible light feature set by a hybrid pooling operation to obtain a compressed visible light enhanced feature set; and compressing each SAR feature in the SAR feature set by a hybrid pooling operation to obtain a compressed SAR enhanced feature set; Flattening the compressed visible light enhancement features and the compressed SAR enhancement features in the compressed visible light enhancement feature set and the compressed SAR enhancement feature set respectively to obtain a flattened visible light enhancement feature set and a flattened SAR enhancement feature set; The flattened SAR enhancement feature set and a flattened visible light enhancement feature in the flattened visible light enhancement feature set The features are sequentially input into the cross-modal feature enhancement module for feature coupling to obtain the enhanced SAR modal features of the corresponding layer. The flattened visible light enhancement feature in the flattened visible light enhancement feature set and flattening the SAR enhancement features in the flattened SAR enhancement feature set The features are sequentially input into the cross-modal feature enhancement module for feature coupling to obtain the enhanced visible light modal features of the corresponding layers. According to the enhanced SAR modal features of each layer and the enhanced visible light modal characteristics Obtain visible light enhancement feature set and SAR enhancement feature set.

7. The cross-modal feature fusion target detection method based on visible light / SAR according to claim 6 is characterized in that: The flattened SAR enhancement feature set T i SAR and the flattened visible light enhancement feature in the flattened visible light enhancement feature set The features are sequentially input into the cross-modal feature enhancement module for feature coupling to obtain the enhanced SAR modal features of the corresponding layer. include: Flatten SAR enhancement features Projection to the first matrix and the second matrix and flattening the visible light enhancement features Projection to the third matrix Among them, the first matrix and the first parameter W of the trained cross-modal feature enhancement module V Related, the second matrix The second parameter W of the trained cross-modal feature enhancement module K Related, the third matrix The third parameter of the trained cross-modal feature enhancement module Related; According to the second matrix and the third matrix A correlation matrix is ​​obtained by using dot product calculation, and the correlation matrix is ​​normalized to obtain a fusion correlation matrix; The fusion correlation matrix and the first matrix Multiply them together to get the cross-modal feature vector of the i-th layer For the cross-modal feature vector Perform a linear transformation, perform a weighted summation with the input SAR features of the i-th layer, and obtain the enhanced SAR modal features of the i-th layer based on the weighted summation result 8. The cross-modal feature fusion target detection method based on visible light / SAR according to claim 6, characterized in that: The enhanced visible light modality features It is expressed as: The enhanced SAR modal characteristics It is expressed as: Among them, CAFE-opt (·) represents the visible light enhancement unit in the cross-modal feature enhancement module; represents the flattened visible light enhancement feature of the i-th layer; represents the flattened SAR enhancement feature of the i-th layer; represents the enhanced visible light modal features of the i-th layer; Π CAFE-SAR (·) represents the SAR enhancement unit in the cross-modal feature enhancement module; represents the enhanced visible light modality features of the i-th layer.

9. The cross-modal feature fusion target detection method based on visible light / SAR according to claim 1, characterized in that: The step of fusing the visible light enhancement features and the SAR enhancement features of the corresponding layers in the visible light enhancement feature set and the SAR enhancement feature set to obtain the fused features corresponding to each layer comprises: Adding each visible light enhancement feature in the visible light enhancement feature set and the visible light features inputted at each layer to obtain a visible light fusion feature set; Adding each SAR enhancement feature in the SAR enhancement feature set and the SAR features inputted at each layer to obtain a SAR fusion feature set; The visible light fusion features in the visible light fusion feature set and the SAR fusion features in the SAR fusion feature set of the same layer are fused according to the feature fusion formula to obtain the fusion features corresponding to each layer.

10. The cross-modal feature fusion target detection method based on visible light / SAR according to claim 9, characterized in that: The feature fusion formula is expressed as: in, represents the SAR fusion feature of the i-th layer; represents the i-th layer of visible light fusion features; θ f represents the learnable parameters of the feature fusion function; Φ fusion (·) represents the feature fusion function; Represents the fusion features of the i-th layer.

Images