A multimodal environment perception feature level adaptive fusion method

Abstract

The application discloses a kind of multi-mode environment perception feature level adaptive fusion methods, belong to multi-modal information fusion technical field;Including the following steps: S1, the cloud and fog area mask of the infrared image to be fused is extracted;S2, the radar SAR image available degree of the radar SAR image to be fused is extracted;S3, extraction and, to enhance and construct infrared interactive attention enhancement module;S4, to enhance and construct radar SAR image interactive attention enhancement module;S5, feature level adaptive scale fusion is carried out, and fusion feature is obtained;S6, decoupling is carried out to fusion feature, and multi-mode environment perception feature level adaptive fusion is completed;The application adopts the above method, realizes the accurate identification to target under complex environment.

Description

Technical Field

[0001] This invention belongs to the field of multimodal information fusion technology, and in particular relates to a multimodal environmental perception feature-level adaptive fusion method. Background Technology

[0002] With the development of multi-source perception systems, using images of different modalities for information complementarity has become a key way to improve scene understanding. At present, most feature-level fusion methods adopt the method of extracting modal features separately, splicing them and then directly fusing and recognizing them. This lacks high-frequency interaction in the feature extraction process and removal of unfavorable information in the fusion process, resulting in poor adaptability and insufficient proportion of effective information in the fusion results. Summary of the Invention

[0003] The purpose of this invention is to provide a multimodal environment-aware feature-level adaptive fusion method, which judges the availability of multimodal images, displays and guides feature enhancement of available areas, and gradually enhances highly relevant target features through multi-scale intermodal information interaction during feature extraction, supporting information completion enhancement in the final feature fusion, and achieving accurate target identification in complex environments.

[0004] To achieve the above objectives, this invention provides a multi-modal environment-aware feature-level adaptive fusion method, comprising the following steps: S1. Extract the cloud and fog mask from the infrared image to be fused. ; S2. Extract the usability of the radar SAR images to be fused. ; S3. Extract infrared image features and radar SAR image features And through the Enhance the infrared interactive attention enhancement module; S4, through the Enhance the interactive attention enhancement module for radar SAR images; S5. Input the infrared image and radar SAR image to be fused into the feature extractor, the infrared interactive attention enhancement module and the radar SAR image interactive attention enhancement module respectively, and perform feature-level adaptive scale fusion to obtain the fused features. S6. Decouple the fused features to complete the multi-modal environmental perception feature-level adaptive fusion.

[0005] Preferably, the process of S1 is as follows: S11. Input the infrared image to be fused into the extreme judgment model for judgment. If the image is either completely black or completely white, the entire image is judged to be unusable and the infrared image to be fused is reselected. Otherwise, it is judged to be a usable infrared image. S12. Input the available infrared image into the morphological cloud and fog region extraction model. For normal morphological scenes, use morphological judgment and connected region extraction to confirm the cloud and fog region. For morphological failure scenarios, adaptive threshold segmentation is used to confirm the foreground and background segmentation thresholds in the image and extract the cloud and fog regions. S13. Combining the cloud and fog region with the application scenario, adjust the weight of the perceived cloud and fog mask, reducing the weight of the cloud and fog mask at the center of the slice and keeping the weight unchanged at the edge of the slice, to obtain the cloud and fog region mask. .

[0006] Preferably, the process of S2 is as follows: S21. Input the radar SAR image to be fused into the all-black image determination model for determination. If the image is an all-black image, the entire image is determined to be unusable and the radar SAR image to be fused is reselected. Otherwise, it is determined to be a non-all-black image. S22. Set the pixel number difference threshold, input the non-completely black image into the unusable state determination model, and by statistically analyzing the histogram of each gray level interval, if the difference in the number of pixels in each gray level interval is less than the pixel number difference threshold, then the entire image is determined to be unusable, and a new radar SAR image to be fused is selected and returned to S21; otherwise, it is determined to be a non-unusable image. S23. The signal-to-noise ratio (SNR) confirms the usability of radar SAR images by using the ratio of the average gray level to the maximum gray level within the image for both non-completely black and unusable images. .

[0007] Preferably, the process of S3 is as follows: S31. Extract the infrared image features from the infrared image to be fused and the radar SAR image respectively. and radar SAR image features ; S32. Calculate the initial infrared-to-infrared mapping features. and radar to infrared preliminary mapping features The process is as follows: ; ; in This represents the weight matrix for the infrared-to-infrared feature mapping. This represents the weight matrix for mapping radar features to infrared features; S33. Calculate the infrared feature attention weights. The process is as follows: ; ; in Represents the infrared high correlation feature matrix. express The transpose of the matrix, Represents a sigmoid mapping function. Indicates a convolutional layer; S34. Calculate the features of the enhanced infrared image. The infrared interactive attention enhancement module was constructed as follows: ; in This indicates that the attention weights of infrared features are multiplied pixel-by-pixel by the infrared image features.

[0008] Preferably, the process of S4 is as follows: S41. Calculate the preliminary mapping features of infrared-to-radar (SAR) radar. and radar to radar SAR preliminary mapping features The process is as follows: ; ; in This represents the infrared-to-radar SAR feature mapping weight matrix. This represents the weight matrix for mapping radar features to radar SAR features; S42. Calculate radar SAR attention weights The process is as follows: ; ; in This represents the radar SAR high correlation feature matrix. express The transpose of the matrix; S43. Calculate the features of the enhanced radar SAR image. The interactive attention enhancement module for radar SAR images was completed as follows: .

[0009] Preferably, the process of S5 is as follows: S51. Input the infrared image and radar SAR image to be fused into the feature extractor, the infrared interactive attention enhancement module, and the radar SAR image interactive attention enhancement module, respectively. S52. Extract the infrared image features at the first scale respectively. and first-scale radar SAR image features The process is as follows: ; ; in This represents the infrared feature extractor at the first scale. This indicates a feature concatenation operation. This represents the infrared images to be fused. This represents the radar SAR feature extractor at the first scale. Represents the radar SAR images to be fused; S53, the result obtained in S52 and Multiple downsampling operations were performed to obtain the fourth-scale infrared image features. and fourth-scale radar SAR image features ; S54, through fourth-scale infrared image features and fourth-scale radar SAR image features Computational fusion features .

[0010] Preferably, the process of S53 is as follows: S531. Calculate intermediate features of second-scale infrared images. Intermediate features of second-scale radar SAR images The process is as follows: ; ; in This represents the second-scale infrared feature extractor. This indicates a spatial downsampling operation. This represents a radar SAR feature extractor at the second scale. S532, Through and Calculate second-scale infrared image features Second-scale radar SAR image features The process is as follows: ; ; in This indicates the second-scale infrared interactive attention enhancement module. This indicates the second-scale radar SAR interactive attention enhancement module; S533, Calculate intermediate features of third-scale infrared images Intermediate features of third-scale radar SAR images The process is as follows: ; ; in Third-scale infrared feature extractor A third-scale radar SAR feature extractor; S534, Through and Calculate the features of the third-scale infrared image and third-scale radar SAR image features The process is as follows: ; ; in For third-scale infrared interactive attention enhancement modules, For third-scale radar SAR interactive attention enhancement module; S535. Calculate intermediate features of fourth-scale infrared images. Intermediate features of fourth-scale radar SAR images The process is as follows: ; ; in For fourth-scale infrared feature extractors, For fourth-scale radar SAR feature extractors; S536, Through and Calculate the features of the fourth-scale infrared image and fourth-scale radar SAR image features The process is as follows: ; ; in For the fourth-scale infrared interactive attention enhancement module, This is a fourth-scale radar SAR interactive attention enhancement module.

[0011] Preferably, the process of S54 is as follows: S541, obtained through S53 Calculating multi-receptive field features of infrared images The process is as follows: ; ; in This represents the infrared features extracted by a 1×1 convolution kernel. This represents the infrared features extracted by a 3×3 convolution kernel. This represents the infrared features extracted by a 5×5 convolution kernel. It is a 1×1 convolutional layer. It is a 3×3 convolutional layer. It is a 5×5 convolutional layer. Indicates the flattening operation; S542, via S53 Calculating multi-receptive field features of radar SAR images The process is as follows: ; ; in This represents the radar SAR features extracted by a 1×1 convolution kernel. This represents the radar SAR features extracted by a 3×3 convolution kernel. This represents the radar SAR features extracted by a 5×5 convolution kernel; S543. Calculate infrared query matrix features using multi-receptive field features of infrared images. Infrared Key Matrix Features and infrared value matrix features The process is as follows: ; in For infrared query mapping matrix, For infrared key mapping matrix, For infrared value mapping matrix; S544. Calculate the Query Matrix Features of Radar SAR Images Using Multi-Receptive Field Features. Key matrix features of radar SAR images Features of the Value Matrix of Radar SAR Images The process is as follows: ; in This is the radar SAR image query mapping matrix. This is the key mapping matrix for radar SAR images. The value mapping matrix for radar SAR images; S545. Calculate the fusion features based on the results of S543 and S544. The process is as follows: .

[0012] Preferably, the process of S6 is as follows: S61. Obtain the infrared gradient and radar SAR gradient of the infrared image and radar SAR image to be fused. S62. Calculate the cosine similarity between the infrared gradient and the radar SAR gradient, and determine whether it is greater than 0.4. If it is, proceed to S63. S63. Decouple the gradient as follows: ; in The gradient after decoupling. The gradient before decoupling. This represents the vector dot product operation. Represents the vector magnitude operation. This is the gradient of another mode.

[0013] Preferably, the adaptive fusion method is implemented through an adaptive fusion system, including an infrared image pixel-level environment perception module, a radar SAR image environment perception module, an interactive attention enhancement module, a multi-branch enhancement complementary fusion module, and a training optimization module; The infrared image pixel-level environmental perception module includes an extreme judgment model and a morphological cloud and fog region extraction model. It generates mask perception of cloud and fog regions in infrared images through morphological filtering and threshold segmentation. The radar SAR image environment perception module includes a completely black image determination model, an unusable state determination model, and a signal-to-noise ratio (SNR) confirmation model. The usability of the radar SAR image is determined by calculating the SNR. The interactive attention enhancement module includes an infrared interactive attention enhancement module and a radar SAR image interactive attention enhancement module, which extracts highly correlated features based on the similarity between infrared images and radar SAR images. The multi-branch enhanced complementary fusion module includes an end feature extraction model, a multi-scale feature extraction model, and a multi-receptive field feature aggregation model. It interactively increases the proportion of target information in infrared modal and radar SAR modal features through multi-scale fusion, while designing a mutual mapping strategy to ensure the integrity of information. The training optimization module includes a gradient decoupling and orthogonal projection training model and a course learning modality masking model. Multimodal gradient decoupling is achieved through gradient monitoring to accelerate optimization, and a course learning strategy is designed to ensure the balance of multimodal feature learning.

[0014] Therefore, this invention adopts the above-mentioned multi-modal environment perception feature-level adaptive fusion method, designs a dual-branch interactive feature fusion enhancement architecture, unifies the available state prior information at the pixel level into the network learning paradigm, automatically enhances the proportion of target region features, and ensures the integrity of features through mutual mapping, effectively improving the target accuracy recognition capability in complex environments.

[0015] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0016] Figure 1 This is an overall flowchart of a multi-modal environmental perception feature-level adaptive fusion method according to the present invention; Figure 2 This is a system diagram of a multi-modal environmental perception feature-level adaptive fusion method according to the present invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Specific model specifications need to be selected and determined according to the actual specifications of the device, etc. The specific selection calculation method adopts existing technology in the art, and therefore will not be described in detail.

[0018] Example like Figure 1 and Figure 2 As shown, this invention provides a multi-modal environment-aware feature-level adaptive fusion method, comprising the following steps: S1. Extract the cloud and fog mask from the infrared image to be fused. ; The infrared image pixel-level environmental perception module identifies cloud and fog regions within an image through extreme judgment, morphological cloud and fog region extraction, and threshold segmentation. S11. Input the infrared image to be fused into the extreme judgment model for judgment. If the image is either completely black or completely white, the entire image is judged to be unusable and the infrared image to be fused is reselected. Otherwise, it is judged to be a usable infrared image. S12. Input the available infrared image into the morphological cloud and fog region extraction model. For normal morphological scenes, use morphological judgment and connected region extraction to confirm the cloud and fog region. For morphological failure scenarios, adaptive threshold segmentation is used to confirm the foreground and background segmentation thresholds in the image and extract the cloud and fog regions. S13. Combining the cloud and fog region with the application scenario, adjust the weights of the perceived cloud and fog mask. Reduce the weight of the cloud and fog mask at the center of the slice and keep the weight unchanged at the edge of the slice to obtain the cloud and fog region mask. .

[0019] S2. Extract the usability of the radar SAR images to be fused. ; The radar SAR image environment perception module determines the usability of an image through steps such as determining a completely black image, determining an unusable state, and confirming the signal-to-noise ratio. S21. Input the radar SAR image to be fused into the all-black image determination model for determination. If the image is an all-black image, the entire image is determined to be unusable and the radar SAR image to be fused is reselected. Otherwise, it is determined to be a non-all-black image. S22. Set the pixel number difference threshold, input the non-completely black image into the unusable state determination model, and by statistically analyzing the histogram of each gray level interval, if the difference in the number of pixels in each gray level interval is less than the pixel number difference threshold, then the entire image is determined to be unusable, and a new radar SAR image to be fused is selected and returned to S21; otherwise, it is determined to be a non-unusable image. S23. The signal-to-noise ratio (SNR) confirmation model is used for both non-completely black and unusable images, employing the ratio of the average to the maximum gray level within the image as the usability of the radar SAR image. .

[0020] S3. Extract infrared image features and radar SAR image features And through the Enhance the infrared interactive attention enhancement module; The interactive attention enhancement module extracts the associated features of each modality at multiple scales by calculating the similarity matrix between single-modal features and fused features, ensuring the effective extraction of information from each modality. S31. Extract the infrared image features from the infrared image to be fused and the radar SAR image respectively. and radar SAR image features ; S32. Calculate the initial infrared-to-infrared mapping features. and radar to infrared preliminary mapping features The process is as follows: ; ; in This represents the weight matrix for the infrared-to-infrared feature mapping. This represents the weight matrix for mapping radar features to infrared features; S33. Calculate the infrared feature attention weights. The process is as follows: ; ; in Represents the infrared high correlation feature matrix. express The transpose of the matrix, Represents a sigmoid mapping function. Indicates a convolutional layer; S34. Calculate the features of the enhanced infrared image. The infrared interactive attention enhancement module was constructed as follows: ; in This indicates that the attention weights of infrared features are multiplied pixel-by-pixel by the infrared image features.

[0021] S4, through the Enhance the interactive attention enhancement module for radar SAR images; S41. Calculate the preliminary mapping features of infrared-to-radar (SAR) radar. and radar to radar SAR preliminary mapping characteristics The process is as follows: ; ; in This represents the infrared-to-radar SAR feature mapping weight matrix. This represents the weight matrix for mapping radar features to radar SAR features; S42. Calculate radar SAR attention weights The process is as follows: ; ; in This represents the radar SAR high correlation feature matrix. express The transpose of the matrix; S43. Calculate the features of the enhanced radar SAR image. The interactive attention enhancement module for radar SAR images was completed as follows: .

[0022] S5. Input the infrared image and radar SAR image to be fused into the feature extractor, the infrared interactive attention enhancement module and the radar SAR image interactive attention enhancement module respectively, and perform feature-level adaptive scale fusion to obtain the fused features. The multi-branch enhancement and complementary fusion module uses the perceived unavailability mask as input information to guide feature adaptive extraction, and performs multi-scale feature interaction enhancement during semantic feature extraction. Finally, it achieves complementary feature enhancement through multi-receptive field feature aggregation. S51. Input the infrared image and radar SAR image to be fused into the feature extractor, the infrared interactive attention enhancement module, and the radar SAR image interactive attention enhancement module, respectively. S52. Extract the infrared image features at the first scale respectively. and first-scale radar SAR image features The process is as follows: ; ; in This represents the infrared feature extractor at the first scale. This indicates a feature concatenation operation. This represents the infrared images to be fused. This represents the radar SAR feature extractor at the first scale. Represents the radar SAR images to be fused; S53, the result obtained in S52 and Multiple downsampling operations were performed to obtain the fourth-scale infrared image features. and fourth-scale radar SAR image features ; S531. Calculate intermediate features of second-scale infrared images. Intermediate features of second-scale radar SAR images The process is as follows: ; ; in This represents the second-scale infrared feature extractor. This indicates a spatial downsampling operation. This represents a radar SAR feature extractor at the second scale. S532, Through and Calculate second-scale infrared image features Second-scale radar SAR image features The process is as follows: ; ; in This indicates the second-scale infrared interactive attention enhancement module. This indicates the second-scale radar SAR interactive attention enhancement module; S533, Calculate intermediate features of third-scale infrared images Intermediate features of third-scale radar SAR images The process is as follows: ; ; in Third-scale infrared feature extractor A third-scale radar SAR feature extractor; S534, Through and Calculate the features of a third-scale infrared image and third-scale radar SAR image features The process is as follows: ; ; in Third-scale infrared interactive attention enhancement module Third-scale radar SAR interactive attention enhancement module; S535. Calculate intermediate features of fourth-scale infrared images. Intermediate features of fourth-scale radar SAR images The process is as follows: ; ; in For the fourth-scale infrared feature extractor, For fourth-scale radar SAR feature extractors; S536, Through and Calculate the features of the fourth-scale infrared image and fourth-scale radar SAR image features The process is as follows: ; ; in For the fourth-scale infrared interactive attention enhancement module, This is a fourth-scale radar SAR interactive attention enhancement module.

[0023] S54, through fourth-scale infrared image features and fourth-scale radar SAR image features Computational fusion features .

[0024] S541, obtained through S53 Calculating multi-receptive field features of infrared images The process is as follows: ; ; in This represents the infrared features extracted by a 1×1 convolution kernel. This represents the infrared features extracted by a 3×3 convolution kernel. This represents the infrared features extracted by a 5×5 convolution kernel. It is a 1×1 convolutional layer. It is a 3×3 convolutional layer. It is a 5×5 convolutional layer. Indicates the flattening operation; S542, via S53 Calculating multi-receptive field features of radar SAR images The process is as follows: ; ; in This represents the radar SAR features extracted by a 1×1 convolution kernel. This represents the radar SAR features extracted by a 3×3 convolution kernel. This represents the radar SAR features extracted by a 5×5 convolution kernel; S543. Calculate infrared query matrix features using multi-receptive field features of infrared images. Infrared Key Matrix Features and infrared value matrix features The process is as follows: ; in For infrared query mapping matrix, For infrared key mapping matrix, For infrared value mapping matrix; S544. Calculate the Query Matrix Features of Radar SAR Images Using Multi-Receptive Field Features. Key matrix features of radar SAR images Features of radar SAR image value matrix The process is as follows: ; in For radar SAR image query mapping matrix, This is the key mapping matrix for radar SAR images. The value mapping matrix for radar SAR images; S545. Calculate the fusion features based on the results of S543 and S544. The process is as follows: .

[0025] S6. Decouple the fused features to complete the multi-modal environmental perception feature-level adaptive fusion.

[0026] The training optimization module monitors the gradient direction of each mode during backpropagation to control the gradient decoupling of different modes. At the same time, it uses course learning to avoid the optimization from relying too much on a certain mode and to avoid optimizing to a local optimum. S61. Obtain the infrared gradient and radar SAR gradient of the infrared image and radar SAR image to be fused. S62. Calculate the cosine similarity between the infrared gradient and the radar SAR gradient, and determine whether it is greater than 0.4. If it is, proceed to S63. S63. Decouple the gradient as follows: ; in The gradient after decoupling. The gradient before decoupling. This represents the vector dot product operation. Represents the vector magnitude operation. This is the gradient of another mode.

[0027] The adaptive fusion method is implemented through an adaptive fusion system, including an infrared image pixel-level environment perception module, a radar SAR image environment perception module, an interactive attention enhancement module, a multi-branch enhancement complementary fusion module, and a training optimization module. The infrared image pixel-level environmental perception module includes an extreme judgment model and a morphological cloud and fog region extraction model. It generates mask perception of cloud and fog regions in infrared images through morphological filtering and threshold segmentation. The radar SAR image environment perception module includes a completely black image determination model, an unusable state determination model, and a signal-to-noise ratio (SNR) confirmation model. The usability of the radar SAR image is determined by calculating the SNR. The interactive attention enhancement module includes an infrared interactive attention enhancement module and a radar SAR image interactive attention enhancement module, which extracts highly correlated features based on the similarity between infrared images and radar SAR images; The multi-branch enhanced complementary fusion module includes an end feature extraction model, a multi-scale feature extraction model, and a multi-receptive field feature aggregation model. It interactively increases the proportion of target information in infrared modal and radar SAR modal features through multi-scale fusion, while designing a mutual mapping strategy to ensure the integrity of information. The training optimization module includes a gradient decoupling and orthogonal projection training model and a course learning modality masking model. Multimodal gradient decoupling is achieved through gradient monitoring to accelerate optimization, and a course learning strategy is designed to ensure the balance of multimodal feature learning.

[0028] Therefore, this invention adopts a multi-modal environment-aware feature-level adaptive fusion method, designs a dual-branch interactive feature fusion enhancement architecture, unifies the available state prior information at the pixel level into the network learning paradigm, automatically enhances the proportion of target region features, and ensures the integrity of features through mutual mapping, effectively improving the target accuracy recognition capability in complex environments.

[0029] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A multi-modal environmental perception feature-level adaptive fusion method, characterized in that, Includes the following steps: S1. Extract the cloud and fog mask from the infrared image to be fused. ; S2. Extract the usability of the radar SAR images to be fused. ; S3. Extract infrared image features and radar SAR image features And through the Enhance the infrared interactive attention enhancement module; S4, through the Enhance the interactive attention enhancement module for radar SAR images; S5. Input the infrared image and radar SAR image to be fused into the feature extractor, the infrared interactive attention enhancement module and the radar SAR image interactive attention enhancement module respectively, and perform feature-level adaptive scale fusion to obtain the fused features. S6. Decouple the fused features to complete the multi-modal environmental perception feature-level adaptive fusion.

2. The multi-modal environmental perception feature-level adaptive fusion method according to claim 1, characterized in that, The process of S1 is as follows: S11. Input the infrared image to be fused into the extreme judgment model for judgment. If the image is either completely black or completely white, the entire image is judged to be unusable and the infrared image to be fused is reselected. Otherwise, it is judged to be a usable infrared image. S12. Input the available infrared image into the morphological cloud and fog region extraction model. For normal morphological scenes, use morphological judgment and connected region extraction to confirm the cloud and fog region. For morphological failure scenarios, adaptive threshold segmentation is used to confirm the foreground and background segmentation thresholds in the image and extract the cloud and fog regions. S13. Combining the cloud and fog region with the application scenario, adjust the weight of the perceived cloud and fog mask, reducing the weight of the cloud and fog mask at the center of the slice and keeping the weight unchanged at the edge of the slice, to obtain the cloud and fog region mask. .

3. The multi-modal environmental perception feature-level adaptive fusion method according to claim 2, characterized in that, The process of S2 is as follows: S21. Input the radar SAR image to be fused into the all-black image determination model for determination. If the image is an all-black image, the entire image is determined to be unusable and the radar SAR image to be fused is reselected. Otherwise, it is determined to be a non-all-black image. S22. Set the pixel number difference threshold, input the non-completely black image into the unusable state determination model, and by statistically analyzing the histogram of each gray level interval, if the difference in the number of pixels in each gray level interval is less than the pixel number difference threshold, then the entire image is determined to be unusable, and a new radar SAR image to be fused is selected and returned to S21; otherwise, it is determined to be a non-unusable image. S23. The signal-to-noise ratio (SNR) confirms the usability of radar SAR images by using the ratio of the average gray level to the maximum gray level within the image for both non-completely black and unusable images. .

4. The multi-modal environmental perception feature-level adaptive fusion method according to claim 3, characterized in that, The process of S3 is as follows: S31. Extract the infrared image features from the infrared image to be fused and the radar SAR image respectively. and radar SAR image features ; S32. Calculate the initial infrared-to-infrared mapping features. and radar to infrared preliminary mapping features The process is as follows: ； ； in This represents the weight matrix for the infrared-to-infrared feature mapping. This represents the weight matrix for mapping radar features to infrared features; S33. Calculate the infrared feature attention weights. The process is as follows: ； ； in Represents the infrared high correlation feature matrix. express The transpose of the matrix, Represents a sigmoid mapping function. Indicates a convolutional layer; S34. Calculate the features of the enhanced infrared image. The infrared interactive attention enhancement module was constructed as follows: ； in This indicates that the attention weights of infrared features are multiplied pixel-by-pixel by the infrared image features.

5. The multi-modal environmental perception feature-level adaptive fusion method according to claim 4, characterized in that, The process of S4 is as follows: S41. Calculate the preliminary mapping features of infrared-to-radar (SAR) radar. and radar to radar SAR preliminary mapping features The process is as follows: ； ； in This represents the infrared-to-radar SAR feature mapping weight matrix. This represents the weight matrix for mapping radar features to radar SAR features; S42. Calculate radar SAR attention weights The process is as follows: ； ； in This represents the radar SAR high correlation feature matrix. express The transpose of the matrix; S43. Calculate the features of the enhanced radar SAR image. The interactive attention enhancement module for radar SAR images was completed as follows: 。 6. The multi-modal environmental perception feature-level adaptive fusion method according to claim 5, characterized in that, The process of S5 is as follows: S51. Input the infrared image and radar SAR image to be fused into the feature extractor, the infrared interactive attention enhancement module, and the radar SAR image interactive attention enhancement module, respectively. S52. Extract the infrared image features at the first scale respectively. and first-scale radar SAR image features The process is as follows: ； ； in This represents the infrared feature extractor at the first scale. This indicates a feature concatenation operation. This represents the infrared images to be fused. This represents the radar SAR feature extractor at the first scale. Represents the radar SAR images to be fused; S53, the result obtained in S52 and Multiple downsampling operations were performed to obtain the fourth-scale infrared image features. and fourth-scale radar SAR image features ; S54, through fourth-scale infrared image features and fourth-scale radar SAR image features Computational fusion features .

7. The multi-modal environmental perception feature-level adaptive fusion method according to claim 6, characterized in that, The process of S53 is as follows: S531. Calculate intermediate features of second-scale infrared images. Intermediate features of second-scale radar SAR images The process is as follows: ； ； in This represents the second-scale infrared feature extractor. This indicates a spatial downsampling operation. This represents a radar SAR feature extractor at the second scale. S532, Through and Calculate second-scale infrared image features Second-scale radar SAR image features The process is as follows: ； ； in This indicates the second-scale infrared interactive attention enhancement module. This indicates the second-scale radar SAR interactive attention enhancement module; S533, Calculate intermediate features of third-scale infrared images Intermediate features of third-scale radar SAR images The process is as follows: ； ； in For third-scale infrared feature extractors, For third-scale radar SAR feature extractors; S534, Through and Calculate the features of the third-scale infrared image and third-scale radar SAR image features The process is as follows: ； ； in For third-scale infrared interactive attention enhancement modules, For third-scale radar SAR interactive attention enhancement module; S535. Calculate intermediate features of fourth-scale infrared images. Intermediate features of fourth-scale radar SAR images The process is as follows: ； ； in For fourth-scale infrared feature extractors, For fourth-scale radar SAR feature extractors; S536, Through and Calculate the features of the fourth-scale infrared image and fourth-scale radar SAR image features The process is as follows: ； ； in For the fourth-scale infrared interactive attention enhancement module, This is a fourth-scale radar SAR interactive attention enhancement module.

8. The multi-modal environmental perception feature-level adaptive fusion method according to claim 7, characterized in that, The process of S54 is as follows: S541, obtained through S53 Calculating multi-receptive field features of infrared images The process is as follows: ； ； in This represents the infrared features extracted by a 1×1 convolution kernel. This represents the infrared features extracted by a 3×3 convolution kernel. This represents the infrared features extracted by a 5×5 convolution kernel. It is a 1×1 convolutional layer. It is a 3×3 convolutional layer. It is a 5×5 convolutional layer. Indicates the flattening operation; S542, via S53 Calculating multi-receptive field features of radar SAR images The process is as follows: ； ； in This represents the radar SAR features extracted by a 1×1 convolution kernel. This represents the radar SAR features extracted by a 3×3 convolution kernel. This represents the radar SAR features extracted by a 5×5 convolution kernel; S543. Calculate infrared query matrix features using multi-receptive field features of infrared images. Infrared Key Matrix Features and infrared value matrix features The process is as follows: ； in For infrared query mapping matrix, For infrared key mapping matrix, For infrared value mapping matrix; S544. Calculate the Query Matrix Features of Radar SAR Images Using Multi-Receptive Field Features. Key matrix features of radar SAR images Features of the Value Matrix of Radar SAR Images The process is as follows: ； in This is the radar SAR image query mapping matrix. This is the key mapping matrix for radar SAR images. The value mapping matrix for radar SAR images; S545. Calculate the fusion features based on the results of S543 and S544. The process is as follows: 。 9. The multi-modal environmental perception feature-level adaptive fusion method according to claim 8, characterized in that, The process of S6 is as follows: S61. Obtain the infrared gradient and radar SAR gradient of the infrared image and radar SAR image to be fused. S62. Calculate the cosine similarity between the infrared gradient and the radar SAR gradient, and determine whether it is greater than 0.

4. If it is, proceed to S63. S63. Decouple the gradient as follows: ； in The gradient after decoupling. The gradient before decoupling. This represents the vector dot product operation. Represents the vector magnitude operation. This is the gradient of another mode.

10. The multi-modal environmental perception feature-level adaptive fusion method according to claim 9, characterized in that: The adaptive fusion method is implemented through an adaptive fusion system, including an infrared image pixel-level environment perception module, a radar SAR image environment perception module, an interactive attention enhancement module, a multi-branch enhancement complementary fusion module, and a training optimization module. The infrared image pixel-level environmental perception module includes an extreme judgment model and a morphological cloud and fog region extraction model. It generates mask perception of cloud and fog regions in infrared images through morphological filtering and threshold segmentation. The radar SAR image environment perception module includes a completely black image determination model, an unusable state determination model, and a signal-to-noise ratio (SNR) confirmation model. The usability of the radar SAR image is determined by calculating the SNR. The interactive attention enhancement module includes an infrared interactive attention enhancement module and a radar SAR image interactive attention enhancement module, which extracts highly correlated features based on the similarity between infrared images and radar SAR images. The multi-branch enhanced complementary fusion module includes an end feature extraction model, a multi-scale feature extraction model, and a multi-receptive field feature aggregation model. It interactively increases the proportion of target information in infrared modal and radar SAR modal features through multi-scale fusion, while designing a mutual mapping strategy to ensure the integrity of information. The training optimization module includes a gradient decoupling and orthogonal projection training model and a course learning modality masking model. Multimodal gradient decoupling is achieved through gradient monitoring to accelerate optimization, and a course learning strategy is designed to ensure the balance of multimodal feature learning.

Images