Flying bird scene multi-modal image fusion method based on combination of frequency domain and spatial domain

Abstract

The invention discloses a multi-modal image fusion method for a flying bird scene based on combination of a frequency domain and a spatial domain, and belongs to the technical field of multi-modal image fusion, and the method comprises the steps: constructing a multi-source heterogeneous data set of the flying bird scene; constructing a dual-light image fusion network model fusing a frequency domain and a space domain; respectively inputting the visible light image and the infrared image into the dual-light image fusion network model, and respectively outputting a visible light fusion feature map and an infrared fusion feature map; and carrying out adaptive fusion on the visible light fusion feature map and the infrared fusion feature map by adopting a cross-modal attention mechanism to obtain a multi-modal fusion image. According to the method, various frequency domain enhancement technologies and spatial domain processing methods are fused, and high-precision fusion of the visible light image and the infrared image of the bird is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of multimodal image fusion, and in particular relates to a multimodal image fusion method of a flying bird scene based on the combination of frequency domain and spatial domain. Background Art

[0002] In recent years, with the gradual relaxation of airspace management regulations, the economic value of low-altitude airspace has become increasingly prominent. However, with the increase in low-altitude economic activities, ensuring the safety of low-altitude airspace has become a pressing issue for relevant departments, especially bird strikes. Bird strikes can not only damage aircraft but also lead to serious flight accidents, threatening personnel safety. To ensure the safety of low-altitude airspace and promote the development of the low-altitude economy, aircraft are often equipped with binocular cameras for accurate bird detection to prevent collisions. However, existing binocular cameras struggle to fully and effectively identify and track high-speed birds due to limited image quality, insufficient detection accuracy, and poor adaptability to varying lighting conditions. These shortcomings not only increase the risk of false alarms and missed detections, but can also prevent timely response at critical moments, threatening the safe operation of aircraft. To address these issues, image fusion technology has been introduced into binocular camera systems to improve the accuracy and reliability of bird detection. Specifically, image fusion technology integrates information from multiple sensors or image sources to effectively overcome the limitations of a single camera under specific conditions.

[0003] Most traditional multimodal image fusion methods rely solely on single spatial or frequency domain processing, and suffer from shortcomings such as incomplete information fusion (multimodal images each contain different types of information, and single-domain fusion methods find it difficult to effectively integrate two complementary information), degraded fused image quality (single-domain fusion methods are prone to introduce noise or artifacts during the fusion process), and limited feature extraction capabilities (lack of multi-level and multi-dimensional feature information). Summary of the Invention

[0004] The purpose of the present invention is to address the above-mentioned deficiencies in the prior art and provide a multimodal image fusion method for flying bird scenes based on the combination of frequency domain and spatial domain, so as to solve the problems of incomplete information fusion, degraded fused image quality and limited feature extraction capability in the existing multimodal image fusion methods.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is: A multimodal image fusion method for bird scenes based on the combination of frequency domain and spatial domain includes the following steps: S1. Construct a multi-source heterogeneous dataset of flying bird scenes; S2. Construct a dual-light image fusion network model that integrates frequency domain and spatial domain; S3, inputting the visible light image and the infrared image into the visible light branch neural network structure and the infrared branch neural network structure in the dual-light image fusion network model respectively, and outputting a visible light fusion feature map and an infrared fusion feature map respectively; S4. Adopting the cross-modal attention mechanism to adaptively fuse the visible light fusion feature map and the infrared fusion feature map to obtain a multimodal fusion image; S5. Calculate the image fusion loss function of the bi-optical image fusion network model; S6. Evaluate the fusion quality of multimodal fusion images based on multi-source heterogeneous datasets.

[0006] Furthermore, in S2, the dual-light image fusion network model includes a visible light branch neural network structure and an infrared branch neural network structure with the same network structure.

[0007] Furthermore, in S3, the visible light image is input into the visible light branch neural network structure in the dual light image fusion network model, and a visible light fusion feature map is output, which specifically includes the following sub-steps: S31, performing Laplacian pyramid residual feature extraction and frequency domain multi-scale feature extraction on the visible light image, respectively, to obtain a multi-resolution feature stream output by the Laplacian pyramid unit and a multi-scale frequency domain deep feature stream; S32. Perform step-by-step feature fusion on the multi-resolution feature stream and the multi-scale frequency domain deep feature stream output by the corresponding Laplacian pyramid to obtain a visible light fusion feature map.

[0008] Furthermore, in S31, performing Laplacian pyramid residual feature extraction on the visible light image includes the following steps: A11, the input layer of the Laplacian pyramid accepts the aligned visible light image stream (3, H, W), where H is the height of the image and W is the width of the image; A12, the first stage of encoding, converts the visible light image stream (3, H, W) passes through a downsampling convolution module with a stride of 1 and upsampling normalization module Generating primary feature streams (32, H, W); A13, the second stage coding to the fifth stage coding, the primary feature flow (32, H, W) is sequentially passed through four downsampling convolution modules with a stride of 2 and three upsampling normalization modules , generating deep feature flows step by step (32, H / 2, W / 2), deep feature flow (32, H / 4, W / 4), deep feature flow (32, H / 8, W / 8), deep feature flow (32, H / 16, W / 16); A14. Use the Laplace pyramid residual formula to calculate the high-frequency detail features of the visible light image step by step: Where, For the x Multi-resolution feature stream output by the layer Laplacian pyramid unit; For the x Feature flow of the layer; Indicates that Bicubic upsampling interpolation is performed on the feature stream of the x+1th layer to restore the feature map resolution.

[0009] Furthermore, in S31, frequency domain multi-scale feature extraction is performed on the visible light image, including the following steps: A21, the input layer uses wavelet transform convolution layer Generate initial frequency domain feature flow from visible light image (32, H, W); A22, using three-level wavelet transform convolution layer Initial frequency domain feature flow (32, H, W) performs wavelet transform convolution processing: Where, For the x Output frequency domain deep feature flow at each scale; For the x Input frequency domain feature stream of scales; i is the total number of feature maps contained in the input frequency feature stream; for i Wavelet transform kernel function in the input feature map; for i The corresponding learnable convolution kernel weight matrix in the input feature map; is the convolution operation.

[0010] Furthermore, in said S32, a visible light fusion feature map is obtained ,include: The first x The output frequency domain deep feature flow at each scale Hedi x The output frequency domain deep feature flow at each scale Perform channel dimension stitching: Where, is the fusion feature flow; For splicing operation; It is the visible light fusion feature map.

[0011] Furthermore, the S4 includes the following sub-steps: S41, perform feature channel decomposition on the visible light fusion feature map and the infrared fusion feature map respectively, and obtain the visible light n Single channel feature sub-map and infrared n Single-channel feature subgraph; S42. Perform one-dimensional discrete cosine transform on each single-channel feature subgraph: Where, is the result of performing one-dimensional discrete cosine transform on each single-channel sub-feature map; is the one-dimensional discrete cosine transform; is a single-channel feature subgraph; For the i The channel feature sub-graph is located in the spatial position k The value at is the DCT i Basis functions at positions k The weight value of is the spatial length of the feature subgraph; S43, stack the low-frequency components of each channel to obtain the frequency domain feature tensor , frequency domain feature tensor Generate channel attention weight vector through fully connected layer ; S44, the channel attention weight vector Multiply the channel dimension with the fusion feature map to obtain the frequency domain enhanced feature map: Where, Enhanced feature map for visible light frequency domain; Enhanced feature map for infrared frequency domain; is the attention weight vector of the visible light channel; is the infrared channel attention weight vector; S45, respectively enhance the feature map of visible light frequency domain and infrared frequency domain enhanced feature map Perform multi-fine-grained aggregation processing and use convolutional layers to fuse multi-fine-grained features to generate a spatial attention weight map: Where, is the spatial attention weight map; is the Sigmoid activation function; Conv is the 3*3 convolution kernel operation; concat is the splicing operation; AvgPool is the average pooling operation; MaxPool is the maximum pooling operation; is the frequency domain enhancement feature map, which represents the visible light frequency domain enhancement feature map Or infrared frequency domain enhanced feature map ; S46. Combine the spatial attention weight map with the visible light frequency domain enhanced feature map and infrared frequency domain enhanced feature map Perform pixel-by-pixel multiplication: Where, It is the visible light branch feature map after frequency-spatial domain attention modulation; It is the infrared branch feature map after frequency-spatial domain attention modulation; is the visible light space attention weight map; is the infrared spatial attention weight map; S47. Calculate cross-modal fusion weights based on the learnable parameter matrix: Where, is the cross-modal fusion weight; A probability distribution is generated by weighting the concatenated feature map; is the learnable weight matrix; S48, based on cross-modal fusion weights , calculate the multimodal fusion image: Where, It is a multimodal fusion image.

[0012] Furthermore, in S5, the image fusion loss function of the bi-optical image fusion network model is: Where, is the image fusion loss function; is the gradient loss; For strength loss.

[0013] Furthermore, the gradient loss for: Strength loss for: Where, is the gradient amplitude of each pixel in the multimodal fusion image; is the gradient magnitude of each pixel in the original image; i , j Represents the rows and columns of the image; A loss function for evaluating the error between two images; It is the difference between the gradient magnitude of each pixel in the multimodal fusion image and the gradient magnitude of the corresponding pixel in the original image.

[0014] Furthermore, in S6, evaluating the fusion quality of the multimodal fusion image includes: Peak signal-to-noise ratio : Structural similarity index : Peak signal-to-noise ratio VIF : Where MAX is the maximum possible value of the image pixel; MSE is the mean square error of the image pixel; 、 are the mean values ​​of a and b in the image blocks, respectively. a and b are the reference image and the image to be compared, respectively. and are the variances of a and b in the image blocks respectively; is the covariance between a and b in the image block; C1 and C2 are constants; Information in the generated multimodal fusion image; is the information in the original image.

[0015] The multimodal image fusion method for bird scenes based on the combination of frequency domain and spatial domain provided by the present invention has the following beneficial effects: 1. This paper improves target detection accuracy in complex environments by constructing a multi-source heterogeneous dataset and an innovative network architecture. First, it integrates 7,500 multi-scene visible-light-infrared images from an internationally recognized public dataset, and combines them with 3,686 bi-optical data of single and flock birds collected by a self-built infrared pod. After alignment, it forms an RGB-T bi-optical dataset covering a variety of targets. Then, it designs a dual-branch neural network structure and uses a cross-modal attention mechanism to collaboratively extract visible light texture details and infrared thermal radiation features, generating a fused image with both spatial resolution and thermal target saliency. Finally, it evaluates the model's superiority through an indicator function.

[0016] 2. Compared with the traditional image direct addition fusion strategy, this invention improves the target detection accuracy (mAP) in complex lighting scenes by 14.3%, and the inference time is only increased by 1.2ms; 3. Compared with traditional fusion methods, this method improves the edge preservation index (EPI) by 31.2%, reduces the feature alignment error by 19.8%, and only increases the number of parameters by 1.7%; 4. The present invention effectively solves the problem of feature loss caused by modal differences in the existing technology through multi-scale residual feature extraction and frequency domain compensation mechanism, and improves the PSNR and SSIM indicators of the fused image by 18.7% and 22.3% respectively. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Schematic diagram of the MSRS fusion results of the public dataset.

[0018] Figure 2 Schematic diagram of the fusion results of the public dataset M3FD.

[0019] Figure 3 Schematic diagram of the fusion results of the public dataset LLVIP.

[0020] Figure 4 Schematic diagram of the fusion results of the bird scene dataset.

[0021] Figure 5 Schematic diagram of the dual-light image fusion network model framework of the present invention.

[0022] Figure 6 Schematic diagram of the wavelet transform convolution module of the present invention.

[0023] Figure 7 Schematic diagram of the cross-modal attention mechanism of the present invention.

[0024] Figure 8 This is a flow chart of the multimodal image fusion method of flying bird scenes based on the combination of frequency domain and spatial domain of the present invention. DETAILED DESCRIPTION

[0025] The specific embodiments of the present invention are described below to facilitate understanding of the present invention by those skilled in the art. However, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the appended claims, these changes are obvious, and all inventions and creations utilizing the concepts of the present invention are protected.

[0026] Example 1 The multimodal image fusion method of flying bird scenes based on the combination of frequency domain and spatial domain in this embodiment aims to achieve high-precision fusion of bird visible light images and infrared images, so as to solve the shortcomings of traditional multimodal image fusion such as incomplete information fusion, degraded fused image quality and limited feature extraction capability. This embodiment can significantly improve the accuracy and reliability of bird detection. Figure 8 , which specifically includes the following: Step S1: construct a multi-source heterogeneous dataset of flying bird scenes; refer to Figure 1 、 Figure 2 、 Figure 3 and Figure 4 To achieve the model's collaborative adaptability under different imaging devices and complex scenarios, this embodiment adopts a multi-source heterogeneous dataset joint training scheme: a cross-domain fusion of generalized training datasets (including three public datasets: MSRS, M3FD, and LLVIP) and a self-built binocular infrared pod dataset. The specific fusion method is as follows: ① Training set merging: 6,000 training samples from the public dataset were combined with 2,949 flying bird scene training samples from the self-built dataset, forming a multi-source training set of 8,949 samples. ② Test set retention: 1,500 test samples from the public dataset and 737 test samples from the self-built dataset were kept separate, respectively used to evaluate the performance of the trained model in standard scenarios and in actual binocular infrared pod scenarios.

[0027] Step S2: construct a dual-light image fusion network model that fuses the frequency domain and the spatial domain. The dual-light image fusion network model of this embodiment includes a visible light branch neural network structure and an infrared branch neural network structure with the same network structure.

[0028] refer to Figure 5 The core architecture of the dual-light image fusion network model of this embodiment achieves high-precision fusion through multi-scale frequency domain compensation and cross-modal feature interaction. The dual-light image fusion network model adopts a dual-input, single-output structure, receiving registered and aligned visible light and thermal infrared image pairs. It decomposes and extracts multi-level spatial features through a Laplacian pyramid structure, while introducing wavelet transform convolution (WTConv) to suppress high-frequency noise and feature loss in the frequency domain. Furthermore, a cross-modal Transformer cross-attention mechanism is deployed in the feature fusion stage to dynamically associate the texture details of the visible light image with the thermal radiation features of the thermal infrared image, and adaptively allocate weights to generate a fused image with both high resolution and target saliency. Furthermore, the dual-light image fusion network model adopts an unsupervised training paradigm, requiring only paired input dual-light images for end-to-end optimization, ultimately outputting a fusion result that meets detection requirements through a single output port.

[0029] Step S3: inputting the visible light image and the infrared image into the visible light branch neural network structure and the infrared branch neural network structure in the dual-light image fusion network model respectively, and outputting the visible light fusion feature map and the infrared fusion feature map respectively; It should be noted that, since the visible light branch neural network structure and the infrared branch neural network structure of this embodiment are exactly the same, this embodiment only describes the process of the visible light branch neural network structure; Specifically, the specific process of inputting the visible light image into the visible light branch neural network structure in the dual light image fusion network model is as follows: Step S31: performing Laplacian pyramid residual feature extraction and frequency domain multi-scale feature extraction on the visible light image, respectively, to obtain a multi-resolution feature stream output by the Laplacian pyramid unit and a multi-scale frequency domain deep feature stream; Among them, Laplacian pyramid residual feature extraction is performed on the visible light image. The encoding stage of the visible light image constructs a multi-resolution feature stream through a five-level downsampling operation, which specifically includes the following steps: Step A11: The input layer of the Laplacian pyramid accepts the aligned visible light image stream (3, H, W), where H is the height of the image and W is the width of the image; Step A12: First stage encoding, converting the visible light image stream (3, H, W) passes through a downsampling convolution module with a stride of 1 and upsampling normalization module Generating primary feature streams (32, H, W); Step A13, from the second stage encoding to the fifth stage encoding, the primary feature stream (32, H, W) is sequentially passed through four downsampling convolution modules with a stride of 2 and three upsampling normalization modules , generating deep feature flows step by step (32, H / 2, W / 2), deep feature flow (32, H / 4, W / 4), deep feature flow (32, H / 8, W / 8), deep feature flow (32, H / 16, W / 16); Step A14: Calculate the high-frequency detail features of the visible light image step by step using the Laplace pyramid residual formula: Where, For the x Multi-resolution feature stream output by the layer Laplacian pyramid unit; For the xFeature flow of the layer; Indicates that Bicubic upsampling interpolation is performed on the feature stream of the x+1th layer to restore the feature map resolution.

[0030] Perform frequency domain multi-scale feature extraction on visible light images, refer to Figure 6 , including the following steps: Step A21: The input layer uses wavelet transform convolution layer Generate initial frequency domain feature flow from visible light image (32, H, W); Step A22: Use three-level wavelet transform convolution layer Initial frequency domain feature flow (32, H, W) is processed by wavelet transform convolution to obtain multi-scale frequency domain deep feature flow (32, H / 2, W / 2), (32, H / 4, W / 4), (32, H / 8, W / 8); Where, For the x Output frequency domain deep feature flow at each scale; For the x Input frequency domain feature stream of scales; i is the total number of feature maps contained in the input frequency feature stream; for i Wavelet transform kernel function in the input feature map; for i The corresponding learnable convolution kernel weight matrix in the input feature map; is the convolution operation.

[0031] Step S32: performing step-by-step feature fusion on the multi-resolution feature stream and the multi-scale frequency domain deep feature stream output by the corresponding Laplacian pyramid to obtain a visible light fusion feature map, which specifically includes: The first x The output frequency domain deep feature flow at each scale Hedi x The output frequency domain deep feature flow at each scale Perform channel dimension stitching: Where, is the fusion feature flow; For splicing operation; It is the visible light fusion feature map.

[0032] This embodiment uses the inverse wavelet transform layer ( Restore high-frequency detail information by dynamically weighting visible light texture features and thermal infrared thermal radiation features; upsample step by step based on the Laplace pyramid reconstruction formula, and finally output a high-resolution fused image, namely the fused feature stream, whose spatial resolution is consistent with the input image.

[0033] In another branch, the infrared image is input into the infrared branch neural network structure, and the output is: Where, is the infrared fusion feature map, It is the infrared fusion feature flow.

[0034] Step S4: Use cross-modal attention mechanism to perform visible light fusion feature map and infrared fusion feature map Adaptive fusion is performed to obtain a multimodal fusion image, which specifically includes the following steps: Step S41, feature channel decomposition; The visible light fusion feature map and the infrared fusion feature map are decomposed into feature channels respectively to obtain the visible light n Single channel feature sub-map and infrared n Single-channel feature subgraph; Step S42: extracting frequency domain components; Perform one-dimensional discrete cosine transform (DCT) on each single-channel feature subgraph and extract its low-frequency component as the frequency domain feature basis: Where, is the result of performing one-dimensional discrete cosine transform (DCT) on each single-channel sub-feature map; is the one-dimensional discrete cosine transform; is a single-channel feature subgraph; For the i The channel feature sub-graph is located in the spatial position k The value at is the DCT i Basis functions at positions k The weight value of is the spatial length of the feature subgraph; Step S43: generating frequency domain weights; The low-frequency components of each channel are stacked to obtain the frequency domain feature tensor : Frequency domain feature tensor Generate channel attention weight vector through fully connected layer , ; Step S44: recalibrating channel features; The channel attention weight vector Multiply the channel dimension with the fusion feature map to obtain the frequency domain enhanced feature map: Where, Enhanced feature map for visible light frequency domain; Enhanced feature map for infrared frequency domain; is the attention weight vector of the visible light channel; is the infrared channel attention weight vector; Step S45: multi-granularity aggregation and spatial weight generation; Enhance the feature map of visible light frequency domain and infrared frequency domain enhanced feature map Perform multi-fine-grained aggregation processing and use convolutional layers to fuse multi-fine-grained features to generate a spatial attention weight map: Where, is the spatial attention weight map; is the Sigmoid activation function; Conv is the 3*3 convolution kernel operation; concat is the splicing operation; AvgPool is the average pooling operation; MaxPool is the maximum pooling operation; is the frequency domain enhancement feature map, which represents the visible light frequency domain enhancement feature map Or infrared frequency domain enhanced feature map ; Step S46: spatial feature modulation; The spatial attention weight map is respectively combined with the visible light frequency domain enhanced feature map and infrared frequency domain enhanced feature map Perform pixel-by-pixel multiplication: Where, It is the visible light branch feature map after frequency-spatial domain attention modulation; It is the infrared branch feature map after frequency-spatial domain attention modulation; is the visible light space attention weight map; is the infrared spatial attention weight map; Step S47: dynamic weight allocation; Calculate cross-modal fusion weights based on the learnable parameter matrix: Where, is the cross-modal fusion weight; is a probability distribution generated by weighting the concatenated feature map, where each element value represents the fusion weight of a certain modality feature; is a learnable weight matrix used to map concatenated features to the weight space; Step S48: Based on cross-modal fusion weights , calculate the multimodal fusion image: Where, It is a multimodal fusion image.

[0035] Step S5: To accurately capture edge structure information and texture features of the visible light and infrared images, this embodiment proposes an image fusion loss function consisting of two parts: intensity loss and gradient loss. This loss function aims to simultaneously optimize the brightness information and edge structure of the fused image, achieving a balance between brightness consistency and detail preservation in the fused image.

[0036] Loss of strength; The intensity loss is a smooth L1 loss function that measures the difference in pixel intensity between the multimodal fusion image and the visible light image and infrared image: in, l is the error between the predicted value and the true value (i.e. ,in, is the predicted value, is the true value).

[0037] Gradient loss; By using the Sobel operator to extract the gradient information of each image, the difference in gradient information between the multimodal fusion image and the visible light image and infrared image is calculated. The convolution kernel is fixed in the horizontal and vertical directions. The horizontal convolution kernel (Kx) is as follows: The vertical convolution kernel (Ky) is: The formula for calculating the horizontal gradient (Gx) is: The vertical gradient (Gx) is calculated as: in, Indicates that the image is at position Pixel value of and They are convolution kernels and In position The weight of .

[0038] By calculating the horizontal and vertical gradients, we can get the gradient magnitude G and gradient direction of each pixel: This gives the representation of the gradient loss: in, Represents the gradient amplitude calculated by the gradient calculation formula; is the gradient magnitude of each pixel in the original image; i , j Represents the rows and columns of the image; It is a common loss function used to evaluate the error between two images.

[0039] The image fusion loss function of the bi-optical image fusion network model is: Where, is the image fusion loss function; is the gradient loss; For strength loss.

[0040] Step S6: Evaluate the fusion quality of the multimodal fusion image based on the multi-source heterogeneous dataset. Specifically, this embodiment uses the structural similarity index (SSIM), visual information fidelity (VIF), and peak signal-to-noise ratio (PSNR) as key indicators for evaluating image fusion quality. Peak signal-to-noise ratio : Structural similarity index : Peak signal-to-noise ratio VIF : Where MAX is the maximum possible value of the image pixel; MSE is the mean square error of the image pixel; 、 are the mean values ​​of a and b in the image blocks, respectively, which are used to reflect the average intensity information of the two images; a and b are the reference image and the image to be compared; and are the variances of a and b in the image blocks, respectively, which are used to reflect the contrast information of the image; is the covariance between a and b in the image block, reflecting the similarity between the two images; C1 and C2 are constants; Information in the generated multimodal fusion image; is the information in the original image.

[0041] This embodiment uses the above three evaluation indicators to verify the performance of the bi-optical image fusion network model on the test set, and saves the model with the best performance.

[0042] Although the specific embodiments of the invention are described in detail in conjunction with the accompanying drawings, this should not be construed as limiting the scope of protection of this patent. Within the scope described by the claims, various modifications and variations that can be made by those skilled in the art without creative work still fall within the scope of protection of this patent.

Claims

1. A multimodal image fusion method for flying bird scenes based on the combination of frequency domain and spatial domain, characterized in that: The following steps are involved: S1. Construct a multi-source heterogeneous dataset of flying bird scenes; S2. Construct a dual-light image fusion network model that integrates frequency domain and spatial domain; S3, inputting the visible light image and the infrared image into the visible light branch neural network structure and the infrared branch neural network structure in the dual-light image fusion network model respectively, and outputting a visible light fusion feature map and an infrared fusion feature map respectively; S4. Adopting the cross-modal attention mechanism to adaptively fuse the visible light fusion feature map and the infrared fusion feature map to obtain a multimodal fusion image; S5. Calculate the image fusion loss function of the bi-optical image fusion network model; S6. Evaluate the fusion quality of multimodal fusion images based on multi-source heterogeneous datasets.

2. The multimodal image fusion method for bird scenes based on the combination of frequency domain and spatial domain according to claim 1, characterized in that: In S2, the dual-light image fusion network model includes a visible light branch neural network structure and an infrared branch neural network structure with the same network structure.

3. The multimodal image fusion method for flying bird scenes based on the combination of frequency domain and spatial domain according to claim 1, characterized in that: In S3, the visible light image is input into the visible light branch neural network structure in the dual light image fusion network model, and a visible light fusion feature map is output, which specifically includes the following sub-steps: S31, performing Laplacian pyramid residual feature extraction and frequency domain multi-scale feature extraction on the visible light image, respectively, to obtain a multi-resolution feature stream output by the Laplacian pyramid unit and a multi-scale frequency domain deep feature stream; S32. Perform step-by-step feature fusion on the multi-resolution feature stream and the multi-scale frequency domain deep feature stream output by the corresponding Laplacian pyramid to obtain a visible light fusion feature map.

4. The method for multimodal image fusion of flying bird scenes based on the combination of frequency domain and spatial domain according to claim 3, characterized in that: In the step S31, Laplacian pyramid residual feature extraction is performed on the visible light image. The following steps are included: A11, the input layer of the Laplacian pyramid accepts the aligned visible light image stream (3, H, W), where H is the height of the image and W is the width of the image; A12, the first stage of encoding, converts the visible light image stream (3, H, W) passes through a downsampling convolution module with a stride of 1 and upsampling normalization module Generating primary feature streams (32, H, W); A13, the second stage coding to the fifth stage coding, the primary feature flow (32, H, W) is sequentially passed through four downsampling convolution modules with a stride of 2 and three upsampling normalization modules , generating deep feature flows step by step (32, H / 2, W / 2), deep feature flow (32, H / 4, W / 4), deep feature flow (32, H / 8, W / 8), deep feature flow (32, H / 16, W / 16); A14. Use the Laplace pyramid residual formula to calculate the high-frequency detail features of the visible light image step by step: ； Where, For the x Multi-resolution feature stream output by the layer Laplacian pyramid unit; For the x Feature flow of the layer; Indicates that Bicubic upsampling interpolation is performed on the feature stream of the x+1th layer to restore the feature map resolution.

5. The method for multimodal image fusion of flying bird scenes based on the combination of frequency domain and spatial domain according to claim 3, characterized in that: In said S31, frequency domain multi-scale feature extraction is performed on the visible light image. The following steps are included: A21, the input layer uses wavelet transform convolution layer Generate initial frequency domain feature flow from visible light image (32, H, W); A22, using three-level wavelet transform convolution layer Initial frequency domain feature flow (32, H, W) performs wavelet transform convolution processing: ； Where, For the x Output frequency domain deep feature flow at each scale; For the x Input frequency domain feature stream of scales; i is the total number of feature maps contained in the input frequency feature stream; for i Wavelet transform kernel function in the input feature map; for i The corresponding learnable convolution kernel weight matrix in the input feature map; is the convolution operation.

6. The method for multimodal image fusion of flying bird scenes based on the combination of frequency domain and spatial domain according to claim 3, characterized in that: In the step S32, a visible light fusion feature map is obtained. ,include: The first x The output frequency domain deep feature flow at each scale Hedi x The output frequency domain deep feature flow at each scale Perform channel dimension stitching: ； ； Where, is the fusion feature flow; For splicing operation; It is the visible light fusion feature map.

7. The method for multimodal image fusion of flying bird scenes based on the combination of frequency domain and spatial domain according to claim 3, characterized in that: The S4 comprises the following sub-steps: S41, perform feature channel decomposition on the visible light fusion feature map and the infrared fusion feature map respectively, and obtain the visible light n Single channel feature sub-map and infrared n Single-channel feature subgraphs; S42. Perform one-dimensional discrete cosine transform on each single-channel feature subgraph: ； Where, is the result of performing one-dimensional discrete cosine transform on each single-channel sub-feature map; is the one-dimensional discrete cosine transform; is a single-channel feature subgraph; For the i The channel feature sub-graph is located in the spatial position k The value at is the DCT i Basis functions at positions k The weight value of is the spatial length of the feature subgraph; S43, stack the low-frequency components of each channel to obtain the frequency domain feature tensor , frequency domain feature tensor Generate channel attention weight vector through fully connected layer ; S44, the channel attention weight vector Multiply the channel dimension with the fusion feature map to obtain the frequency domain enhanced feature map: ； ； Where, Enhanced feature map for visible light frequency domain; Enhanced feature map for infrared frequency domain; is the attention weight vector of the visible light channel; is the infrared channel attention weight vector; S45, respectively enhance the feature map of visible light frequency domain and infrared frequency domain enhanced feature map Perform multi-fine-grained aggregation processing and use convolutional layers to fuse multi-fine-grained features to generate a spatial attention weight map: ； Where, is the spatial attention weight map; is the Sigmoid activation function; Conv is the 3*3 convolution kernel operation; concat is the splicing operation; AvgPool is the average pooling operation; MaxPool is the maximum pooling operation; is the frequency domain enhancement feature map, which represents the visible light frequency domain enhancement feature map Or infrared frequency domain enhanced feature map ; S46. Combine the spatial attention weight map with the visible light frequency domain enhanced feature map and infrared frequency domain enhanced feature map Perform pixel-by-pixel multiplication: ； ； Where, It is the visible light branch feature map after frequency-spatial domain attention modulation; It is the infrared branch feature map after frequency-spatial domain attention modulation; is the visible light space attention weight map; is the infrared spatial attention weight map; S47. Calculate cross-modal fusion weights based on the learnable parameter matrix: ； Where, is the cross-modal fusion weight; A probability distribution is generated by weighting the concatenated feature map; is the learnable weight matrix; S48, based on cross-modal fusion weights , calculate the multimodal fusion image: ； Where, It is a multimodal fusion image.

8. The method for multimodal image fusion of flying bird scenes based on the combination of frequency domain and spatial domain according to claim 1, characterized in that: In S5, the image fusion loss function of the bi-optical image fusion network model is: ； Where, is the image fusion loss function; is the gradient loss; For strength loss.

9. The method for multimodal image fusion of flying bird scenes based on the combination of frequency domain and spatial domain according to claim 8, characterized in that: The gradient loss for: ； Strength loss for: ； Where, is the gradient amplitude of each pixel in the multimodal fusion image; is the gradient magnitude of each pixel in the original image; i , j Represents the rows and columns of the image; A loss function for evaluating the error between two images; It is the difference between the gradient magnitude of each pixel in the multimodal fusion image and the gradient magnitude of the corresponding pixel in the original image.

10. The method for multimodal image fusion of flying bird scenes based on the combination of frequency domain and spatial domain according to claim 1, characterized in that: In S6, evaluating the fusion quality of the multimodal fusion image includes: Peak signal-to-noise ratio : ； Structural similarity index : ； Peak signal-to-noise ratio VIF : ； Where MAX is the maximum possible value of the image pixel; MSE is the mean square error of the image pixel; 、 are the mean values ​​of a and b in the image blocks, respectively. a and b are the reference image and the image to be compared, respectively. and are the variances of a and b in the image blocks respectively; is the covariance between a and b in the image block; C1 and C2 are constants; Information in the generated multimodal fusion image; is the information in the original image.

Images