Multi-scale image fusion method based on multi-attention synergistic effect

Abstract

The invention discloses a multi-scale image fusion method based on a multi-attention synergistic effect, and belongs to the field of image processing. According to the method, multi-scale decomposition is carried out in combination with HWT-EFC, key features under different scales are captured and optimized, and the capability of capturing detail information is improved; in order to further enhance the consistency of the fused image and the source image, the method uses self-channel attention to preliminarily purify a feature set and remove redundant information and noise, then uses an intercommunication channel and space attention mechanism to generate an accurate cross-channel attention map, guides the feature fusion process, enhances the information expression of a key area, and improves the image fusion accuracy. The fusion result is more accurate and natural; according to the image block generation mode combined with the GAN, the scale and the offset of each image block are dynamically adjusted, so that the final fusion image is kept consistent on each detail level, and generation of block artifacts is avoided. The image fusion method provided by the invention realizes the fused image with high quality and rich details, and has wide applicability.

Description

Technical Field

[0001] This application relates to the field of image processing technologies, and in particular, to a multi-scale image fusion method based on the collaborative action of multiple attentions. Background Art

[0002] In the fields of digital image processing and computer vision, image fusion technology aims to integrate multiple images from different sensors or different time points into a comprehensive image to improve the quality, information richness, and robustness of the image. Traditional multi-image fusion methods mainly include pixel-level fusion methods and transform-domain fusion methods. Among them, pixel-level fusion methods directly operate at the pixel level of the image, which is simple to calculate and easy to implement, but it is difficult to distinguish the importance of different regions and is prone to detail loss. Transform-domain fusion methods process and then fuse images by converting them into the frequency domain or other transform domains. They can handle complex multi-scale information while maintaining image details, but the computational complexity is relatively high. In addition, traditional image fusion methods usually rely on manually designed features and algorithms, have limited adaptability to environmental changes, and cannot be dynamically adjusted to cope with diverse fusion challenges. When facing complex and changing environments, it is often difficult to maintain sufficient robustness.

[0003] In recent years, multi-image fusion algorithms based on deep learning (DL) have become a more popular processing method. Their core advantage lies in the ability to automatically learn and optimize multi-level feature representations by leveraging the powerful feature extraction and pattern recognition capabilities of deep neural networks. Different from traditional methods that rely on manual design, DL-based methods do not require manual intervention. They continuously improve the performance of the model using loss functions and backpropagation mechanisms. Through training on a large amount of data, they have higher stability when facing various noises, environmental changes, and different data types. In addition, DL-based methods can flexibly introduce existing mathematical models in traditional methods, not only inheriting the powerful feature extraction ability of deep learning but also leveraging the mature mathematical models and optimization techniques in traditional methods, thus realizing the complementary advantages of both. Therefore, DL-based multi-image fusion algorithms provide an effective solution for future multi-image fusion tasks.

[0004] To give full play to the advantages of the DL-based multi-image fusion algorithm, more scale and deeper feature analysis are crucial. In terms of multi-scale analysis, modules represented by pyramid pooling and dilated convolution capture image details and context information at different scales, improving the adaptability and robustness of the model to targets of different sizes. However, the downsampling methods introduced by multi-scale analysis are irreversible and cause information loss, resulting in obvious texture detail loss in the fused image compared with the source image. On the other hand, the acquisition of deeper features inevitably introduces some redundant and even negatively effective channel information. Due to the lack of specialized optimization, these adverse channel information participate in the feature fusion process with almost the same weight as the valuable feature information, affecting the expression of the real information in the source image. Summary of the Invention

[0005] This application provides a multi-scale image fusion method based on the synergistic effect of multi-attention, aiming to solve the common problems of information loss and negative effects of deep features in the multi-scale analysis process. The method proposed in this application uses the wavelet transform technology with reversibility and losslessness, combines CNN (Convolutional Neural Network) and multi-attention mechanism to achieve efficient and accurate multi-scale analysis and feature extraction.

[0006] The technical solution adopted in this application is: a multi-scale image fusion method based on the synergistic effect of multi-attention, the method includes:

[0007] Step 1, perform multi-scale image feature extraction processing on each frame of the image sequence of the input image pair to be fused by using a multi-scale wavelet analysis algorithm, and enhance the extracted image features through a convolutional network to obtain the image features of each frame of the image;

[0008] Step 2, perform feature fusion on the image features of each image pair to be fused in the image sequence based on the synergistic effect of multi-attention to obtain fused image features, where the spatial scale of the fused image features is the same as that of the image features of a single frame of the image;

[0009] Step 3, perform image feature splicing on the image features of each image pair to be fused in the image sequence according to the channel dimension to obtain global image features; then generate the global adaptive norm of the global image features of the specified dimension through the global adaptive norm algorithm, which is used to adjust the scale and bias of the local feature blocks of the fused image features;

[0010] Step 4, perform image block generation processing based on the global adaptive norm, and reconstruct the fused image of each image pair to be fused based on the generated image blocks.

[0011] A multi-scale image fusion method based on the collaborative action of multiple attentions proposed in this application extracts features of an image at different resolutions through multi-scale analysis, and then guides the fusion of features through the collaborative work of multiple attentions, so as to accurately capture and fuse the features of the input image. Finally, a high-quality fused image is reconstructed reversely using the optimized comprehensive feature map, significantly improving the performance of the output image in aspects such as detail retention, contrast enhancement, and visual effect improvement.

[0012] Further, in step 1, the feature extraction network for obtaining the image features of each frame of image is set as:

[0013] This feature extraction network includes feature extraction blocks based on Haar wavelet transform, convolution blocks, and upsampling layers based on inverse Haar wavelet transform; where

[0014] wherein, each convolution block sequentially includes a convolution operation, a batch normalization operation, and a ReLU activation operation;

[0015] The feature extraction blocks based on Haar wavelet transform form -level feature extraction structures in a stacked structure. Each feature extraction block based on Haar wavelet transform includes an HWT module for extracting the low-frequency feature F l and the high-frequency feature F h of a single-frame image based on Haar wavelet transform, and after feature splicing of the low-frequency feature F l and the high-frequency feature F h is sent into a convolution block to obtain the output feature map of this feature extraction block;

[0016] The feature extraction network obtains the initial extraction feature F 0 of a single-frame image through a convolution block, and then sends F 0 into the first-level feature extraction block to obtain the first-level extraction feature F 1 based on the output feature map of the feature extraction block. Then, the extraction feature of the current level is sequentially sent into the feature extraction blocks of the next level, and the extraction feature corresponding to the level is obtained based on its output feature map;

[0017] Taking the extraction feature of the th level as the output feature of the th level Starting from the Starting from level 0, the output features of the current level are upsampled through the upsampling layer, and then feature concatenation is performed with the extracted features of the previous level through the concatenation layer. The result of feature concatenation is passed through a convolutional block to obtain the output features of the previous level. Finally, the output features obtained based on the initial extracted features F 0 are used as the image features of each frame of the image.

[0018] Furthermore, the HWT module is specifically set as follows: The input image of the HWT module is convolved through a 2D convolutional layer, and then four components: A, H, V, and D are extracted through the Haar wavelet transform processing layer; then the A component passes through a 2D convolutional layer to obtain the low-frequency feature F l , and after concatenating the three components of H, V, and D and then passing through a 2D convolutional layer, the high-frequency feature F h ;

[0019] Among them, the A component is the low-frequency component of the Haar wavelet transform; H, V, and D are the high-frequency horizontal, vertical, and diagonal components of the Haar wavelet transform respectively.

[0020] Correspondingly, the upsampling layer adopted in this application is the inverse operation of the HWT module.

[0021] That is, in this application, the feature extraction network includes performing multiple wavelet transforms on the input image, decomposing the image into sub-band images at different resolutions in a reversible manner to obtain multi-scale image features, and using a convolutional neural network to perform deep processing and optimization on the features, thereby significantly reducing the information loss caused by resolution transformation sampling and completing the extraction of real and delicate image features. At the same time, by combining wavelet transform and convolutional neural network, dynamic optimization of the high-frequency and low-frequency features of the image is realized, and the effectiveness of feature representation is enhanced.

[0022] Furthermore, step 2 specifically includes:

[0023] For the image features of each pair of images to be fused, it is defined as The subscripts a and b are used to distinguish the two images of the pair of images to be fused;

[0024] The image features are respectively subjected to average pooling and max pooling processing, and then all the pooling processing results are concatenated to obtain the aggregated spatial feature S c ;

[0025] Taking the aggregated spatial feature S c as the input of the one-dimensional convolution, two one-dimensional convolutions are used to extract the self-channel weights of different image features and the Softmax function is used to re-distribute the internal weights of respectively to obtain the self-channel weights and and according to obtain the corresponding preliminary processing feature F κ ′ , where κ ∈ {a, b};

[0026] Perform average pooling and max pooling on each preliminary processing feature F k ′ respectively, and then concatenate all the pooling results to obtain the aggregated feature C s , and then use C s as the input of two-dimensional convolution, and extract the spatial weights of different image features through two two-dimensional convolutions and obtain the cross-space attention through the softmax function and the cross-channel attention

[0027] Finally, based on the cross-space attention, cross-channel attention and preliminary processing features, obtain the fused image features:

[0028]

[0029] where, F fusrd represents the fused image feature of the current image pair to be fused.

[0030] In this application, through the collaborative work of multiple attentions, the negative components in the features are gradually eliminated, and the expression of the real information in the source image is strengthened to ensure the consistency between the generated image and the real image; at the same time, the feature set is initially purified through the self-channel attention mechanism, and then combined with the cross-channel and spatial attention mechanisms to generate an accurate cross-feature attention map to guide the fusion, effectively improving the realism and visual quality of the fused image.

[0031] Further, in step 3, the global adaptive norm of the global image feature with a specified dimension generated by the global adaptive norm algorithm is specifically: gradually reduce the image resolution of the global image feature. When the global image feature is degraded to the preset image resolution, then convert it into a one-dimensional vector form, and then adjust the dimension of this one-dimensional vector to the target dimension through a multi-layer perceptron to obtain the global adaptive norm.

[0032] That is, in the present application, the global adaptive norm algorithm can be implemented through a neural network model. The input of the model can be set as the image features of each pair of images to be fused in the image sequence. Through a concatenation operation, the inputs are concatenated into global image features along the channel dimension. Then, the global image features are successively processed by an adjustment structure based on a convolutional network for reducing the feature resolution to a set image resolution in a progressive manner. Then, they are converted into a one-dimensional vector form. Finally, a multi-layer perceptron is used to adjust the dimension of the one-dimensional vector to the target dimension, and the global adaptive norm is output. Preferably, a basic adjustment unit of the adjustment structure can be composed of a bottleneck structure and an HWT module, and the adjustment structure for reducing the feature resolution in the present application is obtained through a stacked structure of several such basic adjustment units, that is, through the iterative optimization of several basic adjustment units to gradually reduce the spatial dimension of the feature map.

[0033] Furthermore, the global adaptive norm includes 6 elements: α1, α2, β1, β2, γ1, and γ2. These 6 elements are the output of the multi-layer perceptron, and the dimension of each element is adjusted to the target dimension through the multi-layer perceptron. Among them, α1, α2, γ1, and γ2 are used to adjust the scale of the local feature blocks, and β1 and β2 are used to adjust the bias of the local feature blocks.

[0034] In the present application, the scale and bias of each image block can be accurately adjusted based on the obtained global adaptive norm, so as to ensure the continuity and visual consistency between the image blocks. At the same time, the global features are compressed by combining a residual network and a multi-layer perceptron (MLP) to obtain an adaptive norm to guide the generation of image blocks, so as to significantly improve the reliability and robustness when processing complex images.

[0035] Furthermore, step 4 specifically includes:

[0036] The fused image features are segmented according to a preset image block size to obtain a number of non-overlapping local feature blocks; a neural network is used to encode each local feature block to obtain the encoded data in one-dimensional form of each local feature block; and the encoded one-dimensional encoded data of each segment is used as a token, and the dimensions of all tokens are adjusted so that the adjusted dimensions are consistent with the dimension of the global adaptive norm; then, learnable position information is added to each token and all tokens are concatenated.

[0037] The concatenated tokens are input into a first imaging stack structure composed of a number of first imaging processing modules. Among them, the first imaging processing module extracts imaging features through a window multi-head self-attention network module to obtain the context information between local feature blocks, and adjusts the scale and bias of the local feature blocks corresponding to each token through global adaptive normalization; the first imaging processing module then re-encodes the tokens through a re-encoding module to obtain the first output tokens; that is, the first imaging processing module includes a window multi-head self-attention network module and a re-encoding module connected in sequence;

[0038] The first output tokens of the last first imaging processing module of the first imaging stack structure are input into a second imaging stack structure composed of a number of second shaping processing modules. Among them, the stacking numbers of the first imaging processing module and the second imaging processing module are the same; the second imaging processing module extracts imaging features through a sliding window self-attention network module, and adjusts the scale and bias of the local feature blocks corresponding to each token through global adaptive normalization; the second imaging processing module then re-encodes the tokens through a re-encoding module to obtain the second output tokens; that is, the second imaging processing module includes a sliding window self-attention network module and a re-encoding module connected in sequence;

[0039] Based on the second output tokens of the last second imaging processing module, the corresponding local feature blocks are reconstructed and spliced to generate the fused image of each pair of images to be fused, thereby outputting the corresponding fused image sequence.

[0040] Further, the window multi-head self-attention network module of the first imaging processing module sequentially includes: a layer normalization layer, a scale and bias transformation layer, a window multi-head self-attention calculation operation layer, and a scale transformation layer;

[0041] Among them, the input of the window multi-head self-attention network module of the first first imaging processing module is the concatenated tokens, and the input of the window multi-head self-attention network module of the subsequent imaging processing modules is the output of the previous imaging processing module (that is ); among them, the processing process of the window multi-head self-attention network module can be expressed as:

[0042] Z1 = α1 * W-MSA(γ1 * LN(Z in ) + β1) + Z in

[0043] Among them, Z1 represents the intermediate generated tokens in the first stage, W-MSA(·) represents the window multi-head self-attention calculation operation, LN(·) represents the layer normalization operation, and Z in represents the input of the window multi-head self-attention network module.

[0044] Further, the recoding module of the first imaging processing module successively includes a layer normalization layer, a scale and bias transformation layer, an MLP layer, and a scale transformation layer, and its recoding process can be expressed as:

[0045]

[0046] Among them, Z’1 represents the output of the window multi-head self-attention network module, and MLP(·) is a multi-layer perceptron, represents the token output in the first stage, that is, the first output token output by the first imaging processing module.

[0047] Further, the second imaging processing module is similar to the first imaging processing module, that is, the window multi-head self-attention calculation operation layer in the window multi-head self-attention network module of the first imaging processing module is replaced with a sliding window multi-head self-attention calculation operation layer to obtain a corresponding sliding window multi-head self-attention module.

[0048] Among them, the processing process of the sliding window multi-head self-attention module can be expressed as:

[0049]

[0050] Among them, Z2 represents the intermediate generated token in the second stage, SW-MSA(·) represents the sliding window multi-head self-attention calculation operation, and LN(·) represents the layer normalization operation;

[0051] The recoding module of the second imaging processing module successively includes a layer normalization layer, a scale and bias transformation layer, an MLP layer, and a scale transformation layer, and its recoding process can be expressed as:

[0052]

[0053] Among them, Z′2 represents the output of the sliding window multi-head self-attention network module, represents the token output in the second stage, that is, the second output token output by the second imaging processing module.

[0054] In this application, the fused features are divided into blocks according to a preset size, and then an appropriate overlapping area is introduced in each sub-block for optimizing local features, thereby significantly reducing the calculation amount while maintaining the image consistency, and improving the processing efficiency and image quality.

[0055] The technical solution provided by this application at least brings the following beneficial effects:

[0056] (1) Sufficient retention of information: In this application, by fully leveraging the lossless and reversible nature of wavelet transform, image information is encoded into the channel dimension to achieve lossless downsampling operations. On this basis, combined with the learnable ability of the Global Adaptive Norm algorithm (GAN), the high-frequency and low-frequency features of the image are dynamically optimized to intelligently adjust the mechanism for retaining useful information. Additionally, by introducing the multi-scale algorithm, the MWA algorithm (an efficient feature encoding method based on Haar wavelet transform) can extract image information simultaneously in multiple dimensions and use skip connections to prevent information loss, effectively capturing and preserving the multi-level features of the image, including macroscopic structures and microscopic details. This method ensures that rich and useful information is retained during the feature extraction process, and is particularly suitable for tasks that require fine retention of the texture details of the original image.

[0057] (2) High image consistency: Although the feature information after MWA processing is fully retained, there are still significant negative information that affects the quality of the fused image. In this application, by adopting the collaborative effect of multi-attention (i.e., the MAC fusion algorithm), negative information is suppressed synchronously in both the channel and spatial dimensions, while enhancing the expression of real information. Through this two-dimensional optimization mechanism, the image generated reversely based on the fused features can maintain a high degree of consistency with the source image. In addition, the introduction of the GAN method further ensures the internal consistency of the fused image and the continuity between image patches. Through the combined action of the above two, not only is the quality of the fused image significantly improved, but also a high degree of consistency and coherence with the source image is ensured.

[0058] (3) Reducing computational complexity: This application adopts a feature block processing and image generation scheme introducing GAN, which significantly reduces the computational complexity of high-resolution image generation while ensuring a high degree of consistency between image patches. Traditional image generation techniques usually directly convolve to generate high-resolution images or simply perform block processing on the image and then stitch them together. However, the former faces high computational complexity when processing large-sized images, and the latter is prone to discontinuity and visual defects between image patches. In contrast, the method proposed in this application optimizes the generation process of each image block through the GAN-guided block processing mechanism and sliding window technology, ensuring smooth transitions and consistency between image patches while reducing computational complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The above and / or additional aspects and advantages of this application will become apparent and be readily understood from the following description of the embodiments in conjunction with the drawings, where:

[0060] Figure 1 It is a schematic diagram of the processing process of a multi-scale image fusion method based on the collaborative effect of multi-attention provided for the implementation of this application.

[0061] Figure 2 It is a schematic diagram of the specific process of the MWA (Multi-scale wavelet analysis) algorithm.

[0062] Figure 3 It is a schematic diagram of the specific process of the MAC (Multi-attention coordination) fusion algorithm.

[0063] Figure 4 It is a schematic diagram of the specific process of the self-channel attention algorithm in MAC.

[0064] Figure 5 It is a schematic diagram of the specific process of the cross-channel-spatial attention algorithm in MAC.

[0065] Figure 6 It is the block processing of features.

[0066] Figure 7 It is a schematic diagram of the specific process of image generation.

[0067] Figure 8 It is the comparison of the signal intensity and local details of the fused image.

[0068] Figure 9 It is the fused image and its color distribution.

[0069] Figure 10 It is the simulation result of the fused image obtained from unconventional inputs. Detailed implementation manners

[0070] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this application will be described in detail and completely below in conjunction with the accompanying drawings in the embodiments of this application. Obviously, the embodiments described by referring to the accompanying drawings are exemplary and are intended to explain this application, rather than being construed as a limitation to this application.

[0071] The embodiments of this application provide a multi-scale image fusion method based on multi-attention coordination (Multi-attentioncoordination, MAC) to improve the performance of image fusion in complex scenarios while reducing the computational complexity. This method combines a wavelet transform and a multi-scale analysis scheme, a feature enhancement scheme based on multi-attention coordination, and an image block generation scheme based on a global self-adaptive norm. It can not only significantly improve the quality of image fusion under complex environmental conditions but also effectively reduce the computational complexity, and is applicable to various application scenarios that require high-quality image output.

[0072] A multi-scale image fusion method based on multi-attention coordination provided by the embodiments of this application mainly includes:

[0073] (1) Application of multi-scale wavelet analysis algorithm

[0074] The method proposed in the embodiments of this application uses the multi-scale wavelet analysis (MWA) algorithm to extract image features. Among them, the MWA algorithm is a Haar wavelet transform based efficient feature coding (HWT-EFC) method, which can encode image information completely into each channel, realizing lossless downsampling operation and feature extraction. In the iterative process of the MWA algorithm, the algorithm uses a series of wavelets to identify signals and capture the characteristics of signals at different scales and positions, so that the final high- and low-frequency features of the image can accurately reflect the real situation of the image. Because it can accurately extract and retain the key features of the image, it can more effectively integrate information from different sources when performing image fusion, improving the quality and visual effect of the fused image.

[0075] (2) MAC fusion algorithm

[0076] First of all, in the method proposed in the embodiments of this application, for the multi-image features obtained after integrating MWA, the self-channel attention module can be used to analyze the feature distribution inside each input image, identify and suppress the channel information that may have an adverse impact on subsequent processing, thereby initially purifying the feature set. On this basis, the cross-channel and spatial attention mechanisms are further introduced to promote the information exchange between different features and generate accurate attention maps to guide the fusion of features. This algorithm can effectively suppress negative components while retaining useful information, ensuring a high degree of consistency between the generated image and the source image.

[0077] (3) GAN extraction algorithm

[0078] For the integrated features, the embodiments of this application propose a progressive compression method to obtain a one-dimensional vector that can represent the entire image information as a global feature descriptor. The GAN (Global adaptive norm, GAN) extraction algorithm combines the residual structure, feature coding, and MLP to adjust the dimension of the vector, and generates the final one-dimensional GAN according to the information characteristics embedded in it to realize in-depth mining and accurate representation of the global information of the image.

[0079] (4) Block processing of features and image generation

[0080] In the embodiment of the present application, first, based on the features obtained after MAC fusion, the features are processed by block division. Then, combining GAN and the sliding window technique, the scale and bias adjustment of each block feature are guided, and high-quality fused image blocks are generated. This step effectively divides the large-resolution image generation task into multiple small image block generation tasks, significantly reducing the computational complexity. In addition, the introduced GAN and sliding window effectively ensure the smooth transition and consistency between the generated image blocks.

[0081] See Figure 1 , in one embodiment, a multi-scale image fusion method based on the collaborative effect of multiple attentions proposed in the embodiment of the present application includes:

[0082] Step 1, perform multi-scale image feature extraction processing on each frame of the image sequence of the input image pair to be fused by using a multi-scale wavelet analysis algorithm, and enhance the extracted image features through a convolutional network to obtain the image features of each frame of the image;

[0083] Step 2, perform feature fusion on the image features of each image pair to be fused in the image sequence based on the collaborative effect of multiple attentions (i.e., the MAC fusion algorithm) to obtain fused image features, where the spatial scale of the fused image features is the same as that of the image features of a single frame of the image;

[0084] Step 3, perform image feature splicing on the image features of each image pair to be fused in the image sequence according to the channel dimension to obtain global image features; then generate the global adaptive norm of the global image features of the specified dimension through the Global adaptive norm (GAN) algorithm, which is used to adjust the scale and bias of the local feature blocks of the fused image features;

[0085] Step 4, perform image block generation processing based on the global adaptive norm to generate the fused image of each image pair to be fused:

[0086] Perform block division on the fused image features according to the preset image block size to obtain a number of non-overlapping local feature blocks; use a neural network to encode each local feature block to obtain the encoded data in one-dimensional form of each local feature block; and take the encoded one-dimensional encoded data of each block as a token, adjust the dimensions of all tokens so that the adjusted dimensions are consistent with the dimensions of the global adaptive norm; then concatenate all tokens after adding learnable position information to each token;

[0087] The concatenated tokens are input into a first imaging stacking structure composed of a number of first imaging processing modules. Among them, the first imaging processing module extracts imaging features through a window multi-head self-attention network module to obtain the context information between local feature blocks, and adjusts the scale and bias of the local feature blocks corresponding to each token through global adaptive normalization; the first imaging processing module then re-encodes the tokens through a re-encoding module to obtain the first output tokens;

[0088] The first output tokens of the last first imaging processing module of the first imaging stacking structure are input into a second imaging stacking structure composed of a number of second shaping processing modules. Among them, the stacking numbers of the first imaging processing module and the second imaging processing module are the same; the second imaging processing module extracts imaging features through a sliding window self-attention network module, and adjusts the scale and bias of the local feature blocks corresponding to each token through global adaptive normalization; the second imaging processing module then re-encodes the tokens through a re-encoding module to obtain the second output tokens;

[0089] Based on the second output tokens of the last second imaging processing module, the corresponding local feature blocks are reconstructed and spliced to generate the fused image of each pair of images to be fused, thereby outputting the corresponding fused image sequence.

[0090] In one embodiment, step 1 specifically includes:

[0091] The MWA algorithm in the embodiments of the present application includes: the HWT-EFC method and the multi-scale feature extraction network (MFEN), and its processing process is as Figure 2 shown. The former is used to extract the high-frequency and low-frequency features of the image and reduce the image resolution, while the latter realizes high-quality deep feature extraction by combining HWT-EFC.

[0092] The HWT-EFC method uses the Haar wavelet transform layer to effectively reduce the spatial resolution of the feature map while retaining all information. The Haar wavelet transform is a widely recognized, compact, binary, orthogonal transform, and is widely used in the field of image processing due to its losslessness and reversibility when analyzing images. The wavelet basis function and the scaling function of the one-stage one-dimensional Haar transform can be defined as:

[0093]

[0094] Among them, φ j,k (x) can be expressed as k = 0, 1,..., 2 j-1. The parameters j and k represent the stage and order of the Haar basis function respectively. Furthermore, φ 0,0 is defined as:

[0095] φ 0,0 (x) = φ0(x) = u(x) - u(x - 1) (2)

[0096] where u(x) represents the step function. On this basis, the relationship between the Haar transform of the first stage and the Haar basis function of the zero stage can be expressed as:

[0097]

[0098] Through the above transformation, a signal with length L can be decomposed into two signals with length .

[0099] For an image, when the Haar wavelet transform is used separately in the horizontal and vertical directions of the image, four components will be generated, and the resolution of each component is half of the source image. Among these four components, one component contains low-frequency information and has a similar structure to the source image. The other three components contain high-frequency information and are used to capture the texture details of the image. Since both the high-frequency and low-frequency components are completely preserved, the source image can be perfectly restored when generating the image back according to these components. Assume the input feature i ∈ {u, o}. In the embodiments of this application, the input pair of images to be fused is a pair of underexposed images and overexposed images, where u represents the underexposed image and o represents the overexposed image. represents the real number field, H×W×C is the image dimension, where C represents the channel, and H×W represents the height and width of the image.

[0100] By using the Haar wavelet transform, four components as derived above will be obtained. They are: the low-frequency component (A), and the detail components in the high-frequency horizontal (H), vertical (V), and diagonal (D) directions. Their dimensions are all This means that when they are spliced together, the number of channels of the feature will become four times the original. By encoding the information in the spatial dimension into the channel dimension, the embodiments of this application achieve the downsampling operation of the image without losing any information. Finally, before splicing, CNN is used to purify the captured high-frequency and low-frequency features and capture the representative features among them. For the convenience of subsequent derivation, denote HWT-EFC as That is Figure 2 the efficient feature coding algorithm based on wavelet transform shown in

[0101] See Figure 2, in the embodiments of the present application, the efficient feature coding algorithm based on wavelet transform (abbreviated as HWT) is specifically as follows: after passing the image frame through a 2D convolutional layer (Conv2D), four components: A, H, V, and D are extracted through the Haar wavelet transform; then the A component passes through Conv2D to obtain the low-frequency feature F l , and after splicing the three components of H, V, and D and then passing through Conv2D to obtain the high-frequency feature F h .

[0102] Based on HWT-EFC, the embodiments of the present application further propose MFEN. Its detailed structure can also be obtained in Figure 2 . MFEM first obtains the deep features of the input image through convolution, that is, the initial extraction features i ∈ {u, o}. Where 0 represents the initial feature, which has the same length and width as the image. Then, by reducing the resolution of the features through LFEB (lossless feature coding block, such as HWT-EFC of the present application), the j-th level extraction feature can be obtained as:

[0103]

[0104] Among them, Conv(·) represents passing the features through CNN, batch normalization, and ReLU activation operations respectively. The maximum number of levels of j can also be adjusted based on actual applications. By continuously repeating this operation, and are respectively obtained. Then, in order to finally obtain features with the same dimension as the image, the output features are obtained through multiple upsampling operations (i.e., inverse wavelet transform):

[0105]

[0106] Among them, Concat(·) and IHWT(·) respectively represent the j'-th level output feature, concatenation operation, and inverse wavelet transform. Different from the independent downsampling and upsampling operations, the embodiments of the present application perform inverse wavelet transform on the high-level features to achieve reversible dimensional transformation in multi-scale analysis. At the same time, to enhance feature reuse and alleviate the problem of gradient disappearance, MFEM introduces skip connections to connect the feature maps in the contraction path with the corresponding levels in the expansion path. Through the above series of processing procedures, a feature extraction network more suitable for the detail retention task in multi-image fusion is finally obtained. In one embodiment, referring to Figure 2 , the feature extraction network for obtaining the image features of each frame of the input image sequence in the present application can be specifically set as: including feature extraction blocks based on Haar wavelet transform (set to 3 in this embodiment), convolution blocks and An upsampling layer; wherein each convolutional block sequentially includes a convolution operation, a batch normalization operation, and a ReLU activation operation; the feature extraction block based on the Haar wavelet transform includes an HWT module for extracting the low-frequency feature F of a single-frame image l and the high-frequency feature F h , and then through a splicing layer, the low-frequency feature F l and the high-frequency feature F h are subjected to feature splicing to obtain a spliced feature map and pass through a convolutional block to obtain the output feature map of the feature extraction block based on the Haar wavelet transform, that is. In the feature extraction network of the image features of the present application, first, the initial extraction feature F of a single-frame image is obtained through convolutional block 1 0 , and then F 0 is sent into the first-level feature extraction block based on the Haar wavelet transform, and the first-level extraction feature F 1 is obtained based on its output feature map. Then, F 1 is sent into the second-level feature extraction block based on the Haar wavelet transform, and the second-level extraction feature F 2 is obtained based on its output feature map, and the -level extraction features are obtained in sequence Regarding as the -level output feature Starting from the -level, the output feature of the current level is upsampled through an upsampling layer (i.e., the inverse operation of the HWT module), and then feature splicing is performed with the extraction feature of the previous level through a splicing layer, and the feature splicing result passes through a convolutional block to obtain the output feature of the previous level, and the final image feature F' 0 (i.e., the output feature of the 0th level) is obtained based on the extraction feature corresponding to the initial extraction feature.

[0107] In one embodiment, participating in Figure 3 , the processing flow of the MAC fusion algorithm of the present application includes: first, the self-channel attention mechanism is adopted to identify and suppress the channel information that may have an adverse impact on subsequent processing by analyzing the feature distribution inside each input image, thereby initially purifying the feature set. On this basis, the inter-channel and spatial attention mechanisms are further introduced to promote the information exchange between different features and generate accurate cross-channel attention maps. These attention maps work together with the self-channel attention mechanism to continuously eliminate the negative components in the features and at the same time strengthen the expression of the true color information in the source image.

[0108] Specifically, the MAC algorithm of the embodiments of the present application consists of two main functional modules, namely the self-channel attention module and the cross-channel and spatial attention module. First, the MAC algorithm analyzes the feature distribution within each input image through the self-channel attention module, identifies and suppresses the channel information that may have an adverse impact on subsequent processing, thereby initially purifying the feature set, and its processing process is as Figure 4 shown. In the branch of the self-channel attention module, the input features aggregate spatial information through global pooling across spatial dimensions. This aggregation process can be expressed as:

[0109]

[0110] where S c represents the aggregated spatial features, and Avg(·) and Max(·) represent average pooling and max pooling respectively. Then, the aggregated spatial features pass through two one-dimensional convolutions to obtain the self-channel attention weights through cross-feature information interaction. The two channel weights can be expressed as:

[0111]

[0112] where Conv1D1(·) and Conv1D2(·) both represent one-dimensional convolution. respectively represent the self-channel weights of different image features. To make the sum of the weights of each self-channel equal to 1, the Softmax function is used to and reallocate the intrinsic weights to obtain and Subsequently, the self-channel weights are multiplied by the input features, and a residual connection is introduced to obtain the preliminarily processed features. The output features can be expressed as:

[0113]

[0114] On this basis, a cross-channel and spatial attention mechanism is further introduced to promote information exchange between different features and generate an accurate attention map to guide feature fusion. Similar to the above method, when obtaining the cross-spatial attention, first, global pooling is performed on the preliminarily processed features in the channel dimension to aggregate the channel dimension information, as Figure 5 shown. This aggregation process can be expressed as:

[0115] C s = Concat(Avg(F′ u ), Max(F′ u ), Avg(F′ o ), Max(F′ o )) (9)

[0116] where Cs Represents the aggregated channel features. The aggregated channel features are passed to two 2D convolutions to determine the spatial attention maps of different images. The two spatial weights can be expressed as:

[0117]

[0118] where Conv2D1(·) and Conv2D2(·) both represent 2D convolutions. and represent the spatial weights of different image features respectively. Different from the sum of the intrinsic elements of self-channel attention being equal to 1, the cross-channel spatial attention is such that the sum of the corresponding elements between the two weights is equal to 1. This method can be expressed by using the softmax function as:

[0119]

[0120] where, and represent the cross-space attention finally used to guide the fusion of different input images respectively, and represent the cross-channel attention finally used to guide the fusion of different input images respectively.

[0121] Finally, the cross-channel and cross-space attentions are aggregated and multiplied by the pre-processed features to fuse the features of different images with high quality. The output can be expressed as:

[0122]

[0123] where F fused represents the finally fused features, that is, the fused image features. By the collaborative work of self-attention and cross-attention mechanisms, DCS-FM gradually eliminates the negative components in the features, suppressing the influence of redundant and negative-effect channel information on the finally generated image. At the same time, since the sum of the attention weights is equal to 1, while suppressing the negative components, the retention of useful information is achieved, ensuring a high degree of consistency between the generated image and the original image.

[0124] To fully exploit the global information of an image, an innovative GAN extraction algorithm is proposed in the embodiments of this application. By gradually reducing the image resolution, a one-dimensional vector representing the entire image information is finally formed as a global feature descriptor. This process not only preserves the key content of the image but also provides an efficient feature representation for subsequent processing steps. In the subsequent global-local feature fusion stage, the obtained adaptive norm mechanism serves as reference information and gradually interacts with local features through scale adjustment and bias correction strategies to guide the generation of the fused image. First, the features from different images are spliced and integrated, and then processed by the Bottleneck structure (bottleneck structure) and HWT-EFC. Among them, each Bottleneck is composed of two-branch convolutions, and the output can be expressed as: Then, the resolution of the output features is reduced by HWT-EFC, and its output result can Through multiple iterative optimizations, the spatial dimension of the feature map is gradually reduced.

[0125] When the feature map is degraded to the preset resolution, it is converted into a one-dimensional vector form. Finally, the dimension of the vector is further adjusted by a multi-layer perceptron, and the final one-dimensional adaptive norm is generated according to the embedded information characteristics.

[0126] On the other hand, this application introduces a sliding window mechanism and combines it with the GAN extraction algorithm to design an image block generation scheme, the structure of which is as Figure 7 shown, where the left dotted box is the first imaging processing module and the right dotted box is the second imaging processing module. The sliding window mechanism allows the model to focus on different local regions of the image, while the adaptive norm enhances the realism and visual quality of the final output image by finely adjusting these local features. Specifically, first, the input fused feature F fused is processed through feature chunking as shown in Figure 6 This processing divides the feature into multiple non-overlapping chunks. Each chunk is encoded and the encoded one-dimensional data is regarded as a "token", and each token contains all the information in the chunk. In the embodiments of this application, the size of each chunk is defined as 8×8×C fused = 64C fused . Where C fused is the fused feature F fusedChannel dimension. To facilitate the stretching and offset operations of the GAN on tokens, a linear encoding layer is introduced to make the tokens have the same dimension as the GAN, and its dimension size is denoted as the embedding size. Then, the encoded tokens are added with learnable position information, position embedding, to ensure that the network can perceive the relative position information of each token. At the end of the chunk processing, all the tokens are concatenated to prepare for the subsequent attention calculation and the integration of the GAN.

[0127] After chunk processing, image generation is divided into two stages. In the first stage, by partitioning the tokens into windows, only the multi-head self-attention within the windows is calculated to capture the context information between local feature chunks. At the same time, the global adaptive norm is fused into the tokens through scaling and offset after each step of operation. Finally, the tokens are re-encoded using an MLP as the input for the second stage. The second stage has a similar processing flow to the first stage, but instead of using window multi-head self-attention, sliding window attention is adopted to overcome the problem of no information interaction between different windows.

[0128] Specifically, the adaptive norm as the input contains six elements, specifically α1, α2, β1, β2, γ1, and where α1, α2, γ1, and γ2 are used to scale the tokens, and β1 and β2 are used to guide the offset of the tokens. The tokens are first processed by layer normalization, then fused with γ1 and β1 and the self-attention is calculated in the window-based multi-head self-attention (W-MSA). This process can be expressed as:

[0129] Z1 = α1 * W-MSA(γ1 * LN(Z in ) + β1) + Z in (13)

[0130] where, Z in , LN(·), W-MSA(·), and Z1 are the input, layer normalization operation, window multi-head self-attention calculation operation, and the tokens generated midway in the first stage, respectively. Then, an MLP is applied to re-encode Z1 to obtain the output of the first stage of GLP-FM Its calculation process can be expressed as:

[0131]

[0132] Among them, MLP(·) is a multi-layer perceptron. The second-stage processing is similar to the first stage, except that W-MSA is replaced by shifted window based multi-head self-attention (SW-MSA), and its process can be expressed as:

[0133]

[0134] Among them, Z2 and are the tokens generated in the middle of the second stage and the tokens finally output respectively. After multiple iterations (that is, Figure 7 the two imaging processing modules shown by the dotted boxes on the left and right in are stacked multiple times, and the stacking numbers of both are kept the same), the corresponding image patch splicing is reconstructed using the finally output tokens to generate the final fused image.

[0135] Embodiment

[0136] Considering that the image fusion task involves a wide range, in this embodiment, the performance of the proposed MAC-based multi-scale image fusion method is simulated and analyzed by taking the multi-exposure image fusion high-dynamic range image in the field of image fusion as an example. Regarding the simulation details: in this embodiment, the Bottleneck structure and the image patch generation module are stacked 101 times and 3 times respectively, and the embedding size is set to 64. In the image patch generation module, the size of the feature patch represented by each token is set to 8×8, the multi-head attention coefficient is set to 4, and the shifted window size is set to 16. In terms of network training, 440 groups of images with different exposures are selected for the training set, including various real-world scenes captured by seven consumer cameras. The images are presented only in RGB format, and the resolutions mainly range from 3000×2000 to 6000×4000 pixels. The image sequences of indoor scenes usually consist of 7 to 18 frames, while the image sequences of outdoor scenes consist of 3 to 5 frames, and each group of images contains a real reference image. To unify the input resolution, the resolution of the input images is adjusted to 512×512, and the Adam optimizer is used to optimize the parameters with a dynamic learning rate of 1e-4. The network is trained for 1000 epochs.

[0137] Next, this embodiment shows the performance of the fused image in terms of texture details, signal strength, and color distribution through the simulation results. In terms of texture details and signal performance, as Figure 8As shown, this embodiment benefits from the feature extraction algorithm of HWT-EFC and demonstrates excellent performance in capturing and preserving the texture details of the input image. By magnifying the details, it can be clearly observed that even in the case of significant texture detail loss in overexposed images, the method proposed in this application embodiment can still successfully capture the relevant information in underexposed images and effectively integrate it into the finally generated image. In addition, by comparing the signal intensities of the fused image and the real image, it can be found that this application ensures the richness of texture details in the final image while guaranteeing authenticity, demonstrating its superior performance in image fusion. This ability makes this application have important practical value and broad application prospects in application scenarios that require fine texture reproduction, such as medical imaging, remote sensing image processing, and high-definition video processing. Subsequently, the performance of the fused image method proposed in this application in terms of color distribution was simulated, and the results are as Figure 9 shown. Through the combined action of the multi-channel attention strategy, the expression of negative channel information is effectively suppressed, making the color distribution of the fused image highly approximate to that of the real reference image.

[0138] Finally, the generalization ability of this embodiment for different inputs was simulated, and the results are as Figure 10 shown. This embodiment uses five source images with different exposure values (EV) as inputs, including 2 underexposed images and 3 overexposed images. By combining underexposed and overexposed images, six different fusion results were generated. By observing the fused images, it can be obtained that this application does not show obvious overexposure and underexposure under the condition of various different exposure ratios as inputs, and the fused images all have rich and real color and detail information. The method proposed in this application embodiment can still effectively adapt and maintain stable processing effects and reliable output quality when dealing with unconventional inputs. This high degree of robustness not only improves the applicability and reliability of the method in various application scenarios, but also demonstrates its strong adaptability to different types of input data, further proving its practical value and broad application prospects in complex and changing environments.

[0139] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the application and are not intended to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A multi-scale image fusion method based on the collaborative action of multiple attentions, characterized in that, It includes the following steps: Step 1: Use the multi-scale wavelet analysis algorithm to perform multi-scale image feature extraction processing on each frame of the image sequence of the input image pair to be fused, and enhance the extracted image features through a convolutional network to obtain the image features of each frame of the image; Step 2: Based on the synergistic effect of multi-attention, perform feature fusion on the image features of each image pair to be fused in the image sequence to obtain fused image features, where the spatial scale of the fused image features is the same as that of the image features of a single frame of the image; Step 3: Perform image feature splicing on the image features of each image pair to be fused in the image sequence according to the channel dimension to obtain global image features; then generate the global adaptive norm of the global image features of the specified dimension through the global adaptive norm algorithm, which is used to adjust the scale and bias of the local feature blocks of the fused image features; Step 4: Perform image block generation processing based on the global adaptive norm, and reconstruct the fused image of each image pair to be fused based on the generated image blocks.

2. The method according to claim 1, characterized in that, In Step 1, the feature extraction network for obtaining the image features of each frame of the image is set as: The feature extraction network includes feature extraction blocks based on Haar wavelet transform, convolution blocks, and upsampling layers based on inverse Haar wavelet transform; among them, Among them, each convolutional block sequentially includes a convolution operation, a batch normalization operation, and a ReLU activation operation; A feature extraction block based on Haar wavelet transform is constructed in a stacked structure Level feature extraction structure. Each feature extraction block based on Haar wavelet transform includes an HWT module for extracting the low-frequency feature F and high-frequency feature F of a single-frame image, and after feature splicing of the low-frequency feature F and high-frequency feature F, it is sent to a convolutional block to obtain the output feature map of the feature extraction block; l And high-frequency feature F i Of the HWT module, and for the low-frequency feature F l And high-frequency feature F h After feature splicing, it is sent to a convolutional block to obtain the output feature map of the feature extraction block; The feature extraction network obtains the initial extracted feature F of a single-frame image through a convolutional block 0 , and then sends F 0 to the first-level feature extraction block, and obtains the first-level extracted feature F based on the output feature map of the feature extraction block 1 , and then sequentially sends the extracted feature of the current level to the next-level feature extraction block, and obtains the corresponding-level extracted feature based on its output feature map; Regard the extracted features of the th level as the output features of the th level Output features Starting from the th level, upsample the output features of the current level through an upsampling layer, then perform feature concatenation with the extracted features of the previous level through a concatenation layer, and obtain the output features of the previous level by passing the feature concatenation result through a convolutional block. Finally, use the output features obtained based on the initial extracted feature F 0 as the image features of each frame of the image.

3. The method according to claim 2, wherein The HWT module is specifically configured as follows: The input image of the HWT module is convolved through a 2D convolutional layer, and then four components: A, H, V, and D are extracted through a Haar wavelet transform processing layer; then the A component passes through a 2D convolutional layer to obtain low-frequency feature F l , and after concatenating the three components of H, V, and D and then passing through a 2D convolutional layer to obtain high-frequency feature F h ; Among them, the A component is the low-frequency component of the Haar wavelet transform; H, V, and D are the components in the horizontal, vertical, and diagonal directions of the high frequency of the Haar wavelet transform, respectively.

4. The method according to claim 3, wherein The upsampling layer is the inverse operation of the HWT module.

5. The method according to claim 1, wherein Step 2 specifically includes: The image features of each pair of images to be fused are defined as The subscripts a and b are used to distinguish the two images in the pair of images to be fused; Perform average pooling and max pooling on the image features respectively and then concatenate all the pooling results to obtain the aggregated spatial feature S c ; With the aggregated spatial feature S c as the input of the one-dimensional convolution, the self-channel weights of different image features are extracted through two one-dimensional convolutions respectively and the Softmax function is used to re-distribute the internal weights of and to obtain the self-channel weights and the corresponding preliminary processed feature F κ ′ , where κ ∈ {a, b}; Perform average pooling and max pooling on each preliminary processing feature F k ′ respectively, and then concatenate all the pooling results to obtain the aggregated feature C s , and then use C s as the input of two-dimensional convolution, and extract the spatial weights of different image features through two two-dimensional convolutions and obtain the mutual space attention through the softmax function and the mutual channel attention Finally, obtain the fused image features based on cross-space attention, cross-channel attention, and the preliminarily processed features: Among them, F fused represents the fused image feature of the currently to-be-fused image pair.

6. The method according to claim 1, wherein In Step 3, the specific method for generating the global adaptive norm of the global image features of the specified dimension through the global adaptive norm algorithm is: gradually reduce the image resolution of the global image features. When the global image features are degraded to the preset image resolution, then convert them into a one-dimensional vector form, and then adjust the dimension of this one-dimensional vector to the target dimension through a multi-layer perceptron to obtain the global adaptive norm.

7. The method according to claim 1, wherein Step 4 specifically includes: Perform block processing on the fused image features according to the preset image block size to obtain a number of non-overlapping local feature blocks; use a neural network to encode each local feature block to obtain the encoded data in one-dimensional form of each local feature block; and take the encoded one-dimensional encoded data of each block as a token, adjust the dimensions of all tokens so that the adjusted dimensions are consistent with the dimensions of the global adaptive norm; then add learnable position information to each token and concatenate all tokens; Input the concatenated tokens into the first imaging stacking structure composed of several first imaging processing modules. Among them, the first imaging processing module extracts imaging features through the window multi-head self-attention network module, and adjusts the scale and bias of the local feature blocks corresponding to each token through the global adaptive norm; then re-encode the tokens through the re-encoding module to obtain the first output tokens; Input the first output token of the last first imaging processing module in the first imaging stack structure into the second imaging stack structure composed of several second shaping processing modules, where the stacking numbers of the first imaging processing modules and the second imaging processing modules are the same; the second imaging processing module extracts imaging features through a sliding window self-attention network module, and adjusts the scale and bias of the local feature blocks corresponding to each token through global adaptive norm adjustment; then re-encode the token through a re-encoding module to obtain a second output token. Reconstruct the corresponding local feature blocks based on the second output token of the last second imaging processing module for stitching to generate the fused image of each pair of images to be fused.

8. The method according to claim 7, wherein The window multi-head self-attention network module of the first imaging processing module sequentially includes: a layer normalization layer, a scale and bias transformation layer, a window multi-head self-attention calculation operation layer, and a scale transformation layer.

9. The method according to claim 7, wherein The re-encoding module of the first imaging processing module sequentially includes a layer normalization layer, a scale and bias transformation layer, an MLP layer, and a scale transformation layer.

10. The method according to claim 7, characterized in that The sliding window multi-head self-attention module of the second imaging processing module sequentially includes: a layer normalization layer, a scale and bias transformation layer, a sliding window multi-head self-attention calculation operation layer, and a scale transformation layer.

Images