Multi-modal image fusion method for spatial perception multi-scale feature learning

Abstract

The invention relates to the technical field of digital image processing, and discloses a multi-modal image fusion method for spatial perception multi-scale feature learning, which comprises the following steps of: processing a registered multi-modal image pair by adopting a hierarchical expansion convolution structure so as to obtain multi-scale features with different receptive field ranges in parallel; processing the multi-scale features by using a two-layer four-way recurrent neural network, and introducing an attention mechanism to integrate local and global feature information of the image to obtain spatial direction sensing features; by introducing an attention coordination mechanism and jump connection, shallow details and deep semantic features are fused, and a fused image is reconstructed and generated through convolution operation. According to the method, through parallel multi-scale feature extraction, global spatial information perception and cross-level feature compensation, the performance of the fusion image in the aspects of information fidelity, structural integrity and detail definition is effectively improved, and a fusion result with a prominent target and rich textures can be generated.

Description

Technical Field

[0001] This invention relates to the field of digital image processing technology, specifically to a multimodal image fusion method based on spatially perceptual multi-scale feature learning. Background Technology

[0002] This invention relates to the field of digital image processing technology, and in particular to a method for fusing multimodal images. Multimodal image fusion technology aims to integrate multiple images of the same scene acquired from different sensors to generate a single image that is richer in information and easier for human visual perception or machine analysis. In many application scenarios, such as night surveillance, target detection, and autonomous driving, the fusion of infrared and visible light images is a typical and crucial application.

[0003] Infrared imaging, by capturing the thermal radiation of objects, can effectively highlight targets concealed or camouflaged in complex backgrounds, and is largely unaffected by lighting conditions and weather changes. However, its drawbacks include typically low spatial resolution and a lack of texture details, making scene understanding difficult. Complementing this, visible light images offer high spatial resolution, providing rich background texture and detail information, conforming to human visual habits. However, visible light sensors struggle to function effectively under conditions of insufficient light, smoke, or severe weather, resulting in a significant deterioration in image quality. Therefore, effectively fusing the two to generate an image that both highlights thermal targets and preserves clear background texture has significant research value and broad application prospects.

[0004] In recent years, with the development of deep learning technology, image fusion methods based on convolutional neural networks have gradually become the mainstream of research, showing significant performance advantages compared to traditional methods such as multi-scale transformation. However, existing deep learning-based fusion methods still have some inherent limitations in practical applications.

[0005] On the one hand, most network models rely primarily on standard convolutional operations during feature extraction. Their fixed receptive field size makes it difficult to effectively adapt to the feature extraction needs of targets at different scales in images. This results in the inability to simultaneously and accurately capture macroscopic structural information and microscopic texture details when facing complex scenes. On the other hand, in terms of feature fusion strategies, many methods tend to use simple element-wise addition or channel concatenation for information integration. These fixed fusion rules lack the ability to dynamically evaluate the importance of information from different modalities of the image and cannot adaptively allocate feature weights according to the image content. When the source image contains noise or conflicting information, it can easily lead to the weakening of key information or the generation of artifacts.

[0006] Furthermore, due to the local connectivity of convolutional neural networks, their ability to model long-range dependencies in image features is limited, making it difficult to fully extract global contextual information from image content. This may affect the overall structural consistency of the fused image. Simultaneously, in deep network structures, as information is abstracted layer by layer during forward propagation, the rich shallow details in the source image inevitably decay. Without an effective compensation mechanism, the final reconstructed fused image often loses some texture details and lacks clarity.

[0007] Therefore, how to design a fusion method that can efficiently extract and integrate multi-scale features, make full use of spatial context information, and effectively preserve the details of the source image is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0008] To address the shortcomings of existing technologies, this invention provides a multimodal image fusion method based on spatially perceptual multi-scale feature learning. This method solves the problem that existing technologies suffer from blurred details and insufficient information complementarity in fused images due to limitations in feature extraction and integration mechanisms during the multimodal image fusion process.

[0009] To achieve the above objectives, the present invention provides a multimodal image fusion method for spatially perceptual multi-scale feature learning, comprising the following steps: processing the registered multimodal image pairs using a hierarchical dilated convolutional structure to obtain multi-scale features with different receptive field ranges; processing the multi-scale features using a two-layer four-way recurrent neural network to integrate the local and global feature information of the multimodal images to obtain spatial orientation perceptual features; and reconstructing the spatial orientation perceptual features through skip connections and convolution operations to generate a fused image.

[0010] In one optional implementation, the processing steps using a layered dilated convolutional structure specifically involve: processing the input image through three parallel convolutional paths with different dilation rates, and aggregating the features output from the three paths. Dilated convolution expands the receptive field of the convolutional kernel by inserting zero-value weights into the kernel without increasing the number of model parameters. The formula for calculating the receptive field of a single-layer convolution is as follows:

[0011] ;

[0012] in, The size of the receptive field of the current layer. The size of the receptive field of the previous layer. The size of the convolution kernel. This represents the expansion rate.

[0013] In one optional implementation, when processing the multi-scale features using the two-layer four-way recurrent neural network, an attention fusion mechanism is introduced to dynamically assign fusion weights to the features in the four spatial directions (up, down, left, and right). The attention fusion weights are obtained by performing a three-layer cascaded convolution operation on the multi-scale features and decomposing them into four independent weight maps. , , , .

[0014] In one optional implementation, in the two-layer four-way recurrent neural network, the first layer of the four-way recurrent neural network is used to propagate information along four spatial directions and extract spatially adjacent local features; the second layer of the four-way recurrent neural network receives the output of the first layer network and is used to extract contextual features with global perception capabilities.

[0015] In one optional implementation, the spatial orientation-aware features are first processed using a coordinated attention mechanism before reconstruction. This coordinated attention mechanism decouples the two-dimensional global pooling operation into two independent one-dimensional feature encoding processes along the vertical and horizontal directions, to collaboratively model inter-channel dependencies and spatial location information. For the input feature map... The encoding process in its horizontal and vertical directions is as follows:

[0016] ;

[0017] ;

[0018] in, For the first Input of each channel, and These are the outputs encoded along the horizontal and vertical directions, respectively. The two encoded feature maps are concatenated, convolved, and non-linearly activated to generate attention maps along the two spatial directions, which are then applied to the input feature map.

[0019] In one alternative implementation, the skip connection directly transmits the shallow features obtained during multi-scale feature extraction to the deep network, and integrates them with the high-level features of the spatial orientation perception features after processing by a coordinated attention mechanism.

[0020] In an optional implementation, the method further includes the step of training and optimizing the fusion network model using a composite loss function, wherein the composite loss function is a weighted average of pixel loss and detail loss, and its calculation formula is as follows:

[0021] ;

[0022] in, For compound loss, For pixel loss, For the loss of details, and These are the weighting coefficients.

[0023] In one alternative implementation, the pixel loss Used to constrain the consistency of pixel intensity distribution between the fused image and the source image, determined by infrared image pixel loss. and visible light image pixel loss The composition and calculation formula are as follows:

[0024] ;

[0025] ;

[0026] in, To merge images, Source infrared image, For the source visible light image, and These are the height and width of the image, respectively. This represents the L1 norm.

[0027] The loss of detail The gradient difference between the fused image and the source image is calculated to enhance the consistency of the fused image in terms of edge and texture information, which is achieved by reducing the detail loss of the infrared image. and visible light image detail loss The composition and calculation formula are as follows:

[0028] ;

[0029] ;

[0030] in, This is the Sobel gradient operator.

[0031] A second aspect of the present invention provides a multimodal image fusion system for spatially perceptive multi-scale feature learning, comprising:

[0032] The multi-scale feature extraction module is used to process the registered multimodal image pairs using a hierarchical dilated convolutional structure to obtain multi-scale features with different receptive field ranges.

[0033] The spatial orientation sensing module is used to process the multi-scale features using a two-layer four-way recurrent neural network to integrate the local and global feature information of the multimodal image and obtain spatial orientation sensing features.

[0034] The feature reconstruction module is used to reconstruct the spatial orientation-aware features through skip connections and convolution operations to generate a fused image.

[0035] This invention provides a multimodal image fusion method based on spatially perceptual multi-scale feature learning. It offers the following advantages:

[0036] 1. This invention, by employing a hierarchical dilated convolutional structure, can acquire feature information of multimodal images under different receptive fields in parallel, overcoming the limitations of traditional single receptive field in feature extraction. This allows the network to simultaneously consider both local details and macroscopic structure of the image, providing more comprehensive and richer source information for subsequent feature fusion, thereby improving the information fidelity of the fused image.

[0037] 2. This invention utilizes a two-layer four-way recurrent neural network to process multi-scale features, effectively capturing the long-range dependencies of features in the spatial dimension. This structure propagates and aggregates local and global contextual information from four spatial directions, enhancing the model's ability to understand the spatial layout of image content and helping to maintain the integrity of the target structure and the logical consistency of the scene in the fusion result.

[0038] 3. This invention achieves effective integration of high-level semantic features and shallow detail features by introducing a coordinated attention mechanism and skip connections in the feature reconstruction stage. The coordinated attention mechanism enables the network to adaptively focus on the salient features of each modality, while the skip connections directly supplement the texture details that may be attenuated in deep networks. The two work together to ensure that the final fused image can highlight key targets while retaining rich background details, thus improving the overall visual quality of the image. Attached Figure Description

[0039] Figure 1 This is a flowchart of the method of the present invention;

[0040] Figure 2 This is a system architecture diagram of the present invention. Detailed Implementation

[0041] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0042] Example:

[0043] Please see the appendix Figure 1 This invention provides a multimodal image fusion method based on spatially perceptive multi-scale feature learning, comprising the following steps:

[0044] S1: The registered multimodal image pairs are processed using a hierarchical dilated convolutional structure to obtain multi-scale features with different receptive field ranges;

[0045] In this embodiment, to address the technical problem that traditional convolutional networks, due to their use of single-size convolutional kernels, have limited feature extraction capabilities in complex scenes, this invention designs a hierarchical dilated convolutional structure in the initial stage of feature extraction. This structure aims to extract feature information at different spatial scales from the input registered multimodal image pairs in a parallel and efficient manner.

[0046] Specifically, for each input image, the structure sets up multiple parallel convolution processing paths. In a preferred embodiment, three parallel convolution paths are used. The key difference between these paths lies in the different dilation rates used for the convolution operations, thus allowing each path to naturally focus on a different range of image regions.

[0047] At least one path can use standard convolution, while the others use dilated convolution. Dilated convolution, also known as attenuated convolution, is implemented by strategically inserting zero values ​​between the weights of the convolution kernel. This design can effectively expand the actual receptive field of the convolution kernel without introducing additional computational parameters or significantly increasing the computational load.

[0048] The dilation rate is a key parameter controlling the extent of receptive field expansion. By configuring increasing dilation rates for different parallel paths, the network can simultaneously obtain multiple receptive fields of different sizes at the same layer. For example, one path can be set to standard convolution, while the other two paths can be set to larger dilation rates.

[0049] To more clearly characterize the effect of dilated convolution on expanding the receptive field, its single-layer effect can be described by the following formula:

[0050] ;

[0051] In this formula, This represents the size of the receptive field of the output feature map after this convolutional layer; This represents the size of the receptive field of the input feature map; Refers to the size of the convolution kernel; while This represents the set expansion rate. When the expansion rate... When this happens, the formula degenerates into the receptive field calculation of standard convolution, indicating that it is a special case of dilated convolution.

[0052] In this way, different convolutional paths can capture: high-frequency detail information such as local texture and edges of the image perceived by a smaller receptive field, and low-frequency structural information such as macroscopic contours and regional context of the image perceived by a larger receptive field.

[0053] After each parallel path completes the convolution operation and generates its own feature map, this invention aggregates these feature maps from different receptive fields. The aggregation operation can be performed using methods such as feature concatenation or element-wise addition, to fuse these feature mapping vectors containing complementary information into a unified and more comprehensive multi-scale feature set.

[0054] Ultimately, the aggregated features output by this hierarchical dilated convolutional structure achieve a comprehensive representation of the source image from local details to global structure. This comprehensive feature information provides rich and hierarchical input for subsequent spatial orientation sensing processing and the final image reconstruction step, and is the technical foundation for improving the quality of the final fused image.

[0055] S2: A two-layer four-way recurrent neural network is used to process multi-scale features to integrate local and global feature information of multimodal images and obtain spatial orientation perception features.

[0056] In this embodiment, after obtaining multi-scale features rich in information through the aforementioned hierarchical dilated convolutional structure, to effectively overcome the inherent limitations of traditional convolutional networks in mining deep spatial correlations of image features, especially long-range dependencies, this invention introduces a unique spatial orientation-aware processing scheme. The core of this scheme is to utilize a specially designed two-layer four-directional recurrent neural network structure to perform deep processing and information aggregation on multi-scale features.

[0057] Specifically, the scheme aims to comprehensively propagate and integrate feature information from the four basic spatial directions of top, bottom, left, and right by simulating how the human visual system perceives spatial information, thereby enabling the network to simultaneously capture the local fine structure and global contextual contours of an image.

[0058] In a preferred embodiment, to achieve adaptive weighting and selective enhancement of feature information from different directions, this scheme first introduces an attention fusion mechanism. This mechanism uses a weight generation network composed of several cascaded convolutional layers to analyze the input multi-scale features and adaptively learn to generate a set of shared attention fusion weights. This weight generation network receives multi-scale features as input, first compressing their channel number to an intermediate dimension (e.g., 16) through a convolutional layer, then processing them with a ReLU activation function, and finally expanding the channel number to 4 through another convolutional layer. This four-channel output is the attention fusion weight matrix. The matrix was then decomposed into independent weighted graphs corresponding to the four spatial directions: up, down, left, and right. , , , Since these weights are dynamically generated based on the input features, they can be flexibly adjusted according to changes in the image content.

[0059] Then, the processing flow proceeds to a two-layer four-way recurrent neural network:

[0060] The first layer of the four-way recurrent neural network primarily functions to extract and aggregate local features. It takes multi-scale features as input and independently propagates information cyclically along four directions: top-down (U), bottom-up (D), left-to-right (L), and right-to-left (R). Each recurrent neural network unit operates at a specific time step. Calculate its hidden state At that time, the current input will be taken into account. The hidden state of the previous time step This process aims to capture the close relationships between each pixel in the feature map and its spatial neighbors. Specifically, for any direction (e.g., top to bottom), the propagation process can be represented as:

[0061] ;

[0062] in It is a location Input features at the location, It is a location It is currently hidden from top to bottom. This represents a cyclic unit operation in the top-to-bottom direction. Similarly, there are corresponding cyclic propagation processes in the other three directions.

[0063] Crucially, after propagation in each direction is complete, the output features in each direction (e.g.) This will be compared with the previously generated corresponding directional attention weight map (e.g., features propagated from top to bottom). and ) Perform element-wise multiplication and weighted fusion. For example, for weighted features in the top-to-bottom direction. The calculation method is as follows:

[0064] ;

[0065] in This represents element-wise multiplication. This step enables directional filtering of local information, allowing the network to focus on important features in specific directions. Then, the weighted features from these four directions are concatenated or summed via channel concatenation to form the aggregated features of the first layer of the network. .

[0066] The second layer, a four-way recurrent neural network, shifts its core task to extracting global contextual features. This layer integrates the weighted and channel-integrated features from the first layer. As its input. Preferably, this layer reuses the attention fusion weights learned from the first layer of the recurrent neural network. , , , This weight-sharing mechanism ensures that the network follows a unified and consistent feature selection criterion when processing local and global information. The four-directional cyclic propagation process of the second layer is similar to that of the first layer, but its input is... The update of its hidden state also considers the current input and the hidden state of the previous step. Since the input of the second layer network is already a pre-aggregated wide-area feature, four-way information propagation is performed again on it, allowing information in the network to traverse a longer spatial distance, thereby effectively modeling global context dependencies. For example, the calculation method for propagation in the top-to-bottom direction is as follows:

[0067] ;

[0068] Finally, the four directional features output by the second-layer network are also weighted and fused with the shared attention fusion weights, and then channel concatenation or summation is performed to form the final aggregated features. .

[0069] Through the aforementioned two-layer stacked structure, this invention realizes a hierarchical spatial information extraction paradigm from local to global. Since each computational unit of the second layer network depends on the wide-area output of the first layer network, the final generated feature map has significant positional sensitivity, meaning that each pixel on the feature map contains its global contextual information within the entire image.

[0070] Finally, the features output by the second layer network After final integration and refinement by subsequent convolutional layers, the spatial orientation-aware features are obtained. These features retain precise local details while incorporating a broad global structure, laying a solid information foundation for the subsequent feature reconstruction module to generate high-quality fused images.

[0071] S3: Spatial orientation-aware features are reconstructed through skip connections and convolution operations to generate a fused image.

[0072] In this embodiment, after the spatial orientation perception features have undergone the aforementioned processing and fully integrated the local and global information of the image, the final image reconstruction stage begins. This stage aims to accurately restore and fuse these deeply processed, abstract feature representations into a high-quality fused image that retains the key information of the source image while conforming to human visual perception habits. To achieve this, this invention designs a feature reconstruction scheme that combines attention mechanisms and skip connections.

[0073] Specifically, before performing the final convolutional reconstruction, this scheme first employs a coordinated attention mechanism to refine the spatial orientation perception features of the input. The purpose of this step is to further enhance the network model's ability to focus on and propagate key features in images of different modalities, such as locating thermally salient targets in infrared images and perceiving rich texture details in visible light images.

[0074] In a preferred embodiment, the core of the coordinated attention mechanism lies in its unique spatial information encoding method. It innovatively decouples the traditional two-dimensional global pooling operation into two independent one-dimensional feature encoding processes, namely, global pooling along the vertical and horizontal directions of the feature map, respectively. For an input feature map... Its encoding process can be represented by the following formula:

[0075] Encoding along the horizontal direction:

[0076] ;

[0077] Encoding along the vertical direction:

[0078] ;

[0079] In this way, the mechanism can capture long-range dependencies along one spatial direction while preserving precise location information along another spatial direction. Subsequently, the feature maps perceived by these two directions are concatenated and then subjected to convolutional transformation, batch normalization, and a non-linear activation function. This generates an intermediate feature map that simultaneously encodes information from two spatial dimensions. :

[0080] ;

[0081] Next, Decompose into two independent tensors, and then perform independent convolution transformations. , With Sigmoid activation function Generate the final attention maps in the horizontal and vertical directions. and :

[0082] ;

[0083] ;

[0084] Finally, these attention maps are applied to the original input features to obtain attention-weighted feature representations that are more salient to the features.

[0085] After completing the attention enhancement steps described above, this scheme introduces the crucial skip connection structure. This structure is designed to address the vanishing gradient and information decay problems prevalent in deep neural networks, especially in image reconstruction tasks, where deep networks easily lose the fine texture details captured by shallower networks.

[0086] The skip connection establishes a direct data path, directly transmitting the shallow features rich in high-frequency details output by the multi-scale feature extraction module in the early stages of the network to the current deep reconstruction unit.

[0087] In this reconstruction unit, the shallow features transmitted through skip connections are effectively integrated with the high-level features, which have undergone spatial orientation perception processing and coordinated attention enhancement and contain rich semantic information. This integration process can be achieved through feature concatenation or element-wise addition, thereby forming a highly comprehensive fused feature representation that includes both macroscopic structure and microscopic texture.

[0088] Finally, this highly integrated feature representation is fed into a reconstructed network consisting of several cascaded convolutional layers. These convolutional layers are responsible for the final learning and transformation of the fused features, mapping them back from the high-dimensional feature space to the image space. To ensure that the generated fused image is completely identical in size to the source image, the convolution operation here preferably sets the padding to keep the input and output sizes constant, and the stride to 1.

[0089] Through the synergistic effect of attention enhancement, skip connection information compensation, and final convolutional reconstruction, this invention can generate a high-quality multimodal fusion image with complete content, clear details, and prominent targets.

[0090] Please see the appendix Figure 2 A spatially perceptive, multi-scale feature learning multimodal image fusion system, comprising:

[0091] The multi-scale feature extraction module is used to process the registered multimodal image pairs using a hierarchical dilated convolutional structure to obtain multi-scale features with different receptive field ranges.

[0092] The spatial orientation sensing module is used to process multi-scale features using a two-layer four-way recurrent neural network to integrate local and global feature information of multimodal images and obtain spatial orientation sensing features.

[0093] The feature reconstruction module is used to reconstruct spatial orientation-aware features through skip connections and convolution operations to generate a fused image.

[0094] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multi-modal image fusion method of spatial-aware multi-scale feature learning, characterized in that, The method comprises the following steps: S1: processing the registered multi-modal image pair using a hierarchical dilated convolution structure to obtain multi-scale features with different receptive field ranges; S2: processing the multi-scale features using a two-layer four-direction recurrent neural network to integrate local and global feature information of the multi-modal images to obtain spatial direction perception features; S3: reconstructing the spatial direction perception features through a skip connection and convolution operation to generate a fusion image.

2. The method of claim 1, wherein, The step of processing using a hierarchical dilated convolution structure is specifically: processing an input image through three parallel convolution paths with different dilation rates, and aggregating the features output by the three paths.

3. The method of claim 1, wherein, When processing the multi-scale features using the two-layer four-direction recurrent neural network, an attention fusion mechanism is introduced to dynamically assign fusion weights to features in four spatial directions.

4. The method of claim 3, wherein, In the two-layer four-direction recurrent neural network, the first layer of four-direction recurrent neural network is used to extract spatially adjacent local features, and the second layer of four-direction recurrent neural network is used to extract context features with global perception ability.

5. The method of claim 1, wherein, Before reconstructing the spatial direction perception features, a coordinated attention mechanism is first used to process the spatial direction perception features to enhance the focusing ability on salient targets and the perception of texture details.

6. The method of claim 5, wherein, The coordinated attention mechanism decouples a two-dimensional global pooling operation into two independent one-dimensional feature encoding processes along the vertical and horizontal directions to cooperatively model inter-channel dependency and spatial position information.

7. The method of claim 1, wherein, The skip connection directly transmits shallow features obtained during multi-scale feature extraction to a deep network to integrate high-level features after fusion of the spatial direction perception features.

8. The method of claim 1, wherein, The method further comprises a step of training and optimizing the fusion network model using a composite loss function, which is composed of a pixel loss and a detail loss.

9. The method of claim 8, wherein, The pixel loss is used to constrain the consistency of the fusion image and the source image in terms of pixel intensity distribution; the detail loss is used to enhance the consistency of the fusion image in terms of edge and texture information by calculating the gradient difference between the fusion image and the source image, and the gradient is obtained through a Sobel gradient operator.

10. A multi-modal image fusion system of spatial-aware multi-scale feature learning, according to the multi-modal image fusion method of spatial-aware multi-scale feature learning of any one of claims 1-9, characterized in that, The method comprises the following modules: A multi-scale feature extraction module for processing the registered multi-modal image pair using a hierarchical dilated convolution structure to obtain multi-scale features with different receptive field ranges; A spatial direction perception module for processing the multi-scale features using a two-layer four-direction recurrent neural network to integrate local and global feature information of the multi-modal images to obtain spatial direction perception features; A feature reconstruction module for reconstructing the spatial direction perception features through a skip connection and convolution operation to generate a fusion image.

Images