A visible light and infrared image fusion method based on a non-euclidean representation of riemannian manifold

Abstract

The application discloses a visible light and infrared image fusion method based on a Riemann manifold non-Euclidean representation, adopts a manifold attention module and a convolution layer cascading structure mode; source image 1 and source image 2 are respectively subjected to block processing to obtain a row vector matrix corresponding to each image block; the row vector matrix is fused into a covariance matrix 1 representing intra-modality and inter-modality covariance through a cross-modal manifold fusion strategy; the covariance matrix 1 enters a symmetric positive definite matrix manifold network module for processing; finally, a covariance matrix 2 used for weighting is trained; the covariance matrix 2 is taken as a weight matrix and is weighted to the row vector matrix through matrix multiplication to obtain a weighted generated feature map 1 and a generated feature map 2; a convolution layer cascade obtains a final fusion image through dimension transformation; the application can fully retain high-frequency information in the image, reveal potential correlations of semantic information in different regions in the image and effectively realize a global correlation relationship fusion learning.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and image processing technology, and specifically relates to an image fusion method based on Riemannian manifolds. Background Technology

[0002] In the early stages, convolutional neural network-based fusion frameworks achieved good results in infrared-visible light fusion tasks. For image data, convolutional neural networks can automatically capture low-level image features through characteristics such as local receptive fields and parameter sharing, and gradually combine them into higher-level, more abstract feature representations. However, this fusion strategy tends to focus on local image features while neglecting long-distance pixel relationships, failing to achieve image-level perception in complex scenes. In 2021, the Vision Transformer emerged, pioneering the use of 16×16 image blocks as model input. This method can effectively uncover long-distance dependencies between image blocks. Subsequently, this method has been widely used in image fusion tasks. However, a significant drawback of this method is its excessive focus on self-attention, neglecting important cross-complementary information between different modalities, which affects the fusion effect.

[0003] Meanwhile, Euclidean metrics primarily consider the Euclidean distance between data points. However, in many cases, the similarity or difference between data points may not be accurately characterized by a simple distance metric. Relationships between data points may involve nonlinear transformations or complex geometric structures, factors that may be ignored by Euclidean metrics, thus failing to accurately capture the similarities and differences between data points. Summary of the Invention

[0004] In view of this, the present invention addresses the problem that traditional Euclidean distance cannot accurately measure the true distribution of image data, and provides a visible light and infrared image fusion method based on the non-Euclidean representation of Riemannian manifold. This method can fully preserve the high-frequency information in the image, strengthen the guidance of relevant information between different modes on the fused image during the infrared and visible light modal fusion process, reveal the potential correlation of semantic information in different regions of the image, and effectively realize a global correlation relationship fusion learning.

[0005] The technical solution for achieving the objective of this invention is as follows:

[0006] A visible light and infrared image fusion method based on Riemannian non-Euclidean representation is proposed, which adopts a manifold attention module and a convolutional layer cascade structure.

[0007] Step 1: Divide source image 1 and source image 2 into blocks to obtain a row vector matrix corresponding to each image block. Each row contains all the pixels of a certain image block in the source image.

[0008] Step 2: The row vector matrix is ​​fused into a covariance matrix 1, representing the covariance within and between modes, through a cross-modal manifold fusion strategy, for use by the Riemannian manifold network;

[0009] Step 3: Covariance matrix 1 is processed by the symmetric positive definite matrix manifold network module. This module consists of a bilinear mapping layer, a real eigenvalue layer, and a logarithmic eigenvalue layer, which respectively implement dimensionality reduction to enhance discriminability, nonlinear activation, and logarithmic domain straightening operations. Finally, the covariance matrix 2 used for weighting is obtained through training, which enhances the discriminative power of the model while retaining the stage information of the covariance matrix.

[0010] Step 4: Using covariance matrix 2 as the weight matrix, weight the generated features by matrix multiplication onto the row vector matrix formed by source image 1 and source image 2 to obtain the weighted generated features. Figure 1 and generating features Figure 2 ;

[0011] Step 5, the convolutional layer cascade includes convolutional layer 1 and convolutional layer 2, to generate features. Figure 1 and generating features Figure 2 The dimension is increased from 1 to 16 through convolutional layer 1, and the two 16-dimensional feature maps are directly added together. Then, the dimension is reduced from 32 dimensions to 1 dimension through convolutional layer 2. Each convolutional layer contains a 3×3 convolutional kernel and a linear rectified activation function to enhance the nonlinear expression. The final fused image fully expresses the weight distribution on the Riemannian manifold.

[0012] A visible light and infrared image fusion method based on the non-Euclidean representation of Riemannian manifolds has the following significant advantages compared with existing technologies:

[0013] 1. The fusion result is better adapted to the non-Euclidean distribution of image data and can capture non-linear features better than previous methods;

[0014] 2. This method uses a novel manifold attention module to model the covariance of image features, which can effectively mine the original pixel characteristics and statistical relationship patterns of images;

[0015] 3. This method introduces a hybrid attention mechanism to fuse features from different modes, thereby enhancing the correlation between infrared and visible light modes.

[0016] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0017] Figure 1 Flowchart of the present invention

[0018] Figure 2 Detailed diagram of cross-modal manifold fusion strategy

[0019] Figure 3 Symmetric positive definite matrix manifold network module architecture diagram Detailed Implementation

[0020] The present invention will now be described in detail with reference to the accompanying drawings. The embodiments listed are only for the purpose of helping to understand the present invention and should not be construed as limiting the scope of protection of the invention. For those skilled in the art, improvements and modifications can be made to the present invention without departing from the principles and ideas of the present invention, and such improvements and modifications also fall within the scope of protection of the claims of the present invention.

[0021] Combination Figure 1 As shown, this invention adopts a manifold attention module and a convolutional layer cascade structure mode; Step 1, the source image 1 and source image 2 are processed into blocks respectively to obtain the row vector matrix corresponding to each image block, and each row contains all the pixels of a certain image block in the source image;

[0022] Step 2: The row vector matrix is ​​fused into a covariance matrix 1, representing the covariance within and between modes, through a cross-modal manifold fusion strategy, for use by the Riemannian manifold network;

[0023] Step 3: Covariance matrix 1 is processed by the symmetric positive definite matrix manifold network module. This module consists of a bilinear mapping layer, a real eigenvalue layer, and a logarithmic eigenvalue layer, which respectively implement dimensionality reduction to enhance discriminability, nonlinear activation, and logarithmic domain straightening operations. Finally, the covariance matrix 2 used for weighting is obtained through training, which enhances the discriminative power of the model while retaining the stage information of the covariance matrix.

[0024] Step 4: Using covariance matrix 2 as the weight matrix, weight the generated features by matrix multiplication onto the row vector matrix formed by source image 1 and source image 2 to obtain the weighted generated features. Figure 1 and generating features Figure 2 ;

[0025] Step 5, the convolutional layer cascade includes convolutional layer 1 and convolutional layer 2, to generate features. Figure 1 and generating features Figure 2 The dimension is increased from 1 to 16 through convolutional layer 1, and the two 16-dimensional feature maps are directly added together. Then, the dimension is reduced from 32 dimensions to 1 dimension through convolutional layer 2. Each convolutional layer contains a 3×3 convolutional kernel and a linear rectified activation function to enhance the nonlinear expression. The final fused image fully expresses the weight distribution on the Riemannian manifold.

[0026] The convolutional kernel size is 3×3, the block size is 16, and the overlap region size is set to 8. Within the manifold attention module, the weights of the bilinear mapping layer are updated using a stochastic gradient descent optimizer on the Steifel manifold with a learning rate of 0.01. On the convolutional layer, the weights are updated using an Adam optimizer with a learning rate of 0.0001.

[0027] Combination Figure 2 As shown, source image 1 and source image 2 are source images from a pair of infrared and visible light modes in a multispectral road scene dataset. The image size is uniformly 256×256 pixels. For source image 1 and source image 2, a row vector matrix is ​​obtained by dividing and flattening each image block. Each row of this matrix represents all pixels of a certain image block in the source image. A structured feature map stacking method is used to arrange the feature maps of the two modes vertically to obtain the corresponding row vector matrix. During the covariance calculation, the row vector matrix formed after flattening each image block of the two modes is calculated to construct a composite covariance matrix, which is covariance matrix 1. Covariance matrix 1 is divided into four quadrant blocks. The upper left and lower right quadrant blocks are calculated through intramodal covariance, representing the features within a single mode. The upper right and lower left quadrant blocks represent cross-modal covariance calculations.

[0028] Combination Figure 3 As shown, Figure 1 The covariance matrix 1 in the model is first passed through two cascaded bilinear mapping layers and a real eigenvalue layer. The bilinear mapping layer performs dimensionality reduction and enhances discriminative power while preserving the symmetric positive definite properties of the matrix. The real eigenvalue layer introduces a nonlinear activation function into the Riemann network, reducing the eigenvalues ​​to a lower threshold to ensure the model learns more complex features. Finally, the logarithmic eigenvalue layer stretches the curved manifold space until it becomes a flat Euclidean space, resulting in the model. Figure 1 The covariance matrix 2 in the figure.

[0029] Extensive experiments demonstrate that the fusion method of this invention outperforms most state-of-the-art methods. Compared with the most popular fusion networks currently available, on the visible and infrared light dataset proposed by Susterberg in the Netherlands, we achieved entropy, mean square error, peak signal-to-noise ratio, sum of differential correlations, and correlation coefficient of 6.939, 0.0457, 62.350, 1.751, and 0.525, respectively, ranking first among all networks compared.

Claims

1. A method for fusing visible light and infrared images based on the non-Euclidean representation of Riemannian manifolds, characterized in that, A manifold attention module and a cascaded convolutional layer structure are adopted; Step 1: Divide source image 1 and source image 2 into blocks to obtain a row vector matrix corresponding to each image block. Each row contains all the pixels of a certain image block in the source image. Step 2: The row vector matrix is ​​fused into a covariance matrix 1, representing the covariance within and between modes, through a cross-modal manifold fusion strategy, for use by the Riemannian manifold network; Step 3: Covariance matrix 1 is processed by the symmetric positive definite matrix manifold network module. This module consists of a bilinear mapping layer, a real eigenvalue layer, and a logarithmic eigenvalue layer, which respectively implement dimensionality reduction to enhance discriminability, nonlinear activation, and logarithmic domain straightening operations. Finally, the covariance matrix 2 used for weighting is obtained through training, which enhances the discriminative power of the model while retaining the stage information of the covariance matrix. Step 4: Use covariance matrix 2 as the weight matrix, and add it to the row vector matrix formed by source image 1 and source image 2 through matrix multiplication to obtain the weighted generated feature map 1 and generated feature map 2. Step 5, the convolutional layer cascade includes convolutional layer 1 and convolutional layer 2. The generated feature map 1 and generated feature map 2 are respectively passed through convolutional layer 1, increasing the dimension from 1 to 16. Then the two 16-dimensional feature maps are directly added together, and then the dimension is reduced from 32 dimensions to 1 dimension through convolutional layer 2. Each convolutional layer contains a 3×3 convolutional kernel and a linear rectified activation function to enhance the nonlinear expression. The final fused image fully expresses the weight distribution on the Riemannian manifold. The covariance matrix 1 is divided into four quadrant blocks. The upper left and lower right quadrant blocks are calculated through internal modal covariance to represent the features within a single mode, while the upper right and lower left quadrant blocks represent cross-modal covariance calculations.

2. The visible light and infrared image fusion method based on the non-Euclidean representation of Riemannian manifolds according to claim 1, characterized in that, The source image 1 and source image 2 are source images of a pair of infrared and visible light modes from the multispectral road scene dataset, and the image size is uniformly 256×256 pixels.

3. The visible light and infrared image fusion method based on the non-Euclidean representation of Riemannian manifolds according to claim 1, characterized in that, The convolution kernel size is 3×3, the block size is 16, and the overlap region size is set to 8.

4. The visible light and infrared image fusion method based on the non-Euclidean representation of Riemannian manifolds according to claim 1, characterized in that, Within the manifold attention module, the weights of the bilinear mapping layer are updated using a stochastic gradient descent optimizer on the Steifel manifold with a learning rate of 0.

01. On the convolutional layer, the weights are updated using an Adam optimizer with a learning rate of 0.0001.

5. The visible light and infrared image fusion method based on the non-Euclidean representation of Riemannian manifolds according to claim 1, characterized in that, In step 3, the covariance matrix 1 first passes through two layers of bilinear mapping and a layer of real eigenvalues. The bilinear mapping layer performs data dimensionality reduction and discriminative enhancement while maintaining the symmetric positive definite properties of the matrix. The real eigenvalue layer introduces a nonlinear activation function into the Riemann network to reduce the eigenvalues ​​to a lower threshold. Finally, the logarithmic eigenvalue layer stretches the curved manifold space to a flat Euclidean space to obtain the covariance matrix 2.

Images