An image fusion method and system based on self-supervised modal interaction

Abstract

The application discloses a kind of based on self-supervision modal interaction's image fusion method and system, belong to multi-sensor hyperspectral image fusion technical field, respectively to low-resolution hyperspectral image and high-resolution multispectral image are carried out feature extraction and fusion, obtain original fusion spectral feature and original fusion spatial feature;Cosine similarity focal loss is used to complete cross-modal feature alignment, and the spatial feature after alignment is decoded to obtain learning abundance matrix by space transformation and normalization, and high-resolution hyperspectral image is reconstructed in combination with end member matrix;Based on image degradation, construct self-supervision total loss function, complete whole process optimization by gradient back propagation, finally output high spatial resolution and high spectral fidelity fusion image.The application solves the problems of spatial information loss, spectral distortion, cross-modal alignment difficulty and dependence on supervision label in traditional spectral fusion, and the image fusion precision and visual effect are better.

Description

Technical Field

[0001] This invention belongs to the field of multi-sensor hyperspectral image fusion technology, specifically relating to an image fusion method and system based on self-supervised modal interaction. Background Technology

[0002] Hyperspectral imaging can continuously acquire detailed spectral information of ground objects, and has important application value in remote sensing monitoring, geological exploration, environmental identification and other fields. However, due to limitations in sensor hardware and imaging principles, a single hyperspectral sensor cannot simultaneously acquire high spatial resolution and high spectral resolution data, resulting in low-resolution hyperspectral images with blurred spatial details. While high-resolution multispectral images have clear spatial textures, they only contain a limited number of spectral bands, which cannot support detailed ground object classification and quantitative inversion. To fully integrate the advantages of both types of data, it is necessary to combine the spatial details of high-resolution multispectral images with the spectral information of low-resolution hyperspectral images to generate high-resolution hyperspectral images that combine high spatial resolution and high spectral fidelity.

[0003] Traditional image fusion methods are mostly based on simple transformations, direct superposition, or shallow models, which are difficult to fully model spatial-spectral correlation characteristics, easily leading to spectral distortion, spatial blurring, and loss of detail in the fusion results. Although deep learning-based fusion methods improve representation capabilities, they are insufficient in feature extraction and cross-modal alignment: Transformer-based methods, while possessing global modeling capabilities, rely on large amounts of labeled data and supervised training, failing to achieve self-supervised fusion and exhibiting weak generalization ability; some unsupervised methods rely solely on convolutional networks to extract features, ignoring non-local dependencies in the image and resulting in insufficient preservation of spatial information.

[0004] Furthermore, existing fusion methods generally require real high-resolution hyperspectral images as supervisory labels, resulting in high data acquisition costs and poor scene adaptability, making it difficult to meet the stable fusion requirements under unlabeled conditions in practical engineering. Therefore, developing a self-supervised fusion method for spectral images that does not require supervisory labels, has accurate cross-modal alignment, and synchronously preserves spatial details and spectral information has become crucial to breaking through existing bottlenecks and improving the usability of multi-source remote sensing data. Summary of the Invention

[0005] To address the aforementioned problems in the prior art, the present invention is achieved through the following technical solution: A first aspect of the present invention provides an image fusion method based on self-supervised modal interaction, comprising the following steps: The basic features, global spectral features, and attention-enhanced features of a low-resolution hyperspectral image are obtained, and then stitched and fused to obtain the original fused spectral features. The basic features, global spatial features, and spatial enhancement features of high-resolution multispectral images are acquired, and then feature splicing and fusion are performed to obtain the original fused spatial features. A cosine similarity focus loss is constructed, and cross-modal alignment constraints are applied to the original fused spectral features and the original fused spatial features based on the cosine similarity focus loss to obtain the aligned fused spectral features and fused spatial features. The aligned fused spatial features are spatially transformed and normalized to obtain the learned abundance matrix; the aligned fused spectral features are spatially transformed and normalized to obtain the degraded abundance matrix; the endmember matrix is ​​obtained based on the degraded abundance matrix and the low-resolution hyperspectral image; the high-resolution hyperspectral image is obtained based on the learned abundance matrix and the endmember matrix; a total loss function is constructed, and the entire process of obtaining the high-resolution hyperspectral image is self-supervised and optimized using the total loss function.

[0006] The basic features, global spectral features, and attention-enhanced features obtained from low-resolution hyperspectral images are concatenated and fused to obtain the original fused spectral features. This also includes: The basic features of the low-resolution hyperspectral image are obtained by projecting a transformation on the low-resolution hyperspectral image. Global attention modeling and feature aggregation are then performed on the basic features of the low-resolution hyperspectral image to obtain global spectral features. The basic features of the low-resolution hyperspectral image are adaptively weighted through a channel attention mechanism to obtain attention-enhanced features. The basic features, global spectral features and attention-enhanced features of the low-resolution hyperspectral image are then concatenated and fused to obtain the original fused spectral features.

[0007] The basic features, global spatial features, and spatial enhancement features of high-resolution multispectral images are acquired, and then feature splicing and fusion are performed to obtain the original fused spatial features. This also includes: The location embedding matrix of the high-resolution multispectral image is obtained. The high-resolution multispectral image is then projected and concatenated with the location embedding matrix to obtain the basic features of the high-resolution multispectral image. Global attention modeling and feature aggregation are performed on the basic features of the high-resolution multispectral image to obtain global spatial features. The basic features of the high-resolution multispectral image are adaptively weighted and enhanced through a channel attention mechanism to obtain spatially enhanced features. The basic features, global spatial features, and spatially enhanced features of the high-resolution multispectral image are then concatenated and fused to obtain the original fused spatial features.

[0008] Obtain the location embedding matrix of the high-resolution multispectral image, including: The high-resolution multispectral image is divided into several p×p square regions, and a unique region index is assigned to each region. With each pixel in the image as the center, an m×m sliding window is used to count the frequency of occurrence of each region index within the window, and the position embedding vector of the corresponding pixel is generated based on the frequency. The position embedding vectors of all pixels are concatenated in order to obtain the position embedding matrix of the high-resolution multispectral image.

[0009] Constructing the cosine similarity focus loss includes: Local region feature vectors are extracted from the original fused spectral features and the original fused spatial features respectively, and the cosine similarity between the local region feature vectors is calculated; the cosine similarity focal loss is obtained based on the cosine similarity between the local region feature vectors.

[0010] Construct the total loss function, including: Based on high-resolution hyperspectral images, reconstruction loss and consistency loss are constructed. Combined with cosine similarity focus loss, the reconstruction loss, consistency loss and cosine similarity focus loss are weighted by loss coefficients to obtain the total loss function.

[0011] Reconstruction loss and consistency loss are constructed based on high-resolution hyperspectral images, including: The high-resolution hyperspectral image is degraded and reconstructed to obtain a low-resolution hyperspectral matrix and a high-resolution multispectral matrix; reconstruction loss and consistency loss are constructed based on the low-resolution hyperspectral matrix and the high-resolution multispectral matrix.

[0012] Degradation and reconstruction of high-resolution hyperspectral images yields low-resolution hyperspectral matrices and high-resolution multispectral matrices, and also includes: The high-resolution hyperspectral image is downsampled and convolutionally decoded to obtain a low-resolution hyperspectral matrix; A spectral degradation matrix is ​​constructed, and the high-resolution hyperspectral image is multiplied with the spectral degradation matrix and decoded to obtain a high-resolution multispectral matrix.

[0013] Constructing the spectral degradation matrix includes: The high-resolution multispectral image is downsampled to make its resolution consistent with that of the low-resolution hyperspectral image. The optimal linear transformation matrix between the downsampled high-resolution multispectral image and the low-resolution hyperspectral image is solved by least squares to obtain the spectral degradation matrix.

[0014] In a second aspect, the present invention provides an image fusion system based on self-supervised modal interaction for performing an image fusion method based on self-supervised modal interaction, the system comprising: The spectral encoder is used to acquire basic features, global spectral features, and attention-enhanced features from low-resolution hyperspectral images, and then stitch and fuse them to obtain the original fused spectral features. A spatial encoder is used to acquire basic features, global spatial features, and spatial enhancement features of high-resolution multispectral images, and then perform feature stitching and fusion to obtain the original fused spatial features. The cosine similarity focus loss module is used to construct the cosine similarity focus loss. Based on the cosine similarity focus loss, cross-modal alignment constraints are applied to the original fused spectral features and the original fused spatial features to obtain the aligned fused spectral features and fused spatial features. The spectral fusion module performs spatial transformation and normalization decoding on the aligned fused spatial features to obtain a learned abundance matrix; it also performs spatial transformation and normalization decoding on the aligned fused spectral features to obtain a degraded abundance matrix, and obtains an endmember matrix based on the degraded abundance matrix and the low-resolution hyperspectral image; it then obtains a high-resolution hyperspectral image based on the learned abundance matrix and the endmember matrix; finally, it constructs a total loss function and performs self-supervised optimization throughout the process of obtaining the high-resolution hyperspectral image using the total loss function.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention designs an encoding strategy of patch-pixel position embedding and triple feature fusion, which fully preserves spatial structure information and enhances the expression of spectral and spatial features. By generating region frequency-based position embeddings for high-resolution multispectral images, it avoids the spatial correlation loss and pixel scrambling problems caused by traditional position encoding. At the same time, it fuses spectral and spatial features separately, which improves the feature expression capability and the spatial detail preservation effect.

[0016] 2. This invention proposes cosine similarity focus loss to achieve cross-modal alignment constraints. It models the matching process of spectral features and spatial features as a local region similarity learning problem. By adaptively focusing on high similarity samples and suppressing low similarity interference, it effectively solves the problems of inaccurate spectral-spatial cross-modal alignment, spectral distortion and spatial blurring, allowing the fused image to be improved in both spectral fidelity and spatial detail.

[0017] 3. This invention constructs a full-process self-supervised optimization framework that can complete model training without high-resolution hyperspectral ground truth labels. It constructs a self-supervised signal through downsampling convolution and spectral degradation matrix, and jointly optimizes the reconstruction loss, consistency loss and cosine similarity focus loss. It can still output high-quality fusion results under unsupervised conditions.

[0018] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0019] Figure 1 This is a flowchart of an image fusion process based on self-supervised modal interaction provided in Embodiment 1 of the present invention; Figure 2 This is a schematic diagram illustrating the principle of obtaining a position embedding matrix according to Embodiment 1 of the present invention; Figure 3 This is a visual representation of the fusion of various methods on the TianGong-1 dataset; Figure 4 This is a visual representation of the fusion of the various methods on the Washington DC Mall dataset. Detailed Implementation

[0020] To further illustrate the technical means and effects adopted by the present invention to achieve the intended purpose, the following detailed description of the solution according to the present invention is provided in conjunction with the accompanying drawings and specific embodiments.

[0021] The foregoing and other technical contents, features, and effects of the present invention will be clearly presented in the following detailed description of specific embodiments in conjunction with the accompanying drawings. Through the description of the specific embodiments, a more in-depth and concrete understanding can be gained of the technical means and effects adopted by the present invention to achieve its intended purpose. However, the accompanying drawings are for reference and illustration only and are not intended to limit the technical solutions of the present invention.

[0022] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations are intended to cover non-exclusive inclusion, such that an article or apparatus comprising a list of elements includes not only those elements but also other elements not expressly listed. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the article or apparatus that includes said element.

[0023] like Figure 1 The diagram shown is a flowchart of an image fusion process based on self-supervised modal interaction provided in Embodiment 1, which includes the following steps: Step 1: Obtain the basic features, global spectral features, and attention enhancement features of the low-resolution hyperspectral image, and then stitch and fuse them to obtain the original fused spectral features.

[0024] Specifically, for low-resolution hyperspectral images acquired by sensors A 3×3 convolutional layer and a ReLU activation function projection transformation are performed to obtain the basic features of the low-resolution hyperspectral image. Where h and w are the height and width of the low-resolution hyperspectral image, respectively, and c is the number of channels in the image. It represents the dimensional space of the real number field.

[0025] Basic features of low-resolution hyperspectral images Global attention is modeled using a multi-head self-attention layer (MHSA) combined with instance normalization (IN):

[0026] in, For the output features of the t-th layer of the spectral encoder, For the output features of the (t-1)th layer of the spectral encoder, For the multi-head self-attention layer, Let t be the projectible learning matrix for the t-th layer. For instance normalization; t is the level index, t=1, 2...g, when t=1, , where g is the total number of multi-head self-attention layers.

[0027] The calculation process for multi-head self-attention can be expressed as follows:

[0028] in, For concatenation functions, This represents the i-th head in the multi-head self-attention system, where the index parameter i ranges from 1 to N.

[0029]

[0030] in, This is the weight normalization function; , For query vector, ; , For key vectors, ; , For value vectors, ; d is the scaling factor, and d is the feature dimension of the spectral encoder; , and These represent the learnable weights of the query vector, key vector, and value vector, respectively.

[0031] To enhance the representation, the outputs of all multi-head self-attention (MHSA) layers are summed, and the global spectral features are obtained through feature aggregation. ;

[0032] in, The aggregated output represents the global spectral features. .

[0033] In the spectral supplementation module, the importance weights of each spectral band are learned through a channel attention mechanism to assess the fundamental features of the hyperspectral image. Adaptive weighted enhancement is performed to obtain attention-enhanced features. .

[0034] Specifically, the fundamental features of low-resolution hyperspectral images Perform global max pooling and global average pooling respectively, and then pass them through a multilayer perceptron (MLP):

[0035]

[0036] in, As the first intermediate feature, As the second intermediate feature, It is a multilayer sensing function. It is the global max pooling function. This is the global average pooling function.

[0037] For the first intermediate feature Second intermediate features Normalization yields attention-enhancing features:

[0038] in, For attention enhancement features, For normalization function, It represents the Hadamardi (or Hadama) stack.

[0039] The basic and global spectral features of the low-resolution hyperspectral image are further extracted using a multilayer perceptron (MLP), and then concatenated and fused with attention-enhanced features to obtain the original fused spectral features. :

[0040] in, The original fusion spectral characteristics, This is a concatenation function.

[0041] Step 2: Obtain the basic features, global spatial features, and spatial enhancement features of the high-resolution multispectral image, and perform feature stitching and fusion to obtain the original fused spatial features.

[0042] In the traditional Transformer model, the input image is first divided into blocks, then each block is flattened into a sequence token, and a sinusoidal position code is added to the token. However, this method has two shortcomings when used for spectral image fusion: first, the image block flattening operation loses the spatial correlation information of pixels within the block; second, the original Transformer directly flattens the image blocks into vectors, scrambling the near and far positions of pixels, resulting in poor spectral fusion performance. To solve these problems, this invention proposes a patch-pixel position embedding method to preserve the high similarity characteristics of adjacent pixels in the spectral image.

[0043] like Figure 2 The diagram shown is a schematic diagram of the principle of obtaining a position embedding matrix according to Embodiment 1 of the present invention.

[0044] High-resolution multispectral images The image is divided into several square patch regions of size p × p. Given that the image height is H and the width is W, the total number of patch regions r is:

[0045] For each patch region that has been partitioned, assign a unique region index in row-major order. This enables spatial calibration of the entire image region, laying the foundation for subsequent pixel-level location correlation mining.

[0046] For each pixel in the image, construct an m×m spatial neighborhood sliding window. Using this m×m sliding window, traverse all pixels within the window and count the indices of each region within the window. The frequency of occurrence of region index k at pixel i is . Based on this frequency Generate the patch-pixel position embedding vector corresponding to pixel i :

[0047] Where r is the total number of patch regions.

[0048] exist Figure 2 middle, Therefore, a sparse vector is obtained. .in, The larger the value, the stronger the correlation between the i-th pixel and the pixel with the k-label. This positional encoding strategy ensures that adjacent pixels have higher similarity.

[0049] Patch all pixels - embed pixel positions into vector By sequentially stitching together, a patch-pixel location embedding matrix B of the high-resolution multispectral image is obtained:

[0050] in, , Indicates matrix transpose. This represents the patch-pixel position embedding vector corresponding to the last pixel of the entire image.

[0051] A 3×3 convolutional layer and ReLU activation function projection transformation are applied to the high-resolution multispectral image X, and then concatenated with the patch-pixel location embedding matrix B to obtain the basic features of the high-resolution multispectral image. :

[0052] in, As a fundamental feature of high-resolution multispectral images, For concatenation functions, This is a 3×3 convolution operation.

[0053] Basic features of high-resolution multispectral images Global attention is modeled using a multi-head self-attention layer (MHSA) combined with instance normalization (IN):

[0054] in, Let be the output features of the t-th layer of the spatial encoder. For the output features of the (t-1)th layer of the spatial encoder, For the multi-head self-attention layer, Let t be the projectible learning matrix for the t-th layer. For instance normalization; t is the level index, t=1, 2...g, when t=1, , where g is the total number of multi-head self-attention layers.

[0055] The calculation process for multi-head self-attention can be expressed as follows:

[0056] in, For concatenation functions, This represents the i-th head in the multi-head self-attention system, where the index parameter i ranges from 1 to N.

[0057]

[0058] in, This is the weight normalization function; , For query vector, ; , For key vectors, ; , For value vectors, ; d is the scaling factor, and d is the feature dimension of the spectral encoder; , and These represent the learnable weights of the query vector, key vector, and value vector, respectively.

[0059] To enhance the representation, the outputs of all multi-head self-attention (MHSA) layers are summed, and the global spatial features are obtained through feature aggregation. ;

[0060] in, The aggregated output represents the global spatial features. .

[0061] In the spatial supplementation module, the fundamental features of the multispectral image are analyzed using a channel attention mechanism. Adaptive weighted augmentation is performed to obtain spatial augmentation features. .

[0062] Basic features of high-resolution multispectral images After performing global max pooling and global average pooling respectively, the data is concatenated; after global max pooling and 3×3 convolution, the data is concatenated; after 3×3 convolution and 5×5 convolution, the data is concatenated, and then linear mapping is applied to each of these operations to obtain three sets of intermediate features:

[0063]

[0064]

[0065] in, It is the third intermediate feature. It is the fourth intermediate feature. The fifth intermediate feature, Indicates global max pooling. Indicates global average pooling. This is a 5×5 convolution operation. The learnable weight matrix consists of the following: , The mapping transformation weights.

[0066] Normalize the three sets of intermediate features separately and compare them with the base features. After performing the Hadamard product and then weighted summation, the spatially enhanced features are obtained. :

[0067] in, For spatial enhancement features, For normalization function, Let i represent the Hadamard product, where i is the index parameter of the intermediate feature, i=1,2,3.

[0068] The basic and global spatial features of the high-resolution multispectral image are further extracted using a multilayer perceptron (MLP), and then concatenated and fused with the spatial enhancement features to obtain the original fused spatial features. :

[0069] in, As the original integrated spatial features, This is a concatenation function.

[0070] Step 3: Construct cosine similarity focus loss, and apply cross-modal alignment constraints to the original fused spectral features and original fused spatial features based on the cosine similarity focus loss to obtain the aligned fused spectral features and fused spatial features.

[0071] Specifically, corresponding local regions are uniformly sampled from the original fused spectral features and the original fused spatial features, respectively, and feature vectors corresponding to the same spatial location in the local regions are extracted. and Calculate the feature vector of the local region and Cosine similarity between ;

[0072] in, Indicates the original fusion spectral characteristics The Middle Local region feature vectors Represents the original fusion spatial characteristics The Middle Local feature region vectors, Representing the eigenvector and Cosine similarity between them This represents the vector dot product operation. This refers to the 2-norm operation for vectors.

[0073] Positive sample pairs are divided according to their cosine similarity. and negative sample pairs When the spectral and spatial local region eigenvectors satisfy Among them, the first empirical threshold At this time, the feature vector of the local region and Positive sample pairs are formed, representing spectral-spatial local region feature vectors corresponding to the same spatial location; when the spectral and spatial local region feature vectors satisfy... Among them, the second empirical threshold At this time, the feature vector of the local region and These constitute negative sample pairs, which represent spectral-spatial local region feature vectors of non-corresponding spatial locations; when the spectral and spatial local region feature vectors satisfy... < < At this time, the feature vector pair is a fuzzy sample pair, and the fuzzy sample pair does not participate in the cosine similarity focus loss. The calculation, in the calculation Discard directly.

[0074] The cosine similarity focus loss is obtained based on the cosine similarity between feature vectors of local regions.

[0075] in, This represents the cosine similarity focus loss. This represents the total number of positive sample pairs. This represents the total number of negative sample pairs. It represents logarithmic operations.

[0076] Cosine similarity focus loss To optimize the target, the original fused spectral features were analyzed. Integrated with original spatial features By applying cross-modal consistency constraints, the system adaptively strengthens the weight of high-similarity regions and suppresses interference from low-similarity regions during backpropagation. At the same time, it automatically corrects the spatial deviation between features and finally outputs the aligned fused spectral features G and the aligned fused spatial features F, providing accurately aligned spatial-spectral joint features for subsequent high-resolution hyperspectral image reconstruction.

[0077] Step 4: Perform spatial transformation and normalization decoding on the aligned fused spatial features to obtain the learning abundance matrix; combine the learning abundance matrix and the endmember matrix to obtain the high-resolution hyperspectral image; construct the total loss function, and perform self-supervised optimization on the entire process of obtaining the high-resolution hyperspectral image through the total loss function.

[0078] The aligned fused spatial features F and aligned fused spectral features G are input into a spatial transformation network (STN) for spatial transformation, and then normalized and decoded using the Softmax function to obtain the learned abundance matrix and the degraded abundance matrix:

[0079]

[0080] in, To learn the abundance matrix, For the degraded abundance matrix, This is a spatial transformation function used to correct geometric deviations between cross-modal features. This is the normalization function.

[0081] Endmember matrix The following objective function can be solved to obtain:

[0082] Among them, low-resolution hyperspectral images , This indicates taking the minimum value. This represents a low-resolution hyperspectral image. This refers to the 2-norm operation for vectors.

[0083] Based on the learning abundance matrix End Element Matrix The combined images yielded a high-resolution hyperspectral image:

[0084] in, This indicates a high-resolution hyperspectral image.

[0085] High-resolution hyperspectral images Perform downsampling convolutional decoding to obtain the reconstructed low-resolution hyperspectral (LR-HSI) matrix. :

[0086] in, This indicates the reconstruction of a low-resolution hyperspectral matrix. This indicates that downsampling convolution operations can be learned. C and R These are the image height and width scaling factors, respectively. , H and W represent the height and width of the high-resolution multispectral image, respectively, while h and w represent the height and width of the low-resolution hyperspectral image, respectively.

[0087] The high-resolution multispectral image X is downsampled to match the resolution of the low-resolution hyperspectral image Y. The optimal linear transformation matrix between the downsampled high-resolution multispectral image and the low-resolution hyperspectral image is then solved using the least squares method to obtain the spectral degradation matrix. :

[0088] in, This is the spectral degradation matrix, used to simulate the spectral dimensionality reduction process of high-resolution multispectral images; Indicates a downsampling operation; Let be the matrix of intermediate variables to be optimized, and the optimal solution is... ; For L2 norm operations; This represents the least squares optimization operator.

[0089] High-resolution hyperspectral images With the spectrum degradation matrix Multiplication decoding yields the reconstructed high-resolution multispectral (HR-MSI) matrix. :

[0090] Based on low-resolution hyperspectral images High-resolution multispectral images Reconstructing low-resolution hyperspectral matrices and reconstructing high-resolution multispectral matrices Construct reconstruction loss and consistency loss, and combine cosine similarity focus loss. The reconstruction loss, consistency loss, and cosine similarity focus loss are weighted using loss coefficients to obtain the total loss function:

[0091] in, For the total loss function, To rebuild the losses, For consistency loss, For reconstruction loss coefficient, The consistency loss coefficient, The cosine similarity focus loss weight coefficient; The original hyperspectral image Y is input into the backbone network, and the output image is directly reconstructed through encoding and decoding. The original multispectral image X is input into the backbone network, and the output image is directly reconstructed through encoding and decoding.

[0092] The entire process of obtaining high-resolution hyperspectral images is self-supervised by using the total loss function. By using gradient backpropagation, the model parameters in the above steps are updated synchronously, continuously reducing the error between the image reconstruction result and the real input, and enhancing the cross-modal feature alignment effect.

[0093] Repeat the above update process until the total loss function converges or the preset maximum number of iterations is reached. Finally, the model can stably output high-resolution hyperspectral images with both high spatial resolution and high spectral fidelity under fully self-supervised conditions.

[0094] Embodiment 2 of the present invention provides an image fusion system based on self-supervised modal interaction, comprising: The spectral encoder is used to acquire basic features, global spectral features, and attention-enhanced features from low-resolution hyperspectral images, and then stitch and fuse them to obtain the original fused spectral features.

[0095] The spatial encoder is used to acquire basic features, global spatial features, and spatial enhancement features of high-resolution multispectral images, and then stitches and fuses these features to obtain the original fused spatial features.

[0096] The cosine similarity focus loss module is used to construct the cosine similarity focus loss. Based on the cosine similarity focus loss, cross-modal alignment constraints are applied to the original fused spectral features and the original fused spatial features to obtain the aligned fused spectral features and fused spatial features.

[0097] The spectral fusion module performs spatial transformation and normalization decoding on the aligned fused spatial features to obtain a learned abundance matrix; it also performs spatial transformation and normalization decoding on the aligned fused spectral features to obtain a degraded abundance matrix, and obtains an endmember matrix based on the degraded abundance matrix and the low-resolution hyperspectral image; it then obtains a high-resolution hyperspectral image based on the learned abundance matrix and the endmember matrix; finally, it constructs a total loss function and performs self-supervised optimization throughout the process of obtaining the high-resolution hyperspectral image using the total loss function.

[0098] Experiments were conducted on two general spectral image fusion datasets, TianGong-1 and Washington DC Mall, comparing the results with methods such as Coupled Nonnegative Matrix Factorization (CNMF), Cooperative Sparse Unmixing (CSU), Enhanced Autoencoder and Degeneracy Learning (EU2ADL), Multi-Level Cross Transformer Network (MCT-Net), and Multi-Scale Spatial Spectral Transformer Network (MSST-Net). Quantitative evaluation metrics included Peak Signal-to-Noise Ratio (PSNR), Spectral Angle Mapper (SAM), Synthetic Dimensionless Global Relative Error (ERGAS), Structural Similarity (SSIM), and Universal Image Quality Index (UIQI). "Proposed" indicates the method in this invention.

[0099] Table 1. Quantitative fusion results of all methods on the TianGong-1 dataset. method SAM PSNR ERGAS SSIM UIQI CNMF 1.771 36.173 0.412 0.971 0.998 CSU 1.547 36.867 0.387 0.972 0.998 EU2ADL 1.387 41.935 0.233 0.987 0.998 MCT-Net 1.602 40.923 0.244 0.984 0.997 MSST-Net 1.174 43.289 0.208 0.987 0.999 Proposed 0.909 46.376 0.155 0.990 0.999 Table 2. Quantitative results of all methods on the Washington DC Mall dataset. method SAM PSNR ERGAS SSIM UIQI CNMF 2.993 34.890 1.623 0.939 0.979 CSU 2.954 34.335 1.621 0.930 0.981 EU2ADL 1.716 38.437 1.602 0.978 0.979 MCT-Net 1.736 37.909 1.292 0.970 0.939 MSST-Net 1.386 40.235 0.945 0.945 0.982 Proposed 0.952 41.639 0.709 0.972 0.993 like Figure 3 and Figure 4 The figures show the fusion visual effects of each method on the TianGong-1 dataset and the Washington DC Mall dataset, respectively. Visual evaluation metrics include the SAM heatmap, the mean relative absolute error (MRAE) heatmap, and the residual heatmap for a specific band. The first row in the figures shows the high-resolution hyperspectral image of the final model output; the second row shows the SAM heatmap; the third row shows the MRAE heatmap; and the fourth row shows the residual heatmap for band 10. As can be seen from the figures, the method proposed in this application has a deeper red area, less blue area, and lower error than other methods, resulting in the best fusion effect.

[0100] This invention proposes a spectral image fusion method and system based on self-supervised modal interaction. It extracts and fuses features from low-resolution hyperspectral images and high-resolution multispectral images respectively, obtaining original fused spectral features and original fused spatial features. Cross-modal feature alignment is achieved by constructing a cosine similarity focal loss, resulting in precisely matched fused spectral features and fused spatial features. The aligned fused spatial features are then spatially transformed and normalized to obtain a learning abundance matrix, which is combined with the endmember matrix to reconstruct the high-resolution hyperspectral image. A total loss function is constructed based on image degradation, and the entire process is optimized in a self-supervised manner, ultimately outputting a fusion result that combines high spatial resolution and high spectral fidelity.

[0101] This invention effectively solves the technical challenges of spatial information loss, severe spectral distortion, difficulty in cross-modal alignment, and reliance on supervised labels in traditional spectral image fusion, while balancing spatial detail preservation and spectral information fidelity. By introducing patch-pixel position embedding to preserve the spatial correlation between adjacent pixels, and employing cosine similarity focal loss to achieve adaptive cross-modal alignment, it eliminates the dependence on high-resolution hyperspectral ground truth labels through full-process self-supervised optimization. This improves fusion accuracy and visual effects while enhancing the model's generalization and robustness in real-world scenarios. Compared to existing methods, it exhibits superior fusion performance, lower fusion error, and clearer details, providing an efficient and reliable technical solution for multi-sensor remote sensing spectral fusion tasks.

[0102] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. An image fusion method based on self-supervised modal interaction, characterized in that, Includes the following steps: The basic features of the low-resolution hyperspectral image are obtained by performing projection transformation on the low-resolution hyperspectral image. Global attention modeling and feature aggregation are then performed on the basic features of the low-resolution hyperspectral image to obtain global spectral features. Attention-enhanced features are obtained by adaptively weighting the basic features of low-resolution hyperspectral images using a channel attention mechanism. The basic features, global spectral features, and attention-enhanced features of a low-resolution hyperspectral image are obtained, and then stitched and fused to obtain the original fused spectral features. The high-resolution multispectral image is divided into several square regions of size p×p, and a unique region index is assigned to each region. With each pixel in the image as the center, an m×m sliding window is used to count the frequency of occurrence of each region index within the window, and the position embedding vector of the corresponding pixel is generated based on the frequency. The position embedding vectors of all pixels are concatenated in order to obtain the position embedding matrix of the high-resolution multispectral image. The location embedding matrix of the high-resolution multispectral image is obtained, the high-resolution multispectral image is subjected to projection transformation, and then concatenated with the location embedding matrix to obtain the basic features of the high-resolution multispectral image. Global attention modeling and feature aggregation are performed on the basic features of high-resolution multispectral images to obtain global spatial features; adaptive weighted enhancement of the basic features of high-resolution multispectral images is performed through channel attention mechanism to obtain spatial enhancement features; the basic features, global spatial features and spatial enhancement features of high-resolution multispectral images are spliced ​​and fused to obtain the original fused spatial features. The basic features, global spatial features, and spatial enhancement features of high-resolution multispectral images are acquired, and then feature splicing and fusion are performed to obtain the original fused spatial features. A cosine similarity focus loss is constructed, and cross-modal alignment constraints are applied to the original fused spectral features and the original fused spatial features based on the cosine similarity focus loss to obtain the aligned fused spectral features and fused spatial features. The aligned fused spatial features are spatially transformed and normalized to obtain the learned abundance matrix; the aligned fused spectral features are spatially transformed and normalized to obtain the degraded abundance matrix; the endmember matrix is ​​obtained based on the degraded abundance matrix and the low-resolution hyperspectral image; the high-resolution hyperspectral image is obtained based on the learned abundance matrix and the endmember matrix; a total loss function is constructed, and the entire process of obtaining the high-resolution hyperspectral image is self-supervised and optimized using the total loss function.

2. The image fusion method based on self-supervised modal interaction according to claim 1, characterized in that, The construction of the cosine similarity focus loss includes: Local region feature vectors are extracted from the original fused spectral features and the original fused spatial features respectively, and the cosine similarity between the local region feature vectors is calculated; the cosine similarity focal loss is obtained based on the cosine similarity between the local region feature vectors.

3. The image fusion method based on self-supervised modal interaction according to claim 1, characterized in that, The construction of the total loss function includes: Based on high-resolution hyperspectral images, reconstruction loss and consistency loss are constructed. Combined with cosine similarity focus loss, the reconstruction loss, consistency loss and cosine similarity focus loss are weighted by loss coefficients to obtain the total loss function.

4. The image fusion method based on self-supervised modal interaction according to claim 3, characterized in that, The reconstruction loss and consistency loss constructed based on high-resolution hyperspectral images include: The high-resolution hyperspectral image is degraded and reconstructed to obtain a low-resolution hyperspectral matrix and a high-resolution multispectral matrix; reconstruction loss and consistency loss are constructed based on the low-resolution hyperspectral matrix and the high-resolution multispectral matrix.

5. The image fusion method based on self-supervised modal interaction according to claim 4, characterized in that, The process of reconstructing a high-resolution hyperspectral image to obtain a low-resolution hyperspectral matrix and a high-resolution multispectral matrix further includes: The high-resolution hyperspectral image is downsampled and convolutionally decoded to obtain a low-resolution hyperspectral matrix; A spectral degradation matrix is ​​constructed, and the high-resolution hyperspectral image is multiplied with the spectral degradation matrix and decoded to obtain a high-resolution multispectral matrix.

6. The image fusion method based on self-supervised modal interaction according to claim 5, characterized in that, The construction of the spectral degradation matrix includes: The high-resolution multispectral image is downsampled to make its resolution consistent with that of the low-resolution hyperspectral image. The optimal linear transformation matrix between the downsampled high-resolution multispectral image and the low-resolution hyperspectral image is solved by least squares to obtain the spectral degradation matrix.

7. An image fusion system based on self-supervised modal interaction, characterized in that, The system is used to perform the image fusion method based on self-supervised modal interaction according to any one of claims 1-6, the system comprising: A spectral encoder is used to perform projection transformation on low-resolution hyperspectral images to obtain the basic features of the low-resolution hyperspectral images. Global attention modeling and feature aggregation are performed on the basic features of the low-resolution hyperspectral images to obtain global spectral features. The basic features of the low-resolution hyperspectral images are adaptively weighted through a channel attention mechanism to obtain attention-enhanced features. The basic features, global spectral features and attention-enhanced features in the low-resolution hyperspectral images are obtained, and then spliced ​​and fused to obtain the original fused spectral features. A spatial encoder is used to divide a high-resolution multispectral image into several p×p square regions, assigning a unique region index to each region. Centered on each pixel in the image, an m×m sliding window is used to count the frequency of occurrence of each region index within the window, generating a position embedding vector for the corresponding pixel based on this frequency. The position embedding vectors of all pixels are concatenated sequentially to obtain the position embedding matrix of the high-resolution multispectral image. The position embedding matrix of the high-resolution multispectral image is obtained, and a projection transformation is performed on the high-resolution multispectral image, concatenating it with the position embedding matrix to obtain the basic features of the high-resolution multispectral image. Global attention modeling and feature aggregation are performed on the basic features of the high-resolution multispectral image to obtain global spatial features. Adaptive weighted enhancement is performed on the basic features of the high-resolution multispectral image using a channel attention mechanism to obtain spatially enhanced features. The basic features, global spatial features, and spatially enhanced features of the high-resolution multispectral image are concatenated and fused to obtain the original fused spatial features. Finally, the basic features, global spatial features, and spatially enhanced features of the high-resolution multispectral image are concatenated and fused to obtain the original fused spatial features. The cosine similarity focus loss module is used to construct the cosine similarity focus loss. Based on the cosine similarity focus loss, cross-modal alignment constraints are applied to the original fused spectral features and the original fused spatial features to obtain the aligned fused spectral features and fused spatial features. The spectral fusion module performs spatial transformation and normalization decoding on the aligned fused spatial features to obtain a learned abundance matrix; it also performs spatial transformation and normalization decoding on the aligned fused spectral features to obtain a degraded abundance matrix, and obtains an endmember matrix based on the degraded abundance matrix and the low-resolution hyperspectral image; it then obtains a high-resolution hyperspectral image based on the learned abundance matrix and the endmember matrix; finally, it constructs a total loss function and performs self-supervised optimization throughout the process of obtaining the high-resolution hyperspectral image using the total loss function.

Images