Hyperspectral and Multispectral Fusion Methods, Systems, and Media Based on Low-Rank Decomposition

Abstract

This invention discloses a hyperspectral and multispectral fusion method, system, and medium based on low-rank decomposition. The method involves fusing the spectral dimensions of hyperspectral and multispectral images to obtain input features and extracting scoring weights from low-rank information, thereby decomposing the input features into low-rank and non-low-rank features. The low-rank features are then reconstructed using tensor decomposition to obtain a low-rank information matrix. The non-low-rank features, along with their attention and gradient features, are sequentially fused and modeled into a non-low-rank information matrix. Finally, the low-rank information matrix, the non-low-rank information matrix, and the input features are fused and reconstructed into output features, which are then decoded to reconstruct the final high-resolution hyperspectral image. This invention aims to generate images with high spectral and spatial resolution by decoupling and fusing spectral and spatial information through low-rank modeling and feature decoupling, effectively improving detection efficiency and image quality.

Description

Technical Field

[0001] This invention relates to image fusion technology in the field of image processing, specifically to a hyperspectral and multispectral fusion method, system, and medium based on low-rank decomposition. Background Technology

[0002] Hyperspectral images possess continuous and fine spectral resolution, making them widely applicable in target recognition, environmental monitoring, and agricultural remote sensing. However, their spatial resolution is typically low, limited by imaging systems and hardware conditions, which affects image detail and practical application effectiveness. While multispectral images offer high spatial resolution, their spectral dimensions are limited, failing to meet the requirements for spectral accuracy. Therefore, a feasible solution is to fuse low-resolution hyperspectral images with high-resolution multispectral images to achieve information complementarity, thereby generating a fused image with both high spatial and spectral resolution. This method does not rely on high-cost hardware; instead, it achieves a balance between spatial structure and spectral information through algorithms, improving the overall image quality. However, in practical fusion processes, issues such as registration errors, dimensional inconsistencies, and noise interference between images can easily lead to spectral distortion or blurred details, thus affecting the performance of downstream tasks. In recent years, low-rank decomposition methods have been increasingly introduced into image fusion tasks due to their advantages in removing redundant information and extracting core features. However, how to construct a hyperspectral and multispectral image fusion model based on low-rank decomposition to improve the spatial detail of the image while maintaining high-fidelity spectral reconstruction remains a technical problem that urgently needs to be solved. Summary of the Invention

[0003] The technical problem to be solved by this invention is to provide a method, system and medium for fusion of hyperspectral and multispectral information based on low-rank decomposition, which addresses the above-mentioned problems of the prior art. This invention aims to generate images with high spectral and high spatial resolution by fusing spectral and spatial information through low-rank modeling and feature decoupling, thereby effectively improving detection efficiency and image quality.

[0004] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0005] A hyperspectral and multispectral fusion method based on low-rank decomposition includes the following steps: fusing low-resolution hyperspectral images... and high-resolution multispectral images The final high-resolution hyperspectral image is reconstructed using a pre-trained hyperspectral image fusion network:

[0006] S101, hyperspectral image and multispectral images Spectral dimension fusion is performed to obtain input features ;

[0007] S102, Input Features Low-rank information prediction yields the scoring weights of low-rank information. Based on the scoring weights of low-rank information Input features Decomposed into low-rank features Non-low rank features ;

[0008] S103, low-rank features Tensor decomposition and reconstruction yields a low-rank information matrix. From input features Obtain the original non-low-rank features from the unpredicted score weighted mask. For the original non-low-rank features Extracting attention features Input features Space x , y Gradient features are obtained by directional gradient extraction. non-low-rank features Attention characteristics Gradient features Sequence-level fusion modeling is performed to obtain a non-low-rank information matrix. ;

[0009] S104, the low-rank information matrix Non-low-rank information matrix and input features Output features obtained through fusion reconstruction ;

[0010] S105 will output features Information decoding is performed to reconstruct the final high-resolution hyperspectral image. .

[0011] Optionally, the functional expression for spectral dimension fusion in step S101 is:

[0012] ,

[0013] In the above formula, Indicates input features, This represents a convolution operation with a kernel size of . ; Indicates a stacking operation. This indicates a bicubic interpolation upsampling operation. Indicates input features, For hyperspectral images, , ,in Indicates the batch size for training. This represents the number of channels in an image. Indicates the width of the image. Indicates the height of the image.

[0014] Optionally, in step S102, the input features are... Low-rank information prediction yields the scoring weights of low-rank information. This refers to input features Input low-rank score prediction module LRD Low-rank information prediction yields the scoring weights of low-rank information. The low-rank score prediction module LRD It includes a local structure awareness submodule, a global context modeling submodule, and a feature fusion scoring submodule; the local structure awareness submodule is used to process input features. Local structure perception is performed to extract local structural features. The local structure perception submodule includes a first channel compressed convolutional layer, a first nonlinear activation layer, and a channel-wise spatial convolutional layer connected in sequence. The global context modeling submodule is used to process the input features. A global context modeling submodule is used to extract global context features. This submodule comprises a second-channel compressed convolutional layer, a second non-linear activation layer, a first dilated convolutional layer, a third non-linear activation layer, and a second dilated convolutional layer connected in sequence. The feature fusion and scoring submodule is used to fuse local structural features and global context features to obtain low-rank information scoring weights. The feature fusion scoring submodule includes a feature splicing layer, a fusion convolutional layer, a fourth nonlinear activation layer, an output convolutional layer, and a sigmoid activation function connected in sequence.

[0015] Optionally, in step S102, the scoring weights are based on the low-rank information. Input features Decomposed into low-rank features Non-low rank features This includes: rating weights for low-rank information in the form of three-dimensional vectors. Flatten and sort by value in descending order, design a low-rank feature proportion parameter. Controlling the proportion of low-rank features and calculating the number of low-rank samples. The input features are divided into low-rank index sets and non-low-rank index sets, and then decomposed into low-rank features by index according to the following formula. Non-low rank features :

[0016] ;

[0017] ;

[0018] In the above formula, This represents element-wise multiplication. This indicates the operation of decomposing non-low-rank and low-rank features based on the index. This indicates the scoring weight of low-rank information. This indicates the scoring weights for non-low-rank information. Indicates will Flattened, it is a one-dimensional low-rank weighted sequence. Represents a set of low-rank indexes. Represents a set of non-low-rank indexes. Indicates a low-rank feature. Indicates non-low-rank features, where , , Indicates the width of the image. Indicates the height of the image.

[0019] Optionally, in step S103, the low-rank features Tensor decomposition and reconstruction yields a low-rank information matrix. The function expression is:

[0020] ;

[0021] In the above formula, This indicates the low-rank features Convert to shape The low-rank feature tensor This indicates that the feature tensor is transformed back into a low-rank information matrix. Output, This represents the low-rank tensor decomposition module; the low-rank tensor decomposition module Including the first rank tensor building block Second-rank tensor construction unit First stacked channel convolutional layers and convolutional layers, second rank tensor building blocks The input is a first-rank tensor building block. The difference between the input and the output is the first-rank tensor construction unit. Second-rank tensor construction unit The outputs of both are stacked using the first stacked channel convolutional layer and then passed through another convolutional layer to obtain a low-rank tensor decomposition module. The output characteristics of the first rank tensor construction unit Second-rank tensor construction unit Both are rank-tensor building blocks. The rank-tensor construction unit The function expression is:

[0022] ;

[0023] In the above formula, Constructing units for rank tensors The output characteristics, Indicates a refactoring operation. For the channel attention extraction module, For spatial attention extraction module, The channel direction feature vector, The feature vector is in the width direction. This is the feature vector in the height direction.

[0024] Optionally, in step S103, the original non-low-rank features are... Extracting attention features This refers to using the attention feature extraction module SAB For original non-low-rank features Extracting attention features The attention feature extraction module SAB The function expression is:

[0025] ,

[0026] In the above formula, The softmax activation function is used. Original non-low-rank features The key obtained through linear mapping Original non-low-rank features The query obtained through linear mapping For learnable parameters; non-low-rank features Attention characteristics Gradient features Sequence-level fusion modeling is performed to obtain a non-low-rank information matrix. This refers to using a sequence-level fusion module Mamba Non-low rank features Attention characteristics Gradient features Sequence-level fusion modeling is performed to obtain a non-low-rank information matrix. The sequence-level fusion module Mamba The function expression is:

[0027] ;

[0028] ;

[0029] ;

[0030] in, This indicates that the number of channels is restored to its original value through a linear layer. , and The output features are the two parallel processing branches of the input features. This indicates a normalization operation. For the sequence fusion modeling module, For gradient embedding module, The SiLU activation function is used. This indicates that the number of channels is expanded through a linear layer. The gradient embedding module The sequence fusion modeling module consists of a linear layer, a Leak ReLU activation layer, and another linear layer connected in sequence. The function expression is:

[0031] ;

[0032] In the above formula, Sequence fusion modeling module The output characteristics, This is a state space module. The function expression is:

[0033] ;

[0034] ;

[0035] In the above formula, Indicates the input feature sequence. Indicates time step The input feature sequence, Indicates the output feature sequence. Indicates time step The output feature sequence, This represents the state variables at each time step. Indicates time step State variables, Indicates time step State variables, Indicates the attenuation factor. For input mapping, For output mapping, Represents the time for each state, where , , These are learnable parameters.

[0036] Optionally, step S104 includes:

[0037] low-rank information matrix Non-low-rank information matrix and input features Merging and reconstructing to obtain intermediate features :

[0038] ;

[0039] ;

[0040] ;

[0041] ;

[0042] In the above formula, Represented by a low-rank information matrix Non-low rank information matrix The reconstructed features obtained from restoration and reconstruction, This indicates the fusion output module, which simultaneously receives input features. as output features The residual connection, the fusion output module Including one Convolutional layer, two Depth-separable convolutional layers and a final The convolutional layers are connected in series, and a ReLU activation layer is introduced after each layer; This indicates an operation that restores and reconstructs non-low-rank and low-rank features based on the index. This indicates the scoring weight of low-rank information. Represents a one-dimensional low-rank weighted sequence. Indicates the sorting of the low-rank index set. Indicates the sorting of a non-low-rank index set; Represents a set of low-rank indexes. Represents a set of non-low-rank indexes;

[0043] intermediate features Perform convolutional downsampling operation and then convert the downsampled features... The features are then used again as new input features and reconstructed in one round to obtain secondary intermediate features. The convolutional downsampling operation is performed by a A convolution with a stride of 4 and a Convolutions with a stride of 2 are used to convert secondary intermediate features. and intermediate features The output features are obtained by fusing according to the following formula. :

[0044] ;

[0045] In the above formula, Indicates use Convolution operations are performed using convolution kernels. Indicates a stacking operation. This indicates a mixed washing operation. Indicates use The convolution kernel will convert the secondary intermediate features An upsampling operation that increases the number of channels by 64 times.

[0046] Optionally, in step S105, the output features will be... Information decoding is performed to reconstruct the final high-resolution hyperspectral image. The function expression is:

[0047] ;

[0048] In the above formula, This represents the final high-resolution hyperspectral image. Indicates use Convolution operations are performed using convolution kernels. This indicates that the output features will be... As a new input feature The output features obtained by executing steps S102 to S104; This indicates that the output features of the previous round will be used. The input features of this round are represented by The output features obtained from steps S102 to S104, and the stacking times in steps S102 to S104 represent... , Represents hyperspectral image The feature map obtained by bicubic interpolation upsampling is used for residual connection.

[0049] The present invention also provides a hyperspectral and multispectral fusion system based on low-rank decomposition, comprising a microprocessor and a memory interconnected thereto, wherein the microprocessor is programmed or configured to execute the hyperspectral and multispectral fusion method based on low-rank decomposition.

[0050] The present invention also provides a computer-readable storage medium storing a computer program or instructions that are programmed or configured to execute the hyperspectral and multispectral fusion method based on low-rank decomposition by a processor.

[0051] The present invention also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the hyperspectral and multispectral fusion method based on low-rank decomposition via a processor.

[0052] Compared with the prior art, the present invention mainly has the following beneficial effects: The method of the present invention includes processing hyperspectral images and multispectral images Spectral dimension fusion is performed to obtain input features ;right Scoring weights for extracting low-rank information ,in accordance with Will Decomposed into low-rank features Non-low rank features ;Will Tensor decomposition and reconstruction yield ; non-low-rank features and its attention characteristics Gradient features Sequence-level fusion modeling is performed to obtain a non-low-rank information matrix. ;Will , and Output features obtained through fusion reconstruction Output features Information decoding is performed to reconstruct the final high-resolution hyperspectral image. This invention, through low-rank modeling and feature decoupling to fuse spectral and spatial information, can generate images with high spectral and spatial resolution, effectively improving detection efficiency and image quality. Attached Figure Description

[0053] Figure 1 This is a schematic diagram of the basic process of the method in an embodiment of the present invention.

[0054] Figure 2 This is a schematic diagram of the network structure of the hyperspectral and multispectral image fusion network in an embodiment of the present invention.

[0055] Figure 3 This is the low-rank score prediction module in this embodiment of the invention. LRD A schematic diagram of the network structure.

[0056] Figure 4 This is the low-rank tensor decomposition module in this embodiment of the invention. A schematic diagram of the network structure.

[0057] Figure 5 This is a rank-tensor construction unit in an embodiment of the present invention. A schematic diagram of the network structure.

[0058] Figure 6 The attention feature extraction module in this embodiment of the invention SAB A schematic diagram of the network structure.

[0059] Figure 7 The sequence-level fusion module in this embodiment of the invention Mamba A schematic diagram of the network structure. Detailed Implementation

[0060] To enable those skilled in the art to better understand the technical solutions of the present invention, the technical solutions of the present invention will be further described in detail below with reference to the accompanying drawings in the embodiments of the present invention.

[0061] like Figure 1 As shown, the hyperspectral and multispectral fusion method based on low-rank decomposition in this embodiment includes the following steps: fusing low-resolution hyperspectral images... and high-resolution multispectral images The final high-resolution hyperspectral image is reconstructed using a pre-trained hyperspectral image fusion network:

[0062] S101, hyperspectral image and multispectral images Spectral dimension fusion is performed to obtain input features ;

[0063] S102, Input Features Low-rank information prediction yields the scoring weights of low-rank information. Based on the scoring weights of low-rank information Input features Decomposed into low-rank features Non-low rank features ;

[0064] S103, low-rank features Tensor decomposition and reconstruction yields a low-rank information matrix. From input features Obtain the original non-low-rank features from the unpredicted score weighted mask. For the original non-low-rank features Extracting attention features Input features Space x , y Gradient features are obtained by directional gradient extraction. non-low-rank features Attention characteristics Gradient features Sequence-level fusion modeling is performed to obtain a non-low-rank information matrix. ;

[0065] S104, the low-rank information matrix Non-low-rank information matrix and input features Output features obtained through fusion reconstruction ;

[0066] S105 will output features Information decoding is performed to reconstruct the final high-resolution hyperspectral image. .

[0067] Step S101 is used to process the input low-resolution hyperspectral image and input high-resolution multispectral images Spectral dimension fusion is performed to obtain input features .like Figure 2 As shown, the function expression for spectral dimension fusion in step S101 of this embodiment is:

[0068] ,

[0069] In the above formula, Indicates input features, This represents a convolution operation with a kernel size of . ; Indicates stacking operation ( Figure 2 (represented by C in Chinese) This indicates a bicubic interpolation upsampling operation. Indicates input features, For hyperspectral images, , ,in Indicates the batch size for training. This represents the number of channels in an image. Indicates the width of the image. Indicates the height of the image.

[0070] In step S102, the input features are processed. Low-rank information prediction yields the scoring weights of low-rank information. This refers to input features Input low-rank score prediction module LRD Low-rank information prediction yields the scoring weights of low-rank information. Its function expression is:

[0071] ;

[0072] In the above formula, This represents the scoring weight of low-rank information.

[0073] like Figure 3 As shown, the low-rank score prediction module in this embodiment LRD It includes a local structure awareness submodule, a global context modeling submodule, and a feature fusion scoring submodule; the local structure awareness submodule is used to process input features. Local structure perception is performed to extract local structural features. The local structure perception submodule includes a first channel compressed convolutional layer (1×1 convolution), a first nonlinear activation layer (Leak RELU), and a channel-wise spatial convolutional layer (1×1 convolution) connected in sequence. The global context modeling submodule is used to process the input features. A global context modeling submodule is used to extract global context features. This submodule comprises a second channel compressed convolutional layer (1×1 convolution), a second non-linear activation layer (Leak RELU), a first dilated convolutional layer (1×1 convolution), a third non-linear activation layer (Leak RELU), and a second dilated convolutional layer (1×1 convolution) connected sequentially. The feature fusion and scoring submodule is used to fuse local structural features and global context features to obtain low-rank information scoring weights. The feature fusion scoring submodule includes sequentially connected feature splicing layers ( Figure 3 The structure consists of a fusion convolutional layer (1×1 convolution), a fourth nonlinear activation layer (LeakRELU), an output convolutional layer (1×1 convolution), and a Sigmoid activation function.

[0074] In step S102 of this embodiment, the scoring weight is based on the low-rank information. Input features Decomposed into low-rank features Non-low rank features This includes: rating weights for low-rank information in the form of three-dimensional vectors. Flatten and sort by value in descending order, design a low-rank feature proportion parameter. Controlling the proportion of low-rank features and calculating the number of low-rank samples. The input features are divided into low-rank index sets and non-low-rank index sets, and then decomposed into low-rank features by index according to the following formula. Non-low rank features :

[0075] ;

[0076] ;

[0077] In the above formula, This represents element-wise multiplication. This indicates the operation of decomposing non-low-rank and low-rank features based on the index. This indicates the scoring weight of low-rank information. This indicates the scoring weights for non-low-rank information. Indicates will Flattened, it is a one-dimensional low-rank weighted sequence. Represents a set of low-rank indexes. Represents a set of non-low-rank indexes. Indicates a low-rank feature. Indicates non-low-rank features, where , , Indicates the width of the image. Indicates the height of the image.

[0078] like Figure 2 As shown, in step S103 of this embodiment, the low-rank feature... Tensor decomposition and reconstruction yields a low-rank information matrix. To reconstruct the module through low-rank feature tensor decomposition Implementation of a low-rank feature tensor decomposition and reconstruction module Low-rank features Tensor decomposition and reconstruction yields a low-rank information matrix. The function expression is:

[0079] ;

[0080] In the above formula, This indicates the low-rank features Convert to shape The low-rank feature tensor This indicates that the feature tensor is transformed back into a low-rank information matrix. Output, This represents the low-rank tensor decomposition module.

[0081] like Figure 4 As shown, the low-rank tensor decomposition module in this embodiment Including the first rank tensor building block Second-rank tensor construction unit First stacked channel convolutional layer ( Figure 4 (represented by C in the original text) and convolutional layers (1×1 convolutions), with the second-rank tensor building unit. The input is a first-rank tensor building block. The difference between the input and the output is the first-rank tensor construction unit. Second-rank tensor construction unit The outputs of both are stacked using the first stacked channel convolutional layer and then passed through another convolutional layer to obtain a low-rank tensor decomposition module. The output characteristics of the first rank tensor construction unit Second-rank tensor construction unit Both are rank-tensor building blocks. ,like Figure 5 As shown, the rank tensor construction unit The function expression is:

[0082] ;

[0083] In the above formula, Constructing units for rank tensors The output characteristics, Indicates a refactoring operation. For the channel attention extraction module, For spatial attention extraction module, The channel direction feature vector, The feature vector is in the width direction. Let be the feature vector in the height direction, and we have:

[0084] ;

[0085] ;

[0086] ;

[0087] In the above formula, This represents a one-dimensional convolution operation with a size of . convolution kernel, This represents the low-rank feature tensor of the input. This indicates that the low-rank feature tensor In image height and image width A weighted average is calculated across two dimensions. (Channel attention extraction module) The spatial attention extraction module consists of two linear layers and a ReLU activation layer. It consists of two 3×3 convolutional layers and a ReLU activation layer.

[0088] In step S103 of this embodiment, the input features are... Obtain the original non-low-rank features from the unpredicted score weighted mask. The function expression is:

[0089] ,

[0090] In the above formula, Indicates input features, Represents a one-dimensional low-rank weighted sequence. Represents a set of non-low-rank indexes. This indicates that non-low-rank features are removed from the input features based on the index. Decomposition operation This represents the original non-low-rank features without a predicted score weighted mask.

[0091] In step S103 of this embodiment, the original non-low-rank features are... Extracting attention features This refers to using the attention feature extraction module SAB For original non-low-rank features Extracting attention features ,like Figure 6 As shown, the attention feature extraction module SAB The function expression is:

[0092] ,

[0093] In the above formula, The softmax activation function is used. Original non-low-rank features The key obtained through linear mapping Original non-low-rank features The query obtained through linear mapping These are learnable parameters.

[0094] In step S103 of this embodiment, the input features are... Space x , y Gradient features are obtained by directional gradient extraction. The function expression is:

[0095] ;

[0096] In the above formula, Representing input features In space x , y The gradient is calculated using the Sobel operator in the direction.

[0097] like Figure 2 As shown, in step S103 of this embodiment, the non-low-rank features are... Attention characteristics Gradient features Sequence-level fusion modeling is performed to obtain a non-low-rank information matrix. To achieve sequence-level fusion module Mamba To achieve this, non-low-rank features Attention characteristics Gradient features Input sequence level fusion module Mamba The non-low-rank information matrix can be obtained by performing sequence-level fusion modeling. ,like Figure 7 As shown, the sequence-level fusion module Mamba The function expression is:

[0098] ;

[0099] ;

[0100] ;

[0101] in, This indicates that the number of channels is restored to its original value through a linear layer. , and The output features are the two parallel processing branches of the input features. This indicates a normalization operation. For the sequence fusion modeling module, For gradient embedding module, The SiLU activation function is used. This indicates that the number of channels is expanded through a linear layer. The gradient embedding module The sequence fusion modeling module consists of a linear layer, a Leak ReLU activation layer, and another linear layer connected in sequence. The function expression is:

[0102] ;

[0103] In the above formula, Sequence fusion modeling module The output characteristics, This is a state space module. The function expression is:

[0104] ;

[0105] ;

[0106] In the above formula, Indicates the input feature sequence. Indicates time step The input feature sequence, Indicates the output feature sequence. Indicates time step The output feature sequence, This represents the state variables at each time step. Indicates time step State variables, Indicates time step State variables, Indicates the attenuation factor. For input mapping, For output mapping, Represents the time for each state, where , , These are learnable parameters.

[0107] In this embodiment, step S104 includes:

[0108] low-rank information matrix Non-low-rank information matrix and input features Merging and reconstructing to obtain intermediate features :

[0109] ;

[0110] ;

[0111] ;

[0112] ;

[0113] In the above formula, Represented by a low-rank information matrix Non-low rank information matrix The reconstructed features obtained from restoration and reconstruction, This indicates the fusion output module, which simultaneously receives input features. as output features The residual connection, the fusion output module Including one Convolutional layer, two Depth-separable convolutional layers and a final The convolutional layers are connected in series, and a ReLU activation layer is introduced after each layer; This indicates an operation that restores and reconstructs non-low-rank and low-rank features based on the index. This indicates the scoring weight of low-rank information. Represents a one-dimensional low-rank weighted sequence. Indicates the sorting of the low-rank index set. Indicates the sorting of a non-low-rank index set; Represents a set of low-rank indexes. Represents a set of non-low-rank indexes;

[0114] intermediate features Perform convolutional downsampling operation and then convert the downsampled features... The features are then used again as new input features and reconstructed in one round to obtain secondary intermediate features. The convolutional downsampling operation is performed by a A convolution with a stride of 4 and a Convolutions with a stride of 2 are used to convert secondary intermediate features. and intermediate features The output features are obtained by fusing according to the following formula. :

[0115] ;

[0116] In the above formula, Indicates use Convolution operations are performed using convolution kernels. Indicates a stacking operation. This indicates a mixed washing operation. Indicates use The convolution kernel will convert the secondary intermediate features An upsampling operation that increases the number of channels by 64 times.

[0117] In step S105 of this embodiment, the output feature will be... Information decoding is performed to reconstruct the final high-resolution hyperspectral image. The function expression is:

[0118] ;

[0119] In the above formula, This represents the final high-resolution hyperspectral image. Indicates use Convolution operations are performed using convolution kernels. This indicates that the output features will be... As a new input feature The output features obtained by executing steps S102 to S104; This indicates that the output features of the previous round will be used. The input features of this round are represented by The output features obtained from steps S102 to S104, and the stacking times in steps S102 to S104 represent... , Represents hyperspectral image The feature map obtained by bicubic interpolation upsampling is used for residual connection.

[0120] To verify the low-rank decomposition-based hyperspectral and multispectral fusion method of this embodiment, experiments were conducted on a publicly available dataset. The proposed fusion method was compared with five currently effective hyperspectral and multispectral fusion methods (NSSR, FusionMamba, DHIF-Net, DSPNet, and Mog-DCN). Peak signal-to-noise ratio (PSNR), spectral angle (SAM), global image quality index (UIQI), and structural similarity (SSIM) were used as evaluation metrics for the image fusion results. The CAVE dataset was used in this embodiment. Each hyperspectral image in the CAVE dataset has 512×512 pixels and 31 spectral bands. The spectral range of the hyperspectral image set is 400nm to 700nm, with a wavelength interval of 10nm. Hyperspectral images from the CAVE dataset were used as reference images in this embodiment. A 7×7 Gaussian blur was applied to the reference images, and then downsampled by 64 pixels in each spectral dimension to simulate a low-resolution hyperspectral image. The response of a Nikon D700 camera was used to generate a three-band high-resolution multispectral image. Table 1 shows the experimental results of the hyperspectral and multispectral fusion method of this embodiment compared with five different methods on the CAVE dataset, with the best performing method highlighted in bold. Table 2 shows the ablation experiments of attention feature extraction (SAB) and gradient feature (G) in the hyperspectral and multispectral fusion method of this embodiment on the CAVE dataset. Table 3 shows the experimental results of the low-rank feature proportion (r) in the hyperspectral and multispectral fusion method of this embodiment on the CAVE dataset.

[0121] Table 1: Quantitative metrics for testing methods on the CAVE dataset

[0122]

[0123] The best results in Table 1 are marked in bold. As shown in Table 1, the method in this embodiment achieves the best results in terms of Peak Signal-to-Noise Ratio (PSNR), Global Image Quality Index (UIQI), and Structural Similarity (SSIM). In the Spectral Angle (SAM) test on the CAVE dataset, the result of this embodiment ranks third. In summary, the method in this embodiment has superior fusion quality compared to other methods and achieves the best fusion results.

[0124] Table 2: Ablation experiments of attention feature extraction (SAB) and gradient feature (G) on the CAVE dataset.

[0125]

[0126] The best results in Table 2 are marked in bold. As shown in Table 2, for the non-low-rank information processing part, when the ordinary convolution module is used to replace the attention feature extraction (SAB) or gradient feature (G), all evaluation metrics decrease to some extent. For example, the PSNR (peak signal-to-noise ratio) decreases by about 0.20dB and 0.32dB, respectively. This means that in the non-low-rank information part of hyperspectral and multispectral fusion, attention feature extraction (SAB) and gradient feature (G) are needed to enhance the decoding and reconstruction of complex information.

[0127] Table 3: Experiments on setting the proportion of low-rank information on the CAVE dataset

[0128]

[0129] The best results in Table 3 are marked in bold. As shown in Table 3, on the CAVE dataset, the overall average performance of the method in this embodiment is the best when the proportion of low-rank features r=0.5. Increasing or decreasing the proportion of low-rank information will reduce the fusion quality to some extent. This effectively demonstrates the scientific validity and effectiveness of the method in this embodiment in separating low-rank and non-low-rank information in the image.

[0130] In summary, the hyperspectral and multispectral image fusion method of this embodiment includes fusion of the input hyperspectral image... and multispectral images Features are obtained by performing basic fusion. ; for features Low-rank information prediction scoring yields low-rank score weights. ,in accordance with Input features Decomposed into low-rank features Non-low rank features ; low-rank features Tensor decomposition and reconstruction yields a low-rank information matrix. From the characteristics Obtain the original non-low-rank features from the unpredicted score weighted mask. ,right Attention feature extraction is performed to obtain Input features Space x , y Gradient features are obtained by directional gradient extraction. ,Will , , Sequence-level fusion modeling is performed to obtain a non-low-rank information matrix. ; low-rank information matrix Non-low-rank information matrix and input features Output features obtained through fusion reconstruction Finally, what will be obtained Information decoding is performed to reconstruct the final high-resolution hyperspectral image. This invention can fully utilize different fusion strategies for low-rank and non-low-rank information in an image, resulting in high fusion accuracy and robustness in obtaining high-resolution hyperspectral images.

[0131] Furthermore, this embodiment also provides a hyperspectral and multispectral fusion system based on low-rank decomposition, including a microprocessor and a memory interconnected, wherein the microprocessor is programmed or configured to execute the hyperspectral and multispectral fusion method based on low-rank decomposition.

[0132] Furthermore, this embodiment also provides a computer-readable storage medium storing a computer program or instructions that are programmed or configured to execute the hyperspectral and multispectral fusion method based on low-rank decomposition by a processor.

[0133] Furthermore, this embodiment also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the hyperspectral and multispectral fusion method based on low-rank decomposition via a processor.

[0134] Those skilled in the art will understand that the technical solutions provided by this invention may take the form of a method, system, or computer program product. Therefore, this invention may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this invention may take the form of a computer program product embodied on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, produce an implementation of the flowchart... Figure 1 One or more processes and / or boxes Figure 1The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0135] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A hyperspectral and multispectral fusion method based on low-rank decomposition, characterized in that, The process includes the following steps: transforming low-resolution hyperspectral images... and high-resolution multispectral images The final high-resolution hyperspectral image is reconstructed using a pre-trained hyperspectral image fusion network: S101, hyperspectral image and multispectral images Spectral dimension fusion is performed to obtain input features ; S102, Input Features Low-rank information prediction yields the score weights of low-rank information. Based on the scoring weights of low-rank information Input features Decomposed into low-rank features Non-low rank features ; S103, low-rank features Tensor decomposition and reconstruction yields a low-rank information matrix. From input features Obtain the original non-low-rank features from the unpredicted score weighted mask. For the original non-low-rank features Extracting attention features Input features Space x , y Gradient features are obtained by directional gradient extraction. non-low-rank features Attention characteristics Gradient features Sequence-level fusion modeling is performed to obtain a non-low-rank information matrix. ; S104, the low-rank information matrix Non-low-rank information matrix and input features Output features obtained through fusion reconstruction ; S105 will output features Information decoding is performed to reconstruct the final high-resolution hyperspectral image. .

2. The hyperspectral and multispectral fusion method based on low-rank decomposition according to claim 1, characterized in that, The function expression for spectral dimension fusion in step S101 is: ， In the above formula, Indicates input features, This represents a convolution operation with a kernel size of . ; Indicates a stacking operation. This indicates a bicubic interpolation upsampling operation. Indicates input features, For hyperspectral images, , ,in Indicates the batch size for training. This represents the number of channels in an image. Indicates the width of the image. Indicates the height of the image.

3. The hyperspectral and multispectral fusion method based on low-rank decomposition according to claim 1, characterized in that, In step S102, the input features are processed. Low-rank information prediction yields the score weights of low-rank information. This refers to input features Input low-rank score prediction module LRD Low-rank information prediction yields the score weights of low-rank information. The low-rank score prediction module LRD It includes a local structure awareness submodule, a global context modeling submodule, and a feature fusion scoring submodule; the local structure awareness submodule is used to process input features. Local structure perception is performed to extract local structural features. The local structure perception submodule includes a first channel compressed convolutional layer, a first nonlinear activation layer and a channel-wise spatial convolutional layer connected in sequence. The global context modeling submodule is used to process input features. A global context modeling submodule is used to extract global context features. This submodule comprises a second-channel compressed convolutional layer, a second non-linear activation layer, a first dilated convolutional layer, a third non-linear activation layer, and a second dilated convolutional layer connected in sequence. The feature fusion and scoring submodule is used to fuse local structural features and global context features to obtain low-rank information scoring weights. The feature fusion scoring submodule includes a feature splicing layer, a fusion convolutional layer, a fourth nonlinear activation layer, an output convolutional layer, and a sigmoid activation function connected in sequence.

4. The hyperspectral and multispectral fusion method based on low-rank decomposition according to claim 1, characterized in that, In step S102, the scoring weights are based on the low-rank information. Input features Decomposed into low-rank features Non-low rank features This includes: rating weights for low-rank information in the form of three-dimensional vectors. Flatten and sort by value in descending order, design a low-rank feature proportion parameter. Controlling the proportion of low-rank features and calculating the number of low-rank samples. The input features are divided into low-rank index sets and non-low-rank index sets, and then decomposed into low-rank features by index according to the following formula. Non-low rank features : ； ； In the above formula, This represents element-wise multiplication. This indicates the operation of decomposing non-low-rank and low-rank features based on the index. This indicates the scoring weight of low-rank information. This indicates the scoring weights for non-low-rank information. Indicates will Flattened, it is a one-dimensional low-rank weighted sequence. Represents a set of low-rank indexes. Represents a set of non-low-rank indexes. Indicates a low-rank feature. Indicates non-low-rank features, where , , Indicates the width of the image. Indicates the height of the image.

5. The hyperspectral and multispectral fusion method based on low-rank decomposition according to claim 1, characterized in that, In step S103, the low-rank features Tensor decomposition and reconstruction yields a low-rank information matrix. The function expression is: ； In the above formula, This indicates the low-rank features Convert to shape The low-rank feature tensor This indicates that the feature tensor is transformed back into a low-rank information matrix. Output, This represents the low-rank tensor decomposition module; the low-rank tensor decomposition module Including the first rank tensor building block Second-rank tensor construction unit First stacked channel convolutional layers and convolutional layers, second rank tensor building blocks The input is a first-rank tensor building block. The difference between the input and the output is the first-rank tensor construction unit. Second-rank tensor construction unit The outputs of both are stacked using the first stacked channel convolutional layer and then passed through another convolutional layer to obtain a low-rank tensor decomposition module. The output characteristics of the first rank tensor construction unit Second-rank tensor construction unit Both are rank-tensor building blocks. The rank-tensor construction unit The function expression is: ； In the above formula, Constructing units for rank tensors The output characteristics, This indicates a refactoring operation. For the channel attention extraction module, For spatial attention extraction module, The feature vector is the channel direction. The feature vector is in the width direction. This is the feature vector in the height direction.

6. The hyperspectral and multispectral fusion method based on low-rank decomposition according to claim 1, characterized in that, In step S103, the original non-low-rank features are processed. Extracting attention features This refers to using the attention feature extraction module SAB For original non-low-rank features Extracting attention features The attention feature extraction module SAB The function expression is: ， In the above formula, The softmax activation function is used. Original non-low-rank features The key obtained through linear mapping Original non-low-rank features The query obtained through linear mapping For learnable parameters; non-low-rank features Attention characteristics Gradient features Sequence-level fusion modeling is performed to obtain a non-low-rank information matrix. This refers to using a sequence-level fusion module Mamba Non-low rank features Attention characteristics Gradient features Sequence-level fusion modeling is performed to obtain a non-low-rank information matrix. The sequence-level fusion module Mamba The function expression is: ； ； ； in, This indicates that the number of channels is restored to its original value through a linear layer. , and The output features are the two parallel processing branches of the input features. This indicates a normalization operation. For the sequence fusion modeling module, For gradient embedding module, The SiLU activation function is used. This indicates that the number of channels is expanded through a linear layer. The gradient embedding module The sequence fusion modeling module consists of a linear layer, a Leak ReLU activation layer, and another linear layer connected in sequence. The function expression is: ； In the above formula, Sequence fusion modeling module The output characteristics, This is a state space module. The function expression is: ； ； In the above formula, Indicates the input feature sequence. Indicates time step The input feature sequence, Indicates the output feature sequence. Indicates time step The output feature sequence, This represents the state variables at each time step. Indicates time step State variables, Indicates time step State variables, Indicates the attenuation factor. For input mapping, For output mapping, Represents the time for each state, where , , These are learnable parameters.

7. The hyperspectral and multispectral fusion method based on low-rank decomposition according to claim 1, characterized in that, Step S104 includes: converting the low-rank information matrix Non-low-rank information matrix and input features Merging and reconstructing to obtain intermediate features : ； ； ； ； In the above formula, Represented by a low-rank information matrix Non-low rank information matrix The reconstructed features obtained from restoration and reconstruction This indicates the fusion output module, which simultaneously receives input features. as output features The residual connection, the fusion output module Including one Convolutional layer, two Depthwise separable convolutional layer and a final The convolutional layers are connected in series, and a ReLU activation layer is introduced after each layer; This indicates an operation that restores and reconstructs non-low-rank and low-rank features based on the index. This indicates the scoring weight of low-rank information. Represents a one-dimensional low-rank weighted sequence. Indicates the sorting of the low-rank index set. Indicates the sorting of a non-low-rank index set; Represents a set of low-rank indexes. Represents a non-low-rank index set; intermediate features Perform convolutional downsampling operation and then convert the downsampled features... The features are then used again as new input features and reconstructed in one round to obtain secondary intermediate features. The convolutional downsampling operation is performed by a A convolution with a stride of 4 and a Convolutions with a stride of 2 are used to convert secondary intermediate features. and intermediate features The output features are obtained by fusing according to the following formula. : ； In the above formula, Indicates use Convolution operations are performed using convolution kernels. Indicates a stacking operation. This indicates a mixed washing operation. Indicates use The convolution kernel will convert the secondary intermediate features Upsampling operation that increases the number of channels by 64 times; in step S105, the output features are... Information decoding is performed to reconstruct the final high-resolution hyperspectral image. The function expression is: ； In the above formula, This represents the final high-resolution hyperspectral image. Indicates use Convolution operations are performed using convolution kernels. This indicates that the output features will be... As a new input feature The output features obtained by executing steps S102 to S104; This indicates that the output features of the previous round will be used. The input features of this round are represented by The output features obtained from steps S102 to S104, and the stacking times in steps S102 to S104 represent... , Represents hyperspectral image The feature map obtained by bicubic interpolation upsampling is used for residual connection.

8. A hyperspectral and multispectral fusion system based on low-rank decomposition, comprising a microprocessor and a memory interconnected, characterized in that, The microprocessor is programmed or configured to execute the hyperspectral and multispectral fusion method based on low-rank decomposition as described in any one of claims 1 to 7.

9. A computer-readable storage medium storing a computer program or instructions, characterized in that, The computer program or instructions are programmed or configured to execute the hyperspectral and multispectral fusion method based on low-rank decomposition as described in any one of claims 1 to 7 via a processor.

10. A computer program product, comprising a computer program or instructions, characterized in that, The computer program or instructions are programmed or configured to execute the hyperspectral and multispectral fusion method based on low-rank decomposition as described in any one of claims 1 to 7 via a processor.

Images