A hyperspectral and panchromatic image fusion method based on feature enhancement channel shuffle

Abstract

The application discloses a hyperspectral and panchromatic image fusion method based on feature enhancement channel mixing, comprising the following steps: acquiring and preprocessing hyperspectral images and panchromatic images respectively, and obtaining the preprocessed hyperspectral images and the preprocessed panchromatic images; performing feature extraction on the preprocessed hyperspectral images based on a matrix rank constraint hyperspectral image feature enhancement module, and obtaining hyperspectral image features; performing feature extraction on the preprocessed panchromatic images based on a filtering panchromatic image feature extraction module, and obtaining panchromatic image features; splicing the hyperspectral image features and the panchromatic image features to obtain input features of an image feature fusion part; performing feature fusion on the input features based on a multi-scale convolution, extracting features of different scales, and performing in-depth fusion on the features of different scales based on a channel mixing deep and shallow fusion module to obtain a final fusion result. The application further improves the image fusion effect.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing satellite and image fusion technology, and in particular relates to a method for fusion of hyperspectral and panchromatic images based on feature enhancement channel washing. Background Technology

[0002] Deep learning possesses powerful feature extraction capabilities and has made significant progress in the field of computer vision. Therefore, many deep learning-based methods have also been applied to remote sensing image fusion. These methods train networks on a large amount of sample data to learn the mapping relationship between low-resolution hyperspectral observation images and high-resolution panchromatic observation images to an ideal high-resolution hyperspectral fused image.

[0003] Zhuo et al. designed the Hyper-DSNet algorithm, which extracts spatial information from panchromatic images using a high-pass filter operator and corrects it using spectral attention, thus improving spatial blur and spectral distortion. Furthermore, it reduces network parameters by connecting light and dark features. Figure 1 As shown, the overall framework of the Hyper-DSNet algorithm can be mainly divided into three modules: Multi-Detail Extractor (MDE), Deep-Shallow Fusion Module (DSF), and Spectral Attention (SA).

[0004] For the MDE module, the panchromatic image is first processed by five high-pass filtering operators to extract multi-level high-frequency information. This high-frequency information is then concatenated with the PAN image itself to form the input features. The five high-pass operators are the first-order difference operator, RO... b ert operator, Prewitt operator, SO b El operator, Laplacian operator.

[0005] First, the output of the MDE module and the upsampled hyperspectral image are stitched together along the spectral dimension as input to the detail extraction structure. The multi-scale convolution module introduces convolutional kernels of different sizes (e.g., 3×3, 5×5, 7×7) to accommodate feature extraction needs at different scales. The core idea of ​​this multi-scale convolution module is to enable the network to capture information at multiple granularities simultaneously by applying convolutional kernels of multiple scales in parallel, thereby enriching the diversity of feature representations. Hyper-DSNet considers both shallow and deep features to be important. The DSF module concatenates the results of each shallow and deep convolution to preserve the key information of both types at each step.

[0006] Low-resolution hyperspectral imagery is used as input to the spectral SA module. First, a global average pooling layer is used to more easily aggregate spatial information, which outputs a vector. Then, the global spectral information is compressed into a vector of length equal to the number of bands. The output of the DSF module is then weighted using a sigmoid activation function. Finally, the result is corrected using the original hyperspectral imagery.

[0007] However, the Hyper-DSNet network still lacks the ability to process hyperspectral image features that are rich in spectral information, and it does not take into account the dependence of distant similar ground features during fusion. Summary of the Invention

[0008] To address the aforementioned technical problems, this invention provides a method for fusing hyperspectral and panchromatic images based on feature enhancement channel shuffling, comprising:

[0009] Acquire hyperspectral and panchromatic images, and preprocess the hyperspectral and panchromatic images respectively to obtain preprocessed hyperspectral and panchromatic images.

[0010] The hyperspectral image feature enhancement module based on matrix rank constraint performs feature extraction on the preprocessed hyperspectral image to obtain hyperspectral image features.

[0011] The filter-based panchromatic image feature extraction module extracts features from the preprocessed panchromatic image to obtain panchromatic image features.

[0012] The hyperspectral image features and panchromatic image features are stitched together to obtain the input features for the image feature fusion part; the input features are fused based on multi-scale convolution to extract features at different scales, and the features at different scales are further fused based on the channel shuffling deep and shallow fusion module to obtain the final fusion result.

[0013] Preferably, the process of acquiring hyperspectral and panchromatic images, and preprocessing the hyperspectral and panchromatic images respectively to obtain preprocessed hyperspectral and panchromatic images includes:

[0014] The low-resolution hyperspectral image LRHS with spectral number B, LRHS∈R 1 / 4H×1 / 4W×B And a single-channel panchromatic image PAN, P∈R H×W×1 Alignment is performed in the spatial dimension, and LRHS is upsampled to obtain LRHS. U LRHS U ∈R H×W×B .

[0015] Preferably, the hyperspectral image feature enhancement module based on matrix rank constraints extracts features from the preprocessed hyperspectral image to obtain hyperspectral image features, including:

[0016] LRHS U Input the hyperspectral image feature enhancement module based on matrix rank constraints to LRHS U Perform a dimension increase operation to obtain O LRHS U ∈R 1×H×W×B The upper limit of the matrix rank is increased by performing a 1D convolution of size 3 once in each of the three dimensions.

[0017] Preferably, the process of increasing the upper bound of the matrix rank by performing a 1D convolution of size 3 once in each of the three dimensions includes:

[0018] Define A∈R M×C×k×k×k It is the convolution kernel tensor of a 3D convolutional layer, with M 3D convolution kernels of size k×k×k, where C is the number of input feature channels, and it is fed by a hyperspectral feature map HS∈R. C×B×H×W 3D convolutional layer output feature map F∈R M×B′×H′×W′ The 3D convolution process can be equivalently expressed as a 2D matrix multiplication, as follows:

[0019] F = A·HS

[0020] Where F∈R M×B′H′W′ It is the matrix form of F, A∈R M×k3C This means that the 3D kernel matrix is ​​flattened into row vector form, and then F∈R M×B′H′W′ With A∈R M×k3C Stacked vertically together, HS∈R k3C×B′H′W′ This represents the matrix form of HS obtained by iterative expansion along the space-spectral direction;

[0021] Based on the rank of a 2D matrix, which represents the independence and degree of freedom of its elements, and based on the properties of matrix multiplication, we can quantitatively measure the diversity of the feature matrix F by analyzing its rank, and increase the upper limit of Rank(A) by directly changing the network topology.

[0022] The properties based on matrix multiplication are expressed as follows:

[0023] Rank(F)≤min{Rank(A), Rank(HS)}

[0024] Rank(·) returns the rank of the input 2D matrix, M << k3C << B′H′W′, Rank(F) ≤ M.

[0025] Preferably, the panchromatic image feature extraction module based on filtering performs feature extraction on the preprocessed panchromatic image, and before obtaining the panchromatic image features, it further includes:

[0026] The ReConvBlock module's 3D convolutional structure further fuses features from each layer, comprehensively integrating deep features. After each convolutional layer, a ReLU activation function is applied for further aggregation. Finally, the O... LRHS U Dimensionality reduction is performed to bring it back to its original dimensions, H×W×B.

[0027] Preferably, the panchromatic image feature extraction module based on filtering extracts features from the preprocessed panchromatic image, and the process of obtaining panchromatic image features includes:

[0028] The panchromatic image feature extraction module based on filtering comprehensively applies high-pass filtering operators to extract and fuse multi-level high-frequency information. It operates on the panchromatic image PAN using high-pass filtering operators and then stitches the obtained features with the original panchromatic image along the channel direction to obtain O0. P ∈R H×W×8 .

[0029] Preferably, the process of extracting features at different scales by fusing the input features based on multi-scale convolution includes:

[0030] Multi-scale convolution is used for feature fusion. By applying 3×3, 5×5, and 7×7 convolution kernels, features at different scales are extracted. The formula expression is as follows:

[0031]

[0032] O b =[O3O5,O7]

[0033] Among them, O i For feature maps output by convolutional kernels (i×i) of different scales, O b It is O i The final output of this multi-scale convolutional module is obtained by splicing along the channels, where δ(·) represents the ReLU activation function.

[0034] Preferably, the process of using a channel-washing-based deep-shallow fusion module to perform in-depth fusion of features at different scales to obtain the final fusion result includes:

[0035] O b The input is a channel-based deep-shallow fusion module for deep fusion. In the process of extracting deep and shallow features, a channel shuffling operation is added to capture the dependency of long-distance relationships. The deep global information, the shallow local information, and the dependency of long-distance features are comprehensively utilized to perform deep fusion.

[0036] The formula expression for the deep-shallow fusion module is:

[0037]

[0038] O c =[O b O b1 O b2s O b3s O b4 ]

[0039] Among them O bi Represents the output of the i-th convolutional layer, O biS This indicates the convolution output after channel shuffling, W 3i and b 3i These represent the weights and biases of the 3×3 convolution in that layer, respectively.

[0040] Preferably, the channel shuffling includes a horizontal random shuffle and a vertical random shuffle;

[0041] Define the plane as V∈RH×W,

[0042]

[0043] The eigenvectors on the plane are denoted as V. ij ∈R, and i∈{1,2,...,H},j∈{1,2,...,W}; d is the length of the Shuffle, with a value of win_size / 2, where "win size" represents the window size for self-attention, and Spliceh is used for horizontal merging of groups G. k ;

[0044] Subsequently, plane V is divided into (W / d) groups, denoted as G. k , where k∈{1,2,...,(W / d)};

[0045] Then, a shuffle operation is performed on plane V to obtain the horizontal shuffle result V. s Afterwards, regarding V s Perform the same process along the vertical direction to obtain the final Shuffle result.

[0046] Preferably, the method further includes:

[0047] The final output is corrected using a spectral attention module, and then passed through a simple gating mechanism with sigmoid activation, as shown in the following formula:

[0048] s=σ(W2.δ(W1v b ))

[0049] Where the output s∈R B σ represents the Sigmoid activation function, W1∈R C / r ×C and W2∈R C ×C / r is the weight of the two convolutional layers, and the kernel size of the two convolutional layers is 1×1;

[0050] To reduce computation, the number of channels is first reduced by a ratio r, and then the number of channels is expanded back to B using a convolutional layer;

[0051] O Output =F scale (O c ,s)+LRHS U

[0052] =[O c1 s1,O c2 s2, ...,O cb s c ]+LRHS U

[0053] Where Fscale(·,·) represents element-wise dot product, after integrating the Oc channel attention module with the output of the shallow / deep blending module, an initial LRHS is added to the final output. U As a residual;

[0054] The method further includes:

[0055] During training, the L1 loss function is used to optimize the network. The loss function is expressed as follows:

[0056]

[0057] Among them O Output To fuse the network output results, LRHS U The image is an upsampled hyperspectral image, N represents the number of training samples, and |·|1 represents the l1 norm.

[0058] Compared with the prior art, the present invention has the following advantages and technical effects:

[0059] The hyperspectral image feature enhancement module based on matrix rank constraints of this invention enhances the diversity of information by independently applying one-dimensional convolution in three dimensions, significantly improving the feature representation of hyperspectral images and enhancing the utilization of high-dimensional spatial spectral information.

[0060] The present invention provides a panchromatic image feature extraction module based on filtering. By using a variety of high-pass filters, it can deeply mine more spatial detail information of panchromatic images and improve the utilization of high-frequency information of panchromatic images.

[0061] This invention uses a channel-based shimmering-based deep-shallow fusion module to enhance the model's ability to handle long-distance dependencies through channel shimmering operations, and utilizes highly similar regions in the image for complementary enhancement of the fusion effect. Attached Figure Description

[0062] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0063] Figure 1 This is a schematic diagram of the Hyper-DSNet algorithm network structure, which is the background technology of this invention.

[0064] Figure 2 This is a schematic diagram of the network structure according to an embodiment of the present invention;

[0065] Figure 3 This is a schematic diagram of the structure of the hyperspectral image feature enhancement module based on matrix rank constraints according to an embodiment of the present invention;

[0066] Figure 4 This is a schematic diagram of the internal structure of ReConvBlock according to an embodiment of the present invention;

[0067] Figure 5 This is a schematic diagram of the deep and shallow fusion module based on channel shuffling in an embodiment of the present invention. Detailed Implementation

[0068] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0069] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0070] like Figure 2-5 As shown, this embodiment provides a method for fusion of hyperspectral and panchromatic images based on feature enhancement channel shuffling, including:

[0071] Acquire hyperspectral and panchromatic images, and preprocess them respectively to obtain preprocessed hyperspectral and panchromatic images.

[0072] The hyperspectral image feature enhancement module based on matrix rank constraint performs feature extraction on the preprocessed hyperspectral image to obtain hyperspectral image features.

[0073] The filter-based panchromatic image feature extraction module extracts features from the preprocessed panchromatic image to obtain panchromatic image features.

[0074] The hyperspectral image features and panchromatic image features are stitched together to obtain the input features for the image feature fusion part. The input features are then fused based on multi-scale convolution to extract features at different scales. Finally, the features at different scales are further fused based on a channel shuffling deep and shallow fusion module to obtain the final fusion result.

[0075] Furthermore, the overall framework of the present invention is as follows: Figure 2 As shown, the baseline model, based on the Hyper-DSNet algorithm designed by Zhuo et al., comprises four parts: hyperspectral image feature processing, panchromatic image feature processing, image feature fusion, and spectral correction of the fusion result. The network inputs are low-resolution hyperspectral images (LRHS) with spectral number B, where LRHS ∈ R. 1 / 4H×1 / 4W×B And a single-channel panchromatic image (PAN), P∈R H×W×1 Since LRHS and PAN need to be aligned in spatial dimensions, LRHS is upsampled to obtain LRHS. U ∈R H×W×B After that, LRHS U Input a hyperspectral image feature enhancement module based on matrix rank constraints. The module structure is as follows: Figure 3 As shown.

[0076] To better utilize the spectral dimensionality information of hyperspectral images by using 3D convolution, LRHS... U The dimension-increasing operation was performed, resulting in O. LRHS U ∈R 1×H×W×B The upper limit of the matrix rank is increased by performing a 1D convolution of size 3 once in each of the three dimensions, thereby improving feature diversity.

[0077] Assume A∈R M×C×k×k×k It is a typical 3D convolutional layer kernel tensor with M k×k×k 3D convolutional kernels, where C is the number of input feature channels. It takes a hyperspectral feature map HS∈R as input. C×B×H×W 3D convolutional layer output feature map F∈R M×B′×H′×W′ This invention can equivalently express the 3D convolution process as a 2D matrix multiplication, that is:

[0078] F = A·HS

[0079] Where F∈R M×B′H′W′ It is the matrix form of F, A∈R M×k3CThis means that the 3D kernel matrix is ​​flattened into row vector form and then stacked vertically together, where HS∈R k3C×B′H′W′ This represents the matrix form of HS obtained through iterative expansion along the space-spectral direction. Considering that the rank of a 2D matrix represents the independence and degrees of freedom of its elements, this invention quantitatively measures its diversity by analyzing the rank of the characteristic matrix F. Specifically, based on the properties of matrix multiplication:

[0080] Rank(F)≤min{Rank(A), Rank(HS)}

[0081] Rank(·) returns the rank of the input 2D matrix. Since the values ​​of M and C are usually at the same level, we have M << k3C << B′H′W′, thus Rank(F) ≤ M. Therefore, lowering the upper limit of Rank(F) ≤ M, which is essentially limited by the upper limit of Rank(A), is also a prerequisite for ensuring the learning of diverse and powerful features. To this end, this invention proposes to increase the upper limit of Rank(A) by directly changing the network topology.

[0082] The ReConvBlock network, based on specially designed convolutional kernels, employs a multi-layer structure to further integrate and enrich features, as shown in the structure... Figure 4 As shown.

[0083] This module continues to employ a 3D convolutional structure, further fusing features from each layer to ensure comprehensive integration of deep features. Furthermore, after each convolutional layer, the module not only applies the ReLU activation function but also performs an aggregation step. These designs enable the model to significantly improve expressive power and generalization while maintaining computational efficiency. Finally, O... LRHS U We perform dimensionality reduction to bring it back to its original dimensions, H×W×B.

[0084] In the field of remote sensing image fusion, the accurate recovery of high-frequency information (such as details like edges, contours, and textures) has a significant impact on the overall performance of the algorithm. However, the limitations of single-type high-pass filters in extracting detail information fail to fully exploit the subtle spatial details of panchromatic images. To overcome this limitation, this invention proposes a filter-based panchromatic image feature extraction module. This module extracts and fuses multi-level high-frequency information by comprehensively applying multiple high-pass filter operators, which is expected to improve the detail richness and overall quality of the fused image.

[0085] Considering the rich high-frequency information in panchromatic images, a filter-based panchromatic image feature extraction module was designed. Six different high-pass filtering operators were applied to the panchromatic image (PAN), and the resulting features were then stitched together with the original panchromatic image along the channel direction to obtain O0. P ∈R H×W×8Each operator extracts high-frequency information with unique characteristics; some are smoother, while others are more refined. This finding demonstrates the effectiveness of the method described in this invention in extracting multi-level high-frequency information to enhance model performance. The specific descriptions of the above operators are as follows:

[0086] The first-order difference operator is based on the first-order derivatives of an image in the horizontal and vertical directions. This operator reveals the rate of gray-level change in the image in these directions. It is particularly suitable for capturing subtle changes in an image, although it may be relatively sensitive to noise.

[0087] The Robert operator extracts edge information by differentiating along the diagonal of the image, making it particularly suitable for identifying fine edge details. While the Robert operator is relatively accurate in edge localization, it is highly sensitive to noise and may not be suitable for noisy image environments.

[0088] The Prewitt operator uses a large 3×3 template to smooth the image while preserving edge information, thus providing some noise resistance. This operator focuses more on the main edges of the image, while its detail capture is slightly coarser.

[0089] The Scharr operator is an improved method for calculating image gradients, used to more accurately estimate the direction and intensity of image edges. Compared to the commonly used SObel operator, the Scharr operator offers superior rotational symmetry and edge response. This operator is particularly suitable for applications requiring high-precision edge detection, such as in images with complex textures or high structures.

[0090] The Laplacian operator, a second-order differential operator, is used for edge enhancement in images. Based on the second derivative of the image, it is more sensitive to edge responses, but also more susceptible to noise.

[0091] The Canny operator aims to extract useful structural information from images while minimizing false detections. Its design principles are based on optimizing signal-to-noise ratio, localization accuracy, and single-response performance, making it the preferred edge detection tool for complex scenes.

[0092] The output hyperspectral image features and panchromatic image features are stitched together to obtain the input features OP and LRHS of the image feature fusion part. U ∈R H×W×(B+8) Image feature fusion consists of two steps. The first step uses multi-scale convolution for feature fusion. By applying convolution kernels of different sizes, namely 3×3, 5×5 and 7×7, features of different scales are extracted, thereby enriching the diversity of features.

[0093]

[0094] O b =[O3,O5,O7]

[0095] O i For feature maps output by convolutional kernels (i×i) of different scales, O b It is O i The final output of this multi-scale convolutional module is obtained by splicing along the channels, where δ(·) represents the ReLU activation function.

[0096] The second step is to place O b The input is processed by a channel-based shuffling deep-shallow fusion module for deep fusion. The specific structure of this module is as follows: Figure 5 As shown.

[0097] Generally, shallow convolutions focus on local regions with small receptive fields, producing fine-grained features but lacking contextual information. In contrast, deeper layers have larger receptive fields, yielding abstract features with semantic information. During the extraction of features from both shallow and deep layers, a channel shuffling operation is incorporated to capture dependencies between distant relationships, thus comprehensively utilizing global information from deep layers, local information from shallow layers, and dependencies between distant features. The specific operations of the fusion module are mathematically expressed as follows:

[0098]

[0099] O c =[O b O b1 O b2s O b3s O b4 ]

[0100] Among them O bi Represents the output of the i-th convolutional layer, O biS This indicates the convolution output after channel shuffling, W 3i and b 3i These represent the weights and biases of the 3×3 convolution in this layer, respectively. Since shallow and deep convolution kernels with the same number of features can introduce information redundancy, the number of features decreases from high to low as the convolutional layer deepens. The number of output channels above are set to 48, 32, 16, 8, and 8 respectively. Furthermore, the addition of channel shuffling allows for long-distance cross-window connections of features, which also reduces information redundancy to some extent.

[0101] The channel shuffling operation specifically consists of two parts: a horizontal random shuffle and a vertical random shuffle. For simplicity, the plane is defined as V∈RH×W.

[0102]

[0103] The eigenvectors on the plane are denoted as V. ij ∈R, and i∈{1,2,...,H},j∈{1,2,...,W}. In the above formula, d is the length of the Shuffle, which is win_size / 2, where "win size" represents the window size for self-attention, and Spliceh is used for horizontal merging of groups G. k Subsequently, plane V is divided into (W / d) groups, denoted as G. k , where k∈{1,2,...,(W / d)}. Then, a Shuffle operation is performed on plane V, and the result is as follows. Figure 3 As shown in the top right corner. The horizontal Shuffle result V is obtained. s Then, a similar strategy was used on V. s Processing is performed vertically to produce the final Shuffle result. Reshuffle is the inverse process of Shuffle, and the introduction of a window self-attention mechanism can expand the receptive field that was originally limited by the window size.

[0104] To maintain the spectral fidelity of the hyperspectral image, a spectral attention module is used to correct the final output. This module uses a global average pooling layer to more easily aggregate spatial information, a 1×1 kernel convolution to reduce computation, and finally a simple gating mechanism with sigmoid activation. The specific formula is as follows:

[0105] s=σ(W2.δ(W1v b ))

[0106] Where the output s∈R B σ represents the Sigmoid activation function, W1∈R C / r ×C and W2∈R C ×C / r represents the weights of the two convolutional layers, each with a kernel size of 1×1. To reduce computation, the number of channels is first reduced by a scaling factor r, and then expanded back to B using convolutional layers.

[0107] O Output =F scale (O c ,s)+LRHS U

[0108] =[O c1 s1,O c2 s2,...,O cb s c ]+LRHS U

[0109] Where Fscale(·,·) represents element-wise dot product. After integrating the channel attention module with the output of the fusion module mentioned above, the final output still needs to be processed by adding the initial LRHS. U As a residual. In other words, the final output can be seen as the result of LRHS. U Additional detailed information was added to the existing model, thus preserving its original spectral information, preventing overfitting, avoiding performance degradation as network depth increases, and accelerating model convergence. This allows the network to train faster and achieve better results.

[0110] To describe the difference between the network output and the ground truth (GT), this invention employs an L1 loss function to optimize the proposed network during training. The loss function can be expressed as follows:

[0111]

[0112] Among them O Output To fuse the network output results, LRHS U The image is an upsampled hyperspectral image, N represents the number of training samples, and |·|1 represents the l1 norm.

[0113] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for fusing hyperspectral and panchromatic images based on feature enhancement channel shuffling, characterized in that, include: Acquire hyperspectral and panchromatic images, and preprocess the hyperspectral and panchromatic images respectively to obtain preprocessed hyperspectral and panchromatic images. The hyperspectral image feature enhancement module based on matrix rank constraint performs feature extraction on the preprocessed hyperspectral image to obtain hyperspectral image features. The filter-based panchromatic image feature extraction module extracts features from the preprocessed panchromatic image to obtain panchromatic image features. The hyperspectral image features and panchromatic image features are stitched together to obtain the input features for the image feature fusion part; the input features are fused based on multi-scale convolution to extract features at different scales, and the features at different scales are further fused based on the channel shuffling deep and shallow fusion module to obtain the final fusion result; The hyperspectral image feature enhancement module based on matrix rank constraints performs feature extraction on the preprocessed hyperspectral image to obtain hyperspectral image features. The process includes: LRHS U Input a matrix rank-constrained hyperspectral image feature enhancement module to perform LRHS U Perform a dimension increase operation to obtain O LRHS U ∈R 1×H×W×B And by performing a 1D convolution of size 3 once in each of the three dimensions, the upper limit of the matrix rank is increased; The process of fusing the input features based on multi-scale convolution to extract features at different scales includes: Multi-scale convolution is used for feature fusion. By applying 3×3, 5×5, and 7×7 convolution kernels, features at different scales are extracted. The formula expression is as follows: Among them, O i For feature maps output by convolutional kernels of different scales (i×i), O b It is O i The final output of this multi-scale convolutional module is obtained by concatenating the data along the channels, where δ(·) represents the ReLU activation function; P Features of panchromatic images; The channel-washing-based deep-shallow fusion module performs in-depth fusion of features at different scales to obtain the final fusion result. The process includes: O b The input is a channel-based deep-shallow fusion module for deep fusion. In the process of extracting deep and shallow features, a channel shuffling operation is added to capture the dependency of long-distance relationships. The deep global information, the shallow local information, and the dependency of long-distance features are comprehensively utilized to perform deep fusion. The formula expression for the deep-shallow fusion module is: Among them O bi Represents the output of the i-th convolutional layer, O biS This indicates the convolution output after channel shuffling, W 3i and b 3i These represent the weights and biases of the 3×3 convolution in this layer, respectively. The channel shuffle includes horizontal random shuffle and vertical random shuffle; Define the plane as V∈R H×W , The eigenvectors on the plane are denoted as V. ij ∈R, and i∈{1,2,...,H},j∈{1,2,...,W}; d is the length of the Shuffle, with a value of win_size / 2, where "win size" represents the window size for self-attention, and Spliceh is used for horizontal merging of groups G. k； Subsequently, plane V is divided into (W / d) groups, denoted as G. k , where k∈{1,2,...,(W / d)}; Then, a shuffle operation is performed on plane V to obtain the horizontal shuffle result V. s Afterwards, regarding V s Perform the same process along the vertical direction to obtain the final Shuffle result.

2. The hyperspectral and panchromatic image fusion method based on feature enhancement channel shuffling according to claim 1, characterized in that, The process of acquiring hyperspectral and panchromatic images, and preprocessing the hyperspectral and panchromatic images respectively to obtain preprocessed hyperspectral and panchromatic images includes: The low-resolution hyperspectral image LRHS with spectral number B, LRHS∈R 1 / 4H×1 / 4W×B And a single-channel panchromatic image PAN, P∈R H×W×1 Alignment is performed in the spatial dimension, and LRHS is upsampled to obtain LRHS. U LRHS U ∈R H×W×B .

3. The hyperspectral and panchromatic image fusion method based on feature enhancement channel shuffling according to claim 1, characterized in that, The process of increasing the upper bound of a matrix rank by performing a 1D convolution of size 3 once in each of the three dimensions includes: Define A∈R M×k3C It is the kernel tensor of a 3D convolutional layer, with M 3D convolutional kernels of size k×k×k, where C is the number of input feature channels, and it is fed by a hyperspectral feature map HS∈R. k3C×B′H′W′ 3D convolutional layer output feature map F∈R M×B′×H′×W′ The 3D convolution process can be equivalently expressed as a 2D matrix multiplication, as follows: Where F∈R M×B′×H′×W′ It is the matrix form of F, A∈R M×k3C This means that the 3D kernel matrix is ​​flattened into row vector form, and then F∈R M×B′×H′×W′ With A∈R M×k3C Stacked vertically together, HS∈R k3C×B′H′W′ This represents the matrix form of HS obtained by iterative expansion along the space-spectral direction; Based on the rank of a 2D matrix, which represents the independence and degree of freedom of its elements, and based on the properties of matrix multiplication, we can quantitatively measure the diversity of the feature matrix F by analyzing its rank, and increase the upper limit of Rank(A) by directly changing the network topology. The properties based on matrix multiplication are expressed as follows: Rank(·) returns the rank of the input 2D matrix, M≪k3C≪B′H′W′, Rank(F)≤M.

4. The hyperspectral and panchromatic image fusion method based on feature enhancement channel shuffling according to claim 1, characterized in that, The filter-based panchromatic image feature extraction module extracts features from the preprocessed panchromatic image. Before obtaining the panchromatic image features, the module also includes: The ReConvBlock module's 3D convolutional structure further fuses features from each layer, comprehensively integrating deep features. After each convolutional layer, a ReLU activation function is applied for further aggregation. Finally, the O... LRHS U Dimensionality reduction is performed to bring it back to its original dimensions, H×W×B.

5. The hyperspectral and panchromatic image fusion method based on feature enhancement channel shuffling according to claim 1, characterized in that, The filter-based panchromatic image feature extraction module extracts features from the preprocessed panchromatic image. The process of obtaining panchromatic image features includes: The panchromatic image feature extraction module based on filtering comprehensively applies high-pass filtering operators to extract and fuse multi-level high-frequency information. It operates on the panchromatic image PAN using high-pass filtering operators and then stitches the obtained features with the original panchromatic image along the channel direction to obtain O0. P ∈R H×W×8 .

6. The hyperspectral and panchromatic image fusion method based on feature enhancement channel shuffling according to claim 1, characterized in that, The method further includes: The final output is corrected using a spectral attention module, and then passed through a simple gating mechanism with sigmoid activation, as shown in the following formula: Where the output s∈R B σ represents the Sigmoid activation function, W1∈R C / r×C and W2∈R C×C / r These are the weights of two convolutional layers, and the kernel size of the two convolutional layers is 1×1; To reduce computation, the number of channels is first reduced by a ratio r, and then the number of channels is expanded back to B using a convolutional layer; Where Fscale(·,·) represents element-wise dot product, after integrating the Oc channel attention module with the output of the shallow / deep blending module, an initial LRHS is added to the final output. U As a residual; The method further includes: During training, a ℓ1 loss function is used to optimize the network. The loss function is expressed as follows: Among them O Output To fuse the network output results, LRHS U The image is an upsampled hyperspectral image, N represents the number of training samples, and ||·||1 represents the ℓ1 norm.

Images