Hyperspectral and panchromatic image fusion method based on feature enhancement channel shuffling

Abstract

The invention discloses a hyperspectral and panchromatic image fusion method based on feature enhancement channel shuffling, and the method comprises the steps: obtaining a hyperspectral image and a panchromatic image, carrying out the preprocessing of the hyperspectral image and the panchromatic image, and correspondingly obtaining a preprocessed hyperspectral image and a preprocessed panchromatic image; a hyperspectral image feature enhancement module based on matrix rank constraint performs feature extraction on the preprocessed hyperspectral image to obtain hyperspectral image features; a filtering-based panchromatic image feature extraction module performs feature extraction on the preprocessed panchromatic image to obtain panchromatic image features; splicing the hyperspectral image features and the panchromatic image features to obtain input features of an image feature fusion part; and carrying out feature fusion on the input features based on multi-scale convolution, extracting features of different scales, and carrying out deep fusion on the features of different scales based on a deep and shallow fusion module of channel shuffling to obtain a final fusion result. According to the invention, the image fusion effect is further improved.

Description

Technical Field

[0001] The invention belongs to the technical field of remote sensing satellite and image fusion, and in particular relates to a hyperspectral and panchromatic image fusion method based on feature enhancement channel shuffling. Background Art

[0002] Deep learning has powerful feature extraction capabilities and has made significant progress in the field of computer vision. Therefore, many deep learning-based methods are also used in remote sensing image fusion. These methods train the network on a large amount of sample data to learn the mapping relationship between low-resolution hyperspectral observation images and high-resolution panchromatic observation images to ideal high-resolution hyperspectral fusion images.

[0003] Zhuo et al. designed the Hyper-DSNet algorithm, which extracts spatial information from panchromatic images through a high-pass filter operator and corrects it through spectral attention, improving the problems of spatial blur and spectral distortion, and reducing network parameters by connecting deep and shallow features. Figure 1 As shown in the figure, the overall framework of the Hyper-DSNet algorithm can be divided into three modules, namely the Multi-Detail Extractor (MDE), the Deep-Shallow Fusion Module (DSF) module and the Spectral Attention (SA) module.

[0004] For the MDE module, the panchromatic image first passes through five high-pass filter operators to extract multi-level high-frequency information, and then connects this high-frequency information with the PAN image itself to form the input feature. The five high-pass operators are the first-order difference operator, RO b ert operator, Prewitt operator, SO b el operator, Laplacian operator.

[0005] First, the output of the MDE module and the upsampled hyperspectral image are spliced ​​along the spectral dimension as the input of the detail extraction structure. The multi-scale convolution module introduces convolution kernels of different sizes (such as 3×3, 5×5, 7×7) to meet the requirements of feature extraction at different scales. The core idea of ​​the multi-scale convolution module is that by applying convolution kernels of multiple scales in parallel, the network can capture information of multiple granularities at the same time, thereby enriching the diversity of feature representation. Hyper-DSNet believes that both shallow and deep features are important. The DSF module connects each shallow convolution and deep convolution result to maintain these two types of key information in each step.

[0006] The low-resolution hyperspectral image is used as the input of the spectral SA module. First, a global average pooling layer is used to aggregate spatial information more conveniently. This operation will output a vector. After that, the global spectral information is compressed into a vector with a length of the number of bands. The output of the DSF module is weighted by the Sigmoid activation function, and finally the result is corrected by the original hyperspectral image.

[0007] However, the Hyper-DSNet network still lacks the ability to process hyperspectral image features based on their rich spectral information, and does not consider the dependency of long-distance similar ground features during fusion. Summary of the invention

[0008] In order to solve the above technical problems, the present invention provides a method for fusion of hyperspectral and panchromatic images based on feature enhancement channel shuffling, comprising:

[0009] Acquire a hyperspectral image and a panchromatic image, and preprocess the hyperspectral image and the panchromatic image respectively to obtain a preprocessed hyperspectral image and a preprocessed panchromatic image;

[0010] A hyperspectral image feature enhancement module based on matrix rank constraints performs feature extraction on the preprocessed hyperspectral image to obtain hyperspectral image features;

[0011] A filtering-based full-color image feature extraction module performs feature extraction on the pre-processed full-color image to obtain full-color image features;

[0012] The hyperspectral image features and the panchromatic image features are spliced ​​to obtain the input features of the image feature fusion part; the input features are fused based on multi-scale convolution to extract features of different scales, and the features of different scales are deeply fused based on the deep and shallow fusion module of channel shuffling to obtain the final fusion result.

[0013] Preferably, the hyperspectral image and the panchromatic image are acquired, and the hyperspectral image and the panchromatic image are preprocessed respectively. The process of obtaining the preprocessed hyperspectral image and the preprocessed panchromatic image includes:

[0014] The low-resolution hyperspectral image LRHS with spectral number B, LRHS∈R 1 / 4H×1 / 4W×B and a single-channel panchromatic image PAN, P∈R H×W×1 Align in the spatial dimension, upsample the LRHS, and obtain the LRHS U ,LRHS U ∈R H×W×B .

[0015] Preferably, the hyperspectral image feature enhancement module based on matrix rank constraint extracts features from the preprocessed hyperspectral image, and the process of obtaining the hyperspectral image features includes:

[0016] The LRHS U Input the hyperspectral image feature enhancement module based on matrix rank constraint to enhance LRHS U Perform dimension increase operation to obtain O LRHS U ∈R 1×H×W×B , and the upper limit of the matrix rank is increased by performing a 1D convolution of size 3 in each of the three dimensions.

[0017] Preferably, the process of increasing the upper limit of the matrix rank by performing a 1D convolution of size 3 in each of the three dimensions comprises:

[0018] Define A∈R M×C×k×k×k It is the convolution kernel tensor of the 3D convolution layer, with M 3D convolution kernels of size k×k×k, where C is the number of input feature channels, by inputting a hyperspectral feature map HS∈R C×B×H×W , the 3D convolutional layer outputs the feature map F∈R M×B′×H′×W′ The 3D convolution process is equivalently expressed as a 2D matrix multiplication:

[0019] F=A·HS

[0020] where F∈R M×B′H′W′ is the matrix form of F, A∈R M×k3C This means that the 3D kernel matrix is ​​flattened into a row vector form, and then F∈R M×B′H′W′ With A∈R M×k3C Stacked vertically together, HS∈R k3C×B′H′W′ represents the matrix form of HS obtained by iterative expansion along the spatial-spectral direction;

[0021] The rank of a 2D matrix represents the independence and freedom of its elements. Based on the properties of matrix multiplication, the diversity of the feature matrix F is quantitatively measured by analyzing its rank, and the upper limit of Rank(A) is improved by directly changing the network topology.

[0022] Among them, the properties based on matrix multiplication are expressed as:

[0023] Rank(F)≤min{Rank(A), Rank(HS)}

[0024] Where Rank(·) returns the rank of the input 2D matrix, M<<k3C<<B′H′W′, Rank(F)≤M.

[0025] Preferably, the filtering-based panchromatic image feature extraction module extracts features from the preprocessed panchromatic image, and before obtaining the panchromatic image features, it also includes:

[0026] Through the 3D convolution structure of the ReConvBlock module, the features of each layer are fused again to fully integrate the deep features; and after each convolution layer, the ReLU activation function is applied to further perform the aggregation step, and finally the O LRHS U Perform dimensionality reduction to return it to its original dimension, H×W×B.

[0027] Preferably, the filtering-based panchromatic image feature extraction module extracts features from the preprocessed panchromatic image, and the process of obtaining the panchromatic image features includes:

[0028] The filtering-based panchromatic image feature extraction module comprehensively applies high-pass filtering operators to extract and fuse multi-level high-frequency information, operates the panchromatic image PAN through the high-pass filtering operator, and splices the obtained features with the original panchromatic image along the channel direction to obtain O P ∈R H×W×8 .

[0029] Preferably, the process of performing feature fusion on the input features based on multi-scale convolution and extracting features of different scales includes:

[0030] Multi-scale convolution is used for feature fusion. By applying 3×3, 5×5 and 7×7 convolution kernels, features of different scales are extracted. The formula is:

[0031]

[0032] O b =[O3O5,O7]

[0033] Among them, O i is the feature map output by convolution kernels (i×i) of different scales, O b Is O i The final output of the multi-scale convolution module is obtained by concatenating along the channel, where δ(·) represents the activation function ReLU.

[0034] Preferably, the deep and shallow fusion module based on channel shuffling performs deep fusion on features of different scales, and the process of obtaining the final fusion result includes:

[0035] O b Input the deep and shallow fusion module based on channel shuffling for deep fusion. In the process of extracting deep and shallow features, the channel shuffling operation is added to capture the dependency of long-distance relationships. The deep global information, shallow local information and the dependency of long-distance features are comprehensively utilized to perform deep fusion.

[0036] The formula expression of the deep and shallow fusion module is:

[0037]

[0038] O c =[O b ,O b1 ,O b2s ,O b3s ,O b4 ]

[0039] Among them, bi represents the output of the i-th convolution layer, O biS It represents the convolution output after channel shuffle, W 3i and b 3i They represent the weight and bias of the 3×3 convolution layer respectively.

[0040] Preferably, the channel shuffle includes horizontal random shuffle and vertical random shuffle;

[0041] Define the plane as V∈RH×W,

[0042]

[0043] Among them, the eigenvector on the plane is denoted by V ij ∈R, and i∈{1,2,...,H},j∈{1,2,...,W}; d is the length of Shuffle, its value is win_size / 2, “win size” represents the window size of self-attention, and Spliceh is used to merge groups G horizontally k ;

[0044] Then, the plane V is divided into (W / d) groups, denoted as G k , where k∈{1,2,...,(W / d)};

[0045] Then, perform the Shuffle operation on the plane V to obtain the horizontal Shuffle result V s After that, V s The same process is performed along the vertical direction to obtain the final Shuffle result.

[0046] Preferably, the method further comprises:

[0047] The final output is corrected by the spectral attention module and then by a simple gating mechanism with Sigmoid activation. The formula is as follows:

[0048] s=σ(W2.δ(W1v b ))

[0049] where the output s∈R B ,σ represents the Sigmoid activation function, W1∈R C / r ×C and W2∈R C ×C / r is the weight of the two convolutional layers, and the kernel size of the two convolutional layers is 1×1;

[0050] In order to reduce the amount of computation, the number of channels is first reduced by a ratio r, and then the number of channels is expanded back to B through a convolutional layer;

[0051] O Output =F scale (O c ,s)+LRHS U

[0052] =[O c1 s1,O c2 s2,...,O cb s c ]+LRHS U

[0053] Where Fscale(·,·) represents the element-by-element dot product. After integrating the deep and shallow fusion module output Oc into the channel attention module, the initial LRHS is added to the final output U as residual;

[0054] The method further comprises:

[0055] The l1 loss function is used to optimize the network during training. The loss function is expressed as follows:

[0056]

[0057] Among them, Output To fusion network output results, LRHS U is the upsampled hyperspectral image, N represents the number of training samples, and |·|1 represents the l1 norm.

[0058] Compared with the prior art, the present invention has the following advantages and technical effects:

[0059] The hyperspectral image feature enhancement module based on matrix rank constraint of the present invention improves the diversity of information by independently applying one-dimensional convolution in three dimensions, significantly enhances the feature expression of hyperspectral images, and improves the utilization of high-dimensional spatial spectral information.

[0060] The filtering-based panchromatic image feature extraction module of the present invention uses a variety of high-pass filters to deeply mine more spatial detail information of the panchromatic image, thereby improving the utilization of high-frequency information of the panchromatic image.

[0061] The deep and shallow fusion module based on channel shuffling of the present invention enhances the model's processing capability for long-distance dependencies through channel shuffling operations, and utilizes highly similar areas in images for complementarity to improve the fusion effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0063] Figure 1 A schematic diagram of the network structure of the Hyper-DSNet algorithm of the background technology of the present invention;

[0064] Figure 2 A schematic diagram of a network structure of an embodiment of the present invention;

[0065] Figure 3 This is a schematic diagram of the structure of a hyperspectral image feature enhancement module based on matrix rank constraint according to an embodiment of the present invention;

[0066] Figure 4 A schematic diagram of the internal structure of ReConvBlock according to an embodiment of the present invention;

[0067] Figure 5 Schematic diagram of the structure of a deep and shallow fusion module based on channel shuffling according to an embodiment of the present invention. DETAILED DESCRIPTION

[0068] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0069] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0070] like Figure 2-5 As shown, in this embodiment, a method for fusing hyperspectral and panchromatic images based on feature enhancement channel shuffling is provided, comprising:

[0071] Acquire a hyperspectral image and a panchromatic image, and preprocess the hyperspectral image and the panchromatic image respectively, so as to obtain a preprocessed hyperspectral image and a preprocessed panchromatic image accordingly;

[0072] The hyperspectral image feature enhancement module based on matrix rank constraint extracts features from the preprocessed hyperspectral image to obtain the hyperspectral image features;

[0073] The filtering-based panchromatic image feature extraction module extracts features from the preprocessed panchromatic image to obtain panchromatic image features;

[0074] The hyperspectral image features and panchromatic image features are spliced ​​to obtain the input features of the image feature fusion part; the input features are fused based on multi-scale convolution to extract features of different scales, and the features of different scales are deeply fused based on the deep and shallow fusion module of channel shuffling to obtain the final fusion result.

[0075] Furthermore, the overall framework of the present invention is as follows Figure 2 As shown in Figure 1, the benchmark model refers to the Hyper-DSNet algorithm designed by Zhuo et al. The entire network structure consists of four parts: hyperspectral image feature processing, panchromatic image feature processing, image feature fusion, and fusion result spectral correction. The network input is a low-resolution hyperspectral image (LRHS) with a spectral number of B, LRHS∈R 1 / 4H×1 / 4W×B and a single-channel panchromatic image (PAN), P∈R H×W×1 Since LRHS and PAN need to be aligned in the spatial dimension, LRHS is upsampled to obtain LRHS U ∈R H×W×B , then, LRHS U Input the hyperspectral image feature enhancement module based on matrix rank constraint. The module structure is as follows Figure 3 shown.

[0076] In order to use 3D convolution to better utilize the spectral dimension information of hyperspectral images, LRHS U The dimension increase operation is performed to obtain O LRHS U ∈R 1×H×W×B , by performing a 1D convolution of size 3 in each of the three dimensions to increase the upper limit of the matrix rank, thereby improving feature diversity.

[0077] Assume A∈R M×C×k×k×k It is the convolution kernel tensor of a typical 3D convolution layer, with M 3D convolution kernels of size k×k×k, where C is the number of input feature channels. By inputting a hyperspectral feature map HS∈R C×B×H×W , the 3D convolutional layer outputs the feature map F∈R M×B′×H′×W′ The present invention can equivalently express the 3D convolution process in the form of 2D matrix multiplication, namely:

[0078] F=A·HS

[0079] where F∈R M×B′H′W′ is the matrix form of F, A∈R M×k3CThis means that the 3D kernel matrix is ​​flattened into row vector form and then stacked vertically, and HS∈R k3C×B′H′W′ represents the matrix form of HS obtained by iterative expansion along the spatial-spectral direction. Considering that the rank of a 2D matrix represents the independence and degree of freedom of its elements, the present invention quantitatively measures its diversity by analyzing the rank of the feature matrix F. Specifically, based on the properties of matrix multiplication:

[0080] Rank(F)≤min{Rank(A), Rank(HS)}

[0081] Where Rank(·) returns the rank of the input 2D matrix. Since the values ​​of M and C are usually at the same level, M<<k3C<<B′H′W′, so Rank(F)≤M. Therefore, lowering the upper limit of Rank(F)≤M, which is essentially limited by the upper limit of Rank(A), is also a prerequisite for ensuring that diverse and powerful features can be learned. To this end, the present invention proposes to increase the upper limit of Rank(A) by directly changing the network topology.

[0082] The ReConvBlock in the network uses a multi-layer structure based on a specially designed convolution kernel to further integrate and enrich features. Figure 4 shown.

[0083] This module continues to use the 3D convolution structure and fuses the features of each layer again to ensure the full integration of deep features. After each convolution layer, the module not only applies the ReLU activation function, but also further performs the aggregation step. These designs enable the model to significantly improve the expressiveness and generalization while maintaining computational efficiency. Finally, the O LRHS U Perform dimensionality reduction processing to return it to its original dimension, H×W×B.

[0084] In the field of remote sensing image fusion, the accurate recovery of high-frequency information (such as edges, contours, textures and other details) has a significant impact on the overall performance of the algorithm. However, the limitations of a single type of high-pass filter in detail information extraction fail to fully exploit the fine spatial details of panchromatic images. To overcome this limitation, the present invention proposes a filtering-based panchromatic image feature extraction module, which extracts and fuses multi-level high-frequency information by comprehensively applying multiple high-pass filter operators, and is expected to improve the detail richness and overall quality of the fused image.

[0085] Considering that the high-frequency information of panchromatic images is rich, a panchromatic image feature extraction module based on filtering is designed. PAN is operated by six different high-pass filter operators, and the obtained features are spliced ​​with the original panchromatic image along the channel direction to obtain O P ∈R H×W×8, the high-frequency information extracted by each operator has its own unique characteristics, some are smoother and some are more refined. This finding proves the effectiveness of the method of the present invention in extracting multi-level high-frequency information to enhance model performance. The specific introduction of the above operators is as follows:

[0086] The first-order difference operator is based on the first-order differential of the image in the horizontal and vertical directions. This operator reveals the speed of grayscale changes in the image in these directions. It is particularly suitable for capturing subtle changes in the image, although it may be relatively sensitive to noise.

[0087] The RObert operator extracts edge information by differentiating in the diagonal direction of the image, and is particularly suitable for identifying small edge details. Although the RObert operator is relatively accurate in edge positioning, it is highly sensitive to noise and may not be suitable for image environments with more noise.

[0088] The Prewitt operator uses a larger 3×3 template to smooth the image while retaining edge information, thus having a certain noise resistance. This operator focuses more on the main edges of the image and is slightly rough in capturing details.

[0089] The Scharr operator is an improved method for calculating image gradients, which is used to more accurately estimate the direction and strength of image edges. Compared with the commonly used SObel operator, the Scharr operator provides better rotational symmetry and edge response. This operator is particularly suitable for situations where high-precision edge detection is required, such as in complex textures or highly structured images.

[0090] As a second-order differential operator, the Laplacian operator is used for edge enhancement of images. Based on the second-order derivative of the image, it is more sensitive to the edge, but at the same time it is more susceptible to noise.

[0091] The Canny operator is designed to extract useful structural information from images while minimizing the occurrence of false detections. The design criteria of the Canny operator are based on the optimization of signal-to-noise ratio, positioning accuracy, and single response performance, making it the preferred edge detection tool in complex scenes.

[0092] The output hyperspectral image features and panchromatic image features are spliced ​​to obtain the input features OP and LRHS of the image feature fusion part. U ∈R H×W×(B+8) Image feature fusion is divided into two steps. The first step is to use multi-scale convolution for feature fusion. By applying convolution kernels of different sizes, namely 3×3, 5×5 and 7×7, features of different scales are extracted, thereby enriching the diversity of features.

[0093]

[0094] O b =[O3,O5,O7]

[0095] O i is the feature map output by convolution kernels (i×i) of different scales, O b Is O i The final output of the multi-scale convolution module is obtained by concatenating along the channel, where δ(·) represents the activation function ReLU.

[0096] The second step is to b Input the deep and shallow fusion module based on channel shuffling for deep fusion. The specific structure of this module is as follows Figure 5 shown.

[0097] Generally speaking, shallow convolution mainly focuses on local areas with small receptive fields, produces fine-grained features, and lacks contextual information. In contrast, deep layers have larger receptive fields and can obtain abstract features with semantic information. In the process of extracting deep and shallow features, channel shuffling operations are added to capture the dependencies of long-distance relationships, thereby comprehensively utilizing the global information of the deep layer, the local information of the shallow layer, and the dependencies of long-distance features. The specific operation of the fusion module is mathematically expressed as follows:

[0098]

[0099] O c =[O b ,O b1 ,O b2s ,O b3s ,O b4 ]

[0100] Among them, bi represents the output of the i-th convolution layer, O biS It represents the convolution output after channel shuffle, W 3i and b 3i Represent the weight and bias of the 3×3 convolution layer. Since shallow and deep convolution kernels with the same number of features will bring certain information redundancy, the number of features decreases from high to low as the convolution layer deepens. The number of channels of the above outputs is set to 48, 32, 16, 8, and 8 respectively. At the same time, adding channel shuffling operations allows features to be connected across windows over long distances, which also reduces the problem of information redundancy to a certain extent.

[0101] Specifically, the channel shuffle operation consists of two parts: horizontal random shuffle and vertical random shuffle. To simplify the description, the plane is defined as V∈RH×W,

[0102]

[0103] Among them, the eigenvector on the plane is denoted by V ij ∈R, and i∈{1,2,...,H},j∈{1,2,...,W}. In the above formula, d is the length of Shuffle, its value is win_size / 2, "win size" represents the window size of self-attention, and Spliceh is used to merge the grouping G horizontally. k Then, the plane V is divided into (W / d) groups, denoted as G k , where k∈{1,2,...,(W / d)}. Then, the Shuffle operation is performed on the plane V, and the result is as follows Figure 3 As shown in the upper right corner. Get the horizontal shuffle result V s Then, a similar strategy is used to s Processing is performed in the vertical direction to produce the final Shuffle result. Reshuffle is the inverse process of Shuffle. The introduction of the window self-attention mechanism can expand the receptive field originally limited by the window size.

[0104] In order to maintain the spectral fidelity of the hyperspectral image, a spectral attention module is used to correct the final output result. This module uses a global average pooling layer to aggregate spatial information more conveniently, and uses a convolution with a kernel size of 1×1 to reduce the amount of calculation. Finally, a simple gating mechanism with Sigmoid activation is used. The specific formula is as follows:

[0105] s=σ(W2.δ(W1v b ))

[0106] where the output s∈R B ,σ represents the Sigmoid activation function, W1∈R C / r ×C and W2∈R C ×C / r are the weights of the two convolutional layers, and the kernel size of these two convolutional layers is 1 × 1. In order to reduce the amount of computation, the number of channels is first reduced by the ratio r, and then expanded back to B through the convolutional layer.

[0107] O Output =F scale (O c ,s)+LRHS U

[0108] =[O c1 s1,O c2 s2,...,O cb s c ]+LRHS U

[0109] Where Fscale(·,·) represents the element-by-element dot product. After integrating the channel attention module with the output of the fusion module Oc, the final output needs to be added with the initial LRHS U As the residual. In other words, the final output can be regarded as U The method adds additional detail information to the original spectral information, thereby preserving its original spectral information, preventing overfitting, avoiding performance degradation as the network depth increases, and accelerating the convergence of the model. This allows the network to be trained faster and achieve better results.

[0110] In order to describe the difference between the network output and the true value (GT), the present invention uses the l1 loss function to optimize the proposed network during the training process. The loss function can be expressed as follows,

[0111]

[0112] Among them, Output To fusion network output results, LRHS U is the upsampled hyperspectral image, N represents the number of training samples, and |·|1 represents the l1 norm.

[0113] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for fusion of hyperspectral and panchromatic images based on feature-enhanced channel shuffling, characterized in that: include: Acquire a hyperspectral image and a panchromatic image, and preprocess the hyperspectral image and the panchromatic image respectively to obtain a preprocessed hyperspectral image and a preprocessed panchromatic image; A hyperspectral image feature enhancement module based on matrix rank constraints performs feature extraction on the preprocessed hyperspectral image to obtain hyperspectral image features; A filtering-based full-color image feature extraction module performs feature extraction on the pre-processed full-color image to obtain full-color image features; The hyperspectral image features and the panchromatic image features are spliced ​​to obtain the input features of the image feature fusion part; the input features are fused based on multi-scale convolution to extract features of different scales, and the features of different scales are deeply fused based on the deep and shallow fusion module of channel shuffling to obtain the final fusion result.

2. The method for fusion of hyperspectral and panchromatic images based on feature enhancement channel shuffling according to claim 1, characterized in that: The hyperspectral image and the panchromatic image are obtained, and the hyperspectral image and the panchromatic image are preprocessed respectively. The process of obtaining the preprocessed hyperspectral image and the preprocessed panchromatic image includes: The low-resolution hyperspectral image LRHS with spectral number B, LRHS∈R 1 / 4H×1 / 4W×B and a single-channel panchromatic image PAN, P∈R H×W×1 Align in the spatial dimension, upsample the LRHS, and obtain the LRHS U ,LRHS U ∈R H×W×B .

3. The method for fusion of hyperspectral and panchromatic images based on feature enhancement channel shuffling according to claim 1, characterized in that: The hyperspectral image feature enhancement module based on matrix rank constraint extracts features from the preprocessed hyperspectral image, and the process of obtaining the hyperspectral image features includes: The LRHS U Input the hyperspectral image feature enhancement module based on matrix rank constraint to enhance LRHS U Perform dimension increase operation to obtain O LRHS U ∈R 1×H×W×B , and the upper limit of the matrix rank is increased by performing a 1D convolution of size 3 in each of the three dimensions.

4. The method for fusion of hyperspectral and panchromatic images based on feature enhancement channel shuffling according to claim 3, characterized in that: The process of increasing the upper limit of the matrix rank by performing a 1D convolution of size 3 in each of the three dimensions includes: Define A∈R M×C×k×k×k It is the convolution kernel tensor of the 3D convolution layer, with M 3D convolution kernels of size k×k×k, where C is the number of input feature channels, by inputting a hyperspectral feature map HS∈R C×B×H×W , the 3D convolutional layer outputs the feature map F∈R M×B′×H′×W′ The 3D convolution process is equivalently expressed as a 2D matrix multiplication: F=A·HS where F∈R M×B′H′W′ is the matrix form of F, A∈R M×k3C This means that the 3D kernel matrix is ​​flattened into a row vector form, and then F∈R M×B′H′W′ With A∈R M×k3C Stacked vertically together, HS∈R k3C×B′H′W′ represents the matrix form of HS obtained by iterative expansion along the spatial-spectral direction; The rank of a 2D matrix represents the independence and freedom of its elements. Based on the properties of matrix multiplication, the diversity of the feature matrix F is quantitatively measured by analyzing its rank, and the upper limit of Rank(A) is improved by directly changing the network topology. Among them, the properties based on matrix multiplication are expressed as: Rank(F)≤min{Rank(A), Rank(HS)} Where Rank(·) returns the rank of the input 2D matrix, M<<k3C<<B′H′W′, Rank(F)≤M.

5. The method for fusion of hyperspectral and panchromatic images based on feature enhancement channel shuffling according to claim 1, characterized in that: The filtering-based panchromatic image feature extraction module extracts features from the preprocessed panchromatic image, and before obtaining the panchromatic image features, it also includes: Through the 3D convolution structure of the ReConvBlock module, the features of each layer are fused again to fully integrate the deep features; and after each convolution layer, the ReLU activation function is applied to further perform the aggregation step, and finally the O LRHS U Perform dimensionality reduction to return it to its original dimension, H×W×B.

6. The method for fusion of hyperspectral and panchromatic images based on feature enhancement channel shuffling according to claim 1, characterized in that: The filtering-based panchromatic image feature extraction module extracts features from the preprocessed panchromatic image, and the process of obtaining the panchromatic image features includes: The filtering-based panchromatic image feature extraction module comprehensively applies high-pass filtering operators to extract and fuse multi-level high-frequency information, operates the panchromatic image PAN through the high-pass filtering operator, and splices the obtained features with the original panchromatic image along the channel direction to obtain O P ∈R H×W×8 .

7. The method for fusion of hyperspectral and panchromatic images based on feature enhancement channel shuffling according to claim 1, characterized in that: The process of fusing the input features based on multi-scale convolution and extracting features of different scales includes: Multi-scale convolution is used for feature fusion. By applying 3×3, 5×5 and 7×7 convolution kernels, features of different scales are extracted. The formula is: <h2 style=";text-align:left;direction:ltr">O<h2 style=";text-align:left;direction:ltr"> b <h2 style=";text-align:left;direction:ltr"> (O3,O5,O7) Among them, O i is the feature map output by convolution kernels (i×i) of different scales, O b Is O i The final output of the multi-scale convolution module is obtained by concatenating along the channel, where δ(·) represents the activation function ReLU.

8. The method for fusion of hyperspectral and panchromatic images based on feature enhancement channel shuffling according to claim 1, characterized in that: The deep and shallow fusion module based on channel shuffling deeply fuses features of different scales. The process of obtaining the final fusion result includes: O b Input the deep and shallow fusion module based on channel shuffling for deep fusion. In the process of extracting deep and shallow features, the channel shuffling operation is added to capture the dependency of long-distance relationships. The deep global information, shallow local information and the dependency of long-distance features are comprehensively utilized to perform deep fusion. The formula expression of the deep and shallow fusion module is: The c =[O b ,O b1 ,O b2s ,The b3s ,The b4 ] Among them, bi represents the output of the i-th convolution layer, O biS It represents the convolution output after channel shuffle, W 3i and b 3i They represent the weight and bias of the 3×3 convolution layer respectively.

9. The method for fusion of hyperspectral and panchromatic images based on feature enhancement channel shuffling according to claim 8, characterized in that: The channel shuffle includes horizontal random shuffle and vertical random shuffle; Define the plane as V∈RH×W, Among them, the eigenvector on the plane is denoted by V ij ∈R, and i∈{1,2,...,H},j∈{1,2,...,W}; d is the length of Shuffle, its value is win_size / 2, "win size" represents the window size of self-attention, and Spliceh is used to merge groups G horizontally k ; Then, the plane V is divided into (W / d) groups, denoted as G k , where k∈{1,2,...,(W / d)}; Then, perform the Shuffle operation on the plane V to obtain the horizontal Shuffle result V s After that, V s The same process is performed along the vertical direction to obtain the final Shuffle result.

10. The method for fusion of hyperspectral and panchromatic images based on feature enhancement channel shuffling according to claim 8, characterized in that: The method further comprises: The final output is corrected by the spectral attention module and then by a simple gating mechanism with Sigmoid activation. The formula is as follows: s=σ(W2.δ(W1v b )) where the output s∈R B ,σ represents the Sigmoid activation function, W1∈R C / r ×C and W2∈R C ×C / r is the weight of the two convolutional layers, and the kernel size of the two convolutional layers is 1×1; In order to reduce the amount of computation, the number of channels is first reduced by a ratio r, and then the number of channels is expanded back to B through a convolutional layer; O Output =F scale (O c ,s)+LRHS U =[O c1 s1,O c2 s2,..,O cb s c ]+LRHS U Where Fscale(·,·) represents the element-by-element dot product. After integrating the deep and shallow fusion module output Oc into the channel attention module, the initial LRHS is added to the final output U as residual; The method further comprises: The l1 loss function is used to optimize the network during training. The loss function is expressed as follows: Among them, Output To fusion network output results, LRHS U is the upsampled hyperspectral image, N represents the number of training samples, and |·|1 represents the l1 norm.

Images