Image super-resolution reconstruction method and system based on channel-constrained multi-feature fusion

Abstract

The present invention discloses an image super-resolution reconstruction method and system based on channel-constrained multi-feature fusion, which belongs to the field of super-resolution image reconstruction. It includes: obtaining high spatial resolution hyperspectral, low spatial resolution hyperspectral and high spatial resolution multispectral image pairs in the same scene to construct a training set; building a dual-channel super-resolution network, including: a feature extraction module, used to simultaneously extract spatial spectrum features from low spatial resolution hyperspectral and high spatial resolution multispectral images in the same scene; a feature fusion module, used to fuse the spatial information of the multispectral image in the same scene with the spectral information of the hyperspectral image; an image reconstruction module, used to reconstruct a reconstructed image; training the network until the reconstructed image is consistent with the change law of each element in the spectral vector corresponding to the original image; obtaining low spatial resolution hyperspectral and high spatial resolution multispectral images in the scene to be reconstructed, inputting them into the trained network, and obtaining a reconstructed super-resolution hyperspectral image.

Description

Technical Field

[0001] The present invention belongs to the field of super-resolution image reconstruction, and more specifically, to an image super-resolution reconstruction method and system based on channel-constrained multi-feature fusion. Background Art

[0002] In the field of remote sensing images, a certain trade-off must be made between spectral and spatial detail information. In order to ensure the integrity of spectral band information, some spatial detail information has to be sacrificed. However, this will bring greater difficulty to subsequent advanced tasks and limit the application of hyperspectral images. Therefore, improving the spatial quality of images while ensuring spectral resolution so that they can restore scene information to the greatest extent is the focus and difficulty of current research in the field of hyperspectral images.

[0003] There are usually two ways to improve the spatial resolution of hyperspectral images: one is to improve the hardware capabilities of imaging equipment, but this depends on breakthroughs in imaging technology and semiconductor processes. This method not only has a long R&D cycle, but also increases hardware costs. The other way is to solve it at the software level and use algorithms to improve image quality. In recent years, image super-resolution (SR) technology has attracted the attention of many researchers. This technology aims to reconstruct low-resolution (LR) images with less spatial detail information into high-resolution (HR) images with richer detail features, while eliminating the effects of image noise and blur to a certain extent. With the rapid update of computer equipment, the computing power of computers has been significantly improved. With the support of sufficient computing power and a large amount of data, deep learning has achieved very dazzling results in image super-resolution. Compared with traditional machine learning methods, super-resolution methods based on deep learning can avoid manually setting feature rules, automatically extract image and spectral hierarchical features, learn the complex nonlinear mapping relationship between high and low spatial resolution images from a large amount of data, and have better reconstruction effects. Therefore, it is of great significance and value to use deep learning technology to complete the task of super-resolution reconstruction of hyperspectral images.

[0004] Although researchers have achieved certain results in the field of image super-resolution, due to the particularity of the data structure of hyperspectral images, it is still a challenging task to effectively use spectral information and reduce spectral information distortion during the reconstruction process. In view of the above situation, a joint method of hyperspectral and multispectral images based on multi-feature fusion is applied to the field of image super-resolution, and improvements are made based on the problems existing in the actual application of the network model to enhance the algorithm's ability to reconstruct image details and the model's ability to restore spectral information. Summary of the invention

[0005] In view of the defects of the prior art and the need for improvement, the present invention provides an image super-resolution reconstruction method and system based on channel-constrained multi-feature fusion, which aims to solve the problems of frequent distortion in spectral information of images after super-resolution reconstruction of hyperspectral images and great difficulty in image reconstruction when the super-resolution multiple is high.

[0006] To achieve the above object, according to a first aspect of the present invention, a method for image super-resolution reconstruction based on channel-constrained multi-feature fusion is provided, the method comprising:

[0007] Training phase:

[0008] Obtain high spatial resolution hyperspectral images, low spatial resolution hyperspectral images and high spatial resolution multispectral images of the same scene, and use low spatial resolution hyperspectral images as training samples and corresponding high spatial resolution hyperspectral images as labels to construct a training set;

[0009] A dual-channel super-resolution network is constructed, wherein the dual-channel super-resolution network includes: a feature extraction module for simultaneously extracting spatial features and spectral features from a low spatial resolution hyperspectral image and a high spatial resolution multispectral image of the same scene; a feature fusion module for fusing the spatial-spectral features of the multispectral image and the hyperspectral image of the same scene; and an image reconstruction module for reconstructing the fused features to obtain a reconstructed high spatial resolution hyperspectral image;

[0010] The dual-channel super-resolution network is trained using the training set until the change rules of each element in the corresponding spectral vector of the reconstructed high spatial resolution hyperspectral image and the original high spatial resolution hyperspectral image are consistent, thereby obtaining a trained dual-channel super-resolution network;

[0011] Application phase:

[0012] The low spatial resolution hyperspectral image and high spatial resolution multispectral image of the scene to be reconstructed are obtained and input into the trained dual-channel super-resolution network for super-resolution reconstruction to obtain a reconstructed high spatial resolution hyperspectral image.

[0013] Preferably, the low spatial resolution hyperspectral image corresponding to the high spatial resolution hyperspectral image is obtained by: downsampling the original high spatial resolution hyperspectral image by bicubic interpolation, and adding a Gaussian blur with a standard deviation of 0.5;

[0014] The high spatial resolution multispectral image corresponding to the high spatial resolution hyperspectral image is obtained by spectrally downsampling the original high spatial resolution hyperspectral image in combination with the spectral response curve.

[0015] Preferably, the feature extraction module comprises a first branch and a second branch connected in parallel;

[0016] The first branch is used to extract high spatial resolution multispectral image information, which includes a first three-dimensional convolutional layer, a first three-dimensional residual feature aggregation module and a second three-dimensional convolutional layer connected in series, wherein the first three-dimensional convolutional layer is used to extract shallow spatial spectrum features of the multispectral image, the first three-dimensional residual feature aggregation module is used to further extract its spatial dimension features and spectral dimension features, and assign corresponding weights to each feature channel in the obtained feature map, and the second three-dimensional convolutional layer is used to extract high-level features and output them to the feature fusion module;

[0017] The second branch is used to extract low spatial resolution hyperspectral image information, which includes a first three-dimensional deconvolution layer, a third three-dimensional convolution layer, a second three-dimensional residual feature aggregation module and a fourth three-dimensional convolution layer connected in series, wherein the first three-dimensional deconvolution layer is used to adjust the size of the hyperspectral image to the same as the multispectral image, the third three-dimensional convolution layer is used to extract the shallow layer of the hyperspectral image, the second three-dimensional residual feature aggregation module is used to extract its spatial dimension features and spectral dimension features, and assign corresponding weights to each feature channel in the obtained feature map, and the fourth three-dimensional convolution layer is used to extract high-level features and output them to the feature fusion module.

[0018] The present invention preferably uses three-dimensional convolution to extract spatial features and spectral features simultaneously, thereby enhancing the model's utilization of spectral information and improving the model's super-resolution reconstruction accuracy.

[0019] Preferably, the feature fusion module includes a three-dimensional residual module and a three-dimensional convolutional layer connected in series, wherein the three-dimensional residual module is used to fuse the features output by the feature extraction network, and the fusion adopts a jump connection method; the three-dimensional convolutional layer is used to fuse the feature map output by each residual module with the feature map of all residual features, and the N feature maps are connected together in a cascade form to form a feature map group M, which are transmitted together to the image reconstruction module.

[0020] Beneficial effect: The preferred fusion module of the present invention adds the residual features in each residual module during the feature fusion process to strengthen the reconstruction of image detail information. The fusion module adopts a jump connection method to connect the feature map output by each residual module and the feature map after convolution of all residual features to the module output end, and connects each feature map together in a cascade form and transmits it to the image reconstruction module, which ensures that the features of each layer are fully utilized to the greatest extent and reduces the information loss caused by convolution.

[0021] Preferably, the image reconstruction module comprises a feature layer weighted constraint module, a first three-dimensional convolutional layer, a residual feature aggregation module, a second three-dimensional convolutional layer and a third three-dimensional convolutional layer connected in series;

[0022] Among them, the feature layer weighted constraint module is used to perform weighted constraints on feature maps of different images and different levels in the feature map group output by the feature fusion module; the first three-dimensional convolution layer module is used to further extract features from the feature maps after weighted constraints; the residual feature aggregation module is used to extract the fused spatial spectrum features and assign corresponding weights; the second three-dimensional convolution layer and the third three-dimensional convolution layer are used for nonlinear mapping to generate the final hyperspectral image.

[0023] Beneficial effects: The present invention applies a feature layer weighted constraint module (Layer Attention Module, LAM) to the hyperspectral image reconstruction task, and automatically assigns different weight values ​​to feature maps from different depths through network learning, thereby improving the expressive power of feature extraction.

[0024] Preferably, the residual feature aggregation module consists of four three-dimensional residual modules, the first three of which are stacked together in a conventional manner, and the last one removes the identity mapping part and only retains the 3D-CRB part; the feature maps output by the four 3D-CRBs are weighted fused in a cascade manner.

[0025] Preferably, the weighted constrained feature map group The calculation process is expressed as:

[0026]

[0027]

[0028] Where ρ represents the scaling factor, M j Represents the input feature map group, M i Represents the feature map group after dimension conversion, w i,j represents the correlation coefficient between the i-th group of features and the j-th group of features, δ(·) and They represent Softmax and dimension conversion operations respectively, and N represents the number of feature maps that constitute the feature map group.

[0029] Beneficial effect: The present invention further calculates the fused features, adaptively assigns weights to features of different images and different levels, and reconstructs their feature maps into high spatial resolution hyperspectral images, so that the advantages of the two can fully complement each other, thereby completing the reconstruction task under higher super-resolution multiples and higher image accuracy requirements.

[0030] Preferably, during the training process, the space-spectrum joint constraint loss function is calculated, and the training stops when the loss value reaches an expected range or no longer decreases.

[0031] Preferably, the space-spectrum joint constraint loss function is:

[0032] L=L MSE +αLspecral +βL diff

[0033]

[0034] Among them, L represents the total loss function, L MSE represents the mean square error loss function, L specral represents the spectral difference loss function, L diff represents the spectral difference feature error loss function, α, β are the coefficients used to balance the loss function, H, W represent the length and width of the image space direction respectively, Δx ij Represents the one-dimensional spectral difference vector of the original image in the i-th row and j-th column of the space, Δy ij Represents the one-dimensional spectral difference vector of the reconstructed image in the i-th row and j-th column of the space.

[0035] Beneficial effects: In order to solve the problem of spectral distortion that is easy to occur during the reconstruction process, the present invention proposes a spatial-spectral joint constraint loss function to reduce the distortion of spectral information. In order to ensure the authenticity of spectral information while improving the spatial resolution of the image, the spectral difference is added as a constraint in the loss function; in order to prevent the adjacent elements of the spectral vector of the reconstructed image from jumping abnormally and inconsistent with the original image, the spectral difference characteristic error is also added in the design of the loss function.

[0036] To achieve the above object, according to a second aspect of the present invention, there is provided an image super-resolution reconstruction system based on channel-constrained multi-feature fusion, the system comprising: a computer-readable storage medium and a processor;

[0037] The computer-readable storage medium is used to store executable instructions;

[0038] The processor is used to read the executable instructions stored in the computer-readable storage medium to execute the image super-resolution reconstruction method based on channel-constrained multi-feature fusion described in the first aspect.

[0039] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects:

[0040] In view of the low utilization rate of spectral information in the super-resolution reconstruction process, the present invention proposes to improve the multi-level feature expression ability of the model by fusing the spatial-spectral features of multispectral images (high spatial resolution) and hyperspectral images (low spatial resolution) in the same scene, combining the spatial information advantages of the former with the spectral information advantages of the latter, and then using the spatial-spectral information of the two to jointly reconstruct a high-spatial-resolution hyperspectral image, so that the spatial-spectral features of the two are fully integrated to meet the reconstruction accuracy requirements under higher super-resolution multiples. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A flow chart of an image super-resolution reconstruction method based on channel-constrained multi-feature fusion provided by the present invention;

[0042] Figure 2 A schematic diagram of the dual-channel super-resolution network structure based on multi-feature fusion provided by the present invention;

[0043] Figure 3 A schematic diagram of the structure of the feature fusion module provided by the present invention;

[0044] Figure 4 A schematic diagram of the structure of an image reconstruction module provided by the present invention;

[0045] Figure 5 A schematic diagram of the structure of a feature layer weighted constraint module provided by the present invention;

[0046] Figure 6 It is an error diagram based on the root mean square error in the Chikusei data set for an implementation case of the present invention;

[0047] Figure 7 The results of different super-resolution reconstruction algorithms in the ground object classification task in the implementation cases of the present invention are compared;

[0048] Figure 8 It is a statistical scatter plot of the classification accuracy of various types of objects by different super-resolution reconstruction algorithms in the implementation cases of the present invention;

[0049] Fig. 9 It is a statistical histogram of the overall classification accuracy of ground objects of different super-resolution reconstruction algorithms in the implementation cases of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0051] For the joint super-resolution reconstruction task of hyperspectral image and multispectral image, the present invention proposes a dual-channel image Super-Resolution networkbased on Multi-Feature Fusion (DSR-MFF) based on multi-feature fusion, which simultaneously extracts spatial features and spectral features through three-dimensional convolution, enhances the utilization rate of spectral information of the model, improves the super-resolution reconstruction accuracy of the model, and combines the spatial information advantage of the former with the spectral information advantage of the latter by fusing the multi-level feature information of high spatial resolution multispectral image (HR-MSI) and low spatial resolution hyperspectral image (LR-HSI) in the same scene, and uses the spatial-spectral information of the two to jointly reconstruct high spatial resolution hyperspectral image (HR-HSI), thereby improving the multi-level feature expression ability of the model. For the spectral distortion problem that is easy to occur in the reconstruction process, the present invention proposes a spatial-spectral joint constraint loss function to reduce the distortion of spectral information.

[0052] The present invention proposes an image super-resolution reconstruction method based on channel-constrained multi-feature fusion, comprising the following steps:

[0053] Step (1), create a data set, select some hyperspectral images, and use them as original high spatial resolution image input.

[0054] Step (2) obtains a low spatial resolution hyperspectral image by downsampling, adding blur and noise, etc.

[0055] Step (3), performing spectral downsampling operation on the original image in combination with the spectral response curve to obtain a high spatial resolution multispectral image.

[0056] Step (4), constructing a dual-channel super-resolution network based on multi-feature fusion, inputting a low spatial resolution hyperspectral image and a high spatial resolution multispectral image, adjusting the two images to the same spatial size, and extracting the multi-level features of the input multispectral image and hyperspectral image through a feature fusion module and fusing them.

[0057] The dual-channel super-resolution network for multi-feature fusion in step (4) is specifically implemented as follows:

[0058] A dual-channel super-resolution network-based image super-resolution reconstruction method is proposed. The high spatial resolution multispectral image HR-MSI with rich spatial information and the low spatial resolution hyperspectral image LR-HSI with rich spectral information in the same scene are jointly super-reconstructed so that the spatial and spectral features of the two are fully integrated to meet the reconstruction accuracy requirements at higher super-resolution multiples.

[0059] The feature fusion module in step (4) is specifically implemented as follows:

[0060] The fusion module adds the residual features of each residual module to strengthen the reconstruction of image detail information during the feature fusion process. The fusion module uses a jump connection method to connect the feature map output by each residual module and the feature map after convolution of all residual features to the module output end. Each feature map is connected together in a cascade form and transmitted to the image reconstruction module to ensure that the features of each layer are fully utilized to the greatest extent and reduce the information loss caused by convolution. The overall structure of the fusion module is as follows: Figure 3 shown.

[0061] The feature map cascade used by the feature fusion module in step (4) is specifically implemented as follows:

[0062] The cascade of feature maps needs to ensure that the spatial size of each feature map is consistent, and the spatial size of the image feature map should not change during the convolution process. To this end, the convolution stride in each channel weighted constrained residual module in the feature fusion module is set to 1, and the feature map is padded in the SAME way to ensure that the feature maps output by each layer have the same spatial size.

[0063] Step (5), outputting the fused features to the image reconstruction module, further calculating the fused features, adaptively assigning weights to features of different images and different levels, and reconstructing the feature maps into high spatial resolution hyperspectral images.

[0064] The image reconstruction module in step (5) is specifically implemented as follows:

[0065] The feature layer weighted constraint module (Layer Attention Module, LAM) is applied to the hyperspectral image reconstruction task. Different weight values ​​are automatically assigned to feature maps from different depths through network learning to improve the expressiveness of feature extraction. For the feature map group M composed of N cascaded feature maps input into the image fusion module, it is first converted from the dimension H×W×NC to the dimension N×HWC to obtain a new two-dimensional matrix. The matrix calculates the correlation between different layers by performing matrix multiplication operations with its corresponding transposed matrix. The calculation process can be expressed as:

[0066]

[0067] Among them, w i,j represents the correlation coefficient between the i-th group of features and the j-th group of features, δ(·) and They represent Softmax and dimension conversion operations respectively. Finally, the feature map group after dimension conversion is multiplied by the correlation matrix coefficient and the scaling factor α, and the result is added to the input feature map group to obtain the weighted constrained feature map group. The calculation process is expressed as:

[0068]

[0069] Step (6), train the constructed model, calculate the space-spectrum joint constraint loss function, and stop model training when the loss value reaches the expected range or no longer decreases.

[0070] The spatial spectrum joint constraint loss function in step (6) is:

[0071] For hyperspectral images, adjacent bands have strong correlation. If pixel loss is considered alone, it is not conducive to the expression of spectral features of hyperspectral images, which will lead to distortion of spectral information of reconstructed images. Therefore, in the hyperspectral image reconstruction task, in order to improve the spatial resolution of the image while ensuring the authenticity of the spectral information, the spectral difference is added as a constraint in the loss function:

[0072]

[0073] Among them, H and W represent the length and width of the image in the spatial direction, L spectrum represents the spectral difference loss function, x ij With y ij Represent the one-dimensional spectral vectors of the original image and the reconstructed image in the i-th row and j-th column of the space respectively.

[0074] At the same time, in addition to constraining the overall spectral information, in order to ensure that the elements of the spectral vectors of the original image and the reconstructed image maintain consistent change rules and prevent the adjacent elements of the spectral vector of the reconstructed image from jumping abnormally inconsistent with the original image, the spectral differential feature error is also added to the design of the loss function. ij With y ij , the spectral difference at band k (2≤k≤C) is expressed as:

[0075] Δx ij (k) = x ij (k)-x ij (k-1)

[0076] Δy ij (k) = y ij (k)-yij (k-1)

[0077] The spectral difference vector can be expressed as:

[0078] Δx ij =(Δx ij (2),Δx ij (3),…,Δx ij (C) T

[0079] Δy ij =(Δy ij (2),Δy ij (3),…,Δy ij (C) T

[0080] Then the spectral difference characteristic error loss function L diff It can be expressed as:

[0081]

[0082] Total loss function:

[0083] L=L MSE +αL specral +βL diff

[0084] Among them, α and β are coefficients used to balance the loss function. In this embodiment, α and β are both set to 0.5.

[0085] Step (7), using the trained model to perform super-resolution calculation on the input high spatial resolution multispectral image and the low spatial resolution hyperspectral image, and reconstruct a high spatial resolution hyperspectral image.

[0086] The present invention verifies the image reconstruction quality of the two models from multiple angles in terms of visual effects, parameter indicators and impact on ground object classification performance in a public data set.

[0087] The embodiment comprises the following steps:

[0088] 1. Dataset preparation and analysis

[0089] This embodiment uses the CAVE and Chikusei datasets. The CAVE dataset contains 32 indoor scene hyperspectral images with a spatial size of 512×512, with 31 spectral bands from 400nm to 700nm, and the interval between the bands is 10nm. The Chikusei dataset is a remote sensing hyperspectral image taken in Japan, which contains 128 bands and an image size of 2517pixel×2335pixel.

[0090] In the preparation of the training set, the original image is used as the high spatial resolution label, and the corresponding low spatial resolution image is generated by downsampling through bicubic interpolation, and a Gaussian blur with a standard deviation of 0.5 is added. In the CAVE dataset, 20 images are randomly selected to make the training set of the model, while for the Chikusei dataset with only a single image, five 320×320 areas in the image are selected as the test set, and the remaining areas are cut into partially overlapping small image blocks for making the training set. For the Pavia Centre and University dataset, in order to ensure that there is no interference with the image classification task, the entire Pavia Centre image is selected as the training set, and multiple training images are generated by image cutting.

[0091] In the verification experiment of improving the effect of super-resolution reconstruction on the task of object classification, the Pavia Centre and University dataset, which is commonly used in both classic super-resolution and object classification tasks, was selected. For the Pavia Centre and University dataset, in order to ensure that it does not interfere with the image classification task, the entire Pavia Centre image was selected as the training set, and multiple training images were generated by image cropping.

[0092] The model is supervised. DSR-MFF requires low spatial resolution hyperspectral images (LR-HSI) and corresponding high spatial resolution hyperspectral images (HR-HIS) as training data. It also requires corresponding high spatial resolution multispectral images (HR-MSI). The HR-MSI required by the DSR-MFF model is synthesized by HR-HSI and the curves of the three RGB bands in the corresponding spectral response curve. The spectral response curve of the Nikon D700 camera is used in the CAVE dataset, and the spectral response curve of the Landsat-8 satellite sensor is used in the Chikusei and PaviaCentre and University datasets.

[0093] Before the experiment, the image pixel values ​​need to be normalized to adjust each pixel value to between 0 and 1. In the experiment of this embodiment, each image is normalized by dividing the image by the maximum pixel value of the image:

[0094] I′=I / max(I)

[0095] Where I′ represents the normalized image, I represents the original image, and max(·) represents the maximum value of all pixels in the image.

[0096] Perform sample augmentation on the images in the training set: perform scaling processing on each group of samples in the training set by 0.5, 0.75, and 1, and rotate and flip the images to 0, 90°, and 270°, respectively, and flip them horizontally to obtain more training data.

[0097] 2. Network training process

[0098] The present invention provides a method for joint super-resolution reconstruction of multi-spectral and hyperspectral images based on multi-feature fusion. The overall process is as follows: Figure 1 As shown in the figure, the specific process of network training of this method mainly includes three parts: network construction, loss function design, and network training.

[0099] Network construction

[0100] The dual-channel super-resolution network process proposed in this invention can be divided into two steps: feature fusion and image reconstruction. First, the feature fusion module extracts and fuses the features of the two input images, and outputs the fused feature map to the image reconstruction module. Then, the image reconstruction module reconstructs the fused features to obtain the final HR-HIS. The overall network structure is as follows: Figure 2 shown.

[0101] Dual-channel feature fusion module

[0102] Traditional image fusion methods can be divided into three levels according to different information levels: pixel-level fusion, feature-level fusion, and decision-level fusion. The present invention designs an image feature fusion module with dual input channels to extract and fuse image features. Generally speaking, deeper network models can bring better nonlinear expression capabilities. High-level networks have stronger global mapping capabilities, but their feature map resolution is low and lacks the ability to express the details of image geometric features. The feature maps extracted by low-level networks usually contain strong geometric feature information, but lack semantic information. In order to combine the advantages of both, the feature fusion module inputs the lower-level features and the higher-level features into the subsequent reconstruction module to improve model performance.

[0103] First, before fusion, in order to adjust the two images to the same spatial size, a deconvolution layer is placed at the input of LR-HSI to expand the image size.

[0104] The fusion module adds the residual features of each residual module to strengthen the reconstruction of image detail information during the feature fusion process. The feature map output by each residual module and the feature map after convolution of all residual features are connected to the module output end by skip connection. Each feature map is connected together in a cascade form and transmitted to the image reconstruction module together, ensuring that the features of each layer are fully utilized to the greatest extent and reducing the information loss caused by convolution. The overall structure of the fusion module is as follows: Figure 3 shown.

[0105] The cascading of feature maps needs to ensure that the spatial size of each feature map is consistent, and the spatial size of the image feature map should not change during the convolution process. To this end, the convolution stride in each CRM in the feature fusion module is set to 1, and the feature map is padded in the SAME way to ensure that the feature maps output by each layer have the same spatial size.

[0106] Image reconstruction module

[0107] The image reconstruction module uses the multi-level spatial spectrum feature map group output by the image fusion module as input. The importance of these feature maps from different images and different levels to image reconstruction is not consistent. In addition, there is correlation between feature maps at different levels. Simply splicing feature maps at different levels cannot effectively reflect the correlation and difference between layers.

[0108] The present invention proposes to apply the feature layer weighted constraint module to the hyperspectral image reconstruction task, and automatically assign different weight values ​​to feature maps from different depths through network learning, thereby improving the expressive power of feature extraction.

[0109] After the feature map group output by the image fusion module is input into the image reconstruction module, the feature maps of different images and different levels are first weighted and constrained by LAM. Then, the feature maps after weighted constraints are nonlinearly mapped and reconstructed through the convolution layer and the residual feature aggregation module composed of four CRMs to generate the final hyperspectral image. The structure of the image reconstruction module is as follows: Figure 4 The following is an introduction to the implementation process of the feature layer weighted constraint module. The specific structure is as follows: Figure 5 shown.

[0110] For the feature map group M formed by cascading N feature maps input into the image fusion module, first convert it from the dimension H×W×NC to the dimension N×HWC to obtain a new two-dimensional matrix. This matrix calculates the correlation between different layers by performing matrix multiplication with its corresponding transposed matrix. The calculation process can be expressed as:

[0111]

[0112] Among them, w i,j represents the correlation coefficient between the i-th group of features and the j-th group of features, δ(·) and They represent Softmax and dimension conversion operations respectively. Finally, the feature map group after dimension conversion is multiplied by the correlation matrix coefficient and the scaling factor α, and the result is added to the input feature map group to obtain the weighted constrained feature map group. The calculation process can be expressed as:

[0113]

[0114] Loss function design

[0115] To address the problem that spectral dimension information is easily distorted when multispectral images and hyperspectral images are jointly reconstructed, the present invention proposes a space-spectrum joint constrained loss function that combines spectral difference, spectral differential feature error and mean square error, which can effectively reduce spectral distortion during the reconstruction process.

[0116] Hyperspectral image mean square error loss function:

[0117]

[0118] Spectral difference loss function:

[0119] In order to ensure the authenticity of spectral information while improving the spatial resolution of the image, the spectral difference is added as a constraint in the loss function:

[0120]

[0121] Among them, H and W represent the length and width of the image in the spatial direction, L spectrum represents the spectral difference loss function, x ij With y ij Represent the one-dimensional spectral vectors of the original image and the reconstructed image in the i-th row and j-th column of the space respectively. Using the spectral difference, the spectral vector between the original image and the reconstructed image can be constrained to reduce the overall spectral difference between the two images.

[0122] At the same time, in addition to constraining the overall spectral information, in order to ensure that the elements of the spectral vectors of the original image and the reconstructed image maintain consistent change rules and prevent the adjacent elements of the spectral vector of the reconstructed image from jumping abnormally inconsistent with the original image, the spectral differential feature error is also added to the design of the loss function. ij With y ij , the spectral difference at band k (2≤k≤C) is expressed as:

[0123] Δx ij (k) = x ij (k)-x ij (k-1)

[0124] Δy ij (k) = y ij (k)-y ij (k-1)

[0125] The spectral difference vector can be expressed as:

[0126] Δx ij=(Δx ij (2),Δx ij (3),…,Δx ij (C) T

[0127] Δy ij =(Δy ij (2),Δy ij (3),…,Δy ij (C) T

[0128] Spectral difference feature error loss function

[0129] By utilizing the spectral difference characteristic error, the difference between adjacent elements in the spectral vector between reconstructed images can be constrained, so that the change rules of each element in the corresponding spectral vector of the reconstructed image and the original image are consistent, and abnormal jumps in adjacent elements of the spectral vector of the reconstructed image can be prevented.

[0130] Spectral difference feature error loss function L diff It is expressed as:

[0131]

[0132] Total loss function

[0133] In order to reduce the pixel error of the reconstructed image and enhance the spectral similarity, the spectral difference loss function, the spectral difference feature error loss function and the mean square error loss function are combined to obtain the total loss function:

[0134] L=L MSE +αL specral +βL diff

[0135] Wherein, α and β are coefficients used to balance the loss function. In this embodiment, α and β are both 0.5.

[0136] Network training

[0137] The network training process is to input the prepared training samples into the built network for training, calculate the loss function, perform back propagation gradients, and update the network. The training steps are as follows:

[0138] Initialize the dual-channel feature fusion network and image reconstruction network;

[0139] The low spatial resolution hyperspectral image y and the high spatial resolution multispectral image z are respectively sent to the dual-channel feature fusion network to extract the content features and style features of the multispectral image and hyperspectral image;

[0140] Apply geometric transformation to the content features, which can be rotation, mirroring, etc., and then send the geometrically transformed content features and the original style features together to the image reconstruction network for image generation;

[0141] Compare the original image with the generated image and calculate the spatial-spectral joint constraint loss;

[0142] Enter the next iteration process to generate a high spatial resolution hyperspectral image.

[0143] Fusion model performance analysis

[0144] In order to comprehensively evaluate the overall performance of DSR-MFF, this paper compares DSR-MFF with four latest hyperspectral and multispectral image joint super-resolution reconstruction algorithms on the Chikusei dataset, with super-resolution multiples set to 16 and 32. These four methods include NSSR, HySure, DHSIS and uSDN. Since the numerical distribution of each pixel in the Chikusei dataset is relatively wide, the error map based on the root mean square error is used here. The reconstruction results of different methods in the Chikusei dataset are shown in Figure 2. Figure 6 As shown in Figure 1, the left side is DSR-MFF and the right side is the true value image. The comparison results of objective evaluation indicators are shown in Table 1.

[0145] Table 1

[0146]

[0147] We will further explore the impact of the reconstruction task on the application value of hyperspectral images in actual scenarios. In the application scenario of hyperspectral images, the classic hyperspectral image task - ground object classification is selected as the test task. The DSR-MFF network model is used to reconstruct LR-HSI, and the super-resolution magnification is set to 4.

[0148] After the model is trained, the LR image, the images reconstructed by the two models, and the true value image before downsampling are classified and tested. The classification results are as follows: Figure 7 As shown in the figure, the left side is the DER-MFF reconstruction classification result, the middle is the classification result of the original image, and the right side is the labeled ground object pair. Figure 8 and Fig. 9 shown.

[0149] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An image super-resolution reconstruction method based on channel-constrained multi-feature fusion, characterized in that: The method includes: Training phase: Obtain high spatial resolution hyperspectral images, low spatial resolution hyperspectral images and high spatial resolution multispectral images of the same scene, and use low spatial resolution hyperspectral image pairs as training samples and corresponding high spatial resolution hyperspectral images as labels to construct a training set; A dual-channel super-resolution network is constructed, wherein the dual-channel super-resolution network includes: a feature extraction module for simultaneously extracting spatial features and spectral features from a low spatial resolution hyperspectral image and a high spatial resolution multispectral image of the same scene; a feature fusion module for fusing the spatial-spectral features of the multispectral image and the hyperspectral image of the same scene; and an image reconstruction module for reconstructing the fused features to obtain a reconstructed high spatial resolution hyperspectral image; The dual-channel super-resolution network is trained using the training set until the change rules of each element in the corresponding spectral vector of the reconstructed high spatial resolution hyperspectral image and the original high spatial resolution hyperspectral image are consistent, thereby obtaining a trained dual-channel super-resolution network; The image reconstruction module includes a series of feature layer weighted constraint modules, a first three-dimensional convolutional layer, a residual feature aggregation module, a second three-dimensional convolutional layer and a third three-dimensional convolutional layer; Among them, the feature layer weighted constraint module is used to perform weighted constraints on the feature maps of different images and different levels in the feature map group output by the feature fusion module; the first three-dimensional convolution layer module is used to further extract features from the feature map after weighted constraints; the residual feature aggregation module is used to extract the fused spatial spectrum features and assign corresponding weights; the second three-dimensional convolution layer and the third three-dimensional convolution layer are used for nonlinear mapping to generate the final hyperspectral image; Application phase: The low spatial resolution hyperspectral image and high spatial resolution multispectral image of the scene to be reconstructed are obtained and input into the trained dual-channel super-resolution network for super-resolution reconstruction to obtain a reconstructed high spatial resolution hyperspectral image.

2. The method according to claim 1, characterized in that The low spatial resolution hyperspectral image corresponding to the high spatial resolution hyperspectral image is obtained by: downsampling the original high spatial resolution hyperspectral image by bicubic interpolation and adding a Gaussian blur with a standard deviation of 0.5; The high spatial resolution multispectral image corresponding to the high spatial resolution hyperspectral image is obtained by spectrally downsampling the original high spatial resolution hyperspectral image in combination with the spectral response curve.

3. The method according to claim 1, characterized in that The feature extraction module includes a first branch and a second branch connected in parallel; The first branch is used to extract high spatial resolution multispectral image information, which includes a first three-dimensional convolutional layer, a first three-dimensional residual feature aggregation module and a second three-dimensional convolutional layer connected in series, wherein the first three-dimensional convolutional layer is used to extract shallow spatial spectrum features of the multispectral image, the first three-dimensional residual feature aggregation module is used to further extract its spatial dimension features and spectral dimension features, and assign corresponding weights to each feature channel in the obtained feature map, and the second three-dimensional convolutional layer is used to extract high-level features and output them to the feature fusion module; The second branch is used to extract low spatial resolution hyperspectral image information, which includes a first three-dimensional deconvolution layer, a third three-dimensional convolution layer, a second three-dimensional residual feature aggregation module and a fourth three-dimensional convolution layer connected in series, wherein the first three-dimensional deconvolution layer is used to adjust the size of the hyperspectral image to the same as the multispectral image, the third three-dimensional convolution layer is used to extract the shallow layer of the hyperspectral image, the second three-dimensional residual feature aggregation module is used to extract its spatial dimension features and spectral dimension features, and assign corresponding weights to each feature channel in the obtained feature map, and the fourth three-dimensional convolution layer is used to extract high-level features and output them to the feature fusion module.

4. The method according to claim 1, characterized in that The feature fusion module includes a three-dimensional residual module and a three-dimensional convolutional layer connected in series, wherein the three-dimensional residual module is used to fuse the features output by the feature extraction network, and the fusion adopts a jump connection method; the three-dimensional convolutional layer is used to fuse the feature map output by each residual module with the feature map of all residual features, and N feature maps are connected together in a cascade form to form a feature map group M that is transmitted together to the image reconstruction module.

5. The method according to claim 1, characterized in that The residual feature aggregation module consists of four three-dimensional residual modules, the first three of which are stacked together in a conventional manner, and the last one removes the identity mapping part and only retains the 3D-CRB part; the feature maps output by the four 3D-CRBs are weighted fused in a cascade manner.

6. The method according to claim 1, characterized in that Weighted constrained feature map group The calculation process is expressed as: Where ρ represents the scaling factor, M j Represents the input feature map group, M i Represents the feature map group after dimension conversion, w i,j represents the correlation coefficient between the i-th group of features and the j-th group of features, δ(·) and They represent Softmax and dimension conversion operations respectively, and N represents the number of feature maps that constitute the feature map group.

7. The method according to claim 1, characterized in that During the training process, the spatial-spectral joint constraint loss function is calculated and the training stops when the loss value reaches the expected range or no longer decreases.

8. The method according to claim 7, characterized in that The spatial-spectral joint constraint loss function is: L=L MSE +αL specral +βL diff Among them, L represents the total loss function, L MSE represents the mean square error loss function, L specral represents the spectral difference loss function, L diff represents the spectral difference feature error loss function, α, β are the coefficients used to balance the loss function, H, W represent the length and width of the image space direction respectively, Δx ij Represents the one-dimensional spectral difference vector of the original image in the i-th row and j-th column of the space, Δy ij Represents the one-dimensional spectral difference vector of the reconstructed image in the i-th row and j-th column of the space.

9. An image super-resolution reconstruction system based on channel-constrained multi-feature fusion, characterized in that: The system includes: a computer-readable storage medium and a processor; The computer-readable storage medium is used to store executable instructions; The processor is used to read the executable instructions stored in the computer-readable storage medium to execute the image super-resolution reconstruction method based on channel-constrained residual feature fusion according to any one of claims 1 to 8.

Images