Multi-spectral image super-resolution reconstruction method based on adaptive spectral feature fusion network

Abstract

The invention discloses a multi-spectral image super-resolution reconstruction method based on an adaptive spectral feature fusion network, relates to the technical field of remote sensing images, and discloses a multi-spectral image super-resolution reconstruction method based on the adaptive spectral feature fusion network, and the method comprises the steps: obtaining a panchromatic image and a corresponding multi-spectral image; the panchromatic image and the multispectral image are fused through a preset spectral feature fusion network to obtain a reconstructed spectral image, the preset spectral feature fusion network comprises an encoder, a bottleneck module and a decoder, the encoder, the bottleneck module and the decoder all comprise feature enhancement feedforward networks, the bottleneck module and the decoder all comprise HRM, the HRM comprises ASTM, the feature enhancement feedforward networks comprise feature enhancement feedforward networks, the feature enhancement feedforward networks comprise feature enhancement feedforward networks, and the feature enhancement feedforward networks comprise feature enhancement feedforward networks. The ASTM is used for identifying effective spectral features and inhibiting extraction of redundant spectral features, and the feature enhancement feedforward network is used for identifying effective spatial features and inhibiting extraction of redundant spatial features. The quality of the reconstructed multispectral image is improved by preventing the multispectral image from containing redundant features.

Description

Technical Field

[0001] The present application relates to the field of remote sensing image technology, and in particular to a multispectral image super-resolution reconstruction method based on an adaptive spectral feature fusion network. Background Art

[0002] Multispectral imagery is a key tool in remote sensing that can efficiently acquire data across multiple spectral bands to capture the unique characteristics of various types of materials.

[0003] In order to enhance the spatial resolution of multispectral images, multispectral images are usually fused with panchromatic images to achieve both high spatial resolution and rich spectral information. At present, adaptive pan-sharpening neural networks are usually used to improve the performance of pan-sharpening to effectively retain the spectral and spatial information in the source image during fusion. However, multispectral images contain data from multiple bands, which may be highly correlated, resulting in redundant spectral features. The current pan-sharpening neural network needs to extract these redundant spectral features, resulting in the inclusion of these redundant spectral features in the reconstructed multispectral image, which in turn leads to a decrease in the quality of the reconstructed multispectral image.

[0004] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0005] The main purpose of this application is to provide a multispectral image super-resolution reconstruction method based on an adaptive spectral feature fusion network, aiming to solve the technical problem that the reconstructed multispectral image contains these redundant spectral features, which leads to a decrease in the quality of the reconstructed multispectral image.

[0006] To achieve the above objectives, the present application proposes a multispectral image super-resolution reconstruction method based on an adaptive spectral feature fusion network, the method comprising:

[0007] Obtain a panchromatic image and a corresponding multispectral image;

[0008] The panchromatic image and the multispectral image are fused through a preset spectral feature fusion network to obtain a reconstructed spectral image. The preset spectral feature fusion network includes an encoder, a bottleneck module and a decoder. The encoder, the bottleneck module and the decoder all include a feature enhancement feedforward network. The bottleneck module and the decoder all include HRM. The HRM includes ASTM. The ASTM is used to identify effective spectral features and suppress the extraction of redundant spectral features. The feature enhancement feedforward network is used to identify effective spatial features and suppress the extraction of redundant spatial features.

[0009] In one embodiment, the bottleneck module further includes an upsampling module;

[0010] The step of fusing the panchromatic image and the multispectral image through a preset spectral feature fusion network to obtain a reconstructed spectral image comprises:

[0011] Extracting effective spatial features from the full-color image through feature enhancement feed-forward network processing in the encoder;

[0012] Extracting features with a resolution that meets preset expectations from the effective spatial features through the upsampling module to obtain spatial features to be merged;

[0013] fusing the spatial features to be merged and the multispectral image through the bottleneck module to obtain an initial fused image;

[0014] The HRM is used to extract and fuse the features of the initial fused image according to the frequency separation to obtain a reconstructed spectral image.

[0015] In one embodiment, the HRM further includes a convolutional layer and a spectral attention module;

[0016] The step of extracting and fusing the features of the initial fused image according to the frequency separation by the HRM to obtain the reconstructed spectral image comprises:

[0017] By means of the HRM, high-frequency information having a wavelength higher than a preset separation wavelength and low-frequency information having a wavelength less than or equal to the preset separation wavelength are separated from the initial fused image;

[0018] By using the ASTM, weighted convolution processing is performed on the high-frequency information to obtain high-frequency features;

[0019] Through the spectral attention module, global features and spatial texture features are extracted from the low-frequency information to obtain supplementary features;

[0020] Through the HRM, the high-frequency features are fused with the supplementary features to obtain a reconstructed spectral image.

[0021] In one embodiment, the ASTM includes a predictor and a token mixer, wherein the predictor includes a token mask module and a spectral attention weight module;

[0022] The step of performing weighted convolution processing on the high-frequency information through the ASTM to obtain high-frequency features includes:

[0023] Preprocessing the high frequency information in the predictor to obtain an intermediate feature map;

[0024] By means of the token mask module, the intermediate feature map is subjected to feature sorting in the spatial dimension to obtain a spectral token mask of each intermediate feature in the intermediate feature map;

[0025] extracting, by the token mixer, key spectral features corresponding to the spectral token mask from each of the intermediate features;

[0026] Through the spectral attention weight module, each of the intermediate features is processed to obtain a spectral fusion feature;

[0027] The key spectral feature is fused with the spectral fusion feature to obtain a high-frequency feature.

[0028] In one embodiment, the step of extracting global features and spatial texture features from the low-frequency information through the spectral attention module to obtain supplementary features includes:

[0029] Through the spectral attention module, the low-frequency information is upsampled according to a preset ratio to obtain global features and spatial texture features;

[0030] Based on the weight of the low-frequency information, supplementary features are selected from the global features and the spatial texture features.

[0031] In one embodiment, the feature enhancement feed-forward network includes a gating mechanism and a pixel attention mechanism;

[0032] The step of extracting effective spatial features from the full-color image by feature-enhanced feed-forward network processing in the encoder comprises:

[0033] By means of the gating mechanism, the full-color image is divided into redundant spatial features and representative spatial features according to the bands;

[0034] Performing group convolution and depth convolution on the representative spatial features through the convolution layer in the feature enhancement feedforward network to obtain representative features;

[0035] Performing deep convolution on the redundant spatial features through the convolution layer in the feature enhancement feed-forward network to obtain supplementary spatial features;

[0036] Through the pixel attention mechanism, the representative features are fused pixel by pixel with the supplementary spatial features to obtain effective spatial features.

[0037] In addition, to achieve the above purpose, the present application also proposes a multi-spectral image super-resolution reconstruction device based on an adaptive spectral feature fusion network, and the multi-spectral image super-resolution reconstruction device based on an adaptive spectral feature fusion network includes:

[0038] An acquisition module, used for acquiring a full-color image and a corresponding multi-spectral image;

[0039] A reconstruction module is used to fuse the panchromatic image and the multispectral image through a preset spectral feature fusion network to obtain a reconstructed spectral image. The preset spectral feature fusion network includes an encoder, a bottleneck module and a decoder. The encoder, the bottleneck module and the decoder all include a feature enhancement feedforward network. The bottleneck module and the decoder all include HRM. The HRM includes ASTM. The ASTM is used to identify effective spectral features and suppress the extraction of redundant spectral features. The feature enhancement feedforward network is used to identify effective spatial features and suppress the extraction of redundant spatial features.

[0040] In addition, to achieve the above-mentioned purpose, the present application also proposes a multispectral image super-resolution reconstruction device based on an adaptive spectral feature fusion network, the device comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, the computer program being configured to implement the steps of the multispectral image super-resolution reconstruction method based on an adaptive spectral feature fusion network as described above.

[0041] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the multispectral image super-resolution reconstruction method based on the adaptive spectral feature fusion network as described above are implemented.

[0042] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the multi-spectral image super-resolution reconstruction method based on the adaptive spectral feature fusion network as described above.

[0043] One or more technical solutions proposed in this application have at least the following technical effects:

[0044] Since the preset spectral feature fusion network is provided with a feature enhancement feedforward network and an ASTM (Adaptive Spectral Token Mixer), after the acquired panchromatic image and the corresponding multispectral image are input into the preset spectral feature fusion network, the feature enhancement feedforward network is used to identify effective spatial features from the panchromatic image and / or the multispectral image and to suppress the extraction of redundant spatial features, and the ASTM is used to identify effective spectral features from the panchromatic image and / or the multispectral image and to suppress the extraction of redundant spectral features, thereby avoiding the output reconstructed multispectral image from containing too many redundant features. By avoiding the appearance of too many pairs of redundant features in the reconstructed multispectral image, the accuracy and detail expression of the reconstructed multispectral image are improved, thereby improving the quality of the reconstructed multispectral image. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0046] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0047] Figure 1 A flowchart diagram of a multi-spectral image super-resolution reconstruction method based on an adaptive spectral feature fusion network provided in the present application;

[0048] Figure 2 This is a schematic diagram of the architecture of a preset spectral feature fusion network in the multi-spectral image super-resolution reconstruction method based on an adaptive spectral feature fusion network in this application;

[0049] Figure 3 A flowchart diagram of a second embodiment of a multi-spectral image super-resolution reconstruction method based on an adaptive spectral feature fusion network is provided for this application;

[0050] Figure 4 A flowchart diagram of a method for super-resolution reconstruction of multi-spectral images based on an adaptive spectral feature fusion network according to Embodiment 3 of the present invention;

[0051] Figure 5 This is a schematic diagram of the architecture of ASTM in the multi-spectral image super-resolution reconstruction method based on an adaptive spectral feature fusion network in this application;

[0052] Figure 6 This is a schematic diagram of the module structure of a multi-spectral image super-resolution reconstruction device based on an adaptive spectral feature fusion network according to an embodiment of the present application;

[0053] Figure 7 Schematic diagram of the device structure of the hardware operating environment involved in the multi-spectral image super-resolution reconstruction method based on the adaptive spectral feature fusion network in the embodiment of the present application.

[0054] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0055] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0056] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0057] The main solution of the embodiment of the present application is: the remote sensing image processor obtains a full-color image and a corresponding multi-spectral image; the full-color image and the multi-spectral image are fused through a preset spectral feature fusion network to obtain a reconstructed spectral image, the preset spectral feature fusion network includes an encoder, a bottleneck module and a decoder, the encoder, the bottleneck module and the decoder all include a feature enhancement feedforward network, the bottleneck module and the decoder all include HRM, the HRM includes ASTM, the ASTM is used to identify effective spectral features and suppress the extraction of redundant spectral features, the feature enhancement feedforward network is used to identify effective spatial features and suppress the extraction of redundant spatial features.

[0058] In this embodiment, for the convenience of description, the remote sensing image processor is used as the execution body for explanation below.

[0059] In order to enhance the spatial resolution of multispectral images, multispectral images are usually fused with panchromatic images to achieve both high spatial resolution and rich spectral information. At present, adaptive pan-sharpening neural networks are usually used to improve the performance of pan-sharpening to effectively retain the spectral and spatial information in the source image during fusion. However, multispectral images contain data from multiple bands, which may be highly correlated, resulting in redundant spectral features. The current pan-sharpening neural network needs to extract these redundant spectral features, resulting in the inclusion of these redundant spectral features in the reconstructed multispectral image, which in turn leads to a decrease in the quality of the reconstructed multispectral image.

[0060] The present application provides a solution. Since a feature enhancement feedforward network and ASTM are provided in a preset spectral feature fusion network, after the acquired panchromatic image and the corresponding multispectral image are input into the preset spectral feature fusion network, the feature enhancement feedforward network is used to identify effective spatial features from the panchromatic image and / or the multispectral image and suppress the extraction of redundant spatial features. ASTM is used to identify effective spectral features from the panchromatic image and / or the multispectral image and suppress the extraction of redundant spectral features, thereby avoiding the output reconstructed multispectral image from containing too many redundant features. By avoiding the appearance of too many pairs of redundant features in the reconstructed multispectral image, the accuracy and detail expression of the reconstructed multispectral image are improved, thereby improving the quality of the reconstructed multispectral image.

[0061] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a remote sensing image processor, etc. The following takes a remote sensing image processor as an example to illustrate this embodiment and the following embodiments.

[0062] Based on this, the embodiment of the present application provides a multi-spectral image super-resolution reconstruction method based on an adaptive spectral feature fusion network, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the multi-spectral image super-resolution reconstruction method based on the adaptive spectral feature fusion network of the present application.

[0063] In this embodiment, the multispectral image super-resolution reconstruction method based on the adaptive spectral feature fusion network includes steps S10 to S20:

[0064] Step S10, obtaining a full-color image and a corresponding multi-spectral image;

[0065] It should be noted that a panchromatic image is a mixed image of the entire visible light wavelength range (generally defined as 0.38-0.76μm) acquired by a remote sensor, usually a single-channel grayscale image. Since a panchromatic image is a single-band image, it is displayed as a grayscale image on the map, and there is no color information to display the color of the object, so the image has little spectral information; generally, it has a high spatial resolution, can provide more spatial details, and display a large amount of spatial information of the object. A multispectral image is an image acquired by a remote sensor in multiple discrete bands (usually within the visible and near-infrared bands), usually containing several to a dozen bands, each band records the light radiation information within a specific wavelength range, so a multispectral image can provide rich color information of the object; because of the need to strike a balance between spectral resolution and spatial resolution, the spatial resolution is usually low.

[0066] It can be understood that since the panchromatic image has a high spatial resolution and the multispectral image has a high spectral resolution, in order to make the multispectral image have a higher spatial resolution, it is necessary to obtain the panchromatic image and the multispectral image.

[0067] In a specific implementation, the panchromatic image and the multispectral image can be received by the remote sensor, and uploaded to the remote sensing image processor, so that the remote sensing image processor can obtain the multispectral image corresponding to the panchromatic image, which is convenient for the subsequent fusion and reconstruction of the panchromatic image and the multispectral image using a neural network.

[0068] Step S20, fusing the panchromatic image and the multispectral image through a preset spectral feature fusion network to obtain a reconstructed spectral image, the preset spectral feature fusion network includes an encoder, a bottleneck module and a decoder, the encoder, the bottleneck module and the decoder all include a feature enhancement feedforward network, the bottleneck module and the decoder all include HRM, the HRM includes ASTM, the ASTM is used to identify effective spectral features and suppress the extraction of redundant spectral features, the feature enhancement feedforward network is used to identify effective spatial features and suppress the extraction of redundant spatial features.

[0069] It should be noted that the preset spectral feature fusion network is a neural network model for fusing panchromatic images and multispectral images. Its purpose is to combine the advantages of panchromatic images and multispectral images to generate reconstructed spectral images with high spatial resolution and accurate spectral information. The encoder is a part of the neural network, responsible for converting input data (such as panchromatic images) into a set of feature representations, which usually have a lower dimension but contain the main information of the input data. The bottleneck module is usually located in the middle part of the neural network. Its function is to compress the input data to a lower dimension and then restore it to the original dimension through the decoder, which helps to extract the main features of the data and remove redundant information. The decoder is responsible for converting the feature representation generated by the encoder back to the dimension of the original data in the neural network. The feature enhancement feedforward network is a neural network structure used to identify and enhance effective features. It can identify effective spatial features in the input data and suppress the extraction of redundant spatial features, thereby improving the quality and information content of the data. HRM (High-frequency Refinement Module) can effectively separate high-frequency information from low-frequency features through fuzzy operations and downsampling operations, thereby improving processing efficiency. ASTM can effectively identify the most important features, filter out redundant information, reconstruct high-resolution multispectral images, and improve the utilization efficiency of spectral information. Effective spectral features are those spectral information in spectral data that can significantly reflect the characteristics of target objects or phenomena. These features usually show obvious responses in specific spectral bands and can be used to distinguish different substances or identify specific objects. Redundant spectral features contain a large amount of repeated or irrelevant information in hyperspectral data. This information does not contribute substantially to target detection and classification analysis, but may increase computational complexity and reduce model performance. Effective spatial features are features that can significantly reflect the spatial structure and morphology of target objects or phenomena in the spatial domain. These features usually include information such as edges, textures, and shapes, and can be used to describe the spatial distribution and mutual relationships of objects. Redundant spatial features are in the spatial dimension. Due to the spatial distribution characteristics of ground objects and the resolution limitations of sensors, a large amount of repeated or irrelevant spatial information is extracted.

[0070] It can be understood that by introducing ASTM into the preset spectral feature fusion network, it is possible to effectively identify effective spectral features that contribute to image reconstruction, while suppressing redundant spectral features that have little or even negative impact on image quality, thereby improving the quality of reconstructed spectral images. ASTM utilizes learnable spectral token masks and deformable spectral attention mechanisms to effectively focus on important feature maps, optimize the allocation of computing resources, and improve the fusion efficiency of the preset spectral feature fusion network for panchromatic images and multispectral images.

[0071] It can be understood that by setting a feature enhancement feedforward network in the preset spectral feature fusion network, it is possible to retain and enhance the effective spatial features in the image, reduce redundant spatial information, and further improve the spatial resolution and detail expression of the reconstructed spectral image. That is, through the feature enhancement feedforward network, the spatial texture information of the panchromatic image can be better migrated to the multispectral image. At the same time, the module adopts a segmentation-enhancement strategy to suppress the extraction of redundant spatial features, promote the flow of valuable signals, and ensure that the complementary information from the multispectral image and the panchromatic image is fully utilized.

[0072] It can be understood that in the encoder and decoder parts of the preset spectral feature fusion network, a feature enhancement feedforward network is set to better realize the transfer of spatial texture information of the full-color image to the multispectral image, and a segmentation enhancement strategy is adopted through the feature enhancement feedforward network to suppress the redundant extraction of spatial features and promote the flow of valuable signals; and in order to fully pay attention to the high-frequency information and low-rank characteristics in the spectral data, a HRM (High-frequency Refinement Module) is set in the preset spectral feature fusion network, and ASTM is set in the HRM to utilize learnable spectral token masks and deformable spectral attention to effectively focus on important feature maps and optimize the allocation of computing resources. Through the integration of these two mechanisms, the preset spectral feature fusion network can effectively identify key information and filter out redundant parts, thereby achieving high-quality reconstruction of multispectral images.

[0073] In the specific implementation, refer to Figure 2Each stage of the encoder is mainly composed of a basic module and a convolutional layer, where the basic module contains N1 feature enhancement feed-forward networks (FEFN); the basic module in the decoder is composed of N2 HRMs and N2 FEFNs to reduce the computational burden and number of parameters of the preset spectral feature network, improve computational efficiency, and promote the flow of low-level features. A bottleneck module is introduced between the encoder and the decoder, which includes an upsampling module, a convolutional layer, N3 HRMs and N3 FEFNs to build a more hierarchical representation, and the input parts of the encoder and decoder are connected to their respective output parts to form a residual connection to prevent the loss of deep information. Finally, a convolutional layer is used to process the output of the decoder and the result of adding the residual to obtain the final output.

[0074] The present embodiment provides a multispectral image super-resolution reconstruction method based on an adaptive spectral feature fusion network. Since a feature enhancement feedforward network and ASTM are provided in the preset spectral feature fusion network, after the acquired panchromatic image and the corresponding multispectral image are input into the preset spectral feature fusion network, the feature enhancement feedforward network is used to identify effective spatial features from the panchromatic image and / or the multispectral image and suppress the extraction of redundant spatial features. ASTM is used to identify effective spectral features from the panchromatic image and / or the multispectral image and suppress the extraction of redundant spectral features, thereby avoiding the output reconstructed multispectral image from containing too many redundant features. By avoiding the appearance of too many pairs of redundant features in the reconstructed multispectral image, the accuracy and detail expression of the reconstructed multispectral image are improved, thereby improving the quality of the reconstructed multispectral image.

[0075] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the first embodiment can be referred to the above description, and no further description will be given later. On this basis, the bottleneck module also includes an upsampling module, please refer to Figure 3 , step S20 also includes steps S21 to S24:

[0076] Step S21, extracting effective spatial features from the full-color image through feature enhancement feed-forward network processing in the encoder;

[0077] Step S22, extracting features whose resolution meets preset expectations from the effective spatial features through the upsampling module to obtain spatial features to be merged;

[0078] Step S23, fusing the spatial features to be merged with the multispectral image through the bottleneck module to obtain an initial fused image;

[0079] Step S24: extracting and fusing the features of the initial fused image according to the frequency separation by using the HRM to obtain a reconstructed spectral image.

[0080] It should be noted that the upsampling module is used to enlarge the low-resolution feature map to a high resolution.

[0081] It can be understood that since the feature enhancement feedforward network is a deep learning model that aims to enhance the effective features in the input data by means of forward propagation, it usually includes multiple convolutional layers and nonlinear activation functions, and can automatically learn and extract important features in the input data while suppressing redundant information. Therefore, effective spatial features can be extracted from the full-color image through the feature enhancement feedforward network processing in the encoder.

[0082] It can be understood that since the upsampling module is used to enlarge the low-resolution feature map to a high resolution, through upsampling, the model can restore detail information, making the output feature map more refined and accurate. Therefore, the upsampling module can be used to extract features with a resolution that meets preset expectations from the effective spatial features.

[0083] It can be understood that since the bottleneck module contains one or more fully connected layers, it can compress high-dimensional features into a low-dimensional space while retaining the most important feature information. It is usually used to reduce feature dimensions and capture the most important feature information. Therefore, the bottleneck module can be used to fuse the spatial features to be merged with the multispectral image to obtain an initial fused image.

[0084] It can be understood that the HRM module can use blurring and downsampling operations to separate high-frequency information from low-frequency information, and use a dual-branch structure to process the two features, and enhance high-frequency features, improve visual contrast and ensure the accuracy of the fusion process, thereby improving the fidelity and quality of the fused image.

[0085] It can be understood that by constructing a deep learning model including a feature enhanced feedforward network, an upsampling module, a bottleneck module and an HRM, effective feature extraction and fusion of panchromatic images and multispectral images are achieved, and finally a high-quality reconstructed spectral image is obtained. That is, when processing high-resolution images, multi-scale feature information can be effectively fused and the high resolution of the output image can be maintained.

[0086] Furthermore, the feature enhancement feedforward network includes a gating mechanism and a pixel attention mechanism; step S21 includes:

[0087] By means of the gating mechanism, the full-color image is divided into redundant spatial features and representative spatial features according to the bands;

[0088] Performing group convolution and depth convolution on the representative spatial features through the convolution layer in the feature enhancement feedforward network to obtain representative features;

[0089] Performing deep convolution on the redundant spatial features through the convolution layer in the feature enhancement feed-forward network to obtain supplementary spatial features;

[0090] Through the pixel attention mechanism, the representative features are fused pixel by pixel with the supplementary spatial features to obtain effective spatial features.

[0091] It should be noted that the gating mechanism can control the input and output of information by introducing one or more gates, thereby achieving more efficient sequence data processing. The pixel attention mechanism is a mechanism that emphasizes or suppresses certain pixel features by assigning a weight to each pixel in the image, allowing the preset spectral feature fusion network to pay more attention to specific areas or objects in the image, thereby improving the accuracy of feature extraction.

[0092] It can be understood that the amount of feature map information is evaluated through the gating mechanism, and the frequency components are refined and optimized using grouped convolution and deep convolution, while pixel-level attention is applied to the spatial dimension of the feature map to enhance the spatial feature representation.

[0093] In the specific implementation, refer to Figure 2 , through the gating mechanism, the feature map X of the input feature enhancement feedforward network is divided into two parts, and the input feature map X is divided into two bands along the channel dimension. The two bands contain β and (1-β) channels respectively, where 0<β<1, and β is a pre-set parameter. Subsequently, two 1×1 convolutional layers (Conv 1×1 ), and get the representative feature X rep and redundant features X red Then, X rep Grouped convolution and depth convolution are applied to perform feature transformation, and finally generate representative features (Y1). The transformation process can be expressed as follows:

[0094]

[0095] in, and is the projection matrix. Input features and output features Represent the input and output of the transformation respectively. For the low-level feature X red , only the additional redundant features are refined into (supplementary spatial features) Y2 using deep convolution to supplement the representative features:

[0096]

[0097] in, is the projection matrix. Features and are the input and output of the low-level branches respectively.

[0098] Then apply the pixel-level attention mechanism to enhance the fineness of the features. By multiplying Y1 and Y2 pixel by pixel, we get Then use the softmax function to refine it. The resulting weights are consistent with Y m Perform pixel-by-pixel multiplication to generate the final output (effective spatial feature) Y out . The process can be expressed as:

[0099] Y m =Y1⊙Y2;μ out =Expand(Softmax(μ c (Y m )))⊙Y m

[0100] in, Represents the intermediate result after pixel-by-pixel multiplication and softmax refinement. Operation μ c (·) is the mean function applied to the channel dimension, is the final output, and ⊙ represents element-by-element multiplication.

[0101] Based on the first and second embodiments of the present application, in the third embodiment of the present application, the same or similar contents as those in the above embodiments can be referred to the above introduction, and no further description will be given later. The HRM also includes a convolutional layer and a spectral attention module. On this basis, please refer to Figure 4 , step S24 also includes steps S01 to S04:

[0102] Step S01, separating high-frequency information with a wavelength higher than a preset separation wavelength and low-frequency information with a wavelength less than or equal to the preset separation wavelength from the initial fused image through the HRM;

[0103] Step S02, performing weighted convolution processing on the high-frequency information through the ASTM to obtain high-frequency features;

[0104] Step S03, extracting global features and spatial texture features from the low-frequency information through the spectral attention module to obtain supplementary features;

[0105] Step S04: By using the HRM, the high-frequency features are fused with the supplementary features to obtain a reconstructed spectral image.

[0106] It should be noted that the spectral attention module is used to enhance the model's attention to the spectral information in the input data to improve the model's ability to capture spectral features. High-frequency information refers to the high-frequency components contained in an image or signal, which usually correspond to features such as details, edges, and noise. Low-frequency information refers to the low-frequency components contained in an image or signal, which usually correspond to features such as smooth areas, large areas of color blocks, and overall shapes. Supplementary features refer to features extracted from low-frequency information, which are usually related to spatial information such as the overall structure, general outline, and smooth areas of the image.

[0107] It can be understood that through the separation strategy of high-frequency and low-frequency information in HRM, as well as blurring and downsampling operations, the input features are processed and selectively routed to two different branches, effectively managing the complex spectral information in multispectral and panchromatic images, and effectively balancing the computational complexity and performance. Through blurring and downsampling operations, high-frequency information is effectively separated from low-frequency features, thereby improving processing efficiency.

[0108] Furthermore, the ASTM includes a predictor and a token mixer, the predictor includes a token mask module and a spectral attention weight module; the step S02 also includes:

[0109] Preprocessing the high frequency information in the predictor to obtain an intermediate feature map;

[0110] By means of the token mask module, the intermediate feature map is subjected to feature sorting in the spatial dimension to obtain a spectral token mask of each intermediate feature in the intermediate feature map;

[0111] extracting, by the token mixer, key spectral features corresponding to the spectral token mask from each of the intermediate features;

[0112] Through the spectral attention weight module, each of the intermediate features is processed to obtain a spectral fusion feature;

[0113] The key spectral feature is fused with the spectral fusion feature to obtain a high-frequency feature.

[0114] It should be noted that the predictor is used to generate token masks, spectral attention weights and position embeddings respectively. The token mixer is used to fuse features from different sources or at different levels to improve the ability of the preset spectral feature fusion network to capture key spectral features.

[0115] It can be understood that through the application of the token mask module, the spatial dimension features in the intermediate feature map can be effectively sorted, so as to more accurately identify useful high-frequency features, and by extracting and fusing key spectral features, unnecessary calculations can be reduced, thereby improving the computational efficiency of the entire feature extraction process.

[0116] Furthermore, step S03 also includes:

[0117] Through the spectral attention module, the low-frequency information is upsampled according to a preset ratio to obtain global features and spatial texture features;

[0118] Based on the weight of the low-frequency information, supplementary features are selected from the global features and the spatial texture features.

[0119] It should be noted that the preset ratio is a preset magnification or reduction multiple when performing upsampling or other image processing operations. Spatial texture features are the spatial distribution patterns of pixel values ​​in a local area of ​​an image, such as edges, corners, textures, etc.

[0120] It can be understood that through the application of the spectral attention module, the low-frequency information can be effectively upsampled, thereby more accurately identifying useful low-frequency features.

[0121] In the specific implementation, given the input initial fusion image The separation operation can be expressed as:

[0122] F h =Conv 3×3 (F), F l =Down2(Blur(F))

[0123] Where Fh and Fl represent high-frequency and low-frequency information respectively. 3×3 Indicates the use of 3×3 convolution operations, Blu(·) refers to the blur operation achieved through depthwise separable dilated convolution, and Down2(·) represents downsampling with a scaling factor of 2. In this method, the high-frequency information processing branch is used as the main path, using N4 ASTMs, where ASTM is combined with a deformable spectral attention mechanism to process representative spectral features. At the same time, the low-frequency information processing branch uses N5 SWABs to integrate low-frequency information. Finally, HRM performs a weighted fusion of the results of the two branches to obtain refined high-frequency information (high-frequency features), which is mathematically expressed as:

[0124]

[0125] in represents the output of HRM, and are the weights assigned to the high-frequency and low-frequency branches, respectively. represents the sequential processing of Fh by N4 ASTMs. Up2(·) represents upsampling with a scaling factor of 2, and It means that N5 SWABs are used to integrate low-frequency information.

[0126] refer to Figure 5 ,The overall structure of ASTM mainly consists of a predictor and a token mixer, which combines the deformable spectral attention mechanism and convolution operation. Given the input features (high-frequency information after convolution), X is first processed through point-by-point convolution and global refinement modules to generate global and local conditions respectively. The local and global conditions are then connected together to form the input of the predictor, which is expressed as:

[0127] P in =Concat(f gr (X),f pwc (X)0

[0128] Among them, fgr(·) represents the global refinement operation, which includes a Conv 5×5 A LeakyReLU activation function and a Conv 3×3 , f pwc (·) represents point-by-point convolution, Concat represents the connection operation, and It is the input of the predictor, and then the intermediate feature map P in Preprocessing is performed to form the common input features of the three submodules inside the predictor. The three submodules use the intermediate feature maps to generate token masks, spectral attention weights, and position embeddings respectively:

[0129] F s =f pre (P in ); M = argsort(softmax(μ s (Conv 3×3 (F s ))))

[0130] W=σ(Conv 3×3 (μ s (F s )));PE=f pwc (ReLU(f gwc (F s )))

[0131] where f pre (·) indicates the preprocessing process, including the connection operation, a Conv 3×3 and a layer normalization. Represents the shared intermediate feature map in the predictor. Function μ s (·) computes the mean over the spatial dimensions, and argsort(·) is a function in PyTorch used to sort tensors. Vector represents the generated spectral token mask, and σ(·) represents the sigmoid function. represents the channel attention weight. Fgec(·) represents the grouped convolution. In addition, Represents positional embedding.

[0132] Then, we sort the matrix M according to the weight value and use M to sample X. Next, we divide the αC features with the highest weight into an information feature map The remaining (1-α)C features are divided into redundant features Where 0<α<1. For Zinf, we apply the deformable spectral attention mechanism for more detailed processing. For Zred, we use a simple Conv 3×3 Next, we perform spectral reorganization to combine the results of the previous step and obtain Finally, by multiplying the channel attention weight W by Zfus and adding the position embedding PE, we obtain the final output The specific equation is:

[0133] M sorted =argsort(M); Z inf ,Z red =Sampling(M sorted ,X)

[0134] Z fus =Shuffle(DCA(Z inf ),Conv 3×3 (Z r e d ))

[0135] Z out =Z fus ·W+PE

[0136] Among them, argsort(·) represents the sorting function provided by PyTorch, and Sampling(·) function selects elements from X according to vector M. DCA(·) function represents the deformable spectral attention mechanism. The spectral recombination operation is performed by Conv 1×1 and Conv 3×3 This spectral tokenization enables ASTM to better focus on key areas in the feature map. By applying attention to the core information areas and using convolution for the redundant parts, the module is able to more efficiently allocate computational resources among different features, thereby improving the balance between complexity and performance.

[0137] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the multispectral image super-resolution reconstruction method based on the adaptive spectral feature fusion network of the present application. More forms of simple transformations based on this technical concept are all within the scope of protection of the present application.

[0138] This application also provides a multi-spectral image super-resolution reconstruction device based on an adaptive spectral feature fusion network, please refer to Figure 6 , the multi-spectral image super-resolution reconstruction device based on the adaptive spectral feature fusion network comprises:

[0139] An acquisition module 10 is used to acquire a full-color image and a corresponding multi-spectral image;

[0140] The reconstruction module 20 is used to fuse the panchromatic image and the multispectral image through a preset spectral feature fusion network to obtain a reconstructed spectral image. The preset spectral feature fusion network includes an encoder, a bottleneck module and a decoder. The encoder, the bottleneck module and the decoder all include a feature enhancement feedforward network. The bottleneck module and the decoder all include HRM. The HRM includes ASTM. The ASTM is used to identify effective spectral features and suppress the extraction of redundant spectral features. The feature enhancement feedforward network is used to identify effective spatial features and suppress the extraction of redundant spatial features.

[0141] Optionally, the bottleneck module further includes an upsampling module;

[0142] The reconstruction module 20 is also used to extract effective spatial features from the full-color image through feature enhancement feedforward network processing in the encoder; extract features whose resolution meets preset expectations from the effective spatial features through the upsampling module to obtain spatial features to be merged; fuse the spatial features to be merged with the multispectral image through the bottleneck module to obtain an initial fused image; and perform feature extraction and fusion on the initial fused image according to frequency separation through the HRM to obtain a reconstructed spectral image.

[0143] Optionally, the HRM further includes a convolutional layer and a spectral attention module;

[0144] The reconstruction module 20 is also used to separate high-frequency information with a band higher than a preset separation band and low-frequency information with a band less than or equal to the preset separation band from the initial fused image through the HRM; to perform weighted convolution processing on the high-frequency information through the ASTM to obtain high-frequency features; to extract global features and spatial texture features from the low-frequency information through the spectral attention module to obtain supplementary features; and to fuse the high-frequency features with the supplementary features through the HRM to obtain a reconstructed spectral image.

[0145] Optionally, the ASTM comprises a predictor and a token mixer, wherein the predictor comprises a token mask module and a spectral attention weight module;

[0146] The reconstruction module 20 is also used to pre-process the high-frequency information through the predictor to obtain an intermediate feature map; to sort the features of the intermediate feature map in the spatial dimension through the token mask module to obtain the spectral token mask of each intermediate feature in the intermediate feature map; to extract the key spectral features corresponding to the spectral token mask from each intermediate feature through the token mixer; to add processing to each intermediate feature through the spectral attention weight module to obtain a spectral fusion feature; and to fuse the key spectral features with the spectral fusion features to obtain a high-frequency feature.

[0147] Optionally, the reconstruction module 20 is also used to upsample the low-frequency information according to a preset ratio through the spectral attention module to obtain global features and spatial texture features; and based on the weight of the low-frequency information, filter supplementary features from the global features and the spatial texture features.

[0148] Optionally, the feature enhancement feed-forward network includes a gating mechanism and a pixel attention mechanism;

[0149] The reconstruction module 20 is also used to divide the full-color image into redundant spatial features and representative spatial features according to the bands through the gating mechanism; perform group convolution and deep convolution on the representative spatial features through the convolution layer in the feature enhancement feedforward network to obtain representative features; perform deep convolution on the redundant spatial features through the convolution layer in the feature enhancement feedforward network to obtain supplementary spatial features; and perform pixel-by-pixel fusion of the representative features and the supplementary spatial features through the pixel attention mechanism to obtain effective spatial features.

[0150] The multispectral image super-resolution reconstruction device based on an adaptive spectral feature fusion network provided by the present application adopts the multispectral image super-resolution reconstruction method based on an adaptive spectral feature fusion network in the above-mentioned embodiment, which can solve the technical problem that the reconstructed multispectral image contains these redundant spectral features, thereby causing the quality of the reconstructed multispectral image to deteriorate. Compared with the prior art, the beneficial effects of the multispectral image super-resolution reconstruction device based on an adaptive spectral feature fusion network provided by the present application are the same as the beneficial effects of the multispectral image super-resolution reconstruction method based on an adaptive spectral feature fusion network provided by the above-mentioned embodiment, and the other technical features in the multispectral image super-resolution reconstruction device based on an adaptive spectral feature fusion network are the same as the features disclosed in the above-mentioned embodiment method, which will not be repeated here.

[0151] The present application provides a multispectral image super-resolution reconstruction device based on an adaptive spectral feature fusion network. The multispectral image super-resolution reconstruction device based on an adaptive spectral feature fusion network includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the multispectral image super-resolution reconstruction method based on the adaptive spectral feature fusion network in the above-mentioned embodiment one.

[0152] Reference below Figure 7 , which shows a schematic diagram of the structure of a multi-spectral image super-resolution reconstruction device based on an adaptive spectral feature fusion network suitable for implementing the embodiment of the present application. The multi-spectral image super-resolution reconstruction device based on an adaptive spectral feature fusion network in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The multi-spectral image super-resolution reconstruction device based on the adaptive spectral feature fusion network is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0153] like Figure 7As shown, the multi-spectral image super-resolution reconstruction device based on the adaptive spectral feature fusion network may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM: Read Only Memory) 1002 or the program loaded from the storage device 1003 to the random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the multi-spectral image super-resolution reconstruction device based on the adaptive spectral feature fusion network are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 can allow the multispectral image super-resolution reconstruction device based on the adaptive spectral feature fusion network to communicate wirelessly or wired with other devices to exchange data. Although the multispectral image super-resolution reconstruction device based on the adaptive spectral feature fusion network with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have alternatively.

[0154] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0155] The multispectral image super-resolution reconstruction device based on an adaptive spectral feature fusion network provided by the present application adopts the multispectral image super-resolution reconstruction method based on an adaptive spectral feature fusion network in the above-mentioned embodiment, which can solve the technical problem that the reconstructed multispectral image contains these redundant spectral features, thereby causing the quality of the reconstructed multispectral image to deteriorate. Compared with the prior art, the beneficial effects of the multispectral image super-resolution reconstruction device based on an adaptive spectral feature fusion network provided by the present application are the same as the beneficial effects of the multispectral image super-resolution reconstruction method based on an adaptive spectral feature fusion network provided by the above-mentioned embodiment, and the other technical features in the multispectral image super-resolution reconstruction device based on an adaptive spectral feature fusion network are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0156] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0157] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0158] The present application provides a computer-readable storage medium having computer-readable program instructions (i.e., computer programs) stored thereon, and the computer-readable program instructions are used to execute the multispectral image super-resolution reconstruction method based on the adaptive spectral feature fusion network in the above-mentioned embodiment.

[0159] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0160] The above-mentioned computer-readable storage medium may be included in a multi-spectral image super-resolution reconstruction device based on an adaptive spectral feature fusion network; or it may exist independently without being assembled into a multi-spectral image super-resolution reconstruction device based on an adaptive spectral feature fusion network.

[0161] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by a multispectral image super-resolution reconstruction device based on an adaptive spectral feature fusion network, the multispectral image super-resolution reconstruction device based on the adaptive spectral feature fusion network: obtains a full-color image and a corresponding multispectral image; fuses the full-color image and the multispectral image through a preset spectral feature fusion network to obtain a reconstructed spectral image, the preset spectral feature fusion network includes an encoder, a bottleneck module and a decoder, the encoder, the bottleneck module and the decoder all include a feature enhancement feedforward network, the bottleneck module and the decoder all include HRM, the HRM includes ASTM, the ASTM is used to identify effective spectral features and suppress the extraction of redundant spectral features, and the feature enhancement feedforward network is used to identify effective spatial features and suppress the extraction of redundant spatial features.

[0162] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0163] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0164] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0165] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned multispectral image super-resolution reconstruction method based on the adaptive spectral feature fusion network, and can solve the technical problem that the reconstructed multispectral image contains these redundant spectral features, thereby causing the quality of the reconstructed multispectral image to decrease. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the multispectral image super-resolution reconstruction method based on the adaptive spectral feature fusion network provided by the above-mentioned embodiment, and will not be repeated here.

[0166] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the multi-spectral image super-resolution reconstruction method based on an adaptive spectral feature fusion network as described above.

[0167] The computer program product provided by the present application can solve the technical problem that the reconstructed multispectral image contains these redundant spectral features, which in turn leads to a decrease in the quality of the reconstructed multispectral image. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the multispectral image super-resolution reconstruction method based on the adaptive spectral feature fusion network provided in the above embodiment, and will not be described in detail here.

[0168] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A multispectral image super-resolution reconstruction method based on an adaptive spectral feature fusion network, characterized in that: The method includes: Obtain a panchromatic image and a corresponding multispectral image; The panchromatic image and the multispectral image are fused through a preset spectral feature fusion network to obtain a reconstructed spectral image. The preset spectral feature fusion network includes an encoder, a bottleneck module and a decoder. The encoder, the bottleneck module and the decoder all include a feature enhancement feedforward network. The bottleneck module and the decoder all include HRM. The HRM includes ASTM. The ASTM is used to identify effective spectral features and suppress the extraction of redundant spectral features. The feature enhancement feedforward network is used to identify effective spatial features and suppress the extraction of redundant spatial features.

2. The method according to claim 1, characterized in that The bottleneck module also includes an upsampling module; The step of fusing the panchromatic image and the multispectral image through a preset spectral feature fusion network to obtain a reconstructed spectral image comprises: Extracting effective spatial features from the full-color image through feature enhancement feed-forward network processing in the encoder; By means of the upsampling module, features with a resolution that meets preset expectations are extracted from the effective spatial features to obtain spatial features to be merged; fusing the spatial features to be merged and the multispectral image through the bottleneck module to obtain an initial fused image; The HRM is used to extract and fuse the features of the initial fused image according to the frequency separation to obtain a reconstructed spectral image.

3. The method according to claim 2, characterized in that The HRM also includes a convolutional layer and a spectral attention module; The step of extracting and fusing the features of the initial fused image according to the frequency separation by the HRM to obtain the reconstructed spectral image comprises: By means of the HRM, high-frequency information having a wavelength higher than a preset separation wavelength and low-frequency information having a wavelength less than or equal to the preset separation wavelength are separated from the initial fused image; By using the ASTM, weighted convolution processing is performed on the high-frequency information to obtain high-frequency features; Through the spectral attention module, global features and spatial texture features are extracted from the low-frequency information to obtain supplementary features; Through the HRM, the high-frequency features are fused with the supplementary features to obtain a reconstructed spectral image.

4. The method according to claim 3, characterized in that The ASTM includes a predictor and a token mixer, wherein the predictor includes a token mask module and a spectral attention weight module; The step of performing weighted convolution processing on the high-frequency information through the ASTM to obtain high-frequency features includes: Preprocessing the high frequency information in the predictor to obtain an intermediate feature map; By means of the token mask module, the intermediate feature map is subjected to feature sorting in the spatial dimension to obtain a spectral token mask of each intermediate feature in the intermediate feature map; extracting, by the token mixer, key spectral features corresponding to the spectral token mask from each of the intermediate features; Through the spectral attention weight module, each of the intermediate features is processed to obtain a spectral fusion feature; The key spectral feature is fused with the spectral fusion feature to obtain a high-frequency feature.

5. The method according to claim 3, characterized in that The step of extracting global features and spatial texture features from the low-frequency information through the spectral attention module to obtain supplementary features comprises: Through the spectral attention module, the low-frequency information is upsampled according to a preset ratio to obtain global features and spatial texture features; Based on the weight of the low-frequency information, supplementary features are selected from the global features and the spatial texture features.

6. The method according to claim 2, characterized in that The feature enhancement feedforward network includes a gating mechanism and a pixel attention mechanism; The step of extracting effective spatial features from the full-color image by feature-enhanced feed-forward network processing in the encoder comprises: By means of the gating mechanism, the full-color image is divided into redundant spatial features and representative spatial features according to the bands; Performing group convolution and depth convolution on the representative spatial features through the convolution layer in the feature enhancement feedforward network to obtain representative features; Performing deep convolution on the redundant spatial features through the convolution layer in the feature enhancement feed-forward network to obtain supplementary spatial features; Through the pixel attention mechanism, the representative features are fused pixel by pixel with the supplementary spatial features to obtain effective spatial features.

7. A multispectral image super-resolution reconstruction device based on an adaptive spectral feature fusion network, characterized in that: The device comprises: An acquisition module, used for acquiring a full-color image and a corresponding multi-spectral image; A reconstruction module is used to fuse the panchromatic image and the multispectral image through a preset spectral feature fusion network to obtain a reconstructed spectral image. The preset spectral feature fusion network includes an encoder, a bottleneck module and a decoder. The encoder, the bottleneck module and the decoder all include a feature enhancement feedforward network. The bottleneck module and the decoder all include HRM. The HRM includes ASTM. The ASTM is used to identify effective spectral features and suppress the extraction of redundant spectral features. The feature enhancement feedforward network is used to identify effective spatial features and suppress the extraction of redundant spatial features.

8. A multispectral image super-resolution reconstruction device based on an adaptive spectral feature fusion network, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the multispectral image super-resolution reconstruction method based on an adaptive spectral feature fusion network as described in any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the multispectral image super-resolution reconstruction method based on an adaptive spectral feature fusion network as described in any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, which, when executed by a processor, implements the steps of the multispectral image super-resolution reconstruction method based on an adaptive spectral feature fusion network as described in any one of claims 1 to 6.

Images