A method for reconstructing a missing area of a high-resolution hyperspectral satellite image

Abstract

The application discloses a high-resolution hyperspectral satellite image missing area reconstruction method, and relates to the technical field of remote sensing image processing. The method comprises the following steps: acquiring a to-be-processed image, and acquiring a low-resolution hyperspectral satellite image containing a missing area of the to-be-processed image; acquiring a pixel area corresponding to the missing area in the low-resolution hyperspectral satellite image, and constructing an extended area based on the pixel area and a preset number of surrounding pixels; inputting a low-resolution feature map corresponding to the extended area into a trained spatio-spectral enhancement Mamba super-resolution network to obtain a super-resolution image; and filling the missing area based on the super-resolution image to obtain a reconstructed high-resolution hyperspectral satellite image. The application can effectively integrate information obtained by different satellite sensors, can improve the accuracy of super-resolution prediction by fully utilizing context information of pixels around the missing area, and can improve the accuracy of the reconstructed high-resolution hyperspectral satellite image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of remote sensing image processing, and particularly relates to a missing area reconstruction method for high-resolution hyperspectral satellite images. BACKGROUND

[0002] The high-resolution hyperspectral satellite image can provide data support for city planning, resource monitoring and environment assessment, etc. For example, it can accurately present details such as buildings and roads, and provide a basis for the rational layout of city planning.

[0003] However, when the satellite image is affected by factors such as cloud cover, sensor failure, unstable satellite attitude, etc. during shooting, it is easy to cause the satellite image hole problem, that is, the missing area appears in the satellite image. These missing areas will destroy the integrity of the image, seriously reduce the data availability, and then cause the subsequent analysis results based on the satellite image to deviate, greatly limiting the application value of the satellite image in actual decision-making.

[0004] At present, in the related art, when the missing area of the satellite image is reconstructed, it is difficult to fully tap the potential value of multi-source satellite data, and it is difficult to effectively integrate the information obtained by different satellite sensors. On the other hand, when processing high-resolution images, the reconstruction accuracy is poor, it is difficult to accurately restore the real information of the missing area, and the efficiency is low, which cannot meet the current demand for quickly obtaining accurate data.

[0005] Therefore, there is an urgent need for a method that can efficiently and accurately reconstruct the missing area of high-resolution hyperspectral satellite images. SUMMARY

[0006] The present application provides a missing area reconstruction method for high-resolution hyperspectral satellite images to solve the defects in the related art. The technical solution is as follows:

[0007] In a first aspect, the present application provides a missing area reconstruction method for high-resolution hyperspectral satellite images, which comprises:

[0008] Obtaining a to-be-processed image, the to-be-processed image being a high-resolution hyperspectral satellite image with a missing area; obtaining a low-resolution hyperspectral satellite image containing the missing area;

[0009] Obtaining a pixel area corresponding to the missing area in the low-resolution hyperspectral satellite image, and constructing an extended area based on the pixel area and a preset number of pixels around the pixel area;

[0010] input the low-resolution feature map corresponding to the extended area into the trained spatial-spectral enhancement Mamba super-resolution network to obtain a super-resolution image, wherein the spatial resolution of the super-resolution image is the same as that of the image to be processed;

[0011] fill the missing area based on the super-resolution image to obtain a reconstructed high-resolution hyperspectral satellite image.

[0012] In an optional implementation of the first aspect, the spatial-spectral enhancement Mamba super-resolution network comprises a spatial enhancement branch, a spectral enhancement branch, a feature fusion module, a reconstruction module, and an up-sampling module.

[0013] The inputting of the low-resolution feature map corresponding to the extended area into the trained spatial-spectral enhancement Mamba super-resolution network to obtain a super-resolution image comprises:

[0014] inputting the low-resolution feature map into the spatial enhancement branch to perform feature extraction on the low-resolution feature map through the spatial enhancement branch to obtain a spatial feature map;

[0015] inputting the low-resolution feature map into the spectral enhancement branch to perform feature extraction on the low-resolution feature map through the spectral enhancement branch to obtain a spectral feature map;

[0016] performing weighted fusion on the spatial feature map and the spectral feature map through the feature fusion module to obtain a fused feature map;

[0017] splicing the spatial feature map, the spectral feature map, and the fused feature map to obtain a spliced feature map, performing feature extraction on the spliced feature map through the reconstruction module to obtain a new feature map with the same dimension as the low-resolution feature map, and performing residual connection on the new feature map and the low-resolution feature map to obtain a reconstructed feature map;

[0018] performing bilinear interpolation on the reconstructed feature map through the up-sampling module to obtain a super-resolution image corresponding to the extended area.

[0019] In an optional implementation of the first aspect, the performing of feature extraction on the low-resolution feature map through the spatial enhancement branch to obtain a spatial feature map comprises:

[0020] performing multi-scale down-sampling on the low-resolution feature map to obtain a plurality of scale down-sampled feature maps;

[0021] The downsampled feature map at each scale is flattened to obtain the corresponding first feature vector. The first feature vector is then processed by the Mamba module to obtain the first intermediate feature vector. The recovery operation is then performed based on the intermediate feature vector to obtain the first spatial feature map at the same scale as the downsampled feature map.

[0022] Perform bilinear interpolation on the first spatial feature map at each scale to obtain a second spatial feature map with the same scale as the low-resolution feature map.

[0023] Each second spatial feature map is weighted and fused to obtain a fused spatial feature map;

[0024] The fused spatial feature map and the low-resolution feature map are subjected to residual connection to obtain the spatial feature map.

[0025] In one alternative embodiment of the first aspect, the step of extracting the corresponding spectral feature map based on the band information of the low-resolution feature map through the spectral enhancement branch includes:

[0026] The low-resolution feature map is divided into multiple feature map groups based on the band information of the low-resolution feature map;

[0027] Each feature map group is flattened to obtain the corresponding second feature vector. The second feature vector is then processed by the Mamba module to obtain the second intermediate feature vector.

[0028] The second intermediate feature vector is fused with the corresponding feature map group to obtain the first fused spectral feature map;

[0029] The first fused spectral feature map corresponding to each feature map group is combined to obtain the second fused spectral feature map;

[0030] The spectral feature map is obtained by performing residual connection based on the second fused spectral feature map and the low-resolution feature map.

[0031] In one alternative embodiment of the first aspect, the step of filling the missing regions based on the super-resolution image to obtain the reconstructed high-resolution hyperspectral satellite image includes:

[0032] The super-resolution region corresponding to the missing region is obtained by cropping from the super-resolution image based on the coordinate range corresponding to the missing region;

[0033] The weight of each pixel is determined based on the distance from each pixel in the super-resolution region to the missing region;

[0034] weighting and fusing the super-resolution region and the image to be processed based on the weight of each pixel to fill the missing region, to obtain a reconstructed high-resolution hyperspectral satellite image.

[0035] In an optional implementation of the first aspect, the training process of the empty-spectrum enhanced Mamba super-resolution network comprises:

[0036] obtaining a sample set, wherein the sample set comprises a plurality of samples, and a sample input of each sample is a low-resolution hyperspectral image block, and a sample label is a high-resolution hyperspectral image block matched with the coordinates of the low-resolution hyperspectral image block;

[0037] inputting the sample set into the empty-spectrum enhanced Mamba super-resolution network, and obtaining a high-resolution image block calculated by the empty-spectrum enhanced Mamba super-resolution network based on each sample input;

[0038] constructing a loss function based on the calculated high-resolution image block and the corresponding sample label, and determining whether the empty-spectrum enhanced Mamba super-resolution network converges according to the loss function;

[0039] determining model parameters of the converged empty-spectrum enhanced Mamba super-resolution network, and outputting the trained empty-spectrum enhanced Mamba super-resolution network.

[0040] In an optional implementation of the first aspect, the obtaining of the sample set comprises:

[0041] obtaining a plurality of high-resolution hyperspectral satellite images, and selecting matched low-resolution hyperspectral satellite images according to position information, band range and shooting time of each high-resolution hyperspectral satellite image;

[0042] extracting a non-cave portion from the high-resolution hyperspectral satellite image to obtain a plurality of high-resolution hyperspectral image blocks, and selecting a low-resolution hyperspectral image block matched with the coordinates of each high-resolution hyperspectral image block from the matched low-resolution hyperspectral satellite image;

[0043] constructing the sample set by taking each low-resolution hyperspectral image block as a sample input and taking a high-resolution hyperspectral image block matched with the coordinates of the sample input as a sample label.

[0044] In a second aspect, the application further provides a missing region reconstruction device for a high-resolution hyperspectral satellite image, comprising:

[0045] a data acquisition module, configured to acquire an image to be processed, the image to be processed being a high-resolution hyperspectral satellite image with a missing region, and acquire a low-resolution hyperspectral satellite image containing the missing region;

[0046] An image processing module is configured to acquire a pixel region corresponding to the missing region in the low-resolution hyperspectral satellite image, and construct an extended region based on the pixel region and a preset number of pixels around the pixel region;

[0047] The image processing module is further configured to input a low-resolution feature map corresponding to the extended region into a trained spectral-enhanced Mamba super-resolution network to obtain a super-resolution image, wherein the super-resolution image has the same spatial resolution as the image to be processed.

[0048] A reconstruction module is configured to fill the missing region based on the super-resolution image to obtain a reconstructed high-resolution hyperspectral satellite image.

[0049] In a third aspect, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method provided in the first aspect or any of the implementation manners of the first aspect.

[0050] In a fourth aspect, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program is executable on a processor to implement the method provided in the first aspect or any of the implementation manners of the first aspect.

[0051] The technical scheme provided by some embodiments of the present application has at least the following beneficial effects:

[0052] The method for reconstructing a missing region of a high-resolution hyperspectral satellite image provided by the present application can effectively integrate information obtained by different satellite sensors by selecting a pixel region corresponding to the missing region in the image to be processed from a low-resolution hyperspectral satellite image;

[0053] By extending a preset number of pixels around the pixel region to obtain an extended region and taking the extended region as the input of the trained spectral-enhanced Mamba super-resolution network, the texture, structure and other context information of the missing region and the pixels around the missing region can be fully utilized, and the accuracy of super-resolution prediction and the authenticity of reconstruction effect can be improved.

[0054] The spectral-enhanced Mamba super-resolution network can not only utilize the spatial features of the low-resolution hyperspectral image, but also utilize the spectral features of the hyperspectral image, and can more accurately restore the real information of the missing region, thereby improving the accuracy of the reconstructed high-resolution hyperspectral satellite image. BRIEF DESCRIPTION OF DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the application or the related art, the following will briefly introduce the drawings needed to be used in the embodiments or the related art description. Obviously, the drawings in the following description are some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0056] Figure 1 is one of the flowcharts of the high-resolution hyperspectral satellite image missing area reconstruction method provided by the embodiments of the application;

[0057] Figure 2 is another flowchart of the high-resolution hyperspectral satellite image missing area reconstruction method provided by the embodiments of the application;

[0058] Figure 3 is a structural schematic diagram of the high-resolution hyperspectral satellite image missing area reconstruction device provided by the embodiments of the application;

[0059] Figure 4 is a structural schematic diagram of the electronic device provided by the embodiments of the application. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solutions and advantages of the application more clear, the technical solutions in the application will be described clearly and completely in the following with reference to the drawings in the application. Obviously, the described embodiments are some of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the application.

[0061] The terms "include" and "have" and any variations thereof in the specification and claims of the application and the above-mentioned drawings are intended to cover the non-exclusive inclusion. For example, the processes, methods, systems, products or devices including a series of steps or modules are not limited to the listed steps or modules, but can optionally include the steps or modules not listed, or can optionally include other steps or modules inherent to these processes, methods, products or devices.

[0062] It should be noted that the terms "first\second" involved in the application are only to distinguish similar objects, and do not represent the specific order of the objects. Understandably, "first\second" can be interchanged in specific order or sequence as allowed. It should be understood that the objects distinguished by "first\second" can be interchanged under appropriate circumstances, so that the embodiments of the application described herein can be implemented in an order other than those described or illustrated herein.

[0063] The application will be described in detail below with reference to specific embodiments.

[0064] Next, a high-resolution hyperspectral satellite image missing area reconstruction method provided by an embodiment of the application is introduced. Figure 1 The method provided by the embodiment of the application is specifically shown in the following. Figure 1 , Figure 1 A flowchart of a high-resolution hyperspectral satellite image missing area reconstruction method provided by an embodiment of the application is shown. As shown in Figure 1 The method comprises the following steps:

[0065] S101, acquiring a to-be-processed image, the to-be-processed image being a high-resolution hyperspectral satellite image with a missing area; and acquiring a low-resolution hyperspectral satellite image containing the missing area.

[0066] S102, acquiring a pixel area corresponding to the missing area in the low-resolution hyperspectral satellite image, and constructing an extended area based on the pixel area and a preset number of pixels around the pixel area.

[0067] S103, inputting a low-resolution feature map corresponding to the extended area into a trained hyperspectral enhancement Mamba super-resolution network to obtain a super-resolution image; wherein the spatial resolution of the super-resolution image is the same as that of the to-be-processed image.

[0068] S104, filling the missing area based on the super-resolution image to obtain a reconstructed high-resolution hyperspectral satellite image.

[0069] Specifically, in S101, the to-be-processed image can be any high-resolution hyperspectral satellite image with a missing area, for example, a satellite image with a missing area can be selected from high-resolution hyperspectral satellite images collected by GF-5. A threshold segmentation method can be used to identify the missing area in the high-resolution hyperspectral satellite image, and the application is not limited in this regard.

[0070] It should be noted that the resolution of the satellite image involved in the embodiments of the application is specifically spatial resolution. Spatial resolution refers to the actual size on the ground represented by the smallest unit (pixel) that can be distinguished in detail in a remote sensing image, and is a key indicator for measuring the ability of a sensor to distinguish ground object details. For example, if the spatial resolution of a remote sensing image is 1 meter, it means that each pixel in the remote sensing image corresponds to a 1m x 1m area on the ground.

[0071] Specifically, the low-resolution hyperspectral satellite image obtained in S101 needs to be able to cover at least the missing area in the image to be processed. A plurality of low-resolution hyperspectral satellite images collected by satellites can be selected, for example, a low-resolution hyperspectral satellite image can be collected by a giant remote sensing satellite cluster.

[0072] In some embodiments, the plurality of low-resolution hyperspectral satellite images can be screened according to the position coordinate range, band information, and shooting time of the image coverage, including:

[0073] The low-resolution hyperspectral satellite image that can cover the missing area is selected, and the low-resolution hyperspectral satellite image that is consistent with the coordinate range covered by the image to be processed is found as much as possible. The low-resolution hyperspectral satellite image with a large overlap degree of the band range and the band range of the image to be processed is selected. The low-resolution hyperspectral satellite image with a small difference value of the shooting time and the shooting time of the image to be processed is selected. The low-resolution hyperspectral satellite image used for reconstructing the missing area can be screened based on the above screening conditions.

[0074] For example, image A covers 30 band ranges in the image to be processed, and image B covers 40 band ranges in the image to be processed. Therefore, from the perspective of band information, image B is better than image A. The shooting time of image C is 10 days different from that of the image to be processed, and the shooting time of image D is the same as that of the image to be processed. Therefore, from the perspective of shooting time, image D is better than image C. It can be understood that the weights of the above three screening conditions can be configured respectively, and the low-resolution hyperspectral satellite image used for reconstructing the missing area can be selected based on the weighted calculation of the result of each screening condition. The low-resolution hyperspectral satellite image can also be selected directly according to actual experience, and the embodiments of the present application are not limited in this regard.

[0075] Further, in S102, the coordinate range of the missing area in the image to be processed can be determined, so that the pixel area corresponding to the coordinate range can be cut from the low-resolution hyperspectral satellite image, that is, the pixel area corresponding to the missing area in S102 is obtained.

[0076] Further, a preset number of pixels around the pixel area in the low-resolution hyperspectral satellite image can be selected, so that the pixel area is expanded, and an expansion area containing the missing area and a certain range of pixels around the missing area is spliced, so that the subsequent model can fully utilize the context information such as texture and structure around the hole. If the number of available pixels around the pixel area is less than the required preset number, the pixels can be generated by difference calculation, so that the expansion area is constructed. The embodiments of the present application are not limited to the preset number, which can be selected according to actual needs.

[0077] In some embodiments, an outer rectangle of the outer edge of the pixel region can be generated, all pixels of the pixel region and the outer edge are located inside the outer rectangle, the same number of pixels, for example, 30 pixels, can be added to the upper, lower, left and right of the outer rectangle, and then an extended region containing the missing region and a plurality of pixels around the missing region can be obtained from the low-resolution hyperspectral satellite image.

[0078] Further, the extended region can be processed in blocks to obtain a plurality of low-resolution feature maps of the same size, that is, the low-resolution feature map corresponding to the extended region is obtained. In this way, the dimension of the image can be reduced, and the trained hyperspectral enhancement Mamba super-resolution network can be processed. For example, the low-resolution feature map can be processed into a 200x200 low-resolution feature map. The low-resolution feature map extracted based on the extended region can be expressed in the form of a vector: , R represents the dimension of the vector space, B represents the batch size, C represents the number of band ranges, h represents the height of each feature map, and w represents the width of each feature map.

[0079] Further, the step of S103 is performed, as shown in Figure 2 , a flowchart for obtaining a super-resolution image based on a hyperspectral enhancement Mamba super-resolution network. The hyperspectral enhancement Mamba super-resolution network includes a spatial enhancement branch, a spectral enhancement branch, a feature fusion module, a reconstruction module, and an upsampling module. S103 specifically includes:

[0080] S1031, inputting the low-resolution feature map into the spatial enhancement branch, and extracting features from the low-resolution feature map through the spatial enhancement branch to obtain a spatial feature map;

[0081] S1032, inputting the low-resolution feature map into the spectral enhancement branch, and extracting a spectral feature map based on the band information of the low-resolution feature map through the spectral enhancement branch.

[0082] S1033, performing weighted fusion on the spatial feature map and the spectral feature map through the feature fusion module to obtain a fused feature map.

[0083] S1034, splicing the spatial feature map, the spectral feature map and the fused feature map to obtain a spliced feature map, extracting features from the spliced feature map through the reconstruction module to obtain a new feature map with the same dimension as the low-resolution feature map, and performing residual connection on the new feature map and the low-resolution feature map to obtain a reconstructed feature map.

[0084] S1035, performing bilinear interpolation on the reconstructed feature map through the upsampling module to obtain a super-resolution image corresponding to the extended region.

[0085] Specifically, S1031 comprises the following steps:

[0086] First, the low-resolution feature map is subjected to multi-scale down-sampling to obtain a plurality of down-sampled feature maps of different scales , and the formula is applied:

[0087] ;

[0088] ;

[0089] A multi-scale feature pyramid is constructed, and the scales from the bottom layer to the top layer of the multi-scale feature pyramid gradually decrease according to the down-sampling ratio. The scale ratio of each layer to the down-sampled feature map of the immediately previous layer is the given down-sampling ratio.

[0090] wherein, represents the low-resolution feature map , and a plurality of down-sampled feature maps of different scales are obtained by down-sampling. The spatial scale gradually decreases from the down-sampled feature map to the down-sampled feature map .

[0091] Further, the down-sampled feature map of each scale is subjected to a flattening operation, and the function is applied to obtain a corresponding first feature vector. The first intermediate feature vector is obtained by processing the first feature vector through the Mamba module. The intermediate feature vector is subjected to a restoration operation, and the function is applied to obtain a first spatial feature map of the same scale as the down-sampled feature map . The formula is applied:

[0092] ;

[0093] wherein, represents a serialization operation, and the down-sampled feature map can be converted into a one-dimensional first feature vector ; represents that the Mamba module processes the serialized feature to obtain a first intermediate feature vector. Through the Mamba module, long-range spatial dependence features between a plurality of pixels can be extracted; represents the first spatial feature map after the spatial structure is restored, , i.e., the scale of the corresponding down-sampled feature map .

[0094] Further, the first spatial feature map respectively, to obtain a second spatial feature map with the same size as the low-resolution feature map , i.e., the first spatial feature map is up-sampled to the original size , and the formula is applied

[0095] ;

[0096] wherein, represents a bilinear interpolation operation, represents the second spatial feature map, and size represents the original size corresponding to the low-resolution feature map.

[0097] Further, the second spatial feature map fused with multi-scale information can determine the weight of each scale according to the spectral-auxiliary Mamba super-resolution network, and the configuration of the weight depends on the model parameters obtained by the spectral-auxiliary Mamba super-resolution network in the training process. According to the weight of each scale, each second spatial feature map is fused with multi-scale information to obtain a fused spatial feature map, and the specific application formula is

[0098] ;

[0099] ;

[0100] wherein, represents a function for generating weights, represents the numerical value of the extracted weight, is the weight of the corresponding second spatial feature map ; represents element-wise multiplication, and the weight satisfies ; represents the fused spatial feature map.

[0101] Finally, the fused spatial feature map and the low-resolution feature map are connected by residual connection to obtain the spatial feature map, and the formula is applied

[0102] ;

[0103] Through the residual connection mode, the original features of the low-resolution feature map can be preserved, and information loss in the processing process can be avoided to obtain the spatial feature map .

[0104] In this way, the spatial feature map is obtained, and the embodiment of the application can efficiently integrate spatial context information of different scales, significantly enhance the recovery ability of high-frequency details, and work collaboratively and optimize each other with the spectral branch to improve the quality of the reconstructed image, thereby providing a better data basis for subsequent analysis and application.

[0105] Specifically, considering that the hyperspectral image contains extremely rich spectral information, however, there is a certain degree of redundancy and correlation between different wavebands, the spectral feature map is extracted through the step of S1032, and the specific steps include the following steps:

[0106] First, the low-resolution feature map is divided into a plurality of feature groupings according to the waveband information of the low-resolution feature map, and the formula is applied:

[0107] ;

[0108] , ;

[0109] Specifically, the waveband information includes the number C of wavebands of the low-resolution feature map and the corresponding waveband range, and the low-resolution feature map corresponding to the waveband range j is , indicates the kth feature grouping, , is the weight of the low-resolution feature map of the jth waveband range in the kth feature grouping, can be given according to the spatial-spectral enhancement Mamba super-resolution network; the value of the weight and the number of groupings depend on the model parameters obtained in the training process of the spatial-spectral enhancement Mamba super-resolution network, and the embodiment of the application does not limit this.

[0110] In this way, the low-resolution feature maps of a plurality of waveband ranges can be divided into a plurality of subgroups according to the weight , which is equivalent to dividing according to the hyperspectral data carried by the low-resolution feature map, and the plurality of feature groupings obtained are beneficial to the effective fusion of cross-group spectral information, can combine wavebands with correlation together, not only effectively reducing the subsequent calculation amount, but also significantly improving the efficiency of feature extraction.

[0111] Further, a flattening operation is performed on each feature grouping to obtain a corresponding second feature vector , and the second feature vector The second intermediate feature vector is obtained by processing, the Mamba module has strong sequence processing capability, can deeply mine and enhance the spectral characteristics of the hyperspectral image, effectively capture the long-term dependence in the spectral information, and the formula is:

[0112] ;

[0113] Further, the second intermediate feature vector is grouped with the corresponding feature map to obtain a first fused spectral feature map, and the formula is:

[0114] ;

[0115] Wherein, is the first fused spectral feature map, is the second intermediate feature vector, indicates a gating mechanism, through which the output of the Mamba module can be fused with the original feature , thereby enhancing important spectral features and effectively suppressing noise interference.

[0116] After that, the first fused spectral feature map processed by each group is combined to obtain a second fused spectral feature map .

[0117] Finally, the second fused spectral feature map is subjected to feature reorganization and fusion through a three-dimensional convolution layer, and is connected in residual with the low-resolution feature map to obtain the spectral feature map, and the formula is:

[0118] ;

[0119] Wherein, is the spectral feature map, represents a three-dimensional convolution operation.

[0120] Specifically, in S1033, the weight of the spatial feature map and the weight of the spectral feature map can be determined according to the model parameters obtained by the spatial-spectral enhanced Mamba super-resolution network in the training process, so as to realize more effective feature fusion, and the process of feature fusion is applied to the formula:

[0121] ;

[0122] Wherein, represents a fused feature map, is a spatial feature map, is a spectral enhanced feature, is a weight of the spatial feature map, is a weight of the spectral feature map, respectively indicating the degree of attention to the spatial enhanced feature and the spectral enhanced feature.

[0123] Specifically, in S1034, firstly, the spatial feature map, the spectral feature map and the fusion feature map are spliced to obtain a spliced feature map, and then the spliced feature map is processed by the reconstruction module, which sequentially includes a two-dimensional convolution layer with a convolution kernel size of 3, an activation function and a two-dimensional convolution layer with a convolution kernel size of 1, and then the spliced feature map is feature extracted to adjust the number of channels and perform feature fusion, so as to obtain a new feature map with the same dimension as the low-resolution feature map , and then the new feature map and the low-resolution feature map are connected in residual to obtain a reconstructed feature map.

[0124] Specifically, the hyperspectral image super-resolution aims to convert a low-resolution hyperspectral image into a high-resolution hyperspectral image. In the processing process, the spatial enhancement branch, the spectral enhancement branch and the feature fusion module will extract and fuse the features of the input low-resolution hyperspectral image, so as to obtain a feature map rich in feature information. However, the spatial resolution of these feature maps is still at a low level. It is necessary to convert the low-resolution feature map into a high-resolution feature map. Therefore, the step of S1035 is executed, which specifically includes:

[0125] Specifically, for the input low-resolution feature map , after the channel is expanded through convolution operation, the following is obtained , where r is the up-sampling factor. Subsequently, the PixelShuffle operation will rearrange it to convert the channel information into the promotion of spatial resolution, and obtain the high-resolution feature map , which is expressed as:

[0126] ;

[0127] In the process of up-sampling from to , a dynamic up-sampling strategy can be adopted, and the number of times of executing the PixelShuffle operation is determined by the conditional branch according to the target magnification . The specific rules are as follows:

[0128] ;

[0129] wherein, the PixelShuffle operation is one of the functions for implementing the bilinear interpolation operation;

[0130] In this way, the size of the feature map can be aligned to the target size by bilinear interpolation, and the spatial resolution of the corresponding low-resolution feature map of the extended region is adjusted to the same spatial resolution as the image to be processed in S101.

[0131] Further, in S104, the missing region is filled based on the super-resolution image, and a reconstructed high-resolution hyperspectral satellite image is obtained.

[0132] According to the coordinate range corresponding to the missing region, a super-division region corresponding to the missing region is cropped from the super-resolution image, that is, a super-division region completely matching the missing region is cropped from the super-resolution image obtained in S103.

[0133] According to the distance of each pixel in the super-division region to the missing region, the weight of each pixel is determined, and the formula is applied:

[0134]

[0135] is the distance of each pixel in the super-division region to the boundary of the missing region, specifically the minimum distance, which can take the parameter pixel, which is not limited in the embodiments of the present application.

[0136] Further, the super-division region and the image to be processed are weighted and fused based on the weight of each pixel to fill the missing region of the image to be processed, thereby obtaining a reconstructed high-resolution hyperspectral satellite image.

[0137] In this way, a complete high-resolution hyperspectral satellite image can be obtained by fusion, and the obtained high-resolution hyperspectral satellite image has no stitching marks and high spectral consistency.

[0138] In some embodiments, the training process of the empty-spectrum enhanced Mamba super-resolution network includes:

[0139] A sample set is obtained; wherein the sample set includes a plurality of samples, and the sample input of each sample is a low-resolution hyperspectral image block, and the sample label is a high-resolution hyperspectral image block matched with the coordinates of the low-resolution hyperspectral image block;

[0140] The sample set is input into the empty-spectrum enhanced Mamba super-resolution network, and a high-resolution image block calculated by the empty-spectrum enhanced Mamba super-resolution network based on each sample input is obtained.

[0141] A loss function is constructed based on the calculated high-resolution image block and the corresponding sample label, and whether the empty-spectrum enhanced Mamba super-resolution network converges is determined according to the loss function.​​​

[0142] determine the model parameters of the converged spectral enhanced Mamba super-resolution network, and output the trained spectral enhanced Mamba super-resolution network.

[0143] so that the spectral enhanced Mamba super-resolution network learns the mapping relationship between the low-resolution hyperspectral image block and the high-resolution hyperspectral image block.

[0144] The process of establishing the sample set includes:

[0145] A plurality of high-resolution hyperspectral satellite images are obtained, and matched low-resolution hyperspectral satellite images are obtained according to the position information, band range and shooting time of each high-resolution hyperspectral satellite image. The selection of the low-resolution hyperspectral satellite image can refer to the description in S101, which will not be repeated here.

[0146] The non-cave part is extracted from the high-resolution hyperspectral satellite image to obtain a plurality of high-resolution hyperspectral image blocks, and a low-resolution hyperspectral image block matched with the coordinates of each high-resolution hyperspectral image block is selected from the matched low-resolution hyperspectral satellite image.

[0147] Each low-resolution hyperspectral image block is input as a sample, and a high-resolution hyperspectral image block matched with the sample input coordinates is used as a sample label to construct the sample set.

[0148] In some embodiments, the training data can also be expanded by data enhancement techniques such as random flipping, rotation, cropping, etc., which are not limited by the embodiments of the present application.

[0149] In some embodiments, assuming that the original non-cave GF-5 image is , the repaired image is calculated. The Peak Signal-to-Noise Ratio (PSNR) and the Structural Similarity Index Measure (SSIM) in the missing area and the overall image.

[0150] In the missing area, the PSNR value reaches 33.85 dB, which means that the signal intensity of the repaired image in this area is less different from the signal intensity of the original non-cave image, and the image quality is high. The SSIM value is 0.902, indicating that the repaired area and the original non-cave image perform well in structural similarity, and the structural information is well preserved.

[0151] Further, the complete high-resolution hyperspectral image obtained by splicing can be evaluated. The PSNR value of the complete high-resolution hyperspectral image reaches 36.84 dB, indicating that the repair process has little effect on the overall image quality. The SSIM value is 0.942, indicating that the structural similarity of the overall image is not greatly affected by the repair process, and still maintains a high degree of similarity.

[0152] The following is a device embodiment of the present application, which can be used to execute the method embodiment of the present application. For details not disclosed in the device embodiment of the present application, please refer to the method embodiment of the present application.

[0153] Next, please refer to Figure 3 The structure diagram of the high-resolution hyperspectral satellite image missing area reconstruction device provided for an exemplary embodiment of the present application. The device can be realized as all or part of the terminal by software, hardware or a combination of the two, and can also be integrated as an independent module on the server. The high-resolution hyperspectral satellite image missing area reconstruction device in the embodiment of the present application can be applied to the terminal or the cloud, and the device 30 includes a data acquisition module 301, an image processing module 302 and a reconstruction module 303, wherein:

[0154] The data acquisition module 301 is configured to acquire a to-be-processed image, the to-be-processed image being a high-resolution hyperspectral satellite image with a missing area; and acquire a low-resolution hyperspectral satellite image containing the missing area.

[0155] The image processing module 302 is configured to acquire a pixel region corresponding to the missing area in the low-resolution hyperspectral satellite image, and construct an extended region based on the pixel region and a preset number of pixels around the pixel region.

[0156] The image processing module 302 is further configured to input a low-resolution feature map corresponding to the extended region into a trained spectral-spatial enhanced Mamba super-resolution network to obtain a super-resolution image; wherein the spatial resolution of the super-resolution image is the same as that of the to-be-processed image.

[0157] The reconstruction module 303 is configured to fill the missing area based on the super-resolution image to obtain a reconstructed high-resolution hyperspectral satellite image.

[0158] It should be noted that the device 30 provided by the above embodiment is only used as an example to illustrate the division of the above functional modules when the device 30 performs the method for reconstructing the missing area of the high-resolution hyperspectral satellite image. In actual application, the above functions can be completed by different functional modules according to the needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and the method for reconstructing the missing area of the high-resolution hyperspectral satellite image provided by the above embodiment belong to the same concept, and the implementation process is detailed in the method embodiment. Therefore, the detailed description is omitted here.

[0159] The embodiment of the present application further provides an electronic device, including a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor executes the program to realize the steps of the method of any one of the above embodiments.

[0160] Please refer to Figure 4 The structure block diagram of an electronic device provided by the embodiment of the present application is shown in the figure.

[0161] As Figure 4 shown, the electronic device 400 includes a processor 401 and a memory 402.

[0162] In the embodiment of the present application, the processor 401 is the control center of the computer system, which can be the processor of a physical machine or the processor of a virtual machine. The processor 401 can include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 401 can be implemented in at least one of the hardware forms of a DSP (Digital Signal Processing), a FPGA (Field-Programmable Gate Array) and a PLA (Programmable Logic Array).

[0163] The processor 401 can also include a main processor and a coprocessor. The main processor is a processor for processing data in the wake-up state, also known as a CPU (Central Processing Unit). The coprocessor is a low-power processor for processing data in the standby state.

[0164] The memory 402 can include one or more computer-readable storage media. The computer-readable storage media can be non-transitory. The memory 402 can also include high-speed random access memory and can include non-volatile memory, such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. In some embodiments of the present application, the non-transitory computer-readable storage medium in the memory 402 is used to store at least one instruction for being executed by the processor 401 to implement the method in the embodiments of the present application.

[0165] In some embodiments, the electronic device 400 further includes a peripheral device interface 403 and at least one peripheral device 404. The processor 401, the memory 402, and the peripheral device interface 403 can be connected through a bus or a signal line. Each peripheral device 404 can be connected to the peripheral device interface 403 through a bus, a signal line, or a circuit board. Specifically, the peripheral device interface 403 can be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 401 and the memory 402.

[0166] In some embodiments of the present application, the processor 401, the memory 402, and the peripheral device interface 403 are integrated on the same chip or circuit board; in some other embodiments of the present application, any one or two of the processor 401, the memory 402, and the peripheral device interface 403 can be implemented on a separate chip or circuit board. The embodiments of the present application do not make specific limitations in this regard.

[0167] The structure block diagram of the electronic device shown in the embodiments of the present application does not constitute a limitation on the electronic device 400, and the electronic device 400 can include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0168] The embodiments of the present application also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method of any of the preceding embodiments. The computer-readable storage medium can include, but is not limited to, any type of disk including a floppy disk, an optical disk, a DVD, a CD-ROM, a micro-drive, and a magneto-optical disk, a ROM, a RAM, an EPROM, an EEPROM, a DRAM, a VRAM, a flash memory device, a magnetic or optical card, a nanosystem (including molecular memory ICs), or any type of media or device suitable for storing instructions and / or data.

[0169] Those skilled in the art can clearly understand the implementation of the various embodiments by means of software and the necessary general hardware platform from the above description of the embodiments, and of course, the embodiments can also be implemented by hardware. Based on such understanding, the above technical solutions or the part that contributes to the related art can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in the various embodiments or some parts of the embodiments.

[0170] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacement for some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for reconstructing missing regions of high-resolution hyperspectral satellite imagery, characterized in that, The method comprises the following steps: acquire a to-be-processed image, the to-be-processed image being a high-resolution hyperspectral satellite image with a missing area; acquire a low-resolution hyperspectral satellite image containing the missing area; acquire a pixel area corresponding to the missing area in the low-resolution hyperspectral satellite image, and construct an extended area based on the pixel area and a preset number of pixels around the pixel area; input the low-resolution feature map corresponding to the extended area into a trained hyperspectral enhancement Mamba super-resolution network to obtain a super-resolution image; wherein the spatial resolution of the super-resolution image is the same as that of the to-be-processed image; the hyperspectral enhancement Mamba super-resolution network comprises a spatial enhancement branch, a spectral enhancement branch, a feature fusion module, a reconstruction module, and an upsampling module; the steps of obtaining the super-resolution image based on the hyperspectral enhancement Mamba super-resolution network comprise: input the low-resolution feature map into the spatial enhancement branch to extract spatial features from the low-resolution feature map through the spatial enhancement branch; input the low-resolution feature map into the spectral enhancement branch to extract spectral features from the low-resolution feature map based on the band information of the low-resolution feature map through the spectral enhancement branch; fuse the spatial features and the spectral features through the feature fusion module to obtain a fused feature map; splice the spatial feature map, the spectral feature map, and the fused feature map to obtain a spliced feature map, extract features from the spliced feature map through the reconstruction module to obtain a new feature map with the same dimension as the low-resolution feature map, and perform residual connection on the new feature map and the low-resolution feature map to obtain a reconstructed feature map; perform bilinear interpolation on the reconstructed feature map through the upsampling module to obtain a super-resolution image corresponding to the extended area; fill the missing area based on the super-resolution image to obtain a reconstructed high-resolution hyperspectral satellite image.

2. The method of claim 1, wherein the method further comprises: The method of extracting spatial features from the low-resolution feature map through the spatial enhancement branch to obtain a spatial feature map comprises: perform multi-scale down-sampling on the low-resolution feature map to obtain a plurality of down-sampled feature maps; perform flattening operation on each scale of the down-sampled feature map to obtain a corresponding first feature vector, process the first feature vector through a Mamba module to obtain a first intermediate feature vector, and perform restoration operation based on the intermediate feature vector to obtain a first spatial feature map with the same scale as the down-sampled feature map; perform bilinear interpolation operation on each scale of the first spatial feature map to obtain a second spatial feature map with the same scale as the low-resolution feature map; fuse each second spatial feature map to obtain a fused spatial feature map; perform residual connection on the fused spatial feature map and the low-resolution feature map to obtain the spatial feature map.

3. The method of claim 1, wherein the missing region reconstruction method is a method of reconstructing a missing region of a high-resolution hyperspectral satellite image, characterized by, The method of extracting a corresponding spectral feature map based on the band information of the low-resolution feature map through the spectral enhancement branch comprises: divide the low-resolution feature map into a plurality of feature map groups according to the band information of the low-resolution feature map; The flattening operation is performed on each feature map group respectively to obtain a corresponding second feature vector, and the second feature vector is processed by the Mamba module to obtain a second intermediate feature vector; The second intermediate feature vector is fused with the corresponding feature map group to obtain a first fused spectral feature map; The first fused spectral feature map corresponding to each feature map group is combined to obtain a second fused spectral feature map; Residual connection is performed based on the second fused spectral feature map and the low-resolution feature map to obtain the spectral feature map.

4. The method of claim 1, wherein the missing region reconstruction method is a high-resolution hyperspectral satellite image reconstruction method. The filling of the missing area based on the super-resolution image to obtain the reconstructed high-resolution hyperspectral satellite image includes: The super-resolution area corresponding to the missing area is cropped from the super-resolution image according to the coordinate range corresponding to the missing area; The weight of each pixel in the super-resolution area is determined according to the distance of each pixel in the super-resolution area to the missing area; The super-resolution area and the to-be-processed image are weighted fused based on the weight of each pixel to fill the missing area, and a reconstructed high-resolution hyperspectral satellite image is obtained.

5. The method according to any one of claims 1-4, wherein, The training process of the empty-spectrum enhanced Mamba super-resolution network includes: A sample set is obtained; wherein the sample set includes a plurality of samples, and the sample input of each sample is a low-resolution hyperspectral image block, and the sample label is a high-resolution hyperspectral image block matched with the low-resolution hyperspectral image block coordinates; The sample set is input into the empty-spectrum enhanced Mamba super-resolution network, and a high-resolution image block calculated by the empty-spectrum enhanced Mamba super-resolution network based on each sample input is obtained; A loss function is constructed based on the calculated high-resolution image block and the corresponding sample label, and whether the empty-spectrum enhanced Mamba super-resolution network converges is determined according to the loss function; The model parameters of the converged empty-spectrum enhanced Mamba super-resolution network are determined, and the trained empty-spectrum enhanced Mamba super-resolution network is output.

6. The method of claim 5, wherein the missing region reconstruction method of high-resolution hyperspectral satellite imagery is characterized by, The sample set is obtained, including: A plurality of high-resolution hyperspectral satellite images are obtained, and matched low-resolution hyperspectral satellite images are selected according to the position information, band range and shooting time of each high-resolution hyperspectral satellite image; The non-cave part is extracted from the high-resolution hyperspectral satellite image to obtain a plurality of high-resolution hyperspectral image blocks, and a low-resolution hyperspectral image block matched with the coordinates of each high-resolution hyperspectral image block is selected from the matched low-resolution hyperspectral satellite image; Each low-resolution hyperspectral image block is taken as a sample input, and a high-resolution hyperspectral image block matched with the sample input coordinates is taken as a sample label, and the sample set is constructed.

7. An apparatus for reconstructing a missing region of a high-resolution hyperspectral satellite image according to the method of any one of claims 1-6, characterized in that, The device includes: A data acquisition module is configured to acquire a to-be-processed image, wherein the to-be-processed image is a high-resolution hyperspectral satellite image having a missing area; and a low-resolution hyperspectral satellite image containing the missing area is acquired; An image processing module is configured to acquire a pixel region corresponding to the missing area in the low-resolution hyperspectral satellite image, and an extended region is constructed based on the pixel region and a predetermined number of pixels around the pixel region. The image processing module is further configured to input the low-resolution feature map corresponding to the extended area into a trained hyperspectral enhancement Mamba super-resolution network to obtain a super-resolution image, wherein a spatial resolution of the super-resolution image is the same as that of the image to be processed. A reconstruction module is configured to fill the missing area based on the super-resolution image to obtain a reconstructed high-resolution hyperspectral satellite image.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the steps of the method according to any one of claims 1 to 6 when executing the program. 9.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program implements the steps of the method according to any one of claims 1 to 6 when executed by the processor.

Images