Remote sensing image super-resolution reconstruction method and system based on prior diffusion model

Abstract

The present application relates to the field of image processing technology, and specifically discloses a method and system for super-resolution reconstruction of remote sensing images based on a priori diffusion model. The method comprises extracting shallow feature information from a low-resolution image to obtain a shallow feature map, performing multi-scale downsampling and convolution fusion processing on the shallow feature map and the high-resolution image, extracting high-frequency information from the processed feature map and performing adaptive high-frequency enhancement processing to obtain conditional information. In the denoising process of the diffusion model, high-frequency information of the low-resolution image used as the conditional information input is deeply mined, so that the high-frequency information in the low-resolution image gradually approaches the high-resolution image, thereby improving the reconstruction accuracy. In addition, a noise prediction network model is introduced in the reverse diffusion process, and the noise prediction network model guides the gradual removal of noise in the reverse diffusion process, thereby improving the detail restoration capability and accuracy of super-resolution reconstruction of natural disaster remote sensing images.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a remote sensing image super-resolution reconstruction method and system based on a priori diffusion model. Background Art

[0002] At present, in the super-resolution reconstruction of remote sensing images, methods based on diffusion models gradually infer high-resolution images. These methods have significant improvements in detail preservation and global consistency. The introduction of diffusion models provides a new idea for solving the problem of super-resolution reconstruction of remote sensing images, especially in the reconstruction of post-disaster images. It can improve the degree of restoration of building details and provide accurate data support for subsequent disaster assessments.

[0003] However, current super-resolution reconstruction technology faces two major challenges. First, remote sensing images often contain a large number of small targets, especially in post-disaster assessment scenarios, such as collapsed building debris and ground cracks. These small targets have low pixel density, clustered distribution, occlusion, and diverse scales, making high-precision reconstruction particularly difficult. Existing methods have limited ability to model high-frequency details of small targets, making it difficult to effectively restore these fine features, which are crucial for determining the extent of building damage and formulating repair plans.

[0004] Secondly, the denoising process of the diffusion model is usually based on the UNet architecture. The information transmission between the encoder and the decoder in UNet usually adopts the direct channel splicing method, which is prone to mismatch in the process of semantic information transmission, resulting in semantic gap problems. Especially in the reconstruction of complex scenes after disasters, such as the assessment of high-rise buildings and surrounding debris, this information mismatch may introduce blur and distortion in the reconstruction. In addition, UNet usually uses stride convolution in the downsampling process, which leads to the loss of feature information, especially the loss of high-frequency details, which limits the performance of the reconstruction results in detail restoration. This has a direct impact on the detailed assessment of urban infrastructure after disasters, and the lack of details may lead to misjudgment of the extent of damage.

[0005] Therefore, there is an urgent need to design a more sophisticated reconstruction method to improve the performance of super-resolution reconstruction of remote sensing images, especially for infrastructure damage assessment in complex scenarios after natural disasters. Summary of the invention

[0006] The present application aims to solve at least one problem existing in the above-mentioned prior art, and based on this, proposes a remote sensing image super-resolution reconstruction method based on a priori diffusion model.

[0007] In a first aspect, the present application provides a remote sensing image super-resolution reconstruction method based on a priori diffusion model, comprising:

[0008] Acquire a low-resolution image and a high-resolution image, and extract shallow feature information from the low-resolution image to obtain a shallow feature map;

[0009] Performing multi-scale downsampling and convolution fusion processing on the shallow feature map and the high-resolution image respectively to obtain a first multi-scale feature map corresponding to the shallow feature map and a second multi-scale feature map corresponding to the high-resolution image;

[0010] Performing fast Fourier transform on the first multi-scale feature map and the second multi-scale feature map to obtain a first spectrum feature map and a second spectrum feature map, respectively, extracting high-frequency information from the first spectrum feature map and the second spectrum feature map, and performing adaptive high-frequency enhancement processing on the high-frequency information to obtain a first high-frequency feature map and a second high-frequency feature map;

[0011] Building a first loss function according to the spectrum amplitude and phase information of the first high-frequency feature map and the second high-frequency feature map, and determining condition information including an image reconstruction result based on the first loss function;

[0012] Performing forward diffusion processing on the high-resolution image to obtain a final-state image after adding noise, building an initial network model for noise prediction, and training the initial network model based on the final-state image and the condition information to obtain a target network model for noise prediction;

[0013] The final image is subjected to back diffusion processing to obtain a reconstructed high-resolution image, wherein during the back diffusion processing, a noise estimation value of each image iteration is obtained based on the target network model, and the noise estimation value is used to guide the noise processing of the back diffusion process to obtain a high-resolution image.

[0014] In some implementations, the acquiring of a low-resolution image and a high-resolution image, and extracting shallow feature information from the low-resolution image to obtain a shallow feature map includes:

[0015] Direct feature extraction is performed on the low-resolution image to obtain a target feature map, and residual feature extraction is performed to obtain a target residual feature, and the target feature map is residually connected with the target residual feature to obtain the shallow feature map.

[0016] In some implementations, the direct feature extraction of the low-resolution image to obtain a target feature map and the residual feature extraction to obtain a target residual feature, the target feature map and the target residual feature are residually connected to obtain the shallow feature map, and the direct feature extraction includes:

[0017] Extracting initial features from the low-resolution image to obtain an initial feature map;

[0018] Performing a nonlinear transformation on the initial feature map based on an activation function, and performing a convolution operation on the initial feature map after the nonlinear transformation to obtain a first feature map and a second feature map;

[0019] The first branch processing and the second branch processing are respectively performed on the first feature map, wherein the first branch processing includes performing depth convolution and point-by-point convolution on the first feature map to extract spatial and channel information to obtain an output feature map; the second branch processing includes performing feature compression and dimension matching on the first feature map to generate attention weights;

[0020] Performing weighted processing on the output feature map based on the attention weight to generate a weighted feature map;

[0021] Performing a depth-separable convolution operation on the second feature map to extract first feature information, and performing channel fusion and convolution operation on the first feature information and the weighted feature map to obtain second feature information;

[0022] The second feature information is globally residual fused with the initial feature map to generate a target feature map.

[0023] In some implementations, the low-resolution image is subjected to direct feature extraction to obtain a target feature map, and residual feature extraction to obtain a target residual feature, and the target feature map is subjected to residual connection with the target residual feature to obtain the shallow feature map, and the residual feature extraction includes:

[0024] A depth-separable convolution operation is performed on the low-resolution image to extract an initial residual feature, pixels of the initial residual feature are rearranged to obtain an up-sampled feature, and a convolution operation is performed on the up-sampled feature to obtain a target residual feature.

[0025] In some implementations, building a first loss function according to the spectrum amplitude and phase information of the first high-frequency feature map and the second high-frequency feature map, and determining condition information including an image reconstruction result based on the first loss function includes:

[0026] Build an amplitude loss function that measures the difference between the reconstructed image and the real image in the high-frequency region, where the amplitude loss function expression is:

[0027]

[0028] In the formula, , and Respectively represent the height, width and number of channels of the spectrum graph, , and Respectively represent the index position of the spectrum data in the horizontal frequency dimension, vertical frequency dimension and channel, |∙| represents the calculation of the spectrum amplitude, represents the first high-frequency feature map, represents the second high-frequency feature map, is the spectrum feature in Channels, positions plural of

[0029] A phase loss function is constructed to measure the difference between the reconstructed image and the real image in high-frequency structure, wherein the phase loss function expression is:

[0030] ;

[0031] In the formula, Represents the calculation of the spectrum phase;

[0032] According to the amplitude loss function and the phase loss function, a first loss function is obtained, and the expression of the first loss function is:

[0033]

[0034] In the formula, and represents the weight hyperparameter, represents the amplitude loss function, represents the phase loss function.

[0035] In some embodiments, the performing back diffusion processing on the final state image to obtain a reconstructed high-resolution image, wherein during the back diffusion processing, a noise estimation value of each image iteration is obtained based on the target network model, and the noise estimation value is used to guide the noise processing of the back diffusion process to obtain a high-resolution image, comprises:

[0036] Input the final state image and the condition information into the target network model to obtain The noise estimate under ;

[0037] A second loss function corresponding to the target network model is constructed, and the difference between the noise estimation value and the actual noise value is determined based on the second loss function, wherein the second loss function expression is:

[0038]

[0039] In the formula, represents the parameters of the noise prediction network, Represents the time step , noisy image and real noise The expected value of represents the network’s estimation function for noise, Indicates conditional information. represents the square of the 2-norm.

[0040] In some embodiments, the performing back diffusion processing on the final state image to obtain a reconstructed high-resolution image, wherein during the back diffusion processing, a noise estimation value of each image iteration is obtained based on the target network model, and the noise estimation value is used to guide the noise processing of the back diffusion process to obtain a high-resolution image, further comprising:

[0041] The image state is continuously updated based on the step-by-step iterative back diffusion process. The discretization form of the back diffusion process is:

[0042]

[0043] In the formula, represents an un-iterated completely random noise image, represents the discrete time step, Expressed as standard Gaussian noise, Representation and time step The coefficient of correlation, represents the mean, Representation and time step The coefficient of correlation, is the gradient of the conditional probability density, Calculated from the output of the noise prediction network.

[0044] Compared with the prior art, the technical solution provided in the first aspect of the present application includes at least the following beneficial effects or advantages:

[0045] In the above-mentioned remote sensing image super-resolution reconstruction method based on a priori diffusion model, shallow feature information is extracted from a low-resolution image to obtain a shallow feature map, the shallow feature map and the high-resolution image are subjected to multi-scale downsampling and convolution fusion processing, and high-frequency information is extracted and adaptive high-frequency enhancement processing is performed on the processed feature map to obtain prior condition information for input into the diffusion model denoising network. In the denoising process of the diffusion model, high-frequency information of the low-resolution image input as conditional information is deeply mined, so that the high-frequency information in the low-resolution image is gradually approached to the high-resolution image, thereby improving the reconstruction accuracy;

[0046] In addition, by introducing the noise prediction network model in the back diffusion process, the noise prediction network can effectively learn the noise estimation required in the back diffusion process, thereby guiding the model to gradually remove noise and accurately reconstruct high-resolution images. In addition, through the cross-attention module data processing, the feature information transmission and detail restoration capabilities between the encoder and decoder are optimized, effectively alleviating the mismatch problem between the two in the process of semantic information transmission. By using the pixel resampling convolution module to replace the stride convolution in the diffusion model denoising network, the loss of fine-grained feature information is reduced, thereby further improving the detail restoration capability of super-resolution reconstruction and improving the accuracy of super-resolution reconstruction of natural disaster remote sensing images.

[0047] In a second aspect, the present application provides a remote sensing image super-resolution reconstruction system based on a priori diffusion model, comprising:

[0048] A feature extraction module is configured to obtain a low-resolution image and a high-resolution image, extract shallow feature information from the low-resolution image, and obtain a shallow feature map;

[0049] A downsampling convolution module is configured to perform multi-scale downsampling and convolution fusion processing on the shallow feature map and the high-resolution image respectively, to obtain a first multi-scale feature map corresponding to the shallow feature map and a second multi-scale feature map corresponding to the high-resolution image;

[0050] a high-frequency information processing module, configured to perform fast Fourier transform on the first multi-scale feature map and the second multi-scale feature map to obtain a first spectrum feature map and a second spectrum feature map, respectively, extract high-frequency information from the first spectrum feature map and the second spectrum feature map, and perform adaptive high-frequency enhancement processing on the high-frequency information to obtain a first high-frequency feature map and a second high-frequency feature map;

[0051] A loss function building module, configured to build a first loss function according to the spectrum amplitude and phase information of the first high-frequency feature map and the second high-frequency feature map, and determine condition information including an image reconstruction result based on the first loss function;

[0052] The forward diffusion module is configured to perform forward diffusion processing on the high-resolution image to obtain a final image after adding noise,

[0053] A network model building module is configured to build an initial network model for noise prediction, and train the initial network model based on the final state image and the condition information to obtain a target network model for noise prediction;

[0054] The back diffusion module is configured to perform back diffusion processing on the final image to obtain a reconstructed high-resolution image, wherein during the back diffusion processing, a noise estimation value of each image iteration is obtained based on the target network model, and the noise estimation value is used to guide the noise processing of the back diffusion process to obtain a high-resolution image.

[0055] In a third aspect, the present application further provides an electronic device, including:

[0056] at least one processor; and

[0057] a memory communicatively connected to the at least one processor; wherein,

[0058] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the steps of the remote sensing image super-resolution reconstruction method based on the prior diffusion model provided in the first aspect above.

[0059] In a fourth aspect, the present application also provides a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, implement the steps of the remote sensing image super-resolution reconstruction method based on a priori diffusion model provided in the first aspect above.

[0060] It can be understood that the beneficial effects of the technical solutions provided in the second, third and fourth aspects can be found in the relevant description of the first aspect, and will not be repeated here.

[0061] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0063] Figure 1 is a flowchart of a remote sensing image super-resolution reconstruction method based on a priori diffusion model according to an embodiment of the present application;

[0064] Figure 2 is a flowchart of data processing of high-frequency approximation of a low-resolution image to a high-resolution image in a reconstruction method according to an embodiment of the present application;

[0065] Figure 3is a schematic diagram of the structure of a simplified super-resolution convolutional neural network according to an embodiment of the present application;

[0066] Figure 4 is a schematic diagram of the structure of a multi-scale downsampling module in which a low-resolution image is approached to a high-resolution image at high frequency according to an embodiment of the present application;

[0067] Figure 5 It is a structural schematic diagram of a simplified channel attention and a simplified gating method in a multi-scale downsampling method according to an embodiment of the present application;

[0068] Figure 6 is a schematic diagram of the structure of a pixel resampling convolution module in a noise prediction network according to an embodiment of the present application;

[0069] Figure 7 It is a schematic diagram of the structure of the Sigmoid activation function, the SiLU activation function and the GELU activation function according to the embodiment of the present application;

[0070] Figure 8 is a comparison diagram of FID values ​​and parameter quantities of the method of the present application and other existing super-resolution reconstruction models according to an embodiment of the present application;

[0071] Fig. 9 is a block diagram of a remote sensing image super-resolution reconstruction system based on a priori diffusion model according to an embodiment of the present application;

[0072] Fig.10 It is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0073] The embodiments of the present application are described in detail below. The embodiments described with reference to the accompanying drawings are exemplary. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0074] It should be noted that, unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific implementations and are not intended to limit this application. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0075] See also Figures 1 to 7 , Figure 1 A flowchart of a remote sensing image super-resolution reconstruction method based on a priori diffusion model provided by this embodiment is shown; Figure 2 It shows a data processing flow chart of high-frequency approximation of a low-resolution image to a high-resolution image in the reconstruction method provided by this embodiment; Figure 3 A schematic diagram of the structure of a simplified super-resolution convolutional neural network provided in this embodiment is shown; Figure 4 A schematic diagram of the structure of a multi-scale downsampling module in which a low-resolution image is high-frequency approximated to a high-resolution image provided by this embodiment is shown; Figure 5 A schematic diagram of the structure of the simplified channel attention and simplified gating method in the multi-scale downsampling method provided by this embodiment is shown; Figure 6 A schematic diagram of the structure of a pixel resampling convolution module in a noise prediction network provided by this embodiment is shown; Figure 7 The schematic diagram of the structure of the Sigmoid activation function, SiLU activation function and GELU activation function provided by this embodiment is shown.

[0076] like Figures 1 to 7 As shown, this embodiment provides a remote sensing image super-resolution reconstruction method based on a priori diffusion model, including:

[0077] Step S100: acquiring a low-resolution image and a high-resolution image, extracting shallow feature information from the low-resolution image, and obtaining a shallow feature map;

[0078] In this step, first of all, it should be explained that the method of this embodiment is used for super-resolution reconstruction of remote sensing images, especially in post-disaster assessment scenarios, to achieve improved detail restoration capabilities of super-resolution reconstruction of natural disaster remote sensing images, and to effectively restore fine features of low-resolution remote sensing images, such as collapsed building debris and ground cracks. These small targets have the characteristics of low pixel density, clustered distribution, occlusion, and diverse scales, which makes high-precision reconstruction particularly difficult. The method of this embodiment can achieve high-precision reconstruction of remote sensing images.

[0079] In some embodiments, corresponding high-definition remote sensing images are first collected for different types of natural disaster scenes (for example, earthquakes, floods, hurricanes, wildfires, and mudslides). After preprocessing, the collected high-definition images are segmented using the OpenCV library. Specifically, each high-definition image is evenly segmented into sub-images of size 512×512 through OpenCV to ensure that the subsequent super-resolution reconstruction model can better handle small-scale targets and detail features. Then, these sub-images are downsampled using MATLAB, and all 512×512 sub-images are uniformly reduced to a size of 128×128 to generate low-resolution samples as input data in the super-resolution reconstruction task. At this point, through the collection, segmentation, and downsampling of high-definition images, a remote sensing image dataset including high- and low-resolution pairings under different natural disaster scenarios is finally constructed.

[0080] Optionally, after obtaining the corresponding remote sensing image data set including low-resolution images and high-resolution images, directly extract features from the low-resolution images to obtain a target feature map, and extract residual features to obtain target residual features, and perform residual connection between the target feature map and the target residual features to obtain a shallow feature map, wherein the target feature map acquisition process includes: extracting initial features from the low-resolution image to obtain an initial feature map; performing a nonlinear transformation on the initial feature map based on an activation function, and performing a convolution operation on the initial feature map after the nonlinear transformation to obtain a first feature map and a second feature map; respectively performing a convolution operation on the first feature map The first branch and the second branch are processed, wherein the first branch includes performing deep convolution and point-by-point convolution on the first feature map to extract spatial and channel information to obtain an output feature map; the second branch includes performing feature compression and dimension matching on the first feature map to generate attention weights; performing weighted processing on the output feature map based on the attention weights to generate a weighted feature map; performing a deep separable convolution operation on the second feature map to extract the first feature information, performing channel fusion and convolution operations on the first feature information and the weighted feature map to obtain the second feature information; performing global residual fusion on the second feature information and the initial feature map to generate a target feature map.

[0081] Optionally, the process of acquiring the target residual feature includes performing a depth-separable convolution operation on the low-resolution image to extract an initial residual feature, performing pixel rearrangement on the initial residual feature to obtain an upsampled feature, and performing a convolution operation on the upsampled feature to obtain a target residual feature.

[0082] Combination Figure 2 As shown, specifically, a simplified super-resolution convolutional neural network (SSRCNN) is used to perform super-resolution on low-resolution images. Extract shallow feature information and generate feature maps named .

[0083] Combination Figure 3 As shown, it should be noted that the simplified super-resolution convolutional neural network is used for low-resolution images The steps of extracting shallow feature information include: first, low-resolution image Entering the simplified super-resolution convolutional neural network, it is divided into two parallel branches for feature information extraction, namely direct feature extraction and residual feature extraction. Specifically, in the direct feature extraction path, the low-resolution image is first As input, the initial features are extracted through Deep Separable Convolution (DS Conv) to form the initial feature map , and then use the SiLU activation function to activate the feature map Perform nonlinear transformation, and the processed results are stored in tensor matrices. For the SiLU activation function curve, please refer to Figure 7 , the calculation formula of SiLU activation function is:

[0084]

[0085] In the formula, represents the convolution operation, The activation function calculation formula is:

[0086] In the formula, x is the input tensor matrix, which is compressed by Sigmoid. The Sigmoid activation function compresses the data of the tensor matrix to a value in the interval (0, 1) according to its range to ensure normalization. For the curve of the Sigmoid activation function, please refer to Figure 7 ;

[0087] Then the normalized initial feature map It then enters multiple Deep Separable Attention Modules (DSAM) for further processing, where and They are feature maps The height and width of is the number of input channels, then is fed into two parallel The purpose of the convolution operation is to compress or transform the features while keeping the spatial resolution unchanged. The calculation formula for this step is:

[0088]

[0089] in, and They are two different convolution kernels. Represents the convolution operation. Through these two Convolution, generate feature maps respectively and ;

[0090] Next, the first feature map After two branches again, the first branch is further processed by depthwise separable convolution, and the spatial and channel information are extracted by depthwise convolution and pointwise convolution, which are calculated as:

[0091]

[0092] in, represents the spatial coordinates, represents the channel index, Indicates traversal index, For each channel Convolution kernel, is the output of the depthwise convolution;

[0093] Optionally, after the depthwise convolution is completed, the pointwise convolution process is started to mix the channel information. The pointwise convolution converts the depthwise convolution output The features of the channels are linearly combined to obtain output channels, the calculation formula is:

[0094]

[0095] In the formula, the convolution kernel of point-by-point convolution is , is the output feature map of point-wise convolution, Indicates the index of the output channel;

[0096] At the same time, the first characteristic map The second branch is through a Convolution performs feature compression and dimension matching to make the output feature map easier to activate through Sigmoid, thereby generating more accurate attention weights, and then generating an attention weight map through the Sigmoid function. , the Sigmoid function compresses the values ​​in the feature map to the range of 0 to 1, which is used to characterize the weight of each position, where Represents the generated adaptive attention weight map, which is used to adjust the feature map Features in the first branch The weighted features are further extracted through a depth-separable convolution to generate a feature map: ;

[0097] Second path feature map After a depth-separable convolution to extract features, it is combined with the feature map Channel fusion is performed, and the fused feature map passes through a After further feature extraction by convolution, the final feature map is Perform global residual fusion to generate output feature map , while improving the feature expression ability, avoiding information loss and gradient disappearance, and achieving more efficient feature transfer and reconstruction effects by fusing low-resolution input and high-level semantic features;

[0098] After passing through multiple deep separable attention modules for feature enhancement, the network first undergoes a deep separable convolution to further enhance feature extraction in a lightweight manner, and then upsamples the feature map through a pixel shuffle operation to obtain Pixel rearrangement is an efficient upsampling method that improves resolution by rearranging the channel information of the feature map, avoiding the blurring effect introduced by traditional interpolation methods. The upsampled feature map is further The convolution operation refines the details and finally generates an output result that simplifies the direct feature extraction process of the super-resolution convolutional neural network. ;

[0099] Finally, the residual feature extraction process of the network is used to preserve the original input features and enhance the generation of high-resolution images. Extracting initial residual features through depthwise separable convolution , then, the residual features are re-arranged to improve the resolution and generate up-sampled features and through The convolution is further refined. Finally, the residual feature is combined with the output feature of the direct path Perform residual connection to generate the final high-resolution shallow feature map ,Through this residual branch, the network can simultaneously maintain the structural information of the low-resolution image and enhance the details of the generated image.

[0100] Step S200: performing multi-scale downsampling and convolution fusion processing on the shallow feature map and the high-resolution image respectively, to obtain a first multi-scale feature map corresponding to the shallow feature map and a second multi-scale feature map corresponding to the high-resolution image;

[0101] In this step, first, the shallow feature map and high-resolution images Perform multi-scale downsampling operations to generate multiple feature maps of different scales, named and Subsequently, these feature maps of different scales are fused through a 1×1 convolution operation to integrate information from each scale and obtain the first multi-scale feature map containing multi-scale features. and the second multi-scale feature map .

[0102] It should be noted that, combined with Figure 4 As shown, for the shallow feature map and high-resolution images The steps of performing multi-scale downsampling operations include: first, the two inputs are respectively Convolution performs channel compression, thereby reducing computational complexity and unifying the number of channels without changing the spatial resolution. The specific calculation can be expressed as:

[0103]

[0104] in, represents a nonlinear activation function, and The feature map Corresponding Convolution kernel;

[0105] Next, the compressed initial feature map and Enter the multi-scale dilated convolution (DConv) module in parallel. Here, the dilated convolution uses different convolution kernel sizes and dilation rates, which are , , , …, , where the specific kernel size and the number of dilation rates are determined according to the complexity of the task. These multi-scale dilated convolutions can capture more contextual information without significantly increasing the number of parameters by expanding the receptive field. The formula for this process is:

[0106]

[0107]

[0108] in, represents a nonlinear activation function, and With expansion rate The dilated convolution kernel, Represents the size of different convolution kernels, represents the channel index of the feature map, and They represent the reconstruction and high-resolution feature maps after processing with different dilated convolution kernels. The dilated convolution kernel adjusts the expansion rate. To increase the receptive field, so as to better capture features at different scales;

[0109] Then, the feature map at each scale and Feature selection and enhancement are performed through the Simple Gate module. For a schematic diagram of the structure of the simplified gate method, please refer to Figure 5, the simplified gating module divides the input feature map into two parts by channel, and then further enhances the selectivity of key features through element-by-element multiplication and fusion operations. Its calculation formula can be expressed as:

[0110]

[0111] in, represents a nonlinear activation function, and By simplifying the gating module The characteristics after channel division, and By simplifying the gating module The characteristics after channel division, Represents an element-by-element multiplication operation;

[0112] Next, the gated feature map and The channel-level weight adjustment is further performed through the Simple Channel Attention (SCA) module, which first extracts the global statistical information of each channel through global average pooling and then Convolution calculates channel-level attention weights, and then applies the weights to the original feature map to highlight important feature channels. For a simplified schematic diagram of the channel attention module, see Figure 5 , and its calculation formula is:

[0113]

[0114]

[0115] in, is the Sigmoid activation function, and Respectively represent the height and width of the spectrum graph, Represents global average pooling, which calculates the mean of all positions in the feature map; and Represents the pixel index of the feature map in height and width, and the generated channel weight and Used to weight the key channels in the original feature map, the final weighted feature map is:

[0116]

[0117] So far, the multi-scale feature map processed by multi-scale dilated convolution, simplified gating module and channel attention mechanism and Corresponding to the feature map group and , where each and Respectively represent feature maps extracted from different scales:

[0118]

[0119] These feature maps are ready for subsequent high-frequency information separation and image reconstruction operations. This multi-scale processing ensures that the network can fully capture the details and global information of the image under different receptive fields. At the same time, combined with the gating and attention mechanism, it further enhances the flexibility of feature selection and the adaptability of channel information.

[0120] Step S300: performing fast Fourier transform on the first multi-scale feature map and the second multi-scale feature map to obtain a first spectrum feature map and a second spectrum feature map, extracting high-frequency information from the first spectrum feature map and the second spectrum feature map, and performing adaptive high-frequency enhancement processing on the high-frequency information to obtain a first high-frequency feature map and a second high-frequency feature map;

[0121] In this step, the first multi-scale feature map and the second multi-scale feature map Perform a Fast Fourier Transformation (FFT) operation to obtain the corresponding first spectrum feature map And the second spectrum characteristic diagram Then, the two spectrograms are input into the high-frequency and low-frequency information separation module to extract the high-frequency information and obtain the high-frequency spectrum feature graphs. and , at the same time and Adaptive high-frequency enhancement processing is performed through an adaptive Gaussian high-pass filter to obtain the first high-frequency feature map and the second high frequency feature map .

[0122] In the process of reconstructing high-frequency information, we first compare and , for low-resolution images Mining and extracting high-frequency information. The purpose of this process is to reconstruct the implicit high-frequency detail features from the low-resolution image and obtain a feature map with richer high-frequency information. After the high-frequency information is extracted, it is combined with the low-resolution image to form a conditional prior. This conditional prior will serve as the conditional information of the diffusion model denoising network.

[0123] It should be noted that the multi-scale feature maps are respectively and Perform fast Fourier transform operation to obtain the corresponding spectrum feature map and Then, the two spectrograms are input into the high-frequency and low-frequency information separation module to extract the high-frequency information and obtain the high-frequency spectrum feature graphs. and The steps include:

[0124] First, the input multi-scale feature map and The two feature maps obtained by fast Fourier transform and , respectively represent the reconstructed features generated by the network and the spectrum features of the real high-resolution image. These two sets of spectrum feature maps are used as the input of the high- and low-frequency information separation module. First, they are simplified through the channel attention mechanism to enhance the features of each channel. After that, the features of each channel are subjected to the global average pooling operation to calculate the mean of each channel. The mean calculation formula is as follows:

[0125]

[0126]

[0127] in, and Represent the network output features and the real high-resolution image features in the The mean value over the channels, and Represent the height and width of the feature map respectively, and Represents the pixel index in height and width respectively;

[0128] Based on the calculated mean above, the module regards the part above the mean as high-frequency information, and the part below the mean as low-frequency information. The specific separation process can be described as follows: For each channel , The high-frequency and low-frequency features of are defined as:

[0129]

[0130]

[0131] also, The high-frequency and low-frequency feature separation mechanism is similar to that of the CNN. This process ensures that the module can effectively distinguish and extract the high-frequency components in the spectrum, thereby enhancing the key detail information.

[0132] Subsequently, in order to further improve the accuracy of high-frequency information, the module introduces an adaptive gaussian high-pass filter to process the high-frequency part. The calculation process is as follows:

[0133]

[0134] in, Here, it is used as the standard deviation of the Gaussian filter to control the "bandwidth" of the filter, that is, through the channel mean Dynamically adjust the standard deviation to adapt to the high-frequency components of different channels. Specifically, when the mean of a channel When the mean is large (that is, the channel contains more high-frequency information), the enhancement effect of the filter will be weakened to prevent over-amplification of the existing high-frequency components. On the contrary, when the mean is small, the filter will more strongly enhance the high-frequency part of the channel to ensure that more detailed information is captured. is the spatial coordinate of the Gaussian filter kernel function;

[0135] The first high-frequency feature map after adaptive filtering and the second high frequency feature map The calculation is summarized as follows:

[0136]

[0137] in, The high frequency enhancement process and , similarly, Represents the convolution operation, which can highlight the high-frequency information in the feature map while suppressing redundant low-frequency components and noise. This process enables the model to better capture edge and texture details during image reconstruction;

[0138] Step S400: constructing a first loss function according to the spectrum amplitude and phase information of the first high-frequency feature map and the second high-frequency feature map, and determining condition information including an image reconstruction result based on the first loss function;

[0139] In this step, by measuring the spectral amplitude and phase information, a loss function is constructed to guide the network to generate a reconstruction result that is closer to the real high-resolution image. The spectrum amplitude and phase are close to The spectral amplitude and phase of , therefore, the amplitude loss and phase loss are created to constitute the final first loss function.

[0140] Optionally, the amplitude represents the signal strength in the spectrum, and is mainly used to describe the strength of high-frequency components (such as edges and textures). In the super-resolution reconstruction task, if the amplitude of the reconstructed image and the target high-resolution image is closer in the frequency domain, it means that they are more consistent in detail performance and high-frequency features. Therefore, the amplitude loss is used to measure the difference between the two images in the high-frequency region. The calculation formula of the amplitude loss is as follows:

[0141]

[0142] in, , and Respectively represent the height, width and number of channels of the spectrum graph, , and Respectively represent the index position of the spectrum data in the horizontal frequency dimension, vertical frequency dimension and channel, represents the first high-frequency feature map, represents the second high-frequency feature map, is the spectrum feature in Channels, positions plural of It represents the calculation of the spectrum amplitude, that is, the modulus of the complex spectrum element, and the calculation formula is:

[0143]

[0144] in, is the spectrum feature in Channels, positions plural of and Represent the real and imaginary parts of the spectrum respectively;

[0145] Optionally, the phase information describes the relative position relationship between different components in the spectrum, and determines the structure and edge information of the image. Even if the amplitude information is consistent, if the phase is inconsistent, the reconstructed image will be obviously distorted in structure. Therefore, the phase loss is used to measure the difference between the reconstructed image and the real image in high-frequency structure. The calculation formula of the phase loss is as follows:

[0146]

[0147] in, , and Respectively represent the height, width and number of channels of the spectrum graph, , and Respectively represent the index position of the spectrum data in the horizontal frequency dimension, vertical frequency dimension and channel, Represents the calculation of the spectrum phase, and the calculation formula is:

[0148]

[0149] In the formula, and Represent the real and imaginary parts of the spectrum respectively;

[0150] This formula calculates the phase difference between the reconstructed image and the real image pixel by pixel and sums it over all channels and frequency domain positions. By minimizing the phase loss, the network can better maintain the structural integrity and edge consistency of the reconstructed image.

[0151] The final total loss function combines the magnitude and phase losses, and the formula is as follows:

[0152]

[0153] in, and is a weight hyperparameter used to control the influence of the two losses, and its value depends on the specific task.

[0154] Step S500: performing forward diffusion processing on the high-resolution image to obtain a final image after adding noise, building an initial network model for noise prediction, and training the initial network model based on the final image and the condition information to obtain a target network model for noise prediction;

[0155] In this step, all low-resolution remote sensing images in the data set are subjected to forward noise processing. The principle and operation steps are as follows: First, all high-resolution images in the data set are as the initial data distribution, and then gradually add Gaussian random noise that conforms to the normal distribution to the initial data distribution until Then the initial data distribution is transformed into completely random noise with homogeneous distribution , where the diffusion process is defined by the mean reversion stochastic differential equation (SDE), in which, Low-resolution image approximated by bicubic interpolation and Gaussian noise A combination of represents the steady-state variance, where the formula for the forward diffusion process is as follows:

[0156]

[0157] in, represents the standard Wiener process, and are the time-dependent parameters controlling the mean reversion speed and stochastic volatility, respectively. represents the target mean of the data distribution during the diffusion process, Indicates the data status during the diffusion process;

[0158] At the same time, in order to make the equation have a closed solution, set ,in Used to control the random fluctuation of noise intensity, so given the resolution image and time , the middle moment According to the above get The closed form solution is:

[0159]

[0160] in, , Represents the cumulative regression rate in the diffusion process, which is used to control the data state Regression to target mean speed, due to It is obtained by the mean regression and noise accumulation of stochastic differential equations, so Follows Gaussian probability distribution , which is expressed as follows:

[0161]

[0162] in Respectively The mean and variance of can be calculated from the above formula, as the diffusion time , The mean and variance of and , that is, the final state is , which is consistent with the goal of the forward diffusion process;

[0163] It should be noted that the significance of the forward diffusion process is that by constructing a gradually deteriorating process, the network can gradually restore the image in the reverse process. The definition of this process and the choice of equations play a key role in the stability and efficiency of the subsequent reverse diffusion. Therefore, the stochastic differential equation of mean reversion is selected to achieve a stable and effective diffusion process, while ensuring that the final state of the combination of noise and low-resolution image is consistent with the expected noise distribution;

[0164] Step S600: performing back diffusion processing on the final image to obtain a reconstructed high-resolution image, wherein during the back diffusion processing, a noise estimation value of each image iteration is obtained based on the target network model, and the noise estimation value is used to guide the noise processing of the back diffusion process to obtain a high-resolution image.

[0165] In this step, the high-resolution image is forward diffused to obtain the final image after adding noise, and then an initial network model for noise prediction is built. The initial network model is trained based on the final image and conditional information to obtain a target network model for noise prediction. The final image is reverse diffused to obtain a reconstructed high-resolution image, wherein during the reverse diffusion process, a noise estimation value for each image iteration is obtained based on the target network model, and the noise estimation value is used to guide the noise processing of the reverse diffusion process to obtain a high-resolution image.

[0166] In some embodiments, after forward denoising is performed on all high-resolution remote sensing images in the data set, the denoised images are then subjected to a reverse denoising process. The goal of the reverse diffusion process is to obtain the final state. To restore the high-resolution image, the reverse diffusion process can be defined by a stochastic differential equation that simulates the reverse time, as follows:

[0167]

[0168] in, represents the reverse Wiener process, is the true score of the inference phase. In the training phase, since the high-resolution image , so a more accurate conditional score can be used to optimize the model during model training. The formula is as follows:

[0169]

[0170] Furthermore, we can Reparameterize to:

[0171]

[0172] in, is standard Gaussian noise, and its distribution is , so the true value score can be expressed as , and because and are all known, so at this time only a noise prediction target network model is needed to estimate the noise That's it.

[0173] Optionally, for the target network model of noise prediction, the denoising network adopts a U-shaped encoder-decoder architecture and combines an efficient activation block (EAB), a cross attention mechanism and a pixel resampling convolution block (PRCB) to optimize the input intermediate noise image. For efficient feature extraction and step-by-step reconstruction, the input of the network is the time step The noisy image below , the output is the noise By gradually processing the input noise feature map, the network can eventually estimate and remove the noise and reconstruct a high-resolution image.

[0174] First, the input noisy image After entering the denoising network, it passes through a convolutional layer to extract preliminary features and generate a feature map ,in, They represent the number of channels, height and width respectively. The purpose of preliminary feature extraction is to transform the input noisy image into low-level features, laying the foundation for deep feature extraction in the subsequent encoder.

[0175] In the encoder part, there are four layers of feature extraction stages, each of which consists of multiple efficient activation modules and a pixel resampling convolution module. The feature map of each layer After several times of processing by the efficient activation module, the efficient activation module uses deep convolution and simple channel attention mechanism to efficiently extract local nonlinear information, ensuring that key features are extracted at different scales. Then the features are processed by the pixel resampling convolution module to reduce the spatial resolution by half and increase the number of channels of the feature map. For example, in the first stage, the input feature go through After being processed by an efficient activation module, the feature is obtained by a pixel resampling convolution module. .

[0176] In the subsequent feature extraction stage, the feature map , , Passed separately , , The processing of the efficient activation module and the pixel resampling convolution module gradually reduce the spatial dimension, such as the output feature of the third stage The resolution is and number of channels , and the output characteristics of the fourth stage Then further downsample to , the number of channels increases to , at the deepest level, the encoder finally generates features As the deepest semantic information, the pixel resampling convolution module structure diagram in the target network model for noise prediction can be found in Figure 6 .

[0177] Next, the decoder part gradually reconstructs the features compressed by the encoder. The decoder gradually restores the spatial resolution of the feature map through layer-by-layer upsampling operations. In each decoding stage, the input feature map is first upsampled to restore the spatial dimension. At the same time, the feature map is processed by several efficient activation modules to further extract and refine high-frequency information. The decoder does not complete feature reconstruction independently, but uses the cross-attention mechanism for information transmission. Specifically, , , , The encoder features of the layer are passed to the corresponding decoder layer through the cross attention mechanism , , , This process ensures that the decoder can retain different levels of feature details when reconstructing high-resolution images, thereby enhancing the reconstruction effect, especially in detail recovery and noise removal.

[0178] At the same time, time embedding Time embedding plays an important role at all levels of the network encoder and decoder. Encoding is performed, and an embedding vector with the same size as the feature map is generated through a multi-layer perceptron, which is then fused with the features of each layer. Specifically, the time embedding vector adjusts the feature map of each layer so that the network can adapt to the dynamic changes in the diffusion process and flexibly handle different degrees of noise at different time steps.

[0179] The decoder finally outputs the predicted noise , rather than the denoised image, and then by adding the predicted noise From the input noisy image By subtracting it from the image, a cleaner image can be gradually obtained. This process is iteratively executed through the reverse process of the diffusion model to gradually remove the noise and finally obtain a high-quality image.

[0180] It should be noted that the step of transferring information between corresponding layers of the encoder and the decoder through the cross attention mechanism includes:

[0181] First, the input feature maps come from encoder layers of different scales , , , , these feature maps are firstly extracted by 2D average pooling to extract patch information of different scales. Specifically, for the encoder stage, feature map Processed into a size of Patches, where is the number of channels, and are the height and width of the image respectively.

[0182] After pooling, use Convolutional projection and reshaping operations flatten the pooled feature map patches into tokens, generating multiple patch tokens of different scales. , , , , where each The dimension is , is the number of patches, the above process can be expressed by the following calculation formula:

[0183]

[0184] Each patch token generated will then be It is fed into the Layer Normalization module to enhance the distribution stability of the features. The normalization process of each patch token is defined as:

[0185]

[0186] in, and Respectively The mean and standard deviation of and are learnable scaling and translation parameters, and the normalized features Then concatenate along the channel dimension to get , the formula is as follows:

[0187]

[0188] This step combines features of different scales together to provide cross-scale information integration for the subsequent attention mechanism;

[0189] In order to perform cross-attention calculations, we then need to generate queries, keys, and values, respectively, as follows:

[0190] Query generation: Each go through Convolution generates query matrix , whose dimensions are ,in is the dimension reduction of the query vector, which can be defined as:

[0191]

[0192] Key and Value Generation: Concatenating Matrix Also through Convolution generates keys Sum , whose dimensions are and , defined as follows:

[0193]

[0194]

[0195] in, , and are the weight matrices for query, key, and value respectively;

[0196] Then the generated query ,key Sum The similarity between them is calculated by matrix dot product to generate the attention score matrix , the calculation formula is:

[0197]

[0198] Further expanded to:

[0199]

[0200] in, and denote the query and key indices in the patch dimension (i.e., spatial location), respectively. Indicates the number of patches, represents the feature dimensions of the query matrix and the key matrix, Indicates The similarity score matrix between the query of each scale and the key of the entire concatenated feature matrix, and in order to avoid numerical instability caused by excessively large calculated values, a scaling factor is introduced , and normalize the formula to:

[0201]

[0202] In order to convert the attention scores into a probability distribution that can be used for weighting, the Softmax function is then applied to each row. Each element of Convert to the corresponding weight , the calculation formula is as follows:

[0203]

[0204] Among them, the final attention weight matrix It indicates the correlation between the query and the key, and the sum of the weights of each row is 1.

[0205] Next, we use the calculated attention weight matrix Pair Value Perform weighted summation to generate the output feature matrix of multi-scale information fusion Specifically, let each element Indicates The output feature value of each position is calculated as follows:

[0206]

[0207] in, is the attention weight matrix, is the value matrix, and finally the output feature matrix is ​​obtained , which aggregates multi-scale contextual information into the output of each query through weighted summation, providing rich feature expressions for downstream modules.

[0208] Then the output feature matrix Project and reshape the matrix to make it compatible with subsequent modules. Through a The convolutional layer performs linear mapping, transforming the feature dimension from Projection Back The number of channels is calculated as:

[0209]

[0210] Expanded to a specific element expression:

[0211]

[0212] in, is the projection matrix, which transforms the feature dimension from The number of channels mapped back to the input , and finally obtained The original number of patches and channels is maintained, so that the output features can be seamlessly connected to the decoder part;

[0213] In the last step of the cross-attention mechanism, the generated output features are first normalized by the layer, adjusting each element to a standard distribution to reduce the deviation between features, and introducing learnable scaling and translation parameters to adapt to the distribution of different data. These normalized features are then further enhanced through the activation function to enhance the nonlinear expression ability of the model, enabling it to capture more complex patterns. After these two steps, the final output features not only integrate multi-scale information, but also have higher numerical stability and expression ability, providing richer feature representation for subsequent decoding modules.

[0214] In some embodiments, the noise estimate output by the target network model through noise prediction is recorded as , the network uses the noisy image and low-resolution images containing high-frequency features as prior condition information input, aiming to learn The noise distribution under the condition, and in order to train the noise prediction network, a loss function is constructed , which is of the form:

[0215]

[0216] in, represents the parameters of the noise prediction network, Represents the time step , noisy image and real noise The expected value of represents the network’s estimation function for noise, represents the square of the 2-norm.

[0217] The training process includes the following steps: First, Randomly sample time steps To ensure that the model can effectively predict noise over the entire time range, the reparameterized , time step and conditional priors Input the noise prediction network to get the noise estimate , then we can calculate , to measure the real noise With the estimated noise Finally, the back propagation algorithm is used to minimize the loss function through the Adam optimizer to update the network parameters. ;

[0218] It should be noted that the step of reducing the spatial resolution by half through the pixel resampling convolution module includes:

[0219] For the input feature tensor , if its height or Width is an odd number, then the height and width are first made even by zero padding the edges to facilitate subsequent chessboard sampling and convolution operations. Specifically, if If is an odd number, a row of zeros is added to the bottom edge. If is an odd number, a column of zeros is padded on the right edge. After zero padding, a new feature tensor is obtained. ,in:

[0220]

[0221] The mathematical expression of the zero-fill operation is:

[0222]

[0223] Zero-padded feature tensor It has an even height and width in the spatial dimension, so that chessboard sampling can be performed smoothly, and chessboard sampling divides the input feature tensor into 4 smaller non-overlapping sub-feature tensors, denoted as , , , , the size of each tensor is , the specific sampling rules are as follows:

[0224] Select from location The elements of the first even rows and even columns, that is:

[0225]

[0226] Select from location The elements of the first even rows and odd columns, that is:

[0227]

[0228] Select from location The elements of the first odd rows and even columns, that is:

[0229]

[0230] Select from location The elements of the first odd rows and columns, that is:

[0231]

[0232] Through the above sampling rules, the pixels in the original feature tensor are evenly distributed to 4 sub-feature tensors, which effectively reduces the spatial resolution while keeping the number of channels unchanged. Next, these 4 sub-feature tensors are spliced ​​in the channel dimension to obtain a new feature tensor ,Right now:

[0233]

[0234] This concatenation operation connects the four sub-tensors in the channel dimension, making the concatenated feature tensor The number of channels has increased by 4 times, and the feature tensor After the convolution kernel is The size of the output tensor remains the same for the depth convolution of , where the mathematical form of the depth convolution is as follows:

[0235]

[0236] in Indicates that the convolution kernel is at position and Channel The weight on and represents the relative position of the convolution kernel, The output feature tensor is at position and Channel The eigenvalues ​​on Indicated in Middle, distance position Offset The elements in the channel The value on Indicated in The inner layer summation is performed within the window range, and the deep convolution operation processes each channel independently, which greatly reduces the computational complexity and enhances the spatial expression ability of the features.

[0237] After the depth convolution, point-by-point convolution is used to adjust the number of channels of the feature. Point-by-point convolution is used The convolution kernel processes each spatial position, and the number of convolution kernels is the number of target output channels. For the input feature tensor The role of point-by-point convolution is to change the number of channels from Reduce to , thus obtaining the output feature tensor , the calculation formula of point-by-point convolution is as follows:

[0238]

[0239] in, represents the weight of the point-by-point convolution kernel, For the output tensor at position and Channel The point-by-point convolution linearly combines the information of different channels to achieve channel compression and feature fusion.

[0240] Input feature tensor through zero padding, checkerboard sampling, concatenation, depthwise convolution and pointwise convolution. After a series of processing, a feature tensor with higher information integration and lower computational complexity is generated. ,The whole process achieves effective spatial dimensionality reduction and channel integration while retaining the ,input feature information, providing more compact and expressive features for the ,subsequent network layers.

[0241] In some embodiments, by minimizing the above loss function, the target network model for noise prediction can effectively learn the noise estimation required in the back diffusion process, thereby guiding the model to gradually remove noise and accurately reconstruct high-resolution images. In the inference stage of the model, the trained target network model for noise prediction is used to reconstruct the initial completely random noise image. At the beginning, the image state is continuously updated through the step-by-step iterative back diffusion process. The discretization form of the back diffusion process is:

[0242]

[0243] in, represents an un-iterated completely random noise image, represents the discrete time step, is standard Gaussian noise, is a time step The relevant coefficient is used to control the degradation rate of the image state during the back diffusion process. It is also a time step The relevant coefficient is mainly used to adjust the introduction of noise and the influence of gradient. is the gradient of the conditional probability density, which can be calculated from the output of the target network model predicted by noise:

[0244]

[0245] In each iteration step, the model estimates the noise at the current time step using the target network model of noise prediction and combines the conditional information Update the image state. This process effectively guides the model to make full use of the high-frequency information of low-resolution images while denoising, and gradually approach the real distribution of high-resolution images. After iterations, we finally get That is, the reconstructed high-resolution image. This method achieves high-quality super-resolution reconstruction through accurate noise estimation and the introduction of conditional priors.

[0246] In the above steps, by introducing deep mining and enhancement of high-frequency information of low-resolution images in the diffusion model, combined with the optimized noise prediction network and the back diffusion process, accurate reconstruction of high-resolution images is achieved. In addition, the cross-attention module and pixel resampling convolution are used to further optimize the transmission of feature information and the ability to restore details.

[0247] See also Figure 8 , Figure 8 The figure shows the comparison of the FID value and parameter quantity of the method of the present application provided by the present embodiment and other existing super-resolution reconstruction models; it should be noted that the accuracy and parameter quantity of the natural disaster remote sensing image super-resolution reconstruction model based on the conditional prior diffusion model in the above embodiment are compared with the existing advanced reconstruction model. For the comparison results, please refer to Figure 8 , showing the relationship between the average FID value and the number of parameters (unit: million, Mb), where the model of the above embodiment shows an excellent balance between these two indicators, and the average FID value of the model of the present invention is only 28.45, which is significantly lower than most existing models, such as ESRGAN, RCAN and EDSR, etc., which shows that the reconstructed image has better performance in quality and detail preservation;

[0248] At the same time, the parameter volume of the model of the present invention is about 27.94Mb. Compared with models such as SR3 (with a parameter volume of up to 140Mb) and EDSR (with a parameter volume of more than 60Mb), the model of the present invention has significant advantages in computational complexity and storage cost. It can achieve higher reconstruction accuracy with a smaller model scale, which reflects the high efficiency and practicality in super-resolution reconstruction of natural disaster remote sensing images, and is particularly suitable for rapid reconstruction applications under resource-constrained conditions.

[0249] In the remote sensing image super-resolution reconstruction method based on the prior diffusion model of the above-mentioned embodiment, shallow feature information is extracted from the low-resolution image to obtain a shallow feature map, the shallow feature map and the high-resolution image are subjected to multi-scale downsampling and convolution fusion processing, and the processed feature map is subjected to high-frequency information extraction and adaptive high-frequency enhancement processing to obtain prior condition information for input into the diffusion model denoising network. In the denoising process of the diffusion model, the high-frequency information of the low-resolution image input as the conditional information is deeply mined, so that the high-frequency information in the low-resolution image is gradually approached to the high-resolution image, thereby improving the reconstruction accuracy.

[0250] In addition, by introducing the target network model of noise prediction in the back diffusion process, the target network model of noise prediction can effectively learn the noise estimation required in the back diffusion process, thereby guiding the model to gradually remove noise and accurately reconstruct high-resolution images. In addition, through the cross-attention module data processing, the feature information transmission and detail restoration capabilities between the encoder and decoder are optimized, which effectively alleviates the mismatch problem between the two in the process of semantic information transmission. By using the pixel resampling convolution module to replace the stride convolution in the diffusion model denoising network, the loss of fine-grained feature information is reduced, thereby further improving the detail restoration capability of super-resolution reconstruction and improving the accuracy of super-resolution reconstruction of natural disaster remote sensing images.

[0251] See also Fig. 9 , Fig. 9 The block diagram of the remote sensing image super-resolution reconstruction system based on the prior diffusion model provided by the present embodiment is shown. The remote sensing image super-resolution reconstruction system based on the prior diffusion model 200 includes:

[0252] The feature extraction module 210 is configured to obtain a low-resolution image and a high-resolution image, and extract shallow feature information from the low-resolution image to obtain a shallow feature map;

[0253] The downsampling convolution module 220 is configured to perform multi-scale downsampling and convolution fusion processing on the shallow feature map and the high-resolution image respectively, to obtain a first multi-scale feature map corresponding to the shallow feature map and a second multi-scale feature map corresponding to the high-resolution image;

[0254] The high-frequency information processing module 230 is configured to perform fast Fourier transform on the first multi-scale feature map and the second multi-scale feature map to obtain a first spectrum feature map and a second spectrum feature map, extract high-frequency information from the first spectrum feature map and the second spectrum feature map, and perform adaptive high-frequency enhancement processing on the high-frequency information to obtain a first high-frequency feature map and a second high-frequency feature map;

[0255] The loss function building module 240 is configured to build a first loss function according to the spectrum amplitude and phase information of the first high-frequency feature map and the second high-frequency feature map, and determine condition information including an image reconstruction result based on the first loss function;

[0256] The forward diffusion module 250 is configured to perform forward diffusion processing on the high-resolution image to obtain a final image after adding noise.

[0257] A network model building module 260 is configured to build an initial network model for noise prediction, and train the initial network model based on the final state image and the condition information to obtain a target network model for noise prediction;

[0258] The back diffusion module 270 is configured to perform back diffusion processing on the final image to obtain a reconstructed high-resolution image, wherein during the back diffusion processing, a noise estimation value of each image iteration is obtained based on the target network model, and the noise estimation value is used to guide the noise processing of the back diffusion process to obtain a high-resolution image.

[0259] It can be understood that, in the remote sensing image super-resolution reconstruction system 200 based on the prior diffusion model in this embodiment, when implemented, each module runs the above Figures 1 to 8 The steps of a remote sensing image super-resolution reconstruction method based on a priori diffusion model in the corresponding embodiment, and the technical effects that can be achieved can refer to the above Figures 1 to 8 The description of the technical effects achieved in the corresponding embodiments will not be repeated here.

[0260] See also Fig.10 , Fig.10 5 is a block diagram of an electronic device provided in an embodiment of the present application. The server 500 of the electronic device includes: a processor 501, a memory 502, and a computer program 503 stored in the memory 502 and executable on the processor 501, such as a program for a remote sensing image super-resolution reconstruction method based on a priori diffusion model. When the processor 501 executes the computer program 503, the steps of the remote sensing image super-resolution reconstruction method based on a priori diffusion model in the above-mentioned embodiments are implemented, such as Figure 1 The corresponding embodiment of step S100 to step S500. Alternatively, the processor 501 executes the computer program 503 to implement the above Fig. 9 The functions of each module in the corresponding embodiment are, for example, Fig. 9 For details on the functions of the modules (such as feature extraction module 210), please refer to Fig. 9 The relevant descriptions in the corresponding embodiments are not repeated here.

[0261] Exemplarily, the computer program 503 may be divided into one or more units, one or more units are stored in the memory 502, and are executed by the processor 501 to complete the technical solution provided in the above embodiment. One or more units may be a series of computer program instruction segments that can complete specific functions, and the instruction segments are used to describe the execution process of the computer program 503 in the server 500.

[0262] The electronic device may include, but is not limited to, a processor 501 and a memory 502. Those skilled in the art will appreciate that Fig.10It is only an example of the server 500 in the electronic device and does not constitute a limitation of the server 500. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the turntable terminal device may also include an input and output terminal device, a network access terminal device, a bus, etc.

[0263] The processor 501 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0264] The memory 502 may be an internal storage unit of the server 500, such as a hard disk or memory of the server 500. The memory 502 may also be an external storage terminal device of the server 500, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the server 500. Further, the memory 502 may also include both an internal storage unit of the server 500 and an external storage terminal device. The memory 502 is used to store computer programs and other programs and data required by the turntable terminal device. The memory 502 may also be used to temporarily store data that has been output or is to be output.

[0265] In some embodiments, a computer-readable storage medium is also provided, which stores a computer program. When the computer program is executed by a processor, the steps of the remote sensing image super-resolution reconstruction method based on the prior diffusion model as described in the above embodiment are implemented.

[0266] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0267] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Among them, the computer-readable storage medium can be non-volatile or volatile. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0268] The terms "first", "second", "third", etc. in the specification and claims of the present application and the drawings are used to distinguish different objects rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a series of steps or units are included, or optionally, steps or units not listed are included, or optionally, other steps or units inherent to these processes, methods, products or devices are included.

[0269] Only the part relevant to the present application is shown in the accompanying drawings, but not all of the content. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processing or methods depicted as flow charts. Although the flow chart describes each operation (or step) as a sequential process, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of each operation can be rearranged. When its operation is completed, the process can be terminated, but it can also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0270] The terms "component", "module", "system", "unit", etc. used in this specification are used to represent computer-related entities, hardware, firmware, a combination of hardware and software, software, or software in execution. For example, a unit can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or distributed between two or more computers. In addition, these units can be executed from various computer-readable media having various data structures stored thereon. Units can communicate through local and / or remote processes, for example, based on signals having one or more data packets (e.g., data from a second unit interacting with another unit in a local system, a distributed system, and / or a network. For example, the Internet interacts with other systems via signals).

[0271] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example.

[0272] Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Mentioning "embodiment" in this article means that the specific features, structures or characteristics described in conjunction with the embodiment may be included in at least one embodiment of the present embodiment application. The appearance of this phrase in various positions in the specification is not necessarily the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It can be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0273] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

[0274] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the disclosure disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

Claims

1. A remote sensing image super-resolution reconstruction method based on a priori diffusion model, characterized in that: include: Acquire a low-resolution image and a high-resolution image, and extract shallow feature information from the low-resolution image to obtain a shallow feature map; Performing multi-scale downsampling and convolution fusion processing on the shallow feature map and the high-resolution image respectively to obtain a first multi-scale feature map corresponding to the shallow feature map and a second multi-scale feature map corresponding to the high-resolution image; Performing fast Fourier transform on the first multi-scale feature map and the second multi-scale feature map to obtain a first spectrum feature map and a second spectrum feature map, respectively, extracting high-frequency information from the first spectrum feature map and the second spectrum feature map, and performing adaptive high-frequency enhancement processing on the high-frequency information to obtain a first high-frequency feature map and a second high-frequency feature map; Building a first loss function according to the spectrum amplitude and phase information of the first high-frequency feature map and the second high-frequency feature map, and determining condition information including an image reconstruction result based on the first loss function; Performing forward diffusion processing on the high-resolution image to obtain a final image after adding noise, building an initial network model for noise prediction, and training the initial network model based on the final image and the conditional information to obtain a target network model for noise prediction, wherein the initial network model for noise prediction includes adopting a U-shaped encoder-decoder architecture, and combining an efficient activation module, a cross attention mechanism, and a pixel resampling convolution module to construct the initial network model; The final image is subjected to back diffusion processing to obtain a reconstructed high-resolution image, wherein during the back diffusion processing, a noise estimation value of each image iteration is obtained based on the target network model, and the noise estimation value is used to guide the noise processing of the back diffusion process to obtain a high-resolution image.

2. The remote sensing image super-resolution reconstruction method based on a priori diffusion model according to claim 1, characterized in that: The step of acquiring a low-resolution image and a high-resolution image, and extracting shallow feature information from the low-resolution image to obtain a shallow feature map includes: Direct feature extraction is performed on the low-resolution image to obtain a target feature map, and residual feature extraction is performed to obtain a target residual feature, and the target feature map is residually connected with the target residual feature to obtain the shallow feature map.

3. The remote sensing image super-resolution reconstruction method based on a priori diffusion model according to claim 2, characterized in that: The direct feature extraction of the low-resolution image to obtain a target feature map and the residual feature extraction to obtain a target residual feature, and the target feature map and the target residual feature are residually connected to obtain the shallow feature map, wherein the direct feature extraction includes: Extracting initial features from the low-resolution image to obtain an initial feature map; Performing a nonlinear transformation on the initial feature map based on an activation function, and performing a convolution operation on the initial feature map after the nonlinear transformation to obtain a first feature map and a second feature map; The first branch processing and the second branch processing are respectively performed on the first feature map, wherein the first branch processing includes performing depth convolution and point-by-point convolution on the first feature map to extract spatial and channel information to obtain an output feature map; the second branch processing includes performing feature compression and dimension matching on the first feature map to generate attention weights; Performing weighted processing on the output feature map based on the attention weight to generate a weighted feature map; Performing a depth-separable convolution operation on the second feature map to extract first feature information, and performing channel fusion and convolution operation on the first feature information and the weighted feature map to obtain second feature information; The second feature information is globally residual fused with the initial feature map to generate a target feature map.

4. The remote sensing image super-resolution reconstruction method based on a priori diffusion model according to claim 3 is characterized in that: The low-resolution image is subjected to direct feature extraction to obtain a target feature map, and residual feature extraction to obtain a target residual feature, and the target feature map is subjected to residual connection with the target residual feature to obtain the shallow feature map, wherein the residual feature extraction includes: A depth-separable convolution operation is performed on the low-resolution image to extract an initial residual feature, pixels of the initial residual feature are rearranged to obtain an up-sampled feature, and a convolution operation is performed on the up-sampled feature to obtain a target residual feature.

5. The remote sensing image super-resolution reconstruction method based on a priori diffusion model according to claim 1, characterized in that: The step of building a first loss function according to the spectrum amplitude and phase information of the first high-frequency feature map and the second high-frequency feature map, and determining condition information including an image reconstruction result based on the first loss function, comprises: Build an amplitude loss function that measures the difference between the reconstructed image and the real image in the high-frequency region, where the amplitude loss function expression is: In the formula, , and Respectively represent the height, width and number of channels of the spectrum graph, , and Respectively represent the index position of the spectrum data in the horizontal frequency dimension, vertical frequency dimension and channel, represents the calculation of the spectrum amplitude, represents the first high-frequency feature map, represents the second high-frequency feature map, is the spectrum feature in Channels, positions plural of A phase loss function is constructed to measure the difference between the reconstructed image and the real image in high-frequency structure, wherein the phase loss function expression is: ； In the formula, Represents the calculation of the spectrum phase; According to the amplitude loss function and the phase loss function, a first loss function is obtained, and the expression of the first loss function is: In the formula, and represents the weight hyperparameter, represents the amplitude loss function, represents the phase loss function.

6. The remote sensing image super-resolution reconstruction method based on a priori diffusion model according to claim 1, characterized in that: The step of performing a back diffusion process on the final state image to obtain a reconstructed high-resolution image, wherein a noise estimation value of each image iteration is obtained based on the target network model during the back diffusion process, and the noise estimation value is used to guide the noise processing of the back diffusion process to obtain a high-resolution image, comprises: Input the final state image and the condition information into the target network model to obtain The noise estimate under ; A second loss function corresponding to the target network model is constructed, and the difference between the noise estimation value and the actual noise value is determined based on the second loss function, wherein the second loss function expression is: In the formula, represents the parameters of the noise prediction network, Represents the time step , noisy image and real noise The expected value of represents the network’s estimation function for noise, Indicates conditional information. represents the square of the 2-norm.

7. The remote sensing image super-resolution reconstruction method based on a priori diffusion model according to claim 6, characterized in that: The method further comprises: performing a back diffusion process on the final state image to obtain a reconstructed high-resolution image, wherein a noise estimation value of each image iteration is obtained based on the target network model during the back diffusion process, and the noise estimation value is used to guide the noise processing of the back diffusion process to obtain a high-resolution image. The image state is continuously updated based on the step-by-step iterative back diffusion process. The discretization form of the back diffusion process is: ； In the formula, represents an un-iterated completely random noise image, represents the discrete time step, Expressed as standard Gaussian noise, Representation and time step The coefficient of correlation, represents the mean, Representation and time step The coefficient of correlation, is the gradient of the conditional probability density, Calculated from the output of the noise prediction network.

8. A remote sensing image super-resolution reconstruction system based on a priori diffusion model, characterized in that: include: A feature extraction module is configured to obtain a low-resolution image and a high-resolution image, extract shallow feature information from the low-resolution image, and obtain a shallow feature map; A downsampling convolution module is configured to perform multi-scale downsampling and convolution fusion processing on the shallow feature map and the high-resolution image respectively, to obtain a first multi-scale feature map corresponding to the shallow feature map and a second multi-scale feature map corresponding to the high-resolution image; a high-frequency information processing module, configured to perform fast Fourier transform on the first multi-scale feature map and the second multi-scale feature map to obtain a first spectrum feature map and a second spectrum feature map, respectively, extract high-frequency information from the first spectrum feature map and the second spectrum feature map, and perform adaptive high-frequency enhancement processing on the high-frequency information to obtain a first high-frequency feature map and a second high-frequency feature map; A loss function building module, configured to build a first loss function according to the spectrum amplitude and phase information of the first high-frequency feature map and the second high-frequency feature map, and determine condition information including an image reconstruction result based on the first loss function; A forward diffusion module is configured to perform forward diffusion processing on the high-resolution image to obtain a final image after adding noise; A network model building module is configured to build an initial network model for noise prediction, and train the initial network model based on the final state image and the condition information to obtain a target network model for noise prediction, wherein the initial network model for noise prediction includes adopting a U-shaped encoder-decoder architecture, and combining an efficient activation module, a cross attention mechanism, and a pixel resampling convolution module to construct the initial network model; The back diffusion module is configured to perform back diffusion processing on the final image to obtain a reconstructed high-resolution image, wherein during the back diffusion processing, a noise estimation value of each image iteration is obtained based on the target network model, and the noise estimation value is used to guide the noise processing of the back diffusion process to obtain a high-resolution image.

9. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of a remote sensing image super-resolution reconstruction method based on a priori diffusion model as described in any one of claims 1-7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the steps of a remote sensing image super-resolution reconstruction method based on a priori diffusion model described in any one of claims 1 to 7 are implemented.

Images