Remote sensing image super-resolution method based on texture migration guided double diffusion model

Abstract

The present application relates to the technical field of satellite remote sensing image, and more particularly to a remote sensing image super-resolution method based on texture migration guided double diffusion model, low-resolution images, high-definition reference images and blurred reference images are input into a feature extraction module to obtain multi-level features; through a dynamic optimal transport texture planning module, the corresponding relationship between low-resolution and blurred reference features is established based on optimal transport theory, matching textures are aggregated from high-definition reference features to obtain planning texture features; a gating fusion module outputs texture guided features that fuse multi-scale contexts through a top-down path; the low-resolution images after noise addition and degradation are taken as structure conditions, and each level of texture guided features are taken as texture conditions, an adaptive double condition diffusion module learns and predicts noise and residual degradation information, and a reverse sampling module reconstructs a super-resolution image. The present application realizes accurate texture matching through optimal transport, and effectively improves the super-resolution accuracy of remote sensing images.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of satellite remote sensing images, and particularly relates to a remote sensing image super-resolution method based on a texture migration guided double diffusion model. BACKGROUND

[0002] Remote sensing image super-resolution technology is widely used in target detection, semantic segmentation, environmental monitoring, disaster warning and military tasks. Through super-resolution algorithms, the accuracy of downstream tasks is significantly improved, thereby making up for the deficiency of hardware devices and reducing the dependence on imaging devices.

[0003] However, this technology faces many challenges. First, the super-resolution problem is usually considered as an ill-posed problem, as a low-resolution image may correspond to multiple high-resolution images, making it difficult to determine a unique solution, which makes it difficult for super-resolution algorithms to accurately restore detailed information. Second, in large-scale earth observation scenarios, remote sensing images often have complex textures and structures. Traditional methods have obvious shortcomings in restoring details, especially in large-scale data processing, which is prone to loss of high-frequency information.

[0004] With the rapid development of deep learning technology, the progress of super-resolution reconstruction in the field of remote sensing has been significantly improved. However, traditional diffusion model super-resolution networks based on minimization regression objectives have problems such as oversmoothing and artifacts. In addition, although the method based on generative adversarial networks (GAN) optimizes the training process through adversarial training, it still faces the problem of training difficulty and collapse in the process of high-resolution image reconstruction. The super-resolution method based on diffusion model network can generate real images, but due to the randomness of noise diffusion, there is a phenomenon of geometric distortion. The reference super-resolution based method is mostly based on minimization regression and GAN, which still has the above-mentioned problems of oversmoothing and geometric distortion. SUMMARY

[0005] Therefore, the application aims to provide a remote sensing image super-resolution method based on a texture migration guided double diffusion model to solve the problem of inaccurate texture recovery and insufficient detail authenticity of remote sensing image super-resolution results in the prior art.

[0006] To achieve the above-mentioned purpose, the technical solution of the application is as follows:

[0007] The application discloses a remote sensing image super-resolution method based on texture migration guiding a double diffusion model, and comprises the following steps: blurring a high-definition reference image to obtain a blurred reference image; inputting three images into a multi-scale feature extraction module after unifying the resolution of a low-resolution image with the resolution of the high-definition reference image to obtain multi-level features of each image; inputting features with the same level from the obtained three groups of features into a dynamic optimal transmission texture planning module; the dynamic optimal transmission texture planning module uses an optimal transmission theory to establish a corresponding relationship between a low-resolution feature space and a blurred reference feature space, and aggregates texture information most matched with the low-resolution content from the high-definition reference features according to the corresponding relationship to obtain planning texture features corresponding to each level; a gating fusion module adopts a top-down fusion path, uses a gating fusion unit to fuse the planning texture features of multiple levels to output texture guiding features corresponding to each level and fused with multi-scale contexts; and an adaptive double-condition diffusion module is used to learn predicted noise information and residual degradation information by taking the low-resolution image with added noise degradation as a structure condition and taking the generated texture guiding features corresponding to each level as texture conditions.

[0008] Further, the multi-scale feature extraction module adopts a Swin Transformer model, three images are input into the Swin Transformer model, output features of different levels in each image are extracted correspondingly, and multi-level features of each image are obtained.

[0009] Further, the dynamic optimal transmission texture planning module independently processes features of each level to output planning texture features corresponding to each level; the processing process of the three features of a certain level comprises the following steps: performing dimension reduction on the features corresponding to the low-resolution image and the blurred reference image, performing vector expansion on the two features after the dimension reduction, performing optimal solution on a cost matrix between the two expanded feature vectors to obtain a transmission matrix; performing dimension reduction on the feature corresponding to the high-definition reference image, performing vector expansion on the feature after the dimension reduction, and performing weighted aggregation on the expanded vector and the transmission matrix to obtain planning texture features corresponding to the level.

[0010] Further, the cost matrix is the squared Euclidean distance of corresponding elements between the two expanded vectors, and a Sinkhorn-Knopp iterative algorithm is used to perform optimal solution on the cost matrix to obtain the transmission matrix.

[0011] Furthermore, the gated fusion module also includes a multi-scale aggregation unit. In the gated fusion module: the shallowest planning texture features are aggregated by the multi-scale aggregation unit to obtain the texture guidance features of the corresponding scale; after downsampling the obtained texture guidance features, they are fused with the planning texture features of the second scale through the gated fusion unit, and the fused features are then input into the scale aggregation unit for feature aggregation to obtain the texture guidance features of the corresponding scale; the above operation is repeated until the planning texture features of the smallest scale are fused to obtain the texture guidance features of the corresponding scale.

[0012] Furthermore, the feature processing of the gating fusion unit includes: performing pooling operations on the downsampled texture guidance features from the previous scale and the original planned texture features from the current scale, respectively; performing channel concatenation and multilayer perceptron processing on the two pooled features to obtain adaptive gating weights; and performing weighted summation of the downsampled texture guidance features from the previous scale and the original planned texture features from the current scale according to the adaptive gating weights to obtain the output features.

[0013] Furthermore, the feature processing of the multi-scale aggregation unit includes: performing multi-scale kernel fusion on the input features, and then adding the processed features to the input features using residual connections; performing channel attention and spatial attention operations on the features processed by residual connections in sequence, and then adding the processed features to the output features of the previous residual connection using residual connections to obtain the output features; in the multi-scale kernel fusion operation, the input features are separated by channels, and the separated features are subjected to depth convolution operations of different scales, and then all the depth convolution features are integrated; the integrated features are subjected to depth convolution operations of different scales again, and then all the depth convolution features are integrated and point convolution operations are performed to obtain the output features.

[0014] Furthermore, in the adaptive biconditional diffusion module: information from low-resolution and degraded images is extracted through successive multi-level coding layers; each coding layer contains multiple residual blocks and a cross-attention mechanism to encode the output features from the previous coding layer and the temporally encoded feature information; through an intermediate layer containing multiple variance-aware blocks and a cross-attention mechanism, combined with temporal encoding and variance-conditional encoding, the output features of the last coding layer are transitioned to the decoding layer to balance and adjust the feature representation; through a decoding layer containing multiple residual blocks and a cross-attention mechanism, combined with the temporally encoded feature information, the output features of the intermediate layer and the corresponding coding layer are decoded; the decoding layer and the guidance layer obtain information from the coding layer through skip connections; at the end of the decoding layer, noise information and residual information are output through a noise prediction head and a residual prediction head, respectively.

[0015] Further, in the residual block: after group convolution and group normalization of the input features, the feature information is integrated with the time-encoded feature information; after SiLu activation of the integrated features, the output features are obtained after group convolution; in the variance perception block: after group convolution and group normalization of the input features, the feature information is integrated with the time-encoded feature information; after SiLu activation of the integrated features, the first branch is obtained after group convolution and group normalization, and the second branch is obtained after variance calculation and variance coding; the integrated results are obtained after SiLu activation, point convolution and residual connection operations, and the output features are obtained.

[0016] Further, the reverse sampling module adopts the DDIM sampling strategy, diffuses the residual degradation information and noise information through forward diffusion respectively, and uses the predicted noise information and residual information to perform reverse denoising and residual correction through the reverse inference process, and iteratively reconstructs the super-resolution image.

[0017] Compared with the prior art, the present application can achieve the following beneficial effects:

[0018] (1) The present application proposes a texture planning method based on optimal transmission theory, which calculates the transmission plan between low-resolution features and fuzzy reference features to guide the migration of high-definition textures, realizes more accurate and more distribution matching theory of texture corresponding relationship, and overcomes the matching ambiguity problem that may exist in the traditional attention mechanism.

[0019] (2) The present application designs a cross-scale fusion module with adaptive gating weight, which can dynamically adjust the fusion proportion of information from different scales according to the feature content, realizes the intelligent generation of multi-scale texture guided features, and enhances the robustness of feature expression.

[0020] (3) The present application introduces a variance perception conditional encoding mechanism in the diffusion model, extracts and embeds the local spatial variance information of the input features, provides additional spatial structure prior for the generation process, enhances the adaptability of the model to the complexity of image content, and helps to balance in texture-rich areas and smooth areas.

[0021] (4) The present application adopts a diffusion model paradigm driven by double conditions (structure condition and texture condition), and simultaneously predicts noise and residual information, comprehensively utilizes the richness of the generation model and the accuracy of residual learning, and finally improves the visual quality and geometric fidelity of remote sensing image super-resolution. BRIEF DESCRIPTION OF DRAWINGS

[0022] The accompanying drawings, which form a part of this application, are included to provide a further understanding of the application and are incorporated in and constitute a part of this application. The application will be described with additional specificity and detail by reference to the drawings in which:

[0023] Figure 1 A flow chart of the whole process of the remote sensing image super-resolution method based on texture migration guided double diffusion model according to an embodiment of the application;

[0024] Figure 2 A structural schematic diagram of the multi-scale feature extraction module according to an embodiment of the application;

[0025] Figure 3 A structural schematic diagram of the dynamic optimal transmission texture planning module according to an embodiment of the application;

[0026] Figure 4 A whole structural schematic diagram of the gating fusion module according to an embodiment of the application;

[0027] Figure 5 A specific structural schematic diagram of the gating fusion unit according to an embodiment of the application;

[0028] Figure 6 A structural schematic diagram of the multi-scale aggregation unit according to an embodiment of the application;

[0029] Figure 7 A schematic diagram of the channel attention mechanism according to an embodiment of the application;

[0030] Figure 8 A schematic diagram of the spatial attention mechanism according to an embodiment of the application;

[0031] Figure 9 A schematic diagram of the multi-scale kernel fusion according to an embodiment of the application;

[0032] Figure 10 A network structural schematic diagram of the adaptive double conditional diffusion module according to an embodiment of the application;

[0033] Figure 11 (a) A network structural schematic diagram of the residual block according to an embodiment of the application;

[0034] Figure 11 (b) A network structural schematic diagram of the variance perception block according to an embodiment of the application;

[0035] Figure 12 (a) A schematic diagram of the forward diffusion according to an embodiment of the application;

[0036] Figure 12 (b) A schematic diagram of the backward reasoning according to an embodiment of the application. DETAILED DESCRIPTION

[0037] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the present application. In different embodiments, similar elements are denoted by similar reference numerals. In the following embodiments, many details are described in order to make the present application better understood. However, those skilled in the art can easily recognize that some features can be omitted in different cases, or can be replaced by other elements, materials or methods. In some cases, some operations related to the present application are not shown or described in the specification in order to avoid the core part of the present application being overwhelmed by too much description, and it is not necessary for those skilled in the art to describe these related operations in detail according to the description in the specification and general technical knowledge in the art.

[0038] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other to form various embodiments without conflict. Meanwhile, the steps or actions in the method description can also be sequentially adjusted or adjusted in a manner that is obvious to those skilled in the art. Therefore, the various sequences in the specification and the drawings are only for the purpose of clearly describing a certain embodiment, and do not mean a necessary sequence, unless otherwise stated that a certain sequence must be followed.

[0039] In the description of the present application, it should be understood that the terms "center", "longitudinal", "transverse", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the purpose of facilitating the description of the present application and simplifying the description, and do not indicate or imply that the device or element indicated must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second" and the like are only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features limited by "first", "second" and the like can explicitly or implicitly include one or more features. In the description of the present application, unless otherwise stated, the meaning of "a plurality of" is two or more.

[0040] In the description of the present application, it should be noted that unless otherwise explicitly specified and limited, the terms "mounting", "connection", "linking" should be understood in a broad sense, for example, it can be fixed connection, or detachable connection, or integral connection; it can be mechanical connection, or electrical connection; it can be direct connection, or indirect connection through intermediate medium, or internal communication of two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0041] The present application will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0042] As Figure 1 shown, a remote sensing image super-resolution method based on texture migration guiding double diffusion model, the algorithm includes the following steps:

[0043] Step 1: blur the high-definition reference image to obtain a blurred reference image; unify the resolution of the low-resolution image and the resolution of the high-definition reference image, and then input the three images into the multi-scale feature extraction module to obtain the multi-level features of each image.

[0044] Among them, in order to eliminate the scale deviation between the low-resolution image and the high-definition reference image, the low-resolution image is up-sampled to the size of the high-definition reference image by bilinear interpolation, and at the same time, the high-definition reference image is down-sampled and then up-sampled by bilinear interpolation to complete the blurring of the high-definition reference image and obtain the blurred reference image. At this time, the resolutions of the up-sampled low-resolution image, the blurred reference image and the high-definition reference image are consistent. The three processed images are input into the multi-scale feature extraction module to obtain the multi-scale deep feature information of each image.

[0045] In the present application, the multi-scale feature extraction module adopts a Swin Transformer model. Through hierarchical design and shift window attention mechanism, the model can more effectively capture long-range dependencies and global context information in remote sensing images. Specifically, in some embodiments, three images are input into the Swin Transformer model, and output features of different levels in each image are extracted to obtain multi-level features of each image. In the embodiments of the present application, three stages of output features are obtained: the features after the first stage output of the Swin Transformer, the feature map space size of which is 1 / 4 of the input image, has rich details and edge information, and this feature is the shallow feature F1; the features after the second stage output of the Swin Transformer, the feature map space size of which is 1 / 8 of the input image, has stronger semantic information, and this feature is the middle feature F2; the features after the third stage output of the Swin Transformer, the feature map space size of which is 1 / 16 of the input image, has the most abstract high-level semantic information, and this feature is the deep feature F3. It can be further understood that when the up-sampled low-resolution image is input into the Swin Transformer model, the corresponding shallow feature F1, middle feature F2 and deep feature F3 are obtained; when the blurred reference image is input into the Swin Transformer model, the corresponding shallow feature F1, middle feature F2 and deep feature F3 are obtained; when the high-definition reference image is input into the Swin Transformer model, the corresponding shallow feature F1, middle feature F2 and deep feature F3 are obtained.

[0046] Step 2: input the features of the same level in the three groups of features (i.e. the three different levels of features corresponding to the low-resolution image , the three different levels of features corresponding to the blurred reference image , and the three different levels of features corresponding to the high-definition reference image obtained in step 1 into the dynamic optimal transport texture planning module, the dynamic optimal transport texture planning module uses the optimal transport theory to establish the corresponding relationship between the low-resolution feature space and the blurred reference feature space, and according to the corresponding relationship, aggregates the texture information most matched with the low-resolution content from the high-definition reference features to obtain the planning texture features corresponding to each level.

[0047] ​​​​​​​​​In some embodiments, the dynamic optimal transmission texture planning module processes each level of features independently, and outputs the planning texture features corresponding to each level. The processing procedure of three features of a level includes: dimension reduction is performed on the features corresponding to the low-resolution image and the blurred reference image, the two features after dimension reduction are vectorized, and an optimal solution is obtained for the cost matrix between the two vectorized features to obtain a transmission matrix; dimension reduction is performed on the feature corresponding to the high-definition reference image, the feature after dimension reduction is vectorized, and the vector after vectorization is weighted and aggregated with the transmission matrix to obtain the planning texture features of the corresponding level.

[0048] In some embodiments, the cost matrix between the two feature vectors is the squared Euclidean distance between the corresponding elements of the two vectors after vectorization, and the Sinkhorn-Knopp iterative algorithm is used to obtain an optimal solution for the cost matrix to obtain the transmission matrix.

[0049] The dynamic optimal transmission texture planning module provided by the embodiments of the present application is as shown in Figure 3 The processing procedure of three features of a level includes:

[0050] First, the feature corresponding to the low-resolution image and the feature corresponding to the blurred reference image are subjected to layer normalization respectively to stabilize the training process, and then are subjected to dimension reduction by using a point convolution layer, which aims to reduce the amount of calculation and highlight the key features. This operation can be represented by the following formula:

[0051] ;

[0052] ;

[0053] wherein, and represent the two features after dimension reduction respectively, Conv 1×1 represents a 1×1 convolution (i.e. point convolution), and LN represents layer normalization.

[0054] Then, the two features after dimension reduction and are respectively vectorized into vector and vector , each spatial position of vector is regarded as a supply point M, and each spatial position N of vector is regarded as a demand point, each point is allocated with a unit mass of 1 / M and 1 / N. A cost matrix is defined, each element in the cost matrix represents the cost of transporting a unit mass from the mth position of vector to the nth position of vector , and the cost matrix is obtained by using the squared Euclidean distance between the feature vectors:

[0055] ;

[0056] where C(m,n) denotes the element of the cost matrix C with index (m,n).

[0057] where the transport matrix can be obtained by solving the following optimization problem:

[0058] ;

[0059] where T denotes the transport matrix, ε denotes the regularization coefficient, H(T) denotes the quotient regularizer, and is specified as follows:

[0060] .

[0061] The Sinkhorn-Knopp iterative algorithm is preferably used to efficiently solve the above problem, which iteratively converges to an approximately optimal transport matrix by alternately performing row and column scaling, i.e., the feature corresponding to the high-definition reference image The weight for weighted aggregation.

[0062] The feature corresponding to the high-definition reference image is also subjected to layer normalization, point convolution, and flattening, i.e.,

[0063] ;

[0064] where, denotes the feature corresponding to the high-definition reference image The processed flattened vector.

[0065] Next, the transport matrix T is used for aggregation, and for each target position m on the low-resolution feature map, the corresponding planning texture feature P i [m] is obtained by weighting the high-definition reference feature position n, and the weight is T(m,n):

[0066] ;

[0067] Finally, the planning texture feature P i [m] is reshaped to the same spatial size as the feature corresponding to the high-definition reference image to obtain the texture planning feature P i of the current level. Repeating the above operations, the planning texture feature P1 corresponding to the shallow feature F1, the planning texture feature P2 corresponding to the middle feature F2, and the planning texture feature P3 corresponding to the deep feature F3 are obtained.

[0068] Step 3: The gating fusion module adopts a top-down fusion path, and utilizes a gating fusion unit to fuse the multiple levels of planning texture features obtained in step 2, thereby outputting texture guidance features corresponding to each level and fused with multi-scale contexts, to provide accurate texture guidance for a subsequent diffusion model.

[0069] In some embodiments, the gating fusion module further comprises a multi-scale aggregation unit; in the gating fusion module: the planning texture features of the shallowest layer are subjected to feature aggregation by the multi-scale aggregation unit to obtain texture guidance features corresponding to a scale; after the obtained texture guidance features are down-sampled, the down-sampled texture guidance features are fused with the planning texture features of the second scale by the gating fusion unit, and the fused features are input into the multi-scale aggregation unit for feature aggregation to obtain texture guidance features corresponding to the scale; the above operations are repeated until the planning texture features of the smallest scale are fused to obtain texture guidance features corresponding to the scale.

[0070] In the embodiment of the present application, the feature processing process of the gating fusion module is as shown in Figure 4

[0071] First, the planning texture features P1 of the shallowest layer are input into the multi-scale aggregation unit for feature aggregation to obtain texture guidance features G1 corresponding to a scale;

[0072] After the texture guidance features G1 are down-sampled, the down-sampled texture guidance features are fused with the planning texture features P2 of the second scale by the gating fusion unit, and the fused features are input into the multi-scale aggregation unit for feature aggregation to obtain texture guidance features G2 corresponding to a scale;

[0073] After the texture guidance features G2 are down-sampled, the down-sampled texture guidance features are fused with the planning texture features P3 of the third scale by the gating fusion unit, and the fused features are input into the multi-scale aggregation unit for feature aggregation to obtain texture guidance features G3 corresponding to a scale. Thus, the gating fusion module outputs three levels of texture guidance features G1, G2 and G3 fused with multi-scale contexts.

[0074] In some embodiments, the feature processing process of the gating fusion unit comprises: performing a pooling operation on the texture guidance features from the previous scale after down-sampling processing and the original planning texture features of the current scale, respectively, performing channel splicing and multi-layer perception processing on the two pooled features to obtain adaptive gating weights, and performing weighted summation on the texture guidance features from the previous scale after down-sampling processing and the original planning texture features of the current scale according to the adaptive gating weights to obtain output features.

[0075] In the embodiment of the present application, the feature processing process of the gating fusion unit is as shown in Figure 5 i-1 ​​) and the original planning texture feature P of the current scale i The adaptive gating weight a corresponding to the current scale is obtained by respectively performing global average pooling on the two features after pooling, channel splicing and multilayer perception processing i , as follows:

[0076] a i =MLP(Concat(Pool(D(G i-1 )),Pool(P i )));

[0077] Wherein, MLP represents a multilayer perception, including a first fully connected layer, a ReLU activation function layer, a second fully connected layer and a Sigmoid activation function layer along the data processing direction, Concat represents channel splicing, and Pool represents global average pooling;

[0078] The texture guidance feature D(G i-1 ) of the last scale after down-sampling processing and the original planning texture feature P i of the current scale are weighted and summed according to the adaptive gating weight a i to obtain the output feature , as follows:

[0079] .

[0080] The output feature of the gating fusion unit is subjected to feature aggregation processing of the multi-scale aggregation unit to obtain the texture guidance feature G i of the current scale.

[0081] In some embodiments, the feature processing process of the multi-scale aggregation unit includes: performing a multi-scale kernel fusion operation on the input feature, and then adding the processed feature to the input feature by using a residual connection; the feature processed by the residual connection is subjected to channel attention operation and spatial attention operation in turn, and then the processed feature is added to the output feature of the previous residual connection by using a residual connection to obtain the output feature. In the multi-scale kernel fusion operation, the input feature is subjected to channel separation, a plurality of separated features are subjected to deep convolution operation of different scales respectively, and then all the features after deep convolution are integrated; the integrated features are subjected to deep convolution operation of different scales again, and then all the features after deep convolution are integrated and subjected to point convolution operation to obtain the output feature.

[0082] The feature processing process in the multi-scale aggregation unit provided by the embodiment of the application is as shown in Figure 6 , wherein the input feature is first subjected to layer normalization processing, and then input into the multi-scale kernel fusion, and the original information of the input is retained by using a residual connection to the processed feature information In the middle, again Layer normalization operation, channel attention mechanism (its feature processing process is shown in Figure 7 , and the spatial attention mechanism (its feature processing process is shown in Figure 8 ). Using multiple attention mechanisms to adaptively adjust the spatial weights and channel weights of different scale convolution kernels to extract features, and finally using residual connection to retain the original feature information again to get the final output result X out , that is, the final texture guide feature of the current scale .

[0083] Specifically, the feature processing process of multi-scale kernel fusion is shown in Figure 9 , the input feature X after layer normalization is first passed through point convolution, and then the channel dimension of the input feature X is expanded to 3 times the original through chunk channel operation. Then, the input feature X is divided into three sub-tensors along the channel dimension, respectively denoted as X3, X5 and X7, as follows:

[0084] X3,X5,X7=chunk(f 1×1 (LN(X)));

[0085] Where LN represents layer normalization, f 1×1 represents point convolution.

[0086] Subsequently, three sub-tensors X3, X5 and X7 are respectively processed using 3×3, 5×5 and 7×7 three-scale deep convolution, and then respectively activated by ReLU, to obtain three-scale local information 、 and . In order to realize the interaction of different scale information, the three local information 、 and are spliced to obtain the shallow fusion feature X a . Specifically as follows:

[0087] ;

[0088] ;

[0089] Where f 3×3 , f 5×5 and f 7×7 represent 3×3, 5×5 and 7×7 three-scale deep convolution, and Concat represents splicing operation.

[0090] The shallow fusion feature X aAfter the deep convolution of 3 scales of 3x3, 5x5 and 7x7 is used again, the multi-scale local information is further enhanced, and deep features of 3 scales are extracted 、 and . The deep features of 3 scales 、 and are aggregated to obtain deep fusion features Xb. Finally, the channel dimension of the deep fusion features Xb is adjusted back to the channel dimension of the input features X through point convolution to obtain the processed multi-scale deep feature information X', as follows:

[0091] ;

[0092] .

[0093] In the gating fusion module, through multiple refining and fusing of shallow, middle and deep features, three different scale texture guidance features .

[0094] Step 4: The low-resolution image degraded by adding noise is taken as the structure condition, and each level of the texture guidance feature generated in step 3 is taken as the texture condition, and the adaptive double-condition diffusion module is used to learn the predicted noise information and residual degradation information.

[0095] In some embodiments, the adaptive double-condition diffusion module extracts information of the low-resolution image and the degraded image through continuous multi-level encoding layers; each encoding layer includes multiple residual blocks and a cross-attention mechanism, and encodes the output features from the upper-level encoding layer and the time-encoded feature information; the output features of the last-level encoding layer are transitioned to the decoding layer through an intermediate layer including multiple variance perception blocks and a cross-attention mechanism, combined with time encoding and variance condition encoding, to balance and adjust the feature expression; the output features of the intermediate layer and the encoding layer of the corresponding level are decoded through a decoding layer including multiple residual blocks and a cross-attention mechanism, combined with time-encoded feature information; the decoding layer and the guidance layer obtain information in the encoding layer through a skip connection; at the end of the decoding layer, the network branch is a noise prediction head and a residual prediction head, which respectively predict noise information and residual information.

[0096] The adaptive double-condition diffusion module provided in the embodiments of the present application is as follows Figure 10As shown in the figure, after the degraded image and the texture guide feature are convolved and spliced together, the information of the degraded image and the texture guide feature is extracted through 4-level encoding layers. Along the data processing direction, each encoding layer contains 2 consecutive residual blocks and a cross-attention mechanism, which encodes the output features from the upper-level encoding layer and the time-encoded feature information, and then down-samples the output features of the cross-attention mechanism to obtain the output features of the current encoding layer. It can be understood that the input features of the first-level encoding layer are the features obtained by convolving the low-resolution image and the degraded image. Through the intermediate layer, the output features of the last-level encoding layer are transitioned to the decoding layer by combining the time-encoded feature information and the information of the corresponding encoding layer. Along the data processing direction, the intermediate layer includes a variance perception block, a cross-attention mechanism and a variance perception block. Through the decoding layer, the output features of the intermediate layer and the corresponding level of the encoding layer are decoded by combining the time-encoded feature information. Along the data processing direction, the decoding layer includes 2 residual blocks and a cross-attention mechanism, and then the output features of the cross-attention mechanism are up-sampled to obtain the output features of the decoding layer. After the output features of the last-level decoding layer are convolved, they are input into the noise prediction head and the residual prediction head respectively to obtain the noise information and the residual degradation information.

[0097] The cross-attention mechanism is used to extract the relationship between the features, and the feature processing process is as follows:

[0098] ;

[0099] Where Attention represents the cross-attention mechanism, Q c , K c and V c represent the query vector, key vector and value vector corresponding to the input feature respectively, and d represents the dimension size of the key vector K c .

[0100] In the residual block: after group convolution and group normalization are performed on the input feature, the information is integrated with the time-encoded feature information; after SiLu activation is performed on the integrated feature, the output feature is obtained after group convolution.

[0101] In the variance perception block: after group convolution and group normalization are performed on the input feature, the information is integrated with the time-encoded feature information; after SiLu activation is performed on the integrated feature, the output feature is obtained after group convolution and group normalization. The second branch passes through variance calculation and variance encoding to integrate the obtained variance information with the first branch result, and the integration result is subjected to SiLu activation, point convolution and residual connection, etc., to obtain the output feature.

[0102] In the embodiment of the present application, the feature processing process of the residual block is as follows: Figure 11In the (a) shown in the formula, after the group convolution and group normalization of the input feature, the multiplied result is added with the corresponding element of the time-coded feature information, to complete the integration of the input feature and the time-coded information. In the embodiment of the present application, the time step information is embedded in the feature information in the form of cosine position coding. After the integrated feature is activated by SiLu, the output feature is obtained after the group convolution, group normalization and SiLu activation in turn.

[0103] The feature processing process in the variance-aware block provided in the embodiment of the present application is as shown in the (b) of the formula. Figure 11 In the (b) shown in the formula, after the group convolution and group normalization of the input feature, the integrated information is obtained with the time-coded feature information. After the integrated feature is activated by SiLu, the first branch is obtained, which is activated by group convolution and group normalization to obtain the output feature. The second branch is obtained after variance calculation and variance coding, and the integrated result is obtained with the first branch result. In the embodiment of the present application, the variance information is embedded in the feature information in the form of cosine position coding. The integrated result is activated by SiLu, and then the feature channel is adjusted by point convolution. Finally, the output feature is obtained by connecting the residual error with the input feature.

[0104] In the embodiment of the present application, the network composed of the multi-scale feature extraction module, the dynamic optimal transmission texture planning module, the gated fusion module and the adaptive double conditional diffusion module is trained by using the total loss function as follows:

[0105] L = λ1L diff + λ2L pixel + λ3L per ;

[0106] Wherein, L represents the total loss function, L diff represents the diffusion loss function, L pixel represents the pixel loss function, L per represents the perceptual loss function, λ1, λ2 and λ3 represent the loss weight

[0107] Wherein, the specific calculation method of the diffusion loss function L diff , the pixel loss function L pixel and the perceptual loss function L per is as follows:

[0108] ;

[0109] ;

[0110] ;

[0111] Wherein, E represents expectation, and respectively represent predicted residual information and noise information, is a super-resolution result image predicted by the network, and VGG represents a feature output mapping of the fourth layer in the pre-trained VGG19 network model. More specifically, the diffusion loss function L diff is a sum of residual information loss and noise information loss, used to enhance the accuracy of residual and noise prediction; the pixel loss function L pixel is a pixel loss between the predicted image and the ground truth, used to enhance the accuracy of the predicted image; the perceptual loss function L per is a perceptual loss between the predicted image and the ground truth, used to enhance the authenticity of the predicted image.

[0112] According to the specific value of the sampling time step t, the degradation image I t is obtained after t time steps. After that, the adaptive double-condition diffusion module can predict the predicted residual information and the predicted noise information from the texture information of the obtained reference image.

[0113] .

[0114] Step 5: input the predicted noise information and the residual information into the reverse sampling module for image reconstruction to obtain the super-resolution image.

[0115] In some embodiments, the reverse sampling module adopts the DDIM sampling strategy, diffuses the residual degradation information and the noise information through forward diffusion respectively, and uses the predicted noise information and the residual information to perform reverse denoising and residual correction through the reverse inference process, and iteratively reconstructs the super-resolution image.

[0116] In the embodiments of the present application, the forward diffusion process is as shown in (a) of FIG. Figure 12 , wherein the residual degradation information I res is obtained by the following formula:

[0117] I res =I lr -I0;

[0118] , wherein I lr represents the up-sampled low-resolution image, and I0 represents the corresponding high-resolution image.

[0119] The adaptive double-condition diffusion module takes the low-resolution image I lr and the degradation image I t degraded by residual noise as inputs, wherein the degradation image I tThe degraded image I is obtained through a forward diffusion process of the high-resolution image I0 over time steps t. Specifically, I1 is obtained from the high-resolution image I0 through a first diffusion, I2 is obtained through a second diffusion, and finally the degraded image I is obtained. t For simplified calculation, the following formula can be used to deduce the degraded image I from the high-resolution image I0 in one step. t :

[0120] ;

[0121] Among them, I t-1 I t-2 ... I t ε represents the degraded image information at different time steps during the diffusion process. t ε t-1 , ..., ε~N(0,1) represent random Gaussian noise at different time steps. and These represent the final residual diffusion intensity and noise diffusion intensity obtained through reasoning and calculation, respectively.

[0122] During model training, a time step t is randomly sampled to obtain the corresponding residual diffusion intensity. and noise diffusion intensity The degraded image I is obtained in one step according to the above formula. t Image I after noise degradation t Input into the adaptive biconditional diffusion module.

[0123] The noise information and residual degradation information predicted by the dual-conditional diffusion model are used to obtain the reconstructed super-resolution image through the reverse reconstruction process.

[0124] The data processing procedure of the backsampling module is shown in (b) of 12, based on the predicted target image. This leads to the following reverse reasoning formula:

[0125] ;

[0126] Where p represents the degraded image I at the known current time step. t Find the degraded image I at the previous time step. t-1 The conditional probability, where q represents the known degraded image I. t and predict target image and prediction residual information Find the degraded image I at the previous time step. t-1 The conditional probability is given by N, which indicates that the distribution follows a standard normal distribution. ,here , and All known information can be obtained from formulas. and The calculation shows that η represents a manually set coefficient used to control generation capacity. and Let represent the residual and noise intensity at time step t-1, respectively. After parameter renormalization, the final formula for the reverse inference process can be obtained, as follows:

[0127] .

[0128] The above formula can be used to analyze the degraded image I. t Obtain the degraded image I from the previous time step t-1 The final high-resolution image I0 can be obtained by iterating sequentially. To accelerate the inference process, this invention optimizes the sampling process using a non-Markov chain and demonstrates this through mathematical induction. This allows the generation process to perform reverse inference across multiple steps without needing to perform reverse inference step by step. The final accelerated sampling formula is as follows:

[0129] ;

[0130] Among them, I prev The degraded image represents the predicted time step. This indicates the residual strength at the prediction time step. This represents the noise intensity at the prediction time step.

[0131] In this embodiment, to accelerate the reverse reasoning process, we employ a ten-step sampling strategy. Specifically, in the forward process, we set 1000 diffusion processes, and in the reverse reasoning process, we uniformly select a step size of ten. That is, from The final high-resolution image is obtained by using the above formula.

[0132] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this invention disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this invention can be achieved, and this is not limited herein.

[0133] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A super-resolution method for remote sensing images based on a texture transfer-guided dual-diffusion model, characterized in that, include: The high-resolution reference image is blurred to obtain a blurred reference image; After unifying the resolution of the low-resolution image with that of the high-resolution reference image, the three images are input into the multi-scale feature extraction module to obtain multi-level features for each image. The features of the same level among the three sets of features obtained are input into the dynamic optimal transmission texture planning module. The dynamic optimal transmission texture planning module uses the optimal transmission theory to establish the correspondence between the low-resolution feature space and the fuzzy reference feature space. Based on this correspondence, the texture information that best matches the low-resolution content is aggregated from the high-definition reference features to obtain the planned texture features corresponding to each level. The gated fusion module adopts a top-down fusion path, using the gated fusion unit to fuse the planned texture features obtained at multiple levels, and outputs the texture guidance features corresponding to each level, which are fused with multi-scale context. Using the low-resolution image after noise degradation as the structural condition and the texture guidance features corresponding to each generated level as the texture condition, the predicted noise information and residual degradation information are learned through an adaptive biconditional diffusion module. In the adaptive biconditional diffusion module: information from the low-resolution image and the degraded image is extracted through consecutive multi-level coding layers; each coding layer contains multiple residual blocks and a cross-attention mechanism to perform feature encoding on the output features from the previous coding layer and the temporally encoded feature information. By using an intermediate layer containing multiple variance-aware blocks and a cross-attention mechanism, combined with temporal encoding and variance-conditional encoding, the output features of the last-level encoding layer are transitioned to the decoding layer, balancing and adjusting the feature representation; by using a decoding layer containing multiple residual blocks and a cross-attention mechanism, combined with temporally encoded feature information, the output features of the intermediate layer and the corresponding level of the encoding layer are decoded. The decoding layer and the guidance layer obtain information from the coding layer through skip connections; at the end of the decoding layer, noise information and residual information are output through noise prediction head and residual prediction head, respectively; The predicted noise information and residual information are input into the backsampling module for image reconstruction to obtain a super-resolution image.

2. The remote sensing image super-resolution method based on a texture transfer-guided dual diffusion model according to claim 1, characterized in that, The multi-scale feature extraction module uses the Swin Transformer model. The three images are input into the Swin Transformer model respectively, and the output features of different levels in each image are extracted to obtain the multi-level features of each image.

3. The remote sensing image super-resolution method based on a texture transfer-guided dual-diffusion model according to claim 2, characterized in that, The dynamic optimal transfer texture planning module processes the features of each level independently and outputs the planned texture features corresponding to each level. The processing steps for the three features at a certain level include: dimensionality reduction of the features corresponding to the low-resolution image and the blurred reference image; vector expansion of the two dimensionality-reduced features; optimization of the cost matrix between the two expanded feature vectors to obtain the transfer matrix; dimensionality reduction of the features corresponding to the high-resolution reference image; vector expansion of the dimensionality-reduced features; weighted aggregation of the expanded vectors with the transfer matrix to obtain the planned texture features for the corresponding level.

4. The remote sensing image super-resolution method based on a texture transfer-guided dual-diffusion model according to claim 3, characterized in that, The cost matrix is ​​the squared Euclidean distance between corresponding elements of the two expanded vectors. The Sinkhorn-Knopp iterative algorithm is used to optimize the cost matrix to obtain the transfer matrix.

5. The remote sensing image super-resolution method based on a texture transfer-guided dual diffusion model according to claim 1, characterized in that, The gated fusion module also includes a multi-scale aggregation unit. In the gated fusion module, the shallowest planning texture features are aggregated by the multi-scale aggregation unit to obtain texture guidance features of the corresponding scale. After downsampling the obtained texture guidance features, they are fused with the planned texture features of the second scale through a gated fusion unit. The fused features are then input into the scale aggregation unit for feature aggregation to obtain the texture guidance features of the corresponding scale. The above operation is repeated until the planned texture features of the smallest scale are fused to obtain the texture guidance features of the corresponding scale.

6. The remote sensing image super-resolution method based on a texture transfer-guided dual-diffusion model according to claim 5, characterized in that, The feature processing of the gating fusion unit includes: performing pooling operations on the downsampled texture guidance features from the previous scale and the original planned texture features from the current scale respectively; performing channel concatenation and multilayer perceptron processing on the two pooled features to obtain adaptive gating weights; and performing weighted summation of the downsampled texture guidance features from the previous scale and the original planned texture features from the current scale according to the adaptive gating weights to obtain the output features.

7. The remote sensing image super-resolution method based on a texture transfer-guided dual-diffusion model according to claim 5, characterized in that, The feature processing of the multi-scale aggregation unit includes: performing multi-scale kernel fusion on the input features, and then adding the processed features to the input features using residual connections; performing channel attention and spatial attention operations on the features processed by residual connections, and then adding the processed features to the output features of the previous residual connection using residual connections to obtain the output features; in the multi-scale kernel fusion operation, the input features are separated into channels, and the separated features are subjected to depth convolution operations of different scales, and then all the depth convolution features are integrated; the integrated features are subjected to depth convolution operations of different scales again, and then all the depth convolution features are integrated and point convolution operations are performed to obtain the output features.

8. The remote sensing image super-resolution method based on a texture transfer-guided dual diffusion model according to claim 1, characterized in that, In the residual block: after group convolution and group normalization of the input features, information is integrated with the time-encoded features; after SiLu activation of the integrated features, it is then subjected to group convolution to obtain the output features; In the variance-aware block: after group convolution and group normalization of the input features, information is integrated with the time-encoded features; after SiLu activation of the integrated features, branches are derived. The first branch obtains the output features through group convolution and group normalization, and the second branch obtains the variance information through variance calculation and variance encoding, and integrates the obtained variance information with the result of the first branch. The integrated result is then processed by SiLu activation, point convolution, and residual connection to obtain the output features.

9. The remote sensing image super-resolution method based on a texture transfer-guided dual diffusion model according to claim 1, characterized in that, The reverse sampling module adopts the DDIM sampling strategy, which diffuses residual degradation information and noise information through forward diffusion, and uses the predicted noise information and residual information to perform reverse denoising and residual correction through the reverse inference process, and iteratively reconstructs the super-resolution image.

Images