A method, device and terminal for continuous super-resolution enlargement of remote sensing images

Abstract

The present invention discloses a method, device and terminal for continuous super-resolution enlargement of remote sensing images, the method comprising: obtaining a target initial image, encoding and interpolating the target initial image to obtain a target initial hidden representation; iteratively processing the target initial hidden representation based on a target iteration module to obtain a target noise image; obtaining a target diffusion model, denoising the target noise image based on the target diffusion model to obtain a target high-definition image. The present invention solves the problem of high-quality image generation by combining the high-quality image generation capabilities of the iteration module and the diffusion model, and also successfully realizes the task of enlarging super-resolution images of any scale.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal large models, and in particular to a method, a device and a terminal for continuous super-resolution enlargement of remote sensing images. Background Art

[0002] In the prior art, convolutional neural networks (CNNs) have made significant progress in remote sensing super-resolution (SR) tasks, usually using CNNs to enlarge the resolution of images. However, although these methods can achieve impressive results, they still face challenges in recovering high-frequency details and textures in images. Specifically, images enlarged using CNNs tend to appear too smooth and lack realism in visual effects. Moreover, the mainstream remote sensing super-resolution task is to train models with different magnifications separately, which means that a single model can usually only enlarge to a fixed integer multiple, and the magnification is generally small (such as x2, x3, x4).

[0003] For remote sensing images (RSI), continuous super-resolution to multiple and large resolutions is crucial. Therefore, how to achieve efficient, stable and high-quality remote sensing super-resolution reconstruction remains an urgent problem to be solved.

[0004] Therefore, the prior art still needs to be improved and enhanced. Summary of the invention

[0005] In view of the above-mentioned defects of the prior art, the present invention provides a method, device and terminal for continuous super-resolution enlargement of remote sensing images, aiming to solve the problem that the method of enlarging image resolution in the prior art tends to appear too smooth and lack of realism in visual effect.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is as follows:

[0007] A first aspect of the present invention provides a method, device and terminal for continuous super-resolution enlargement of remote sensing images, the method comprising:

[0008] Acquire a target initial image, and perform encoding and interpolation processing on the target initial image to obtain a target initial hidden representation;

[0009] Iteratively process the target initial hidden representation based on the target iteration module to obtain a target noise image;

[0010] A target diffusion model is obtained, and denoising is performed on the target noise image based on the target diffusion model to obtain a target high-definition image.

[0011] In one implementation, encoding and interpolating the target initial image includes:

[0012] Extracting features based on each pixel in the target initial image to obtain a plurality of coded pixels;

[0013] Encoding the plurality of encoded pixel points to a target dimension to obtain initial image features, wherein the target dimension is higher than the dimension of the target initial image;

[0014] The initial image features are interpolated to obtain an initial hidden representation.

[0015] In one implementation, the interpolating the initial image features to obtain an initial hidden representation includes:

[0016] Obtaining a coding feature of a target neighbor point of an interpolation pixel point, and obtaining an interpolation feature of the interpolation pixel point based on the coding feature of the target neighbor point, wherein the interpolation pixel point is a pixel point in the initial image feature other than the coding pixel point, and the target neighbor point is the coding pixel point in the initial image feature adjacent to the interpolation pixel point, wherein there are multiple target neighbor points;

[0017] The coded pixel points and the interpolated pixel points are linearly transformed to obtain the initial hidden representation.

[0018] In one implementation, the target-based iteration module iteratively processes the target initial hidden representation, including:

[0019] A target kernel matrix is ​​obtained, and the target initial hidden representation is iteratively updated based on the target kernel matrix until a target number of times has passed, and then updating is stopped to obtain the target noise image.

[0020] In one implementation, obtaining the target diffusion model includes:

[0021] Acquire a target data set, wherein the target data set includes a plurality of high-definition images;

[0022] The initial diffusion model is trained based on the target data set to obtain the target diffusion model.

[0023] In one implementation, the training of the initial diffusion model based on the target data set to obtain the target diffusion model includes:

[0024] Performing a target round of noise addition processing on the target data set, and recording the image changes of each round, wherein the noise added in each round obeys a Gaussian distribution;

[0025] A target neural network is constructed, and the target neural network is trained in reverse order based on the image changes in each round to obtain a target prediction network.

[0026] In one implementation, the denoising process on the target noise image based on the target diffusion model includes:

[0027] The target noisy image is denoised based on the target prediction network to obtain the target high-definition image.

[0028] A second aspect of the present invention provides a continuous super-resolution enlargement device for remote sensing images, comprising:

[0029] An interpolation module is used to obtain a target initial image, encode and interpolate the target initial image, and obtain a target initial hidden representation;

[0030] An iteration module, used for iteratively processing the target initial hidden representation based on the target iteration module to obtain a target noise image;

[0031] The denoising module is used to obtain a target diffusion model, and perform denoising processing on the target noise image based on the target diffusion model to obtain a target high-definition image.

[0032] According to a third aspect of the present invention, a terminal is provided, comprising a processor and a computer-readable storage medium communicatively connected to the processor, wherein the computer-readable storage medium is suitable for storing a plurality of instructions, and the processor is suitable for calling the instructions in the computer-readable storage medium to execute the steps of the method for continuous super-resolution enlargement of remote sensing images as described in any one of the above items.

[0033] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the continuous super-resolution expansion method of remote sensing images as described in any one of the above items.

[0034] Compared with the prior art, the present invention provides a method, device and terminal for continuous super-resolution enlargement of remote sensing images. The method for continuous super-resolution enlargement of remote sensing images obtains a target initial image, encodes and interpolates the target initial image to obtain a target initial hidden representation, and then iterates the target initial hidden representation based on a target iteration module to obtain a target noise image. Finally, a target diffusion model is obtained, and the target noise image is denoised based on the target diffusion model to obtain a target high-definition image. The method for continuous super-resolution enlargement of remote sensing images proposed by the present invention solves the problem of high-quality image generation by encoding and interpolating low-resolution images, and then combining the high-quality image generation capabilities of the iteration module and the diffusion model, and also successfully realizes the task of enlarging super-resolution images of any scale. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A flow chart of an embodiment of the method for continuous super-resolution enlargement of remote sensing images provided by the present invention;

[0036] Figure 2 A network structure diagram of an embodiment of the method for continuous super-resolution enlargement of remote sensing images provided by the present invention;

[0037] Figure 3 A denoising flow chart of an embodiment of the method for continuous super-resolution enlargement of remote sensing images provided by the present invention;

[0038] Figure 4 A structural principle diagram of an embodiment of the continuous super-resolution enlargement device for remote sensing images provided by the present invention;

[0039] Figure 5 A schematic diagram of the principles of an embodiment of a terminal provided by the present invention. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0041] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0042] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as herein.

[0043] The continuous super-resolution enlargement method of remote sensing images provided by the present invention can be applied to a terminal with computing capability, and the terminal can execute the continuous super-resolution enlargement method of remote sensing images provided by the present invention to improve the resolution of remote sensing images.

[0044] Embodiment 1

[0045] The present embodiment is a method for continuous super-resolution expansion of remote sensing images based on neural operators. In this embodiment, a new model is proposed, a continuous super-resolution diffusion model for remote sensing images based on neural operators, which is a fusion of super-resolution neural operators and diffusion models. The super-resolution neural operator is a deep neural operator framework that can generate high-resolution (HR) remote sensing images of any scale from low-resolution (LR) remote sensing images. Compared with traditional generative adversarial networks, the diffusion model has a more stable training process and is less prone to mode collapse problems. And due to the high-quality image generation capability, strong interpretability, strong generalization ability and other advantages of the diffusion model, it has attracted much attention and popularity.

[0046] In recent years, convolutional neural networks have made significant progress in remote sensing super-resolution tasks. Although these methods can achieve impressive results, they still face challenges in recovering high-frequency details and textures in images, resulting in the generated images appearing too smooth and lacking in realism in visual effects. In addition, existing methods are not robust enough to image noise and distortion. In contrast, generative adversarial networks (GANs) introduce adversarial training mechanisms to enable generative models to capture and generate more realistic high-frequency details and textures. However, the training process of GANs is usually unstable and prone to mode collapse and artifact problems. In addition, mainstream remote sensing super-resolution tasks usually train models with different magnifications separately, which means that a single model can only be magnified by a fixed integer multiple, and the magnification is generally small (such as x2, x3, x4). For remote sensing imagery (RSI), continuous super-resolution to multiple and large resolutions is crucial. Therefore, how to combine the advantages of CNN and GAN while overcoming their respective limitations to achieve efficient, stable and high-quality remote sensing super-resolution reconstruction is an important topic of current research.

[0047] In this embodiment, the low-resolution remote sensing image is first input into the deep neural operator framework, and then the output feature fusion time step is used as a conditional guide to guide the noise image to be denoised step by step, and finally a clear high-resolution remote sensing image is output. Compared with the existing technology, the fusion of the two not only solves the problem of high-quality image generation, but also realizes the super-resolution task of any scale.

[0048] Specifically, Figure 1As shown, in one embodiment of the method for continuous super-resolution enlargement of remote sensing images provided by the present invention, the continuous super-resolution enlargement of remote sensing images comprises the steps of:

[0049] S100: Acquire a target initial image, and perform encoding and interpolation processing on the target initial image to obtain a target initial hidden representation.

[0050] The encoding and interpolation processing of the target initial image comprises:

[0051] S120, extracting features based on each pixel in the target initial image to obtain a plurality of coded pixels;

[0052] S130, encoding the plurality of encoded pixel points to a target dimension to obtain initial image features, wherein the target dimension is higher than a dimension of the target initial image;

[0053] S140: Perform interpolation processing on the initial image features to obtain an initial hidden representation.

[0054] The target drawing image is a low-resolution image LR. When the target initial image is acquired, the target initial image is firstly encoded.

[0055] Reference Figure 2 First, the target initial image (low-resolution image LR) is input into the encoder E (EDSR-baseline or RDN) for feature extraction. Specifically, the feature extraction of the target initial image includes color feature extraction, texture feature extraction, shape feature extraction, deep learning feature extraction, and local / global feature extraction of the target initial image.

[0056] Specifically, color feature is one of the most basic features in an image, which can be obtained by calculating the color histogram of the image, for example, converting the image from RGB space to HSV or LAB space, and then calculating the histogram of each color channel.

[0057] Texture features describe the spatial distribution pattern of pixel values ​​in an image. LBP (local binary pattern) is a commonly used texture feature extraction method. It generates a binary number by comparing the grayscale values ​​of a pixel with its surrounding pixels. This binary number is the LBP feature of the pixel. Other texture feature extraction methods include gray-level co-occurrence matrix (GLCM), wavelet transform, etc.

[0058] Shape features are usually used to describe the outline or boundary of an object in an image. Edge-based methods (such as Sobel and Canny operators) can detect edges in an image to obtain shape features. Corner detection algorithms (such as Harris corner detection and FAST corner detection) can also be used to extract shape features because corners often correspond to the boundary or outline of an object.

[0059] With the development of deep learning technology, convolutional neural networks (CNN) have become the mainstream method for image feature extraction. CNN automatically learns hierarchical feature representations in images through multi-layer convolution and pooling operations. These features contain not only low-level pixel-level information, but also high-level semantic information. Recurrent neural networks (RNN) can also be used to process pixel-level features, especially when the spatial relationship between pixels needs to be considered.

[0060] In addition to extracting features directly from the pixel level, features can also be extracted from local areas of the image or the entire image. Local feature extraction methods (such as SIFT and SURF) focus on extracting features of specific areas in the image, which are robust to local changes and deformations of the image. Global feature extraction methods (such as HOG) extract features from the entire image, which describe the overall content of the image.

[0061] In practical applications, it is necessary to select appropriate feature extraction methods according to specific scenarios and requirements. For example, for face recognition tasks, it may be necessary to consider color, texture, and shape features at the same time; while for object detection tasks, it may focus more on shape and texture features. In addition, with the continuous development of deep learning technology, feature extraction methods based on deep learning are gradually becoming mainstream because they can automatically learn complex feature representations in images and have achieved significant performance improvements in various computer vision tasks.

[0062] In this embodiment, feature extraction is performed based on each pixel in the target initial image, including the position feature, color feature and grayscale feature of each pixel, to obtain multiple coded pixel points. Then, the extracted features are projected to a subspace with a higher dimension. In this embodiment, multiple coded pixel points are encoded to the target dimension to obtain the initial image features.

[0063] After obtaining the initial image features, interpolation processing is performed on the initial image features to obtain the initial hidden representation.

[0064] Specifically, the interpolation processing is performed on the initial image features to obtain an initial hidden representation, including:

[0065] S141. Obtain the coding features of the target neighbor points of the interpolated pixel points, and obtain the interpolation features of the interpolated pixel points based on the coding features of the target neighbor points, wherein the interpolation pixel points are the pixel points in the initial image features except the coding pixel points, and the target neighbor points are the coding pixel points in the initial image features that are adjacent to the interpolation pixel points, wherein there are multiple target neighbor points.

[0066] Reference Figure 2 , the initial image features obtained by the encoder are interpolated by the interpolation module, that is, the Interpolation module, and the low-resolution image LR is encoded into a high-dimensional space to obtain a high-resolution image HR. Specifically, for each high-resolution position, that is, the position x of the interpolated pixel point in the initial image feature, the four target neighbor points of the target interpolated pixel point on the low-resolution grid are found, and the encoded features of the four target neighbor points are obtained. , , , , based on the encoding features of the target neighbor point, the interpolation features of the interpolation pixel point are calculated. Specifically, the interpolation features of the interpolation pixel point are calculated by a first formula.

[0067] The first formula is:

[0068] ;

[0069] in, is the interpolation feature of the difference pixel point; is the interpolation coefficient of the difference pixel point, used to represent the distance weight between the difference pixel point and each neighbor point; Indicates the relative position of the neighboring points of the interpolated pixel x.

[0070] S142: linearly transform the coded pixel points and the interpolated pixel points to obtain the initial hidden representation.

[0071] Specifically, using the linear transformation The encoding feature of the encoding pixel point obtained by the interpolation operation and the interpolation feature of the interpolation pixel point are linearly transformed, and calculated based on the second formula to obtain the initial hidden representation .

[0072] The second formula is:

[0073] ;

[0074] in, , represents the local area in the initial image feature (high resolution image HR), and is the scaling factor; is the feature encoder, is the encoding feature of the target neighbor point, Indicates the relative position of the interpolated pixel point and the target neighbor point.

[0075] S200, iteratively processing the target initial hidden representation based on a target iteration module to obtain a target noise image;

[0076] The target-based iteration module iteratively processes the target initial hidden representation, including:

[0077] A target kernel matrix is ​​obtained, and the target initial hidden representation is iteratively updated based on the target kernel matrix until a target number of times has passed, and then updating is stopped to obtain the target noise image.

[0078] Specifically, after obtaining the initial hidden representation Afterwards, an integral iteration operation will be performed based on the target iteration module. In this embodiment, the target iteration module is an Integral module.

[0079] The Integral module is a mathematical tool for integrating input signals. Usually referred to as the Integrator module in Simulink, it is a tool for integrating input signals. The Integral module outputs a continuous signal by accumulating the value of the input signal. It is widely used in dynamic system modeling and simulation, such as control systems, signal processing and other fields.

[0080] The noise image corresponding to the initial hidden representation of the target The target noisy image is obtained by encoding it into features through convolution operations and then denoising it using the features output by the neural operator as conditional guidance (feature splicing).

[0081] Specifically, in each iteration based on the Integral module, the model updates each feature representation in the initial hidden representation of the target through an attention-based kernel integration operation.

[0082] The iteration method based on the Integral module is similar to the attention mechanism in Transformer, but unlike the standard attention, it focuses more on the linear combination of the feature space, thereby achieving a more detailed capture of the image structure. In other words, in each iteration, the new feature representation will be related to a kernel matrix Multiply.

[0083] Specifically, the core idea of ​​kernel matrix construction is as follows: First, the input features are mapped into three components: query (Q), key (K), and value (V). These three components are transformed through linear layers or 1×1 convolutions to adapt to different dimensions and model requirements. Then, by calculating the relationship between query and key To generate an attention score matrix (the original similarity measure matrix), this matrix represents the mutual influence or correlation between different features.

[0084] In order to make the training more stable, the similarity measure matrix is ​​usually scaled, such as dividing the similarity measure matrix by ,in is the dimension of the key vector, and then the Softmax function is applied for normalization so that all attention weights sum to 1, which helps the model focus on the most relevant features. After obtaining the normalized attention weights, these weights are multiplied with the value (V) matrix to generate the weighted feature representation, which is the kernel matrix.

[0085] Specifically, the target initial hidden representation is updated based on the target feature update formula, and the target feature update formula is:

[0086] ;

[0087] in, represents an operator that combines the result of the kernel matrix with the current feature representation, For the The feature representation after the iteration update is For the The feature representation is obtained by iterative updating. This process will go through multiple iterations, and finally the target noise image is obtained. In this embodiment, the number of iterations is 4 times, that is, , after iterations, the image can be well restored without generating too much data redundancy.

[0088] S300 , obtaining a target diffusion model, and performing denoising processing on the target noise image based on the target diffusion model to obtain a target high-definition image.

[0089] Specifically, after the target noise image is obtained, further denoising processing is performed on the target noise image based on the target diffusion model.

[0090] The core of the diffusion model is to generate new data for step-by-step denoising by gradually adding noise to the data and then learning the denoising process. The diffusion model mainly includes two processes: the forward process and the reverse process. The forward process is also called the diffusion process, which is a process of adding noise to the image, and the reverse process is the denoising process. This embodiment trains the diffusion model to obtain the target diffusion model, and then denoises the target noise image based on the reverse process of the target diffusion model. Figure 3 , Figure 3 This is the process of denoising.

[0091] Specifically, obtaining the target diffusion model includes:

[0092] Acquire a target data set, wherein the target data set includes a plurality of high-definition images;

[0093] The initial diffusion model is trained based on the target data set to obtain the target diffusion model.

[0094] The training of the initial diffusion model based on the target data set to obtain the target diffusion model includes:

[0095] Performing a target round of noise addition processing on the target data set, and recording the image changes of each round, wherein the noise added in each round obeys a Gaussian distribution;

[0096] A target neural network is constructed, and the target neural network is trained in reverse order based on the image changes in each round to obtain a target prediction network.

[0097] The denoising process of the target noise image based on the target diffusion model includes:

[0098] The target noisy image is denoised based on the target prediction network to obtain the target high-definition image.

[0099] Specifically, in the training of the diffusion model, it is first necessary to obtain the target data set, which includes multiple high-definition pictures. The more high-definition pictures there are in the target data set, the more accurate the model will be. However, too many pictures will lead to slow model training and increased data redundancy.

[0100] At the same time, it is also necessary to select a suitable time step T to limit the number of subsequent noise addition and removal processes. After many experiments, in this embodiment, the time step T is set to 1000 times.

[0101] First, the target data set is subjected to a target round of noise addition processing. Specifically, the high-definition image in the target data set is , in the forward process, the image distribution at each moment All of them are determined by the image distribution at the previous moment Add noise to get, where each added noise obeys Gaussian distribution, except for the original high-definition image Except that the image distribution at each moment is recorded at the same time for training the reverse process, and the reverse process has been trained as a target prediction network.

[0102] Specifically, if the distribution of the previous moment is known , then the distribution at the next moment can be obtained by the third formula The third formula is:

[0103] ;

[0104] in, For noise and , ;

[0105] in, is obtained by a linear function, in this embodiment, the linspace() function is used), and The range is from 0.0001 to 0.02. Getting bigger and bigger, It is getting smaller and smaller, and Getting bigger and bigger.

[0106] That is to say, the diffusion process of noise is from small to large, the noise added is small at the beginning, and then gradually increases. And finally it can be deduced and The corresponding target relationship is:

[0107] ;

[0108] In this way, the image distribution at time t, that is, after the tth noise addition, can be calculated in one step. .

[0109] The reverse process is to restore the image by predicting the data distribution at the next moment and randomly sampling from the distribution.

[0110] Since the distribution cannot be predicted directly, the method of predicting noise is used to indirectly achieve the purpose of predicting the distribution.

[0111] Specifically, in the denoising process of training the diffusion model, from each moment t to the next moment In both cases, noise is predicted based on a UNet network.

[0112] That is, the prediction network model is based on the known distribution at time t beg Distribution of time The prediction formula is:

[0113] ;

[0114] Through a series of derivations, the mean and variance of the distribution can be calculated, where the variance is ,because as well as It can be obtained by calculation, so the variance is a constant. The target mean is calculated based on the mean formula, which is:

[0115] ;

[0116] Finally, it can be transformed into the target mean formula:

[0117] ;

[0118] The noise The target prediction network can be used to predict and generate. The mean and variance can be used to predict the current moment. Calculate the next moment The data distribution of , and finally by sampling the predicted distribution, we can calculate Image of the moment.

[0119] The denoising process of the target noise image based on the target diffusion model includes:

[0120] The target noisy image is subjected to noise processing based on the target prediction network to obtain the target high-definition image.

[0121] Specifically, the target noise image is transformed into To express, refer to Figure 3 ,Will By introducing the target prediction network, the target high-definition image can be predicted based on the target mean formula, where: Corresponding Figure 2 In It can be seen that a new model is proposed in this embodiment, which is a fusion of the super-resolution neural operator and the diffusion model. The super-resolution neural operator is a deep neural operator framework that can generate high-resolution (HR) remote sensing images of any scale from low-resolution (LR) remote sensing images. Compared with the traditional generative adversarial network, the diffusion model has a more stable training process and is not prone to mode collapse. And due to the high-quality image generation capability, strong interpretability, strong generalization ability and other advantages of the diffusion model, it has attracted much attention and popularity.

[0122] First, the low-resolution remote sensing image is input into the deep neural operator framework, and then the output feature fusion time step is used as a conditional guide to guide the noise image to be denoised step by step, and finally a clear high-resolution remote sensing image is output. Compared with the existing technology, the fusion of the two not only solves the problem of high-quality image generation, but also realizes the super-resolution task of any scale.

[0123] In this way, the deep neural operator is combined with the diffusion model, integrating the advantages of the two models. The conditional features are combined with the multi-scale feature splicing of the noise image features by fusing the time step T. It can generate high-resolution remote sensing images of any scale from low-resolution remote sensing images. Furthermore, the characteristics of the diffusion model can also generate images with high-frequency details and textures.

[0124] In summary, this embodiment provides a method for continuous super-resolution expansion of remote sensing images, by acquiring a target initial image, encoding and interpolating the target initial image to obtain a target initial hidden representation, then iteratively processing the target initial hidden representation based on a target iteration module to obtain a target noise image, and finally, acquiring a target diffusion model, denoising the target noise image based on the target diffusion model to obtain a target high-definition image. The method for continuous super-resolution expansion of remote sensing images proposed in this embodiment solves the problem of high-quality image generation by encoding and interpolating low-resolution images, and then combining the high-quality image generation capabilities of the iteration module and the diffusion model, and successfully realizes the task of expanding super-resolution images of any scale.

[0125] It should be understood that, although the various steps in the flowcharts given in the accompanying drawings of the present specification are displayed in sequence according to the indications of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless there is a clear description in this article, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0126] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0127] Embodiment 2

[0128] Based on the above embodiments, the present invention also provides a continuous super-resolution enlargement device for remote sensing images. Figure 4 As shown, the continuous super-resolution enlargement device of the remote sensing image comprises:

[0129] An interpolation module, used to obtain a target initial image, encode and interpolate the target initial image, and obtain a target initial hidden representation, as described in the first embodiment;

[0130] An iteration module, used for iteratively processing the target initial hidden representation based on the target iteration module to obtain a target noise image, as specifically described in the first embodiment;

[0131] The denoising module is used to obtain a target diffusion model, and perform denoising processing on the target noise image based on the target diffusion model to obtain a target high-definition image, as specifically described in the first embodiment.

[0132] Embodiment 3

[0133] Based on the above embodiments, the present invention also provides a terminal, such as Figure 5 As shown, the terminal includes a processor 10 and a memory 20. Figure 5Only some components of the terminal are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0134] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal. Furthermore, the memory 20 may also include both an internal storage unit and an external storage device of the terminal. The memory 20 is used to store application software and various types of data installed on the terminal. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, a continuous super-resolution enlargement program 30 for remote sensing images is stored on the memory 20, and the continuous super-resolution enlargement program 30 for remote sensing images can be executed by the processor 10, thereby realizing the continuous super-resolution enlargement method for remote sensing images in the present application.

[0135] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor or other chip, used to run the program code or process data stored in the memory 20, such as executing the continuous super-resolution enlargement method of the remote sensing image.

[0136] In one embodiment, when the processor 10 executes the continuous super-resolution upscaling program 30 of the remote sensing image in the memory 20, the following steps are implemented:

[0137] Acquire a target initial image, and perform encoding and interpolation processing on the target initial image to obtain a target initial hidden representation;

[0138] Iteratively process the target initial hidden representation based on the target iteration module to obtain a target noise image;

[0139] A target diffusion model is obtained, and denoising is performed on the target noise image based on the target diffusion model to obtain a target high-definition image.

[0140] The encoding and interpolation processing of the target initial image includes:

[0141] Extracting features based on each pixel in the target initial image to obtain a plurality of coded pixels;

[0142] Encoding the plurality of encoded pixel points to a target dimension to obtain initial image features, wherein the target dimension is higher than the dimension of the target initial image;

[0143] The initial image features are interpolated to obtain an initial hidden representation.

[0144] The interpolation process of the initial image features to obtain the initial hidden representation includes:

[0145] Obtaining a coding feature of a target neighbor point of an interpolation pixel point, and obtaining an interpolation feature of the interpolation pixel point based on the coding feature of the target neighbor point, wherein the interpolation pixel point is a pixel point in the initial image feature other than the coding pixel point, and the target neighbor point is the coding pixel point in the initial image feature adjacent to the interpolation pixel point, wherein there are multiple target neighbor points;

[0146] The coded pixel points and the interpolated pixel points are linearly transformed to obtain the initial hidden representation.

[0147] The iterative processing of the target initial hidden representation based on the target iteration module includes:

[0148] A target kernel matrix is ​​obtained, and the target initial hidden representation is iteratively updated based on the target kernel matrix until a target number of times has passed, and then updating is stopped to obtain the target noise image.

[0149] Wherein, obtaining the target diffusion model includes:

[0150] Acquire a target data set, wherein the target data set includes a plurality of high-definition images;

[0151] The initial diffusion model is trained based on the target data set to obtain the target diffusion model.

[0152] The step of training the initial diffusion model based on the target data set to obtain the target diffusion model includes:

[0153] Performing a target round of noise addition processing on the target data set, and recording the image changes of each round, wherein the noise added in each round obeys a Gaussian distribution;

[0154] A target neural network is constructed, and the target neural network is trained in reverse order based on the image changes in each round to obtain a target prediction network.

[0155] The denoising process of the target noise image based on the target diffusion model includes:

[0156] The target noisy image is denoised based on the target prediction network to obtain the target high-definition image.

[0157] Embodiment 4

[0158] The present invention also provides a computer-readable storage medium, in which one or more programs are stored. The one or more programs can be executed by one or more processors to implement the steps of the continuous super-resolution expansion method of remote sensing images as described above.

[0159] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for continuous super-resolution enlargement of remote sensing images, characterized in that: The continuous super-resolution enlargement method of the remote sensing image comprises: Acquire a target initial image, and perform encoding and interpolation processing on the target initial image to obtain a target initial hidden representation; Iteratively process the target initial hidden representation based on the target iteration module to obtain a target noise image; Acquire a target diffusion model, and perform denoising on the target noise image based on the target diffusion model to obtain a target high-definition image; The target-based iteration module iteratively processes the target initial hidden representation, including: Obtain a target kernel matrix, iteratively update the target initial hidden representation based on the target kernel matrix until a target number of times has passed, then stop updating to obtain the target noise image; The step of obtaining a target kernel matrix comprises: Map the input features into three components: query, key, and value; Get the attention score matrix based on the query component and the key component; Scaling and normalizing the attention score matrix to obtain a target attention weight; The target kernel matrix is ​​obtained based on the target attention weight and value components.

2. The method for continuous super-resolution enlargement of remote sensing images according to claim 1, characterized in that: The encoding and interpolation processing of the target initial image comprises: Extracting features based on each pixel in the target initial image to obtain a plurality of coded pixels; Encoding the plurality of encoded pixel points to a target dimension to obtain initial image features, wherein the target dimension is higher than the dimension of the target initial image; The initial image features are interpolated to obtain an initial hidden representation.

3. The method for continuous super-resolution enlargement of remote sensing images according to claim 2, characterized in that: The interpolation process is performed on the initial image features to obtain an initial hidden representation, including: Obtaining a coding feature of a target neighbor point of an interpolation pixel point, and obtaining an interpolation feature of the interpolation pixel point based on the coding feature of the target neighbor point, wherein the interpolation pixel point is a pixel point in the initial image feature other than the coding pixel point, and the target neighbor point is the coding pixel point in the initial image feature adjacent to the interpolation pixel point, wherein there are multiple target neighbor points; The coded pixel points and the interpolated pixel points are linearly transformed to obtain the initial hidden representation.

4. The method for continuous super-resolution enlargement of remote sensing images according to claim 1, characterized in that: The obtaining of the target diffusion model comprises: Acquire a target data set, wherein the target data set includes a plurality of high-definition images; The initial diffusion model is trained based on the target data set to obtain the target diffusion model.

5. The method for continuous super-resolution enlargement of remote sensing images according to claim 4, characterized in that: The training of the initial diffusion model based on the target data set to obtain the target diffusion model includes: Performing a target round of noise addition processing on the target data set, and recording the image changes of each round, wherein the noise added in each round obeys a Gaussian distribution; A target neural network is constructed, and the target neural network is trained in reverse order based on the image changes in each round to obtain a target prediction network.

6. The method for continuous super-resolution enlargement of remote sensing images according to claim 5, characterized in that: The denoising process of the target noise image based on the target diffusion model includes: The target noisy image is denoised based on the target prediction network to obtain the target high-definition image.

7. A continuous super-resolution enlargement device for remote sensing images, characterized in that: include: An interpolation module is used to obtain a target initial image, encode and interpolate the target initial image, and obtain a target initial hidden representation; An iteration module, used for iteratively processing the target initial hidden representation based on the target iteration module to obtain a target noise image; A denoising module is used to obtain a target diffusion model, and to perform denoising processing on the target noise image based on the target diffusion model to obtain a target high-definition image; The target-based iteration module iteratively processes the target initial hidden representation, including: Obtain a target kernel matrix, iteratively update the target initial hidden representation based on the target kernel matrix until a target number of times has passed, then stop updating to obtain the target noise image; The step of obtaining a target kernel matrix comprises: Map the input features into three components: query, key, and value; Get the attention score matrix based on the query component and the key component; Scaling and normalizing the attention score matrix to obtain a target attention weight; The target kernel matrix is ​​obtained based on the target attention weight and value components.

8. A terminal, characterized in that: The terminal includes: a processor, a computer-readable storage medium communicatively connected to the processor, the computer-readable storage medium being suitable for storing a plurality of instructions, and the processor being suitable for calling the instructions in the computer-readable storage medium to execute the steps of the method for continuous super-resolution enlargement of remote sensing images as described in any one of claims 1 to 6 above.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the continuous super-resolution expansion method of remote sensing images as described in any one of claims 1-6.

Images