Image super-resolution reconstruction method and related device

Abstract

The invention discloses an image super-resolution reconstruction method and a related device, and relates to the technical field of image processing and artificial intelligence, and the method comprises the steps: obtaining a first image and a reference image; performing feature extraction on the reference image to obtain reference image features; performing feature extraction on the first image to obtain a first image feature; performing fusion processing on the reference image feature and the first image feature; generating a second image based on the fusion feature in the image generation model; the resolution of the second image is higher than that of the first image; the image generation model is generated by performing model training on the basis of the difference between a high-resolution image generated for the low-resolution image sample and a real high-resolution image and the difference between the generated high-resolution image and a corresponding reference image sample. According to the method and the device, the image super-resolution reconstruction process can be accurately guided according to the reference image introduced in the image super-resolution reconstruction, and high-quality super-resolution reconstruction is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing and artificial intelligence, and in particular to an image super-resolution reconstruction method and related device. BACKGROUND

[0002] With the rapid development of remote sensing technology, remote sensing images are increasingly widely used in geographic information systems, disaster monitoring, agricultural assessment, financial risk assessment, and other fields. High-resolution remote sensing images can provide rich ground feature detail information, which is crucial for improving the accuracy of analysis and decision-making. However, due to factors such as satellite payload, shooting conditions, and data transmission costs, the actual acquired remote sensing images often have low resolution, making it difficult to meet the high-precision application requirements. Super-resolution reconstruction technology, as an important means to improve the resolution of remote sensing images, has become a current research hotspot.

[0003] Known technologies rely on low-resolution images to directly infer or abstract priori to achieve super-resolution reconstruction of low-resolution images, but this solution cannot achieve high-quality super-resolution reconstruction and is difficult to meet high-precision application requirements. SUMMARY

[0004] Therefore, the present application provides an image super-resolution reconstruction method and related device, which introduces a reference image in image super-resolution reconstruction to accurately guide the image super-resolution reconstruction process and achieve high-quality super-resolution reconstruction.

[0005] The specific technical solutions are as follows:

[0006] An image super-resolution reconstruction method comprises:

[0007] obtaining a first image and a corresponding reference image;

[0008] performing feature extraction on the reference image to obtain reference image features;

[0009] performing feature extraction on the first image to obtain first image features;

[0010] performing fusion processing on the reference image features and the first image features to obtain fused features;

[0011] generating a second image corresponding to the first image based on the fused features using an image generation model; the resolution of the second image is higher than that of the first image;

[0012] The image generation model is generated based on model training of the difference between the generated high-resolution image and the real high-resolution image for low-resolution image samples, and the difference between the generated high-resolution image and the corresponding reference image sample.

[0013] Optionally, the reference image and the first image satisfy a consistency condition representing consistency of image content, and the resolution of the reference image is higher than that of the first image.

[0014] Optionally, the construction process of the image generation model comprises:

[0015] obtaining a low-resolution image sample and a corresponding reference image sample;

[0016] performing feature extraction on the reference image sample to obtain a reference image sample feature;

[0017] performing feature extraction on the low-resolution image sample to obtain a low-resolution image sample feature;

[0018] fusing the reference image sample feature and the low-resolution image sample feature to obtain a fused sample feature;

[0019] generating, by a generator of a generative adversarial network, a high-resolution image corresponding to the low-resolution image sample based on the fused sample feature;

[0020] adjusting model parameters of the generator based on a difference between the high-resolution image and a corresponding real high-resolution image, a difference between the high-resolution image and the reference image sample, and an adversarial training between the generator and a discriminator in the generative adversarial network until an end condition is met, to obtain the image generation model;

[0021] Optionally, the reference image sample and the low-resolution image sample satisfy a consistency condition representing consistency of image content, and the resolution of the reference image sample is higher than that of the low-resolution image sample.

[0022] Optionally, the low-resolution image sample, the reference image sample, and the real high-resolution image are remote sensing images; performing feature extraction on the reference image sample to obtain a reference image sample feature comprises:

[0023] extracting feature maps of different scales from the reference image sample based on a first feature extraction network with a skip connection; the skip connection enables information in the first feature extraction network to be transmitted across layers;

[0024] performing channel self-attention processing and spatial self-attention processing on a high-level feature map to obtain a feature map carrying attention information;

[0025] performing image style information extraction on the feature map carrying attention information to generate a style vector based on the extracted image style information; the reference image sample feature comprises the style vector;

[0026] The high-level feature map is generated by feature extraction on the feature maps of different scales; the channel self-attention processing is used to enhance the key channel information by learning the importance weight of each channel of the high-level feature map for key texture and edge response; and the spatial self-attention processing is used to enhance the response of the key region by focusing on the contribution of different positions in the image to the key texture and edge.

[0027] Optionally, the low-resolution image sample is subjected to feature extraction to obtain a low-resolution image sample feature, including:

[0028] A second feature extraction network with a residual dense connection structure is used to extract a feature map from the low-resolution image sample to obtain a low-resolution feature map; the residual dense connection structure combines residual connection and dense connection, the residual connection enables the second feature extraction network to perform deep feature learning, and the dense connection captures low-frequency structural information of the low-resolution image.

[0029] The low-resolution feature map is subjected to global context fusion processing to supplement image semantic information to the low-resolution feature map to obtain a semantic-supplemented image feature; and the low-resolution image sample feature includes the semantic-supplemented image feature.

[0030] Optionally, the reference image sample feature and the low-resolution image sample feature are fused to obtain a fused sample feature, including:

[0031] The style vector is injected into the semantic-supplemented image feature to obtain a first sub-fused sample feature;

[0032] The first sub-fused sample feature is subjected to spatial attention processing to obtain a second sub-fused sample feature; the spatial attention processing is used to enhance the feature expression of the ground object contour and the texture significant region.

[0033] The second sub-fused sample feature is subjected to multi-scale fusion to obtain the fused sample feature.

[0034] Optionally, based on the difference between the high-resolution image and the corresponding real high-resolution image, the difference between the high-resolution image and the reference image sample, and the adversarial training between the generator and the discriminator in the generative adversarial network, the model parameters of the generator are adjusted, including:

[0035] A first perception loss and an adversarial loss are determined; the first perception loss is used to measure the difference between the high-resolution image generated by the generator and the corresponding real high-resolution image in the feature space, and the adversarial loss is used to measure the adversarial game between the generator and the discriminator;

[0036] adjusting model parameters of the generator based on the first perceptual loss, the adversarial loss and a first optimization target;

[0037] determining a target similarity between the generated high-resolution image and the corresponding reference image sample, evaluating quality of the generated high-resolution image based on the target similarity, and feeding back to the generator based on the quality evaluation result to determine the adversarial loss based on the feedback result;

[0038] determining a reference loss for measuring difference between the generated high-resolution image and the reference image sample;

[0039] adjusting model parameters of the discriminator based on the reference loss and a second optimization target;

[0040] The first optimization target is used to minimize difference between the high-resolution image generated by the generator and the corresponding real high-resolution image in a feature space, and to enhance adversarial game between the generator and the discriminator; and the second optimization target is used to minimize difference between the high-resolution image generated by the generator and the corresponding reference image sample.

[0041] Optionally, determining a target similarity between the generated high-resolution image and the corresponding reference image sample comprises:

[0042] determining a first similarity in an image style dimension and a second similarity in an image structure dimension between the generated high-resolution image and the corresponding reference image sample;

[0043] determining a reference loss comprises:

[0044] determining a second perceptual loss and a structure consistency loss;

[0045] The second perceptual loss is used to measure difference between the generated high-resolution image and the corresponding reference image sample in an image style dimension, and the structure consistency loss is used to measure difference between the generated high-resolution image and the corresponding reference image sample in an image structure dimension.

[0046] An image super-resolution reconstruction device comprises:

[0047] an acquisition module configured to acquire a first image and a corresponding reference image;

[0048] a first feature extraction module configured to perform feature extraction on the reference image to obtain reference image features;

[0049] a second feature extraction module configured to perform feature extraction on the first image to obtain first image features;

[0050] a feature fusion module configured to fuse the reference image features and the first image features to obtain fused features;

[0051] an image generation module configured to generate a second image corresponding to the first image based on the image generation model and the fused features, the second image having a higher resolution than the first image;

[0052] The image generation model is generated based on a difference between a generated high-resolution image and a real high-resolution image for a low-resolution image sample, and a difference between the generated high-resolution image and a corresponding reference image sample.

[0053] An electronic device includes:

[0054] a memory configured to store a computer program;

[0055] a processor configured to realize any of the image super-resolution reconstruction methods provided above by invoking and executing the computer program in the memory.

[0056] A computer readable medium having a computer program stored thereon, the computer program containing program codes for implementing the image super-resolution reconstruction method disclosed in the embodiments of the present application, and when executed by a processor, the image super-resolution reconstruction method disclosed in the embodiments of the present application can be realized.

[0057] According to the above scheme, the image super-resolution reconstruction method and related device provided by the present application introduces a reference image in the image super-resolution reconstruction, and combines the reference image to perform image super-resolution reconstruction. An image super-resolution reconstruction technology based on a reference image guiding mechanism is proposed and implemented. Based on the technology, the image super-resolution reconstruction process can be accurately guided according to the image information of the introduced reference image, such as accurately guiding the super-resolution reconstruction process from multiple dimensions such as style and structure. Compared with the known technology that relies on low-resolution images to directly infer or abstract priori, the present application can accurately match the feature information of the reference image to realize high-quality image super-resolution reconstruction, thereby effectively improving the quality of image super-resolution reconstruction and meeting the high-precision application requirements. BRIEF DESCRIPTION OF DRAWINGS

[0058] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only embodiments of the present application, and those skilled in the art can obtain other drawings according to the provided drawings without creative labor.

[0059] Figure 1 is a schematic diagram of a construction process of an image generation model provided by the present application;

[0060] Figure 2 is a flowchart of an image super-resolution reconstruction method provided by the present application;

[0061] Figure 3 is a working principle diagram of a remote sensing image super-resolution reconstruction system provided by the present application;

[0062] Figure 4 is a component structure diagram of an image super-resolution reconstruction device provided by the present application;

[0063] Figure 5 is a component structure diagram of an electronic device provided by the present application. DETAILED DESCRIPTION

[0064] For the convenience of reference or clarity, first, the technical terms, abbreviations or terms related to the embodiments of the present application are explained as follows:

[0065] Remote sensing image: an image obtained by detecting the electromagnetic wave information reflected, radiated or scattered by a target object by a remote sensor, such as an instrument sensitive to electromagnetic waves, under the condition of being far away from the target and non-contacting the target object, and performing extraction, determination, processing, analysis and application.

[0066] Super-resolution reconstruction: a technical process of recovering a corresponding high-resolution image from an observed low-resolution image by means of signal processing or machine learning.

[0067] Generative adversarial network: abbreviated as GAN, is a deep learning model composed of a generator and a discriminator, the generator is used to generate simulated data, the discriminator is used to distinguish between real data and generated data, and the two improve performance through adversarial training.

[0068] Feature extraction: an operation of extracting a feature vector or a feature map capable of representing key information (such as texture, structure, color, etc.) of an image from image data.

[0069] Feature fusion: a technical method of integrating image features of different sources or different modalities to obtain more comprehensive and representative image information.

[0070] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor are within the scope of protection of the present application.

[0071] Currently, in the deep learning-based remote sensing image super-resolution reconstruction method, the super-resolution reconstruction method based on the generative adversarial network (GAN) performs relatively outstanding in the image generation quality. It usually takes a low-resolution remote sensing image as the input of the generator, learns the mapping relationship of the low-resolution image to the high-resolution image through the generator, and generates a high-resolution image; the discriminator is responsible for judging whether the generated image is a real high-resolution image or a false image generated by the generator.

[0072] The applicant has found that the existing GAN-based remote sensing image super-resolution reconstruction method has two major problems: first, the details such as texture are missing, and only the low-resolution image itself can generate a high-resolution image, which has limited texture and other detail information, and there is no effective external information to supplement, which makes it difficult for the generator to learn rich and accurate texture and other detail information, and further leads to blurred and unnatural details in the reconstructed image; second, the object shape is distorted, the remote sensing object has a specific shape structure, the known method lacks targeted constraints, the low-resolution image object structure information is incomplete and lacks shape reference, and the generator is difficult to accurately restore the real shape of the object, which leads to a large difference or error in the super-resolution result, and further leads to poor reliability of the application.

[0073] To solve at least part of the above problems, the embodiments of the present application provide an image super-resolution reconstruction method and related device, mainly by introducing a reference image in the image super-resolution reconstruction, based on the details, structures and other aspects of the reference image, to guide the generator to realize detail restoration and structure fidelity, so as to realize high-quality super-resolution reconstruction of remote sensing images.

[0074] The method provided by the embodiments of the present application can be applied to the super-resolution reconstruction scene of remote sensing images, but is not limited to this. And the method provided by the embodiments of the present application needs to be based on the pre-construction of an image generation model for generating high-resolution images based on a generative adversarial network. First, the construction process of the image generation model is described below.

[0075] The image generation model is specifically a trained generator in the generative adversarial network. Referring to Figure 1 The construction process of the image generation model can include the following processing steps 101-106.

[0076] Step 101, obtaining a low-resolution image sample and a corresponding reference image sample.

[0077] Optionally, the low-resolution image sample, the reference image sample, and the real high-resolution image referred to later are remote sensing images.

[0078] Additionally, the reference image sample and the low-resolution image sample satisfy a consistency condition representing image content consistency, and the resolution of the reference image sample is higher than the resolution of the low-resolution image sample, so that the reference image sample can provide effective reference value for high-resolution reconstruction of the low-resolution image sample.

[0079] The consistency condition can be, but is not limited to, set as the content type of the reference image sample being the same as or similar to that of the low-resolution image sample, for example, both of the image contents being buildings, both being persons, or both being trees and the like natural scenery.

[0080] Step 102, feature extraction is performed on the reference image sample to obtain a reference image sample feature.

[0081] Optionally, the feature extraction process for feature extraction on the reference image sample can be further implemented as steps 11)-13) as follows:

[0082] 11) Based on the first feature extraction network with skip connection, different scale feature maps are extracted from the reference image sample.

[0083] The skip connection enables information in the first feature extraction network to be transmitted across layers.

[0084] Optionally, the first feature extraction network is a deep convolutional network with skip connection. The introduction of the skip connection breaks the limitation of information transmission between layers in the traditional convolutional network, so that the features of different layers can be fused with each other. Through the network, multiple different scale feature maps can be extracted from the reference image sample / reference image, for example, four different scale feature maps. Different scale feature maps can capture information of an image at different granularities, wherein a small scale feature map focuses on image details, and a large scale feature map focuses on image overall structure, so that rich feature information is provided for subsequent processing by integrating image details and image overall structure.

[0085] 12) Channel self-attention processing and spatial self-attention processing are performed on the high layer feature map to obtain a feature map carrying attention information.

[0086] The high layer feature map is generated by further feature extraction on the different scale feature maps, and can be, but is not limited to, generated by semantic extraction on the different scale feature maps (such as four different scale feature maps) in a deep network.

[0087] The channel self-attention processing is used for enhancing key channel information by learning importance weights of key texture and edge responses of the high-level feature map; and the spatial self-attention processing is used for enhancing responses of key regions by focusing on contributions of different positions in the image to key texture and edges.

[0088] The embodiment of the present application introduces channel self-attention and spatial self-attention mechanisms in a high-level feature map. The channel self-attention mechanism can automatically learn importance weights of each channel to key texture and edge responses in the channel dimension of the feature map, so as to highlight key channel information; and the spatial self-attention mechanism focuses on contributions of different positions in the image to key texture and edges in the spatial dimension, so as to enhance responses of key regions, thereby effectively improving the capturing ability of key features of the image.

[0089] 13) performing image style information extraction on the feature map carrying attention information, generating a style vector based on the extracted image style information; the reference image sample feature comprises the style vector.

[0090] The image style information can include but is not limited to texture, color / color, light and shadow, lines and other (local) detail information of the image.

[0091] After obtaining the feature map carrying attention information, the feature map carrying attention information can be further analyzed in depth, and relevant statistical quantities for representing image style information can be calculated. Exemplarily, the channel mean and standard deviation of the feature map (carrying attention information) can be calculated, but are not limited to. These statistical quantities reflect the distribution characteristics of the feature map in the channel dimension, and a style vector can be generated based on the calculated relevant statistical quantities. The style vector can be used as a reference image sample feature.

[0092] The generated style vector contains style information of the reference image sample, and can be used for subsequent style transfer and other operations.

[0093] Step 103, performing feature extraction on the low-resolution image sample to obtain low-resolution image sample features.

[0094] Optionally, the feature extraction process for performing feature extraction on the low-resolution image sample can be further implemented as steps 21)-22) as follows:

[0095] 21) extracting a feature map from the low-resolution image sample based on a second feature extraction network with a residual dense connection structure to obtain a low-resolution feature map.

[0096] The low-resolution feature map here is the feature map of the low-resolution image sample.

[0097] The residual dense connection structure is used for combining residual connection and dense connection, the residual connection is used for enabling the second feature extraction network to perform feature learning in a deep level, and the dense connection is used for capturing low-frequency structural information of the low-resolution image.

[0098] For the feature extraction network of the low-resolution image sample / low-resolution image, an unique residual dense connection structure is adopted in the embodiment of the present application. This structure combines residual connection and dense connection. On one hand, the residual connection helps to solve the gradient vanishing problem in the training of a deep network, so as to ensure that the network can perform feature learning in a deep level. On the other hand, the dense connection makes the information flow between layers more sufficient, and can efficiently capture low-frequency structural information of the low-resolution image. These low-frequency structural information is the basic framework of the image, and is crucial for the overall understanding of the image.

[0099] 22) performing global context fusion processing on the low-resolution feature map to supplement image semantic information to the low-resolution feature map, to obtain image features after semantic supplement; the low-resolution image sample feature comprises the image features after semantic supplement.

[0100] In the implementation, the global context fusion processing on the low-resolution feature map can be implemented by adding a global average pooling branch in the deep feature extraction process of the low-resolution feature map. The added global average pooling operation is used for aggregating global information of the entire feature map (such as the low-resolution feature map), to supplement semantic information. In the low-resolution image, local features can have the case of incomplete information. In this way, more comprehensive and rich semantic clues can be provided for subsequent processing.

[0101] The execution sequence of step 102 and step 103 is not limited, one step can be performed first, and the other step can be performed later, or the two steps can be executed in parallel, which can be determined according to actual application.

[0102] Step 104, fusing the reference image sample feature and the low-resolution image sample feature to obtain a fused sample feature.

[0103] Optionally, the process of fusing the reference image sample feature and the low-resolution image sample feature can be further implemented as steps 31)-33) as follows:

[0104] 31) injecting the style vector into the image features after semantic supplement to obtain a first sub-fused sample feature.

[0105] Specifically but not limitedly, the distribution of the LR (Low Resolution) feature can be adjusted according to the style vector by an AdaIN (Adaptive Instance Normalization) technology, that is, the image feature of the low resolution image sample after semantic supplement is adjusted to match the feature distribution of the reference style (the style of the reference image sample). This operation can effectively integrate the style information of the reference image while retaining the content features of the low resolution image, and achieve preliminary fusion of the low resolution image content and the reference image style.

[0106] 32) performing spatial attention processing on the first sub-fusion sample feature to obtain a second sub-fusion sample feature; the spatial attention processing is used to enhance the feature expression of the ground object contour and the texture significant area.

[0107] In implementation, a spatial attention module can be applied to the fused feature map, i.e., the first sub-fusion sample feature. The module is used to automatically learn the importance of different spatial positions in the input feature map, focus on the ground object contour and the texture significant area, and enhance the feature expression of these key areas, so that the fused feature is more consistent with the human visual focus and highlights the important information in the image.

[0108] 33) performing multi-scale fusion on the second sub-fusion sample feature to obtain the fusion sample feature.

[0109] Specifically, the fusion features corresponding to different levels of the second sub-fusion sample feature can be comprehensively processed by a cross-layer fusion technology. Different levels of features contain information of different granularities. Through cross-layer fusion, these information can be integrated to form the final condition feature, i.e., the fusion sample feature, which contains rich details and has global consistency, so as to provide high-quality feature input for subsequent image generation.

[0110] Step 105, generating, by a generator of the generative adversarial network, a high resolution image corresponding to the low resolution image sample based on the fusion sample feature.

[0111] The conditional generative adversarial network enables the generator to learn the distribution characteristics of the real high resolution image through the adversarial training of the generator and the discriminator, so as to generate a high-quality super resolution image.

[0112] In implementation, a conditional generative adversarial network with multiple up-sampling paths can be specifically constructed, for example, a conditional generative adversarial network with four up-sampling paths. The up-sampling operation is a key step to gradually restore the low resolution feature to high resolution. Specifically, the generator of the generative adversarial network can gradually enlarge the size of the feature map (such as the fusion sample feature) through four or more up-sampling paths, to finely restore the high resolution details and generate the high resolution image corresponding to the low resolution image sample.

[0113] Step 106, based on the difference between the high-resolution image and the corresponding real high-resolution image, the difference between the high-resolution image and the reference image sample, and the adversarial training between the generator and the discriminator in the generative adversarial network, adjusting the model parameters of the generator until the end condition is met, obtaining the image generation model.

[0114] Optionally, the step 106 can be further implemented as steps 41)-45) as follows:

[0115] 41) determining a first perceptual loss and an adversarial loss; the first perceptual loss is used to measure the difference between the high-resolution image generated by the generator and the corresponding real high-resolution image in the feature space, and the adversarial loss is used to measure the adversarial game between the generator and the discriminator.

[0116] 42) adjusting the model parameters of the generator based on the first perceptual loss, the adversarial loss and the first optimization target.

[0117] Wherein, the first optimization target is used to minimize the difference between the high-resolution image generated by the generator and the corresponding real high-resolution image in the feature space, and to enhance the adversarial game between the generator and the discriminator.

[0118] In the training process of the generative adversarial network, the embodiments of the present application combine the perceptual loss (such as the perceptual loss based on the VGG feature difference) and the adversarial loss to train the generator. For the convenience of description, the perceptual loss here is called the first perceptual loss, which compares the difference between the high-resolution image generated by the generator and the real high-resolution image in the VGG network feature space to ensure the quality of the generated image in the semantic and visual perception; the adversarial loss makes the data distribution of the generated image consistent with the real image distribution through the game between the generator and the discriminator, so as to simultaneously consider the visual quality and the consistency of the data distribution, and improve the authenticity and credibility of the generated image.

[0119] In the training process, the model parameters of the generator can be adjusted based on the first perceptual loss, the adversarial loss and the first optimization target, and the process is iterated until the end condition is met. By continuously adjusting the model parameters of the generator, the quality of the generated image in the semantic and visual perception, and the consistency of the data distribution of the generated image with the real image distribution are ensured.

[0120] The end condition can be, but is not limited to, setting the iteration number in the training process to reach a threshold value, or the value of the first perceptual loss / adversarial loss is lower than a threshold value.

[0121] 43) the decision maker determines a target similarity between the generated high-resolution image and the corresponding reference image sample, evaluates the quality of the generated high-resolution image based on the target similarity, and feeds back to the generator based on the quality evaluation result to determine the adversarial loss based on the feedback result.

[0122] Optionally, a first similarity in image style dimension and a second similarity in image structure dimension between the generated high-resolution image and the corresponding reference image sample can be determined. The first similarity and the second similarity together constitute the target similarity.

[0123] Correspondingly, the quality of the high-resolution image generated by the generator can be evaluated based on the first similarity and the second similarity.

[0124] In implementation, the discriminator can be designed with parallel input branches for receiving the high-resolution image generated by the generator and for receiving the reference image sample / reference image, and the discriminator calculates the similarity in style dimension and structure dimension between the generated high-resolution image and the reference image by processing the image features of the two branches respectively. The first similarity and the second similarity are obtained, the first similarity measures the similarity in style performance between the generated image and the reference image, and the second similarity focuses on the consistency in object structure layout between the two to comprehensively evaluate the quality of the image generated by the generator.

[0125] 44) determine a reference loss, which is used to measure the difference between the generated high-resolution image and the reference image sample.

[0126] Optionally, a second perceptual loss and a structure consistency loss can be determined, which together constitute the reference loss.

[0127] Among them, the second perceptual loss is used to measure the difference in image style dimension between the generated high-resolution image and the corresponding reference image sample, and the structure consistency loss is used to measure the difference in image structure dimension between the generated high-resolution image and the corresponding reference image sample.

[0128] 45) adjust the model parameters of the discriminator based on the reference loss and a second optimization target.

[0129] Among them, the second optimization target is used to minimize the difference between the high-resolution image generated by the generator and the corresponding reference image sample.

[0130] In the process of adversarial training, the embodiments of the present application introduce a style perception loss term, that is, the second perception loss, which is specifically used to constrain the style of the image generated by the generator to be consistent with the reference image style. Specifically, by minimizing the style perception loss, that is, minimizing the second perception loss, it can be ensured that the image generated by the generator can accurately present the style of the reference image in the style transfer task, avoiding style drift.

[0131] The structural consistency loss can be realized by the feature matching loss and the edge preservation loss. The feature matching loss promotes the features of the image generated by the generator to be consistent with the features of the reference image in the key area, and the edge preservation loss focuses on maintaining the edge information of the ground object and strengthening the accuracy of the ground object form restoration, so as to ensure the rationality and integrity of the generated image in structure.

[0132] After training is completed, the generator at the time of completing training can be used as the image generation model described above and applied to subsequent image super-resolution reconstruction.

[0133] The embodiments of the present application introduce a reference image in image super-resolution reconstruction, extract features of the reference image, obtain features of the reference image in texture, color and other style dimensions and structure dimensions, and use a feature fusion network to deeply fuse the features of the reference image and the features of the low-resolution image. Combined with the GAN (Generative Adversarial Network) architecture, the reference image constraint condition is introduced in the discriminator, so that the generator can accurately match the feature information of the reference image when reconstructing the high-resolution remote sensing image, and the high-quality image super-resolution reconstruction can be realized accordingly.

[0134] After the image generation model is constructed, the image super-resolution reconstruction method provided by the embodiments of the present application can be further realized based on the image generation model. Referring to the image super-resolution reconstruction method flowchart shown in Figure 2 The method can include the following steps 201-205.

[0135] Step 201, obtaining a first image and a corresponding reference image.

[0136] Optionally, the reference image and the first image satisfy the consistency condition representing the consistency of image content described above, and the resolution of the reference image is higher than that of the first image.

[0137] Optionally, the first image and the corresponding reference image can be remote sensing images respectively.

[0138] In actual application, the corresponding reference image can be matched for the first image input by the user to be super-resolution reconstructed by the platform automatically, or the corresponding reference image can be input manually by the user, and no limitation is made in this regard, which can be determined according to actual application.

[0139] Step 202, feature extraction is performed on the reference image to obtain reference image features.

[0140] Optionally, first, different scale feature maps can be extracted from the reference image based on the first feature extraction network with skip connection, then channel self-attention processing and spatial self-attention processing are performed on the high-level feature map to obtain a feature map carrying attention information, on this basis, image style information extraction is performed on the feature map carrying attention information, a style vector is generated based on the extracted image style information, and the generated style vector can be used as the image feature of the reference image.

[0141] The high-level feature map here is generated by further feature extraction on different scale feature maps of the reference image, which can be but is not limited to being generated by semantic extraction on the different scale feature maps (such as four different scale feature maps) in a deep network.

[0142] The feature extraction process of the reference image is consistent with the feature extraction process of the reference image sample, and the more detailed implementation process of the reference image feature extraction can be referred to the related description of the feature extraction process of the reference image sample in the above embodiment, which will not be repeated here.

[0143] Step 203, feature extraction is performed on the first image to obtain first image features.

[0144] Specifically, first, a feature map can be extracted from the first image based on the second feature extraction network with a residual dense connection structure to obtain a low-resolution feature map; then, global context fusion processing is performed on the low-resolution feature map to supplement image semantic information to the low-resolution feature map to obtain image features after semantic supplement, and the image features after semantic supplement can be used as the first image features.

[0145] The feature extraction process of the first image is consistent with the feature extraction process of the low-resolution image sample, and the more detailed implementation process of the first image feature extraction can be referred to the related description of the feature extraction process of the low-resolution image sample in the above embodiment, which will not be repeated here.

[0146] The execution order of step 202 and step 203 is not limited, one step can be performed first and the other step can be performed later, or the two steps can be executed in parallel, which can be determined according to actual application.

[0147] Step 204, fusion processing is performed on the reference image features and the first image features to obtain fusion features.

[0148] Optionally, the content of the first image and the style of the reference image can be preliminarily fused by injecting the style vector of the reference image into the semantic supplemented image feature corresponding to the first image. On this basis, the fusion feature of the reference image feature and the first image feature can be obtained by further performing spatial attention processing and multi-scale feature fusion on the obtained fused feature map. For a more detailed feature fusion implementation process, refer to the description of the reference image sample feature and the low-resolution image sample feature fusion process in the model training stage, which will not be repeated here.

[0149] Step 205, generating a second image corresponding to the first image based on the fusion feature by the image generation model; the resolution of the second image is higher than that of the first image.

[0150] The image generation model is generated based on the difference between the generated high-resolution image and the real high-resolution image for the low-resolution image sample, and the difference between the generated high-resolution image and the corresponding reference image sample.

[0151] After obtaining the fusion feature, the image generation model is used to generate a high-resolution image based on the fusion feature, which is the second image obtained by super-resolution reconstruction of the first image.

[0152] In summary, the image super-resolution reconstruction method provided by the present application introduces a reference image in the image super-resolution reconstruction, combines the reference image for image super-resolution reconstruction, proposes and implements an image super-resolution reconstruction technology based on a reference image guiding mechanism. Based on this technology, the image super-resolution reconstruction process can be accurately guided according to the image information of the introduced reference image, such as accurately guiding the super-resolution reconstruction process from multiple dimensions such as style and structure. Compared with known technologies that rely on low-resolution images for direct inference or abstract priori, the present application can accurately match the feature information of the reference image to achieve high-quality image super-resolution reconstruction, thereby effectively improving the quality of image super-resolution reconstruction and meeting the high-precision application requirements.

[0153] Further, the present application extracts the style features such as real texture and color and the structure features in the reference image, and fuses them with the low-resolution image features to guide the image generation model to reconstruct a high-resolution image, thereby realizing the detail restoration and structure fidelity of the image in the super-resolution reconstruction process. In terms of detail restoration, the high-frequency texture and other style features provided by the reference image effectively solve the problem of blurred details such as texture in traditional super-resolution reconstruction methods. In terms of structure fidelity, the fused structure constraint avoids the distortion and distortion of ground objects, and improves the accuracy of the shape of the ground objects.

[0154] An application example of the method of the present application is provided below.

[0155] In this example, a remote sensing image super-resolution reconstruction system is constructed based on the method of the present application. The constructed system mainly includes an input module, a feature extraction module, a content / style fusion module, a conditional generation module, and a discrimination correction module. The modules cooperate with each other to form a complete processing flow to achieve a specific image super-resolution reconstruction task.

[0156] Among them, the discrimination correction module is applied to the model (image generation model) training stage, and the other modules are applied to the whole stage of model training and inference.

[0157] In combination Figure 3 The working principle diagram of the system is shown. The input module is responsible for receiving the original data such as low-resolution images and corresponding reference images, real high-resolution images, etc. required for input in the model training or inference stage; the feature extraction module is used for feature extraction on the data such as low-resolution images and reference images input by the input module; the content / style fusion module is responsible for fusion processing of different features extracted by the feature extraction module; the conditional generation module includes a generator / image generation model, which is used to generate a high-resolution target image based on the fusion features; the discrimination correction module (not shown in the figure) includes a discriminator, which is used to discriminate and correct the generation result of the conditional generation module in the model training stage to ensure that the output meets the expectation. Figure 3

[0158] The system can accurately guide the image super-resolution reconstruction process according to the reference image introduced in the image super-resolution reconstruction, such as accurately guiding the reconstruction process from multiple dimensions such as style and structure, so as to accurately match the feature information of the reference image and realize high-quality image super-resolution reconstruction, effectively improving the quality of image super-resolution reconstruction.

[0159] Corresponding to the above method, the embodiments of the present application also provide an image super-resolution reconstruction device, referring to Figure 4 the component structure schematic diagram, the device includes:

[0160] The acquisition module 401 is used to acquire a first image and a corresponding reference image.

[0161] The first feature extraction module 402 is used to extract features from the reference image to obtain reference image features.

[0162] The second feature extraction module 403 is used to extract features from the first image to obtain first image features.

[0163] The feature fusion module 404 is used to fuse the reference image features and the first image features to obtain fusion features. ​

[0164] the image generation module 405 is configured to generate a second image corresponding to the first image based on the fusion feature, wherein a resolution of the second image is higher than a resolution of the first image.

[0165] The image generation model is generated based on a difference between a generated high-resolution image and a real high-resolution image for a low-resolution image sample, and a difference between the generated high-resolution image and a corresponding reference image sample.

[0166] In an optional implementation, the reference image and the first image satisfy a consistency condition representing consistency of image content, and a resolution of the reference image is higher than a resolution of the first image.

[0167] In an optional implementation, the device further includes a model construction module configured to:

[0168] obtain a low-resolution image sample and a corresponding reference image sample;

[0169] perform feature extraction on the reference image sample to obtain reference image sample features;

[0170] perform feature extraction on the low-resolution image sample to obtain low-resolution image sample features;

[0171] fuse the reference image sample features and the low-resolution image sample features to obtain fusion sample features;

[0172] a generator of a generative adversarial network generates a high-resolution image corresponding to the low-resolution image sample based on the fusion sample features;

[0173] based on a difference between the high-resolution image and a corresponding real high-resolution image, a difference between the high-resolution image and the reference image sample, and an adversarial training between the generator and a discriminator in the generative adversarial network, adjust model parameters of the generator until a termination condition is met, to obtain the image generation model;

[0174] In an optional implementation, the reference image sample and the low-resolution image sample satisfy a consistency condition representing consistency of image content, and a resolution of the reference image sample is higher than a resolution of the low-resolution image sample.

[0175] In an optional implementation, the low-resolution image sample, the reference image sample, and the real high-resolution image are remote sensing images; when performing feature extraction on the reference image sample to obtain reference image sample features, the model construction module is specifically configured to:

[0176] extracting feature maps of different scales from the reference image sample based on a first feature extraction network with a skip connection; the skip connection enables information in the first feature extraction network to be transmitted across layers;

[0177] performing channel self-attention processing and spatial self-attention processing on the high-level feature map to obtain a feature map carrying attention information;

[0178] extracting image style information from the feature map carrying attention information, and generating a style vector based on the extracted image style information; the reference image sample feature includes the style vector;

[0179] The high-level feature map is generated by performing feature extraction on the feature maps of different scales; the channel self-attention processing is used to enhance key channel information by learning the importance weight of each channel of the high-level feature map for key texture and edge response; and the spatial self-attention processing is used to enhance the response of key regions by focusing on the contribution of different positions in the image to key texture and edges.

[0180] In an optional implementation, the model construction module, when performing feature extraction on the low-resolution image sample to obtain a low-resolution image sample feature, is specifically configured to:

[0181] extracting a feature map from the low-resolution image sample based on a second feature extraction network with a residual dense connection structure, to obtain a low-resolution feature map; the residual dense connection structure is used to combine residual connection and dense connection, the residual connection is used to enable the second feature extraction network to perform deep feature learning, and the dense connection is used to capture low-frequency structural information of the low-resolution image;

[0182] performing global context fusion processing on the low-resolution feature map to supplement image semantic information to the low-resolution feature map, to obtain an image feature after semantic supplementation; the low-resolution image sample feature includes the image feature after semantic supplementation.

[0183] In an optional implementation, the model construction module, when performing fusion on the reference image sample feature and the low-resolution image sample feature to obtain a fusion sample feature, is specifically configured to:

[0184] injecting the style vector into the image feature after semantic supplementation to obtain a first sub-fusion sample feature;

[0185] performing spatial attention processing on the first sub-fusion sample feature to obtain a second sub-fusion sample feature; the spatial attention processing is used to enhance feature expression of ground object contour and texture significant area;

[0186] The second sub-fusion sample features are fused in multiple scales to obtain the fusion sample features.

[0187] In an optional implementation, the model construction module is specifically configured to:

[0188] determine a first perceptual loss and an adversarial loss; the first perceptual loss is used to measure the difference between the high-resolution image generated by the generator and the corresponding real high-resolution image in a feature space, and the adversarial loss is used to measure the adversarial game between the generator and the discriminator;

[0189] adjust the model parameters of the generator based on the first perceptual loss, the adversarial loss, and a first optimization target;

[0190] In the discriminator determines the target similarity between the generated high-resolution image and the corresponding reference image sample, evaluates the quality of the generated high-resolution image based on the target similarity, and feeds back to the generator based on the quality evaluation result to determine the adversarial loss based on the feedback result;

[0191] determine a reference loss, which is used to measure the difference between the generated high-resolution image and the reference image sample;

[0192] adjust the model parameters of the discriminator based on the reference loss and a second optimization target;

[0193] The first optimization target is used to minimize the difference between the high-resolution image generated by the generator and the corresponding real high-resolution image in the feature space, and to enhance the adversarial game between the generator and the discriminator; and the second optimization target is used to minimize the difference between the high-resolution image generated by the generator and the corresponding reference image sample.

[0194] In an optional implementation, the model construction module is specifically configured to:

[0195] determine a first similarity in an image style dimension and a second similarity in an image structure dimension between the generated high-resolution image and the corresponding reference image sample;

[0196] The model construction module is specifically configured to:

[0197] determine a second perceptual loss and a structure consistency loss;

[0198] The second perception loss is used to measure the difference between the generated high-resolution image and the corresponding reference image sample in the image style dimension, and the structural consistency loss is used to measure the difference between the generated high-resolution image and the corresponding reference image sample in the image structure dimension.

[0199] For the image super-resolution reconstruction device disclosed in the embodiments of the present application, since it corresponds to the image super-resolution reconstruction method disclosed in the above method embodiments, the description is relatively simple, and the relevant similarities can be referred to the description of the above method embodiments, which will not be described in detail here.

[0200] The embodiments of the present application also disclose an electronic device, and the component structure of the electronic device is shown as Figure 5 at least comprising:

[0201] The memory 10 is used to store computer programs.

[0202] The processor 20 is used to realize the image super-resolution reconstruction method provided in the above corresponding method embodiments by calling and executing the computer programs in the memory.

[0203] The processor 20 can be a central processing unit (CPU), an application-specific integrated circuit (ASIC), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a neural network processor (NPU), a deep learning processor (DPU) or other programmable logic devices, etc.

[0204] In addition, the electronic device can also include a communication interface, a communication bus and other components. The memory, the processor and the communication interface complete the communication among each other through the communication bus.

[0205] The communication interface is used for communication between the electronic device and other devices. The communication bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc.

[0206] In addition, the present application also provides a computer readable medium, which stores a computer program, the computer program contains program codes for executing the image super-resolution reconstruction method disclosed in the embodiments of the present application, and when executed by a processor, can be used to implement the image super-resolution reconstruction method disclosed in the embodiments of the present application.

[0207] In the context of the present application, the computer readable medium (machine readable medium) can be a tangible medium, which can contain or store programs for use by or in conjunction with an instruction execution system, apparatus or device. The machine readable medium can be a machine readable signal medium or a machine readable storage medium. The machine readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination of the above. More specific examples of machine readable storage media can include one or more wires, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, compact disk read only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above.

[0208] It should be noted that each embodiment in the specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between each embodiment can be referred to each other.

[0209] For the convenience of description, the above system or device is described as various modules or units respectively described in terms of functions. Of course, the functions of each unit can be implemented in the same or multiple software and / or hardware in the implementation of the present application.

[0210] From the description of the above embodiments, those skilled in the art can clearly understand that the present application can be realized by means of software and necessary general hardware platform. Based on such understanding, the technical solutions of the present application can be embodied in the form of software product, which can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including a plurality of instructions for making a computer device (which can be a personal computer, server, or network device, etc.) execute the methods described in the various embodiments or some parts of the embodiments of the present application.

[0211] Finally, it needs to be pointed out that, in this article, the relationship terms such as first, second, third and target, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the term "includes", "contains" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "includes a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0212] The above is only the preferred embodiment of the present application, and it should be pointed out that for ordinary skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can also be made, which should be considered as the protection scope of the present application.

Claims

1. An image super-resolution reconstruction method, characterized in that, include: Obtain the first image and the corresponding reference image; Feature extraction is performed on the reference image to obtain the reference image features; Feature extraction is performed on the first image to obtain the first image features; The reference image features and the first image features are fused together to obtain fused features; The image generation model generates a second image corresponding to the first image based on the fusion features; the resolution of the second image is higher than that of the first image; The image generation model is generated by training a model based on the differences between the high-resolution image generated from the low-resolution image sample and the real high-resolution image, as well as the differences between the generated high-resolution image and the corresponding reference image sample.

2. The image super-resolution reconstruction method of claim 1, wherein, The reference image and the first image satisfy the consistency condition that characterizes the consistency of image content, and the resolution of the reference image is higher than that of the first image.

3. The image super-resolution reconstruction method of claim 1, wherein, The process of constructing the image generation model includes: Obtain low-resolution image samples and corresponding reference image samples; Feature extraction is performed on the reference image sample to obtain the reference image sample features; Feature extraction is performed on the low-resolution image samples to obtain low-resolution image sample features; The features of the reference image samples and the features of the low-resolution image samples are fused to obtain fused sample features; The generator in the generative adversarial network generates a high-resolution image corresponding to the low-resolution image sample based on the features of the fused sample; Based on the differences between the high-resolution image and the corresponding real high-resolution image, the differences between the high-resolution image and the reference image sample, and the adversarial training between the generator and the discriminator in the generative adversarial network, the model parameters of the generator are adjusted until the termination condition is met, and the image generation model is obtained. Wherein, the reference image sample and the low-resolution image sample satisfy the consistency condition characterizing the consistency of image content, and the resolution of the reference image sample is higher than the resolution of the low-resolution image sample.

4. The image super-resolution reconstruction method of claim 3, characterized in that, The low-resolution image sample, the reference image sample, and the real high-resolution image are remote sensing images; Feature extraction is performed on the reference image sample to obtain the reference image sample features, including: Based on a first feature extraction network with skip connections, feature maps of different scales are extracted from the reference image samples; the skip connections enable information in the first feature extraction network to be transferred across layers. Perform channel self-attention processing and spatial self-attention processing on the high-level feature map to obtain a feature map carrying attention information; Image style information is extracted from the feature map carrying attention information, and a style vector is generated based on the extracted image style information; the reference image sample features include the style vector. The high-level feature map is generated by feature extraction on the feature maps of different scales; the channel self-attention processing is used to enhance the key channel information by learning the importance weight of each channel of the high-level feature map for key texture and edge response; and the spatial self-attention processing is used to enhance the response of the key region by focusing on the contribution of different positions in the image to the key texture and edge.

5. The image super-resolution reconstruction method of claim 4, characterized in that, The low-resolution image sample is subjected to feature extraction to obtain a low-resolution image sample feature, including: A second feature extraction network with a residual dense connection structure is used to extract a feature map from the low-resolution image sample to obtain a low-resolution feature map; the residual dense connection structure combines residual connection and dense connection, the residual connection enables the second feature extraction network to perform deep-level feature learning, and the dense connection captures low-frequency structural information of the low-resolution image; The low-resolution feature map is subjected to global context fusion processing to supplement image semantic information to the low-resolution feature map to obtain a semantic-supplemented image feature; the low-resolution image sample feature includes the semantic-supplemented image feature.

6. The image super-resolution reconstruction method of claim 5, characterized in that, The reference image sample feature and the low-resolution image sample feature are fused to obtain a fused sample feature, including: The style vector is injected into the semantic-supplemented image feature to obtain a first sub-fused sample feature; The first sub-fused sample feature is subjected to spatial attention processing to obtain a second sub-fused sample feature; the spatial attention processing is used to enhance the feature expression of the ground object contour and texture significant region; The second sub-fused sample feature is subjected to multi-scale fusion to obtain the fused sample feature.

7. The image super-resolution reconstruction method of any one of claims 3-6, wherein, Based on the difference between the high-resolution image and the corresponding real high-resolution image, the difference between the high-resolution image and the reference image sample, and the adversarial training between the generator and the discriminator in the generative adversarial network, the model parameters of the generator are adjusted, including: A first perception loss and an adversarial loss are determined; the first perception loss is used to measure the difference between the high-resolution image generated by the generator and the corresponding real high-resolution image in the feature space, and the adversarial loss is used to measure the adversarial game between the generator and the discriminator; Based on the first perception loss, the adversarial loss and a first optimization target, the model parameters of the generator are adjusted; The discriminator determines the target similarity between the generated high-resolution image and the corresponding reference image sample, evaluates the quality of the generated high-resolution image based on the target similarity, and feeds back to the generator based on the quality evaluation result to determine the adversarial loss based on the feedback result; A reference loss is determined, which is used to measure the difference between the generated high-resolution image and the reference image sample; Based on the reference loss and a second optimization target, the model parameters of the discriminator are adjusted; The first optimization target is used for minimizing the difference between the high-resolution image generated by the generator and the corresponding real high-resolution image in a feature space and enhancing the adversarial game between the generator and the discriminator; and the second optimization target is used for minimizing the difference between the high-resolution image generated by the generator and the corresponding reference image sample.

8. The image super-resolution reconstruction method of claim 7, characterized in that, The target similarity between the generated high-resolution image and the corresponding reference image sample is determined, including: The first similarity in an image style dimension and the second similarity in an image structure dimension between the generated high-resolution image and the corresponding reference image sample are determined. The reference loss is determined, including: The second perceptual loss and the structure consistency loss are determined. The second perceptual loss is used for measuring the difference between the generated high-resolution image and the corresponding reference image sample in the image style dimension, and the structure consistency loss is used for measuring the difference between the generated high-resolution image and the corresponding reference image sample in the image structure dimension.

9. An image super-resolution reconstruction apparatus, characterized by comprising: including: An acquisition module is configured to acquire a first image and a corresponding reference image; A first feature extraction module is configured to perform feature extraction on the reference image to obtain reference image features; A second feature extraction module is configured to perform feature extraction on the first image to obtain first image features; A feature fusion module is configured to perform fusion processing on the reference image features and the first image features to obtain fused features; An image generation module is configured to generate a second image corresponding to the first image based on the fused features by using an image generation model; the resolution of the second image is higher than that of the first image. The image generation model is generated based on model training of the difference between a high-resolution image generated for a low-resolution image sample and a real high-resolution image and the difference between the generated high-resolution image and a corresponding reference image sample.

10. An electronic device, comprising: including: A memory is configured to store a computer program; A processor is configured to realize the image super-resolution reconstruction method according to any one of claims 1-8 by calling and executing the computer program in the memory.

Images