Image super-resolution reconstruction method and device, computer device and storage medium

Abstract

The present application relates to the field of image processing, and specifically provides an image super-resolution reconstruction method and device, computer equipment and storage medium, including two parts of a degradation model and a super-resolution reconstruction network, the degradation model is used as the general prior knowledge constraint structure of the current deep learning-based super-resolution method, fully considers the unique degradation process of remote sensing imaging relative to the natural image imaging process, adopts the plug-and-play network framework of the form of 'degrader-generator', so that the generalization ability and robustness are enhanced. Secondly, the super-resolution reconstruction network is designed, including two parts of a degrader and a generator, the degrader can learn and integrate the degradation prior knowledge contained in the degradation model, and is used for the constraint of the generator training, and the main task of the generator is to generate a high-quality super-resolution image under the constraint of the degrader, complete the reconstruction, and has the advantages of good visual effect and high fidelity.

Description

Technical Field

[0001] This invention relates to the field of image processing, and in particular to an image super-resolution reconstruction method, apparatus, computer device, and storage medium. Background Technology

[0002] Super-resolution (SR) reconstruction technology for remote sensing images is a research hotspot in the field of remote sensing, and deep learning-based SR reconstruction of remote sensing images has become the mainstream direction of current research on super-resolution of remote sensing images.

[0003] With the development of deep learning networks and the improvement of their learning capabilities, the key factor limiting super-resolution results has gradually shifted from insufficient network learning ability to insufficient utilization of prior knowledge. Therefore, how to transform the degradation factors in the real remote sensing image degradation process into degradation algorithms and incorporate them into the degradation model is an urgent problem to be solved. Summary of the Invention

[0004] In view of this, the present invention provides an image super-resolution reconstruction method, apparatus, computer device, and storage medium.

[0005] In a first aspect, embodiments of the present invention provide an image super-resolution reconstruction method, comprising:

[0006] A pre-configured degradation model for image degradation is used to generate high-resolution HR-low-resolution LR image pairs. The degradation model has a first Gaussian blur kernel, RGB three-channel weighted downsampling, a second Gaussian blur kernel, and noise.

[0007] A super-resolution network is constructed using plug-and-play PNP technology. The super-resolution network includes a degrader, a generator, and a loss function. The generator is used to generate super-resolution images, and the degrader is used to constrain the generator's generation process based on degraded prior information.

[0008] The degradation processor is trained using the HR-LR images, and the degradation information in the degradation model is transferred to the degradation processor.

[0009] Super-resolution target image I target After performing bicubic interpolation, the first super-resolution image I is obtained by inputting it into the generator. SR The first resolution image I SR The super-resolution synthesized image is obtained after inputting into the de-qualityr. The super-resolution target image I is calculated using the loss function. target and the super-resolution synthesized image The loss value is used to constrain the generator;

[0010] Under the constraints of the degradation unit, the generator produces a super-resolution image, thus completing the reconstruction.

[0011] As an optional approach, the degrader employs a CNN network and constructs a supplementary information transmission channel in the form of skip connections. The degrader has two 3×3 convolutional kernels with strides of 1 and 2, a 5-layer recurrent block structure, a first 3×3 convolutional layer with a stride of 1, a BN layer, and an end network. The output of each layer of the 5-layer recurrent block structure is connected to the BN layer and accumulated before being input into the end network. The end network has a first 3×3 convolutional layer with a stride of 2, a first PReLU layer, a second 3×3 convolutional layer with a stride of 2, a second PReLU layer, and a third 3×3 convolutional layer with a stride of 1.

[0012] As an optional approach, the first Gaussian blur kernel adopts both isotropic and anisotropic Gaussian blur kernels;

[0013] The RGB three-channel weighted downsampling is determined by theoretical derivation to determine the weight parameters when downsampling the RGB three channels, and multiplicative weight coefficients are added. The weight coefficients are adjusted during network training.

[0014] The second Gaussian blur kernel is used to simulate the overall degradation effect of the remote sensing imaging platform, and isotropic Gaussian blur kernels with different parameter settings are used.

[0015] The noise used is Gaussian white noise.

[0016] As an alternative approach, the generator's network structure adopts a U-shaped architecture, which extracts cross-scale information through continuous downsampling and upsampling and supplements the cross-scale information into the super-resolution image reconstruction.

[0017] In a second aspect, embodiments of the present invention provide an image super-resolution reconstruction apparatus, characterized in that it comprises:

[0018] The training dataset unit is used to pre-configure a degradation model for image degradation, which generates high-resolution HR-low-resolution LR image pairs. The degradation model has a first Gaussian blur kernel, RGB three-channel weighted downsampling, a second Gaussian blur kernel, and noise.

[0019] A construction unit is used to construct a super-resolution network using plug-and-play PNP technology. The super-resolution network includes a degrader, a generator, and a loss function. The generator is used to generate a super-resolution image, and the degrader is used to constrain the generator's generation process based on degraded prior information.

[0020] The training unit is used to train the degrader using the HR-LR image and to transfer the degradation information in the degradation model to the degrader.

[0021] The loss calculation unit is used to calculate the super-resolution target image I. target After performing bicubic interpolation, the first super-resolution image I is obtained by inputting it into the generator. SR The first resolution image I SR The super-resolution synthesized image is obtained after inputting into the de-qualityr. The super-resolution target image I is calculated using the loss function. target and the super-resolution synthesized image The loss value is used to constrain the generator;

[0022] The reconstruction unit is used to generate a super-resolution image under the constraint of the degrader, thereby completing the reconstruction.

[0023] As an optional approach, the degrader employs a CNN network and constructs a supplementary information transmission channel in the form of skip connections. The degrader has two 3×3 convolutional kernels with strides of 1 and 2, a 5-layer recurrent block structure, a first 3×3 convolutional layer with a stride of 1, a BN layer, and an end network. The output of each layer of the 5-layer recurrent block structure is connected to the BN layer and accumulated before being input into the end network. The end network has a first 3×3 convolutional layer with a stride of 2, a first PReLU layer, a second 3×3 convolutional layer with a stride of 2, a second PReLU layer, and a third 3×3 convolutional layer.

[0024] As an optional approach, the first Gaussian blur kernel adopts both isotropic and anisotropic Gaussian blur kernels;

[0025] The RGB three-channel weighted downsampling is determined by theoretical derivation to determine the weight parameters when downsampling the RGB three channels, and multiplicative weight coefficients are added. The weight coefficients are adjusted during network training.

[0026] The second Gaussian blur kernel is used to simulate the overall degradation effect of the remote sensing imaging platform, and isotropic Gaussian blur kernels with different parameter settings are used.

[0027] The noise used is Gaussian white noise.

[0028] As an alternative approach, the generator's network structure adopts a U-shaped architecture, which extracts cross-scale information through continuous downsampling and upsampling and supplements the cross-scale information into the super-resolution image reconstruction.

[0029] Thirdly, embodiments of the present invention provide a computer device, comprising:

[0030] At least one processor; and

[0031] A memory communicatively connected to the at least one processor; wherein,

[0032] The memory stores instructions that can be executed by the at least one processor, which enables the at least one processor to perform the image super-resolution reconstruction method described above.

[0033] Fourthly, in this embodiment of the invention, a non-transitory computer-readable storage medium storing computer instructions is provided, the computer instructions being used to cause the computer to execute the above-described image super-resolution reconstruction method.

[0034] This invention provides an image super-resolution reconstruction method, apparatus, computer equipment, and storage medium, comprising two parts: a degradation model and a super-resolution reconstruction network. The degradation model serves as a general prior knowledge constraint structure for current deep learning-based super-resolution methods, fully considering the unique degradation process of remote sensing imaging compared to natural image imaging. It selects atmospheric scattering degradation effects and the combined degradation effects of the remote sensing imaging platform, transforming them into specific degradation algorithms incorporated into the degradation model. A plug-and-play network framework in the form of a "degrader-generator" is adopted to achieve unsupervised super-resolution, making it more convenient for practical applications. Generalization ability and robustness are enhanced, and it is also easier to add more degradation algorithms to the degradation model in the future. Secondly, a super-resolution reconstruction network is designed, comprising a degrader and a generator. The degrader can learn and integrate the prior knowledge of degradation contained in the degradation model, which is used as a constraint for generator training. The main task of the generator is to generate high-quality super-resolution images under the constraints of the degrader, completing the reconstruction with advantages of good visual effects and high fidelity. Attached Figure Description

[0035] Figure 1 This is a flowchart of an image super-resolution reconstruction method provided in an embodiment of the present invention;

[0036] Figure 2 This is a schematic diagram illustrating the composition of a degradation model in an image super-resolution reconstruction method provided in an embodiment of the present invention;

[0037] Figure 3 This is a schematic diagram illustrating the framework and implementation process of an image super-resolution reconstruction method provided in an embodiment of the present invention;

[0038] Figure 4 This is a schematic diagram of the degrader portion of the network structure in an image super-resolution reconstruction method provided in this embodiment of the invention;

[0039] Figure 5This is a schematic diagram of the generator part of the network structure in an image super-resolution reconstruction method provided in an embodiment of the present invention;

[0040] Figure 6 This is a schematic diagram illustrating the effect of a comparative experiment between an image super-resolution reconstruction method and other methods in an embodiment of the present invention.

[0041] Figure 7 This is a structural block diagram of an image super-resolution reconstruction device provided in an embodiment of the present invention;

[0042] Figure 8 This invention provides a structural block diagram of a computer device in an embodiment of the invention. Detailed Implementation

[0043] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0044] The terms "first," "second," "third," "fourth," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0045] Combination Figure 1 As shown, this embodiment of the invention provides an image super-resolution reconstruction method, including:

[0046] S101. A degradation model for image degradation is pre-configured, and a high-resolution HR-low-resolution LR image pair is generated using the degradation model. The degradation model has a first Gaussian blur kernel, RGB three-channel weighted downsampling, a second Gaussian blur kernel, and noise.

[0047] Combination Figure 2As shown, the degraded content unique to the image imaging process is selected, extracted, and transformed to obtain a degraded algorithm. This algorithm is then incorporated into a classic degraded model to obtain the degraded model proposed in this invention. As a general prior knowledge constraint structure for current deep learning-based super-resolution methods, the degraded model consists of multiple degraded algorithms. Studying remote sensing image super-resolution from the perspective of degraded mechanism, it is precisely by supplementing more degraded algorithms that conform to the real remote sensing imaging process that the network can obtain better results when reconstructing remote sensing images. Specifically, the First Blur kernel uses both isotropic and anisotropic Gaussian blur kernels; the RGB three-channel weighted downsampling is determined by theoretical derivation, with multiplicative weight coefficients added, and the weight coefficients are adjusted during network training; the Second Blur kernel is used to simulate the comprehensive degradation effect of the remote sensing imaging platform, using isotropic Gaussian blur kernels with different parameter settings. By separating the First and Second Blur kernels, the degradation simulation can be made more refined and closer to the real degradation process; the noise used is Gaussian white noise.

[0048] S102. A super-resolution network is constructed using plug-and-play PNP technology. The super-resolution network includes a degrader, a generator, and a loss function. The generator is used to generate super-resolution images, and the degrader is used to constrain the generation process of the generator based on degraded prior information.

[0049] Specifically, in combination Figure 3 As shown, the super-resolution network adopts a plug-and-play (PNP) construction approach, comprising two parts: a degrader and a generator. The relationship between the generator, degrader, and degrader model in the overall super-resolution method is as follows: Figure 3 The demonstration showcases a plug-and-play network framework in the form of a "degrader-generator," making unsupervised super-resolution more convenient for practical applications. Its generalization ability and robustness are enhanced, and it also facilitates the addition of more degrade algorithms to the degraded model in the future. The degrader plays two roles: first, it learns prior knowledge from the degraded model; second, it constrains the super-resolution process of the generator. The generator is the main structure of the super-resolution task, gradually reconstructing better super-resolution images through iterative training and parameter updates.

[0050] Combination Figure 4As shown, the generator is used to generate super-resolution images, and the degrader is used to constrain the generator's generation process based on degraded prior information. The degrader can employ a CNN network and construct a supplementary information transmission channel in the form of skip connections. The degrader has two types of 3×3 convolutional kernels with strides of 1 and 2, a 5-layer recurrent block structure, a first 3×3 convolutional layer with a stride of 1, a BN layer, and an end network. The output of each layer of the 5-layer recurrent block structure is connected to the BN layer and accumulated before being input into the end network. The end network has a first 3×3 convolutional layer with a stride of 2, a first PReLU layer, a second 3×3 convolutional layer with a stride of 2, a second PReLU layer, and a third 3×3 convolutional layer with a stride of 1.

[0051] S103. The degradation processor is trained using the HR-LR image, and the degradation information in the degradation model is transmitted to the degradation processor.

[0052] Using a degradation model on the original remote sensing image I origin Degradation processing yields synthetic dataset I degra Then use the resulting synthetic dataset I degra The degrader is trained. Within this framework, the degrader is trained first, and then the generator is trained using the degrader as a constraint. A synthetic dataset I is created using the degrader model. degra Synthetic dataset I degra It contains the corresponding HR-LR image pairs and uses the synthetic dataset I degra The degrader is trained by passing the degradation information from the degradation model to the degrader, and then the generator is trained by constraining the degrader to make the first super-resolution image I... SR Gradually moving closer to HR.

[0053] S104, Transfer the super-resolution target image I target After performing bicubic interpolation, the first super-resolution image I is obtained by inputting it into the generator. SR The first resolution image I SR The super-resolution synthesized image is obtained after inputting into the de-qualityr. The super-resolution target image I is calculated using the loss function. target and the super-resolution synthesized image The loss value is used to constrain the generator.

[0054] Specifically, in combination Figure 5As shown, the generator's network structure adopts a U-shaped architecture, extracting cross-scale information through continuous downsampling and upsampling, and then supplementing this cross-scale information into the super-resolution image reconstruction. It includes a downsampling part and an upsampling part, and incorporates a skip connection module to enhance feature extraction. It should be noted that the original remote sensing image I input to the generator network... origin After interpolation at a specified magnification, the image is input into the network. Through downsampling and upsampling feature extraction and reconstruction, the resulting super-resolution image has the same size as the input interpolated image. Then, the first resolution image I... SR Input the pre-trained de-qualityr to obtain a super-resolution synthesized image. The desaturator has learned a large amount of prior desaturation information from the synthetic dataset, which is equivalent to learning the original remote sensing image I. origin and synthetic dataset I degra The mapping relationship between them is thus established, and therefore the loss function is applied to the super-resolution synthesized image. and the original remote sensing image I origin By calculating the loss, we can constrain I. SR Approaching I origin This enables super-resolution reconstruction under the constraint of the degradation device.

[0055] S105. Under the constraint of the degradation unit, the generator generates a super-resolution image and completes the reconstruction.

[0056] The network for training a degrader is trained using HR-LR images. High-resolution original images from the dataset are input into the degrader network, randomly cropped into small HR patches, and then input into the degrader model to obtain corresponding LR patches. The training of the degrader network using HR-LR images ensures that the degradation information in the degrader model is uniformly integrated into the degrader. This allows the degrader to simulate the degradation process from ground image information to remote sensing platform imaging to the greatest extent possible. The constraint process of the degrader is achieved as follows: a generator generates a super-resolution image, which is input into the degrader to obtain a low-resolution image I. LR Then the low-resolution image I LR Image I with target resolution targrt Calculate the loss; the smaller the loss, the lower the resolution of the image I. LE The closer to the low-resolution LR, the better the super-resolution image I becomes. SR The closer the quality is to the high-resolution (HR) image, the better. The generator network is trained, and a degrader is connected after the generator as a constraint: under the constraint of the degrader, through repeated iterative training and parameter updates, the generator produces super-resolution images that are closer in quality to the original high-resolution image.

[0057] This invention provides an image super-resolution reconstruction method, comprising a degradation model and a super-resolution reconstruction network. The degradation model serves as a general prior knowledge constraint structure for current deep learning-based super-resolution methods, fully considering the unique degradation process of remote sensing imaging compared to natural image imaging. It selects atmospheric scattering degradation effects and the combined degradation effects of the remote sensing imaging platform, transforming them into specific degradation algorithms incorporated into the degradation model. A plug-and-play network framework in the form of a "degrader-generator" is adopted, making unsupervised super-resolution more convenient for practical applications, enhancing generalization ability and robustness, and facilitating the addition of more degradation algorithms to the degradation model in the future. Secondly, a super-resolution reconstruction network is designed, comprising a degrader and a generator. The degrader learns and integrates the prior knowledge of degradation contained in the degradation model, serving as a constraint for generator training. The generator's main task is to generate high-quality super-resolution images under the constraints of the degrader, completing the reconstruction with advantages of good visual effects and high fidelity.

[0058] The main steps of the image super-resolution reconstruction method provided in this embodiment of the invention include: (1) setting up a PyTorch environment; (2) preparing a dataset; (3) building and training a network; and (4) obtaining and analyzing experimental results. The comparative experimental results show that the super-resolution images obtained using our method can achieve better evaluation metrics, and the visual effect and fidelity are also the best. The specific implementation includes the following steps:

[0059] Step 1: Setting up the environment

[0060] The hardware platform of this invention is based on an Intel i9-10920x processor with 24 cores, an NVIDIA RTX 3090 24G×2 GPU, and 64G of memory. The experimental platform is based on Ubuntu 22.04, using CUDA 11.3 and PyTorch 1.10 environment, and uses Xavier initialization. Those skilled in the art can flexibly choose other options, and there is no limitation on this.

[0061] Step 2: Prepare the dataset

[0062] The UC Merced Land Use Dataset and NWPU-RESIS45 remote sensing datasets were combined after removing a few images. The datasets contain RGB images, and 15 images were selected for testing, while the remaining images were used for network training. The UC Merced Land Use Dataset images have a pixel resolution of 1 inch and a pixel size of 256*256 pixels, containing 21 scene classes with 100 images per class, totaling 2100 images. The WPU-RESIS45 dataset, created by Northwestern Polytechnical University (NWPU), contains 31,500 images across 45 scenes, with 700 images per scene. It should be noted that dataset selection is optional and not restricted.

[0063] Step 3: Build and train the network

[0064] Remote sensing images are processed using a degradation model to obtain corresponding HR-LR image pairs. The degradation network is then trained using these HR-LR image pairs, integrating the degradation information from the degradation model into the degradation network. This allows the degradation network to simulate the degradation process from ground image information to remote sensing platform imaging to the greatest extent possible. A generator network is then trained, and the degradation network is connected to the generator as a constraint. Under the constraint of the degradation network, through repeated iterative training and parameter updates, the generator produces super-resolution images that are closer in quality to the original high-resolution images.

[0065] Step 4: Experimental Results and Analysis

[0066] Experimental results are as follows Figure 6 As shown, under fair comparison conditions, our experimental method, at 4× super-resolution, ranks second only to RWDM for PNSR, while the results of SSIM and SAM of this invention are superior to all compared supervised and unsupervised methods. At 8× super-resolution, the conclusions are also consistent; comparative experimental results demonstrate that the super-resolution images obtained using the methods provided in this embodiment of the invention can achieve better evaluation metrics, and the visual effect and fidelity are also optimal.

[0067] Combination Figure 7 As shown, correspondingly, this embodiment of the invention provides an image super-resolution reconstruction apparatus, comprising:

[0068] Training dataset unit 601 is used to pre-configure a degradation model for image degradation, which generates high-resolution HR-low-resolution LR image pairs. The degradation model has a first Gaussian blur kernel, RGB three-channel weighted downsampling, a second Gaussian blur kernel, and noise.

[0069] The construction unit 602 is used to construct a super-resolution network using plug-and-play PNP technology. The super-resolution network includes a degrader, a generator, and a loss function. The generator is used to generate a super-resolution image, and the degrader is used to constrain the generation process of the generator based on degraded prior information.

[0070] Training unit 603 is used to train the degrader using the HR-LR image and to transfer the degradation information in the degradation model to the degrader;

[0071] Loss calculation unit 604 is used to calculate the super-resolution target image I target After performing bicubic interpolation, the first super-resolution image I is obtained by inputting it into the generator. SR The first resolution image I SR The super-resolution synthesized image is obtained after inputting into the de-qualityr. The super-resolution target image I is calculated using the loss function. target and the super-resolution synthesized image The loss value is used to constrain the generator;

[0072] The reconstruction unit 605 is used to generate a super-resolution image by the generator under the constraint of the degrader, thereby completing the reconstruction.

[0073] Specifically, the degrader uses a CNN network and constructs a supplementary information transmission channel in the form of skip connections. The degrader has two 3×3 convolutional kernels with strides of 1 and 2, a 5-layer recurrent block structure, a first 3×3 convolutional layer with a stride of 1, a BN layer, and an end network. The output of each layer of the 5-layer recurrent block structure is connected to the BN layer and accumulated before being input into the end network. The end network has a first 3×3 convolutional layer with a stride of 2, a first PReLU layer, a second 3×3 convolutional layer, a second PReLU layer, and a third 3×3 convolutional layer.

[0074] Specifically, the first Gaussian blur kernel uses both isotropic and anisotropic Gaussian blur kernels; the RGB three-channel weighted downsampling is determined by theoretical derivation to determine the weight parameters during RGB three-channel downsampling, and multiplicative weight coefficients are added, with the weight coefficients adjusted during network training; the second Gaussian blur kernel is used to simulate the comprehensive degradation effect of the remote sensing imaging platform, and isotropic Gaussian blur kernels with different parameter settings are used; the noise used is Gaussian white noise.

[0075] Specifically, the generator's network structure adopts a U-shaped architecture, which extracts cross-scale information through continuous downsampling and upsampling and supplements the cross-scale information into the super-resolution image reconstruction.

[0076] This invention provides an image super-resolution reconstruction device, comprising a degradation model and a super-resolution reconstruction network. The degradation model serves as a general prior knowledge constraint structure for current deep learning-based super-resolution methods, fully considering the unique degradation process of remote sensing imaging compared to natural image imaging. It selects atmospheric scattering degradation effects and the combined degradation effects of the remote sensing imaging platform, transforming them into specific degradation algorithms incorporated into the degradation model. A plug-and-play network framework in the form of a "degrader-generator" is adopted, making unsupervised super-resolution more convenient for practical applications, enhancing generalization ability and robustness, and facilitating the addition of more degradation algorithms to the degradation model in the future. Secondly, a super-resolution reconstruction network is designed, comprising a degrader and a generator. The degrader learns and integrates the prior knowledge of degradation contained in the degradation model, which serves as a constraint for generator training. The generator's main task is to generate high-quality super-resolution images under the constraints of the degrader, completing the reconstruction with advantages of good visual effects and high fidelity.

[0077] Accordingly, according to embodiments of the present invention, the present invention also provides a computer device, a readable storage medium, and a computer program product.

[0078] Figure 8 This is a schematic diagram of the structure of a computer device 12 provided in an embodiment of the present invention. Figure 8 A block diagram of an exemplary computer device 12 suitable for implementing embodiments of the present invention is shown. Figure 8 The computer device 12 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.

[0079] like Figure 8 As shown, computer device 12 is represented in the form of a general-purpose computing device. Computer device 12 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0080] The components of the computer device 12 may include, but are not limited to: one or more processors or processing units 16, system memory 28, and bus 18 connecting different system components (including system memory 28 and processing unit 16).

[0081] Bus 18 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. For example, these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0082] Computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by computer device 12, including volatile and non-volatile media, removable and non-removable media.

[0083] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory. Computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be used to read and write non-removable, non-volatile magnetic media (…). Figure 8 Not shown; usually referred to as a "hard drive"). Although Figure 8 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. Memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of the present invention.

[0084] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 42 typically perform the functions and / or methods described in the embodiments of the present invention.

[0085] Computer device 12 can also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), and with one or more devices that enable a user to interact with computer device 12, and / or with any device that enables computer device 12 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 22. Furthermore, computer device 12 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 20. As shown, network adapter 20 communicates with other modules of computer device 12 via bus 18. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with computer device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0086] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing the image super-resolution reconstruction method provided in the embodiments of the present invention.

[0087] This invention also provides a non-transitory computer-readable storage medium storing computer instructions, on which a computer program is stored, wherein the program, when executed by a processor, is the image super-resolution reconstruction method provided in all embodiments of this application.

[0088] The computer storage medium of this invention can be any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. More specific examples (a non-exhaustive list) of computer-readable storage media include: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0089] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0090] The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof. The computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0091] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the image super-resolution reconstruction method described above.

[0092] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this invention can be achieved, and this is not limited herein.

[0093] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. An image super-resolution reconstruction method, characterized in that, include: A pre-configured degradation model for image degradation is used to generate high-resolution HR-low-resolution LR image pairs. The degradation model has a first Gaussian blur kernel, RGB three-channel weighted downsampling, a second Gaussian blur kernel, and noise. A super-resolution network is constructed using plug-and-play PNP technology. The super-resolution network includes a degrader, a generator, and a loss function. The generator is used to generate super-resolution images, and the degrader is used to constrain the generator's generation process based on degraded prior information. The degradation processor is trained using the HR-LR images, and the degradation information in the degradation model is transferred to the degradation processor. Super-resolution target image After performing bicubic interpolation, the image is input into the generator to obtain the first super-resolution image. The first super-resolution image The super-resolution synthesized image is obtained after inputting into the de-qualityr. The super-resolution target image is calculated using the loss function. and the super-resolution synthesized image The loss value is used to constrain the generator; Under the constraint of the degrader, the generator generates a super-resolution image and completes the reconstruction. The first Gaussian blur kernel uses both isotropic and anisotropic Gaussian blur kernels; The RGB three-channel weighted downsampling is determined by theoretical derivation to determine the weight parameters when downsampling the RGB three channels, and multiplicative weight coefficients are added to adjust the weight coefficients during network training. The second Gaussian blur kernel is used to simulate the overall degradation effect of the remote sensing imaging platform, and isotropic Gaussian blur kernels with different parameter settings are used. The noise used is Gaussian white noise.

2. The image super-resolution reconstruction method according to claim 1, characterized in that, The degrader employs a CNN network and constructs supplementary information transmission channels using skip connections. The degrader has two steps with strides of 1 and 2. Convolutional kernel, 5-layer recurrent block structure, stride of 1 The five-layer recurrent block structure, consisting of convolutional layers, batch normalization (BN) layers, and an end network, has its output connected to the BN layer, accumulated, and then input into the end network. The end network has a first step size of 2. Convolutional layer, first PReLU layer, stride of 2 Convolutional layer, second PReLU layer and third layer with stride 1 Convolutional layer.

3. The image super-resolution reconstruction method according to claim 1, characterized in that, The generator's network structure adopts a U-shaped architecture, which extracts cross-scale information through continuous downsampling and upsampling and supplements the cross-scale information into the super-resolution image reconstruction.

4. An image super-resolution reconstruction device, characterized in that, include: The training dataset unit is used to pre-configure a degradation model for image degradation, which generates high-resolution HR-low-resolution LR image pairs. The degradation model has a first Gaussian blur kernel, RGB three-channel weighted downsampling, a second Gaussian blur kernel, and noise. A construction unit is used to construct a super-resolution network using plug-and-play PNP technology. The super-resolution network includes a degrader, a generator, and a loss function. The generator is used to generate a super-resolution image, and the degrader is used to constrain the generation process of the generator based on degraded prior information. The training unit is used to train the degrader using the HR-LR images and to transfer the degradation information in the degradation model to the degrader. Loss calculation unit, used to calculate the super-resolution target image After performing bicubic interpolation, the image is input into the generator to obtain the first super-resolution image. The first super-resolution image The super-resolution synthesized image is obtained after inputting into the de-qualityr. The super-resolution target image is calculated using the loss function. and the super-resolution synthesized image The loss value is used to constrain the generator; The reconstruction unit is used to generate a super-resolution image under the constraint of the degrader, thereby completing the reconstruction. The first Gaussian blur kernel uses both isotropic and anisotropic Gaussian blur kernels; The RGB three-channel weighted downsampling is determined by theoretical derivation to determine the weight parameters when downsampling the RGB three channels, and multiplicative weight coefficients are added to adjust the weight coefficients during network training. The second Gaussian blur kernel is used to simulate the overall degradation effect of the remote sensing imaging platform. Isotropic Gaussian blur kernels with different parameter settings are used. The noise used is Gaussian white noise.

5. The image super-resolution reconstruction apparatus according to claim 4, characterized in that, The degrader employs a CNN network and constructs supplementary information transmission channels using skip connections. The degrader has two steps with strides of 1 and 2. Convolutional kernel, 5-layer recurrent block structure, first step with stride of 1 The five-layer recurrent block structure, consisting of convolutional layers, batch normalization (BN) layers, and an end network, has its output connected to the BN layer, accumulated, and then input into the end network. The end network has a first step size of 2. Convolutional layer, first PReLU layer, stride of 2 Convolutional layer, second PReLU layer and third layer with stride 1 Convolutional layer.

6. The image super-resolution reconstruction apparatus according to claim 4, characterized in that, The generator's network structure adopts a U-shaped architecture, which extracts cross-scale information through continuous downsampling and upsampling and supplements the cross-scale information into the super-resolution image reconstruction.

7. A computer device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the image super-resolution reconstruction method according to any one of claims 1 to 3.

8. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the image super-resolution reconstruction method according to any one of claims 1 to 3.

Images