Super-resolution image reconstruction method and system, electronic equipment and storage medium

Abstract

The invention discloses a super-resolution image reconstruction method and system, electronic equipment and a storage medium, and relates to the technical field of image processing, and the method comprises the steps: constructing a super-resolution image reconstruction model, training the super-resolution image reconstruction model, and obtaining a trained super-resolution image reconstruction model; and reconstructing the low-resolution target image by using the trained super-resolution image reconstruction model to obtain a super-resolution image corresponding to the low-resolution target image. According to the super-resolution image reconstruction method, local information and global context information can be effectively fused through the information fusion unit, and the synergistic effect among all parts in the super-resolution image reconstruction model is enhanced by combining complementary features, so that the overall performance of the reconstructed super-resolution image is remarkably improved, and the image quality of the reconstructed super-resolution image is improved.

Description

Background Art

[0002] With the continuous breakthroughs in remote sensing technology, the application areas of satellite and aerial images are constantly expanding, such as precision agricultural monitoring, natural resource management, and intelligent transportation planning. However, due to the physical constraints of imaging sensors, the complexity of atmospheric conditions, and the limitations of satellite data transmission bandwidth, the resolution of currently acquired remote sensing images is difficult to meet the needs of many high-precision applications. At the same time, the existence of a large number of low-resolution (LR) images also limits the in-depth mining and utilization of earth observation data. Therefore, the development of efficient super-resolution (SR) technology to reconstruct high-resolution (HR) details from low-resolution images has become a research hotspot in the field of remote sensing image processing. As the core direction of this technology, remote sensing image super-resolution can not only make up for the lack of resolution, but also further reveal the key details of surface targets, providing technical support for high-precision information extraction and complex scene analysis.

[0003] In recent years, with the rapid development of deep learning technology, various deep neural network-based architectures have been widely used in the research of remote sensing image super-resolution. Among them, convolutional neural network (CNN) has become the mainstream choice of traditional super-resolution methods due to its superior local feature extraction ability and efficient computing performance. Early models such as SRCNN, VDSR and RCAN have significantly improved the reconstruction quality of super-resolution by continuously deepening the number of network layers and optimizing structural design. However, the effective receptive field problem of these models limits their ability to capture long-range dependencies, especially in large-scale remote sensing scenes that require a wide range of contextual information.

[0004] To solve this problem, methods based on the Transformer architecture have gradually emerged in recent years. Through the self-attention mechanism, Transformer can efficiently model global dependencies and has achieved excellent performance in a variety of computer vision tasks. For example, models such as SwinIR achieve high reconstruction accuracy in remote sensing image super-resolution tasks through local window attention mechanisms and multi-scale feature fusion. However, due to the quadratic complexity of the self-attention mechanism, these models face significant computational cost bottlenecks when processing high-resolution satellite and aerial images. At the same time, although the local windowing strategy reduces the complexity to a certain extent, it also reduces the receptive field, resulting in a decrease in the ability to capture long-distance dependencies.

[0005] In addition to the Transformer architecture, methods based on the state space model (SSM) have also gradually attracted the attention of researchers. Mamba is a typical SSM model that captures long-range dependencies with linear computational complexity, making it an efficient alternative to the Transformer. Studies have shown that Mamba can achieve or even exceed the performance of the Transformer in some scenarios. However, Mamba still faces limitations in two-dimensional image processing, such as the problem of one-way sequence modeling, which makes it insufficient in achieving the optimal receptive field.

[0006] Another emerging architecture that has attracted much attention is RWKV, an efficient model that combines the advantages of RNN and Transformer. Through innovative design, RWKV completely removes the traditional self-attention mechanism and achieves performance comparable to GPT-level large language models (LLM), while significantly reducing computational complexity and memory usage. The core of its architecture revolves around four key parameters: Receptance (R), Weight (W), Key (K), and Value (V), which are used to dynamically adjust the acceptance of historical information, position weight distribution, and information selection and representation update. RWKV not only supports parallel operations during training, has a constant inference speed and ultra-long context processing capabilities, but despite its excellent performance in language modeling and computer vision tasks, its application in remote sensing remains to be explored.

[0007] In summary, although models such as CNN, Transformer, SSM and RWKV have their own advantages, how to effectively balance the global modeling capability and computational complexity, and how to fully solve the limitations of the receptive field and apply them to remote sensing image super-resolution are still research problems that need to be overcome urgently. Summary of the invention

[0008] The technical problem to be solved by the present invention is to address the deficiencies of the prior art and specifically provide a super-resolution image reconstruction method, system, electronic device and storage medium, as follows:

[0009] 1) In the first aspect, the present invention provides a method for reconstructing a super-resolution image, and the specific technical solution is as follows:

[0010] A super-resolution image reconstruction model is constructed. The super-resolution image reconstruction model includes n global information extraction units for extracting global context information, n detail information extraction units for extracting local information, n information fusion units for integrating local information and global context information, and a post-processing module. The inputs of the first global information extraction unit and the first detail information extraction unit are shallow features extracted from the low-resolution sample image, and the shallow features are extracted from the low-resolution sample image through the first convolutional layer; the output of the i-th global information extraction unit is used as the input of the i+1-th global information extraction unit and the i-th information fusion unit respectively; the output of the i-th detail information extraction unit is used as the input of the i-th information fusion unit; the output of the i-th information fusion unit is used as the input of the i+1-th detail information extraction unit; the post-processing module processes the output of the n-th detail information extraction unit to obtain a super-resolution image corresponding to the low-resolution sample image, i and n are both positive integers, and 2≤i≤n-1;

[0011] Training the super-resolution image reconstruction model to obtain a trained super-resolution image reconstruction model;

[0012] The trained super-resolution image reconstruction model is used to reconstruct the low-resolution target image to obtain a super-resolution image corresponding to the low-resolution target image.

[0013] The beneficial effects of the super-resolution image reconstruction method provided by the present invention are as follows:

[0014] The information fusion unit can effectively fuse local information and global context information. By merging complementary features, the synergy between the various parts in the super-resolution image reconstruction model is enhanced, thereby significantly improving the overall performance of reconstructing super-resolution images and improving the image quality of the reconstructed super-resolution images.

[0015] Based on the above solution, the super-resolution image reconstruction method of the present invention can also be improved as follows.

[0016] Further, the post-processing module includes a second convolutional layer, a third convolutional layer, an upsampling layer, and a fourth convolutional layer;

[0017] The second convolutional layer is used to receive and process the output of the nth detail information extraction unit, add the output of the second convolutional layer to the shallow features extracted from the low-resolution sample image, and process the addition result through the third convolutional layer, the upsampling layer and the fourth convolutional layer in sequence to obtain a super-resolution image corresponding to the low-resolution target image.

[0018] Furthermore, each information fusion unit includes two information reorganization units; the information fusion unit is specifically used for: the output of the global information extraction unit and the output of the detail information extraction unit are processed by an information reorganization unit respectively, and then added as the output of the information fusion unit.

[0019] Furthermore, both the low-resolution sample image and the low-resolution target image are remote sensing images.

[0020] 2) In a second aspect, the present invention also provides a super-resolution image reconstruction system, and the specific technical solution is as follows:

[0021] It includes a model building module, a model training module and a super-resolution image reconstruction module;

[0022] The model construction module is used to: construct a super-resolution image reconstruction model, which includes n global information extraction units for extracting global context information, n detail information extraction units for extracting local information, n information fusion units for integrating local information and global context information, and a post-processing module. The inputs of the first global information extraction unit and the first detail information extraction unit are shallow features extracted from the low-resolution sample image, and the shallow features are extracted from the low-resolution sample image through the first convolutional layer; the output of the i-th global information extraction unit is used as the input of the i+1-th global information extraction unit and the i-th information fusion unit respectively; the output of the i-th detail information extraction unit is used as the input of the i-th information fusion unit; the output of the i-th information fusion unit is used as the input of the i+1-th detail information extraction unit; the post-processing module processes the output of the n-th detail information extraction unit to obtain a super-resolution image corresponding to the low-resolution sample image, i and n are both positive integers, and 2≤i≤n-1;

[0023] The model training module is used to: train the super-resolution image reconstruction model to obtain a trained super-resolution image reconstruction model;

[0024] The super-resolution image reconstruction module is used to reconstruct the low-resolution target image using the trained super-resolution image reconstruction model to obtain a super-resolution image corresponding to the low-resolution target image.

[0025] Based on the above solution, the super-resolution image reconstruction system of the present invention can also be improved as follows.

[0026] Further, the post-processing module includes a second convolutional layer, a third convolutional layer, an upsampling layer, and a fourth convolutional layer;

[0027] The second convolutional layer is used to receive and process the output of the nth detail information extraction unit, add the output of the second convolutional layer to the shallow features extracted from the low-resolution sample image, and process the addition result through the third convolutional layer, the upsampling layer and the fourth convolutional layer in sequence to obtain a super-resolution image corresponding to the low-resolution target image.

[0028] Furthermore, each information fusion unit includes two information reorganization units; the information fusion unit is specifically used for: the output of the global information extraction unit and the output of the detail information extraction unit are processed by an information reorganization unit respectively, and then added as the output of the information fusion unit.

[0029] Furthermore, both the low-resolution sample image and the low-resolution target image are remote sensing images.

[0030] 3) In a third aspect, the present invention further provides an electronic device, comprising a processor, the processor being coupled to a memory, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor, so that the electronic device implements any of the above-mentioned super-resolution image reconstruction methods.

[0031] 4) In a fourth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, any of the above-mentioned super-resolution image reconstruction methods is implemented.

[0032] It should be noted that the beneficial effects achieved by the technical solutions of the second to fourth aspects of the present invention and the corresponding possible implementation methods can be found in the above-mentioned technical effects of the first aspect and its corresponding possible implementation methods, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments of the present invention:

[0034] Figure 1 A schematic diagram of a process for reconstructing a super-resolution image according to an embodiment of the present invention;

[0035] Figure 2 Schematic diagram of the network structure of the super-resolution image reconstruction model;

[0036] Figure 3 It is a schematic diagram of the network structure of the detail information extraction unit;

[0037] Figure 4 It is a schematic diagram of the network structure of the global information extraction unit;

[0038] Figure 5It is a schematic diagram of the data processing process of the information reorganization unit;

[0039] Figure 6 A schematic diagram of the structure of a super-resolution image reconstruction system according to an embodiment of the present invention;

[0040] Figure 7 The figure is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The principles and features of the present invention are described below. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0042] The technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0043] like Figure 1 As shown, a super-resolution image reconstruction method according to an embodiment of the present invention comprises the following steps:

[0044] S1. Build a super-resolution image reconstruction model;

[0045] Among them, Figure 2 As shown in FIG. 1 , the super-resolution image reconstruction model (CNN-RWKVDeepSR, the super-resolution image reconstruction model is a remote sensing image super-resolution framework based on deep learning using CNN and RWKV models) includes n global information extraction units for extracting global context information, n detail information extraction units for extracting local information, n information fusion units for integrating local information and global context information, and a post-processing module. The inputs of the first global information extraction unit and the first detail information extraction unit are shallow features extracted from the low-resolution sample image. The shallow features are obtained by the first convolution. The layer is extracted from the low-resolution sample image; the output of the ith global information extraction unit is used as the input of the ith + 1th global information extraction unit and the ith information fusion unit respectively; the output of the ith detail information extraction unit is used as the input of the ith information fusion unit; the output of the ith information fusion unit is used as the input of the ith + 1th detail information extraction unit; the post-processing module processes the output of the nth detail information extraction unit to obtain a super-resolution image corresponding to the low-resolution sample image, i and n are both positive integers, and 2≤i≤n-1, n can be 4, 5 or 6, etc., which can be set according to actual conditions.

[0046] Figure 2In the model, n global information extraction units for extracting global context information form a global information extraction group, n detail information extraction units for extracting local information form a detail information extraction group, and n information fusion units for integrating local information and global context information form an information fusion group. The global extraction branch emphasizes broader structures and smoothes noise, while the local extraction branch focuses on fine-grained textures and edges. These distinct characteristics make the combination of the global extraction branch and the local extraction branch particularly beneficial for improving model performance.

[0047] Among them, Figure 2 As shown, the post-processing module includes a second convolution layer, a third convolution layer, an upsampling layer and a fourth convolution layer;

[0048] The second convolutional layer is used to receive and process the output of the nth detail information extraction unit, add the output of the second convolutional layer to the shallow features extracted from the low-resolution sample image, and process the addition result through the third convolutional layer, the upsampling layer and the fourth convolutional layer in sequence to obtain a super-resolution image corresponding to the low-resolution target image.

[0049] Among them, Figure 3 As shown, each detail information extraction unit includes multiple CNN units arranged in sequence. In every two adjacent CNN units, the output of the previous CNN unit is used as the input of the next CNN unit. The number of CNN units included in each detail information extraction unit is 4, 8 or 12, etc., which can be set according to actual conditions.

[0050] Each CNN unit includes: a fifth convolutional layer, a first LReLU layer and a sixth convolutional layer arranged in sequence. The fifth convolutional layer of the first CNN unit in the i+1th detail information extraction unit receives the output of the i-th global information extraction unit, and after being processed by the first LReLU layer and the sixth convolutional layer in sequence, the product of the output of the i-th global information extraction unit and the parameter α to be trained is added to the output processed by the sixth convolutional layer, and the addition result is used as the input of the fifth convolutional layer of the second CNN unit in the i+1th detail information extraction unit. The output processed by the sixth convolutional layer of the last CNN unit in the i+1th detail information extraction unit is the output in the i+1th detail information extraction unit.

[0051] Among them, the initial value of the parameter α to be trained in each CNN unit can be set to 1 and will be adjusted during the training process.

[0052] Among them, Figure 4As shown, each global information extraction unit includes multiple RWKV units arranged in sequence. In every two adjacent RWKV units, the output of the previous RWKV unit is used as the input of the next RWKV unit. After the input of the last RWKV unit is processed by the seventh convolutional layer, the output of the seventh convolutional layer is added to the input of the first RWKV unit as the output of the corresponding global information extraction unit. The number of RWKV units included in each global information extraction unit can be 4, 6 or 8, etc., which can be set according to actual conditions. Figure 4 In the above equation, α1 and α2 are parameters to be trained.

[0053] S2. Train the super-resolution image reconstruction model to obtain a trained super-resolution image reconstruction model.

[0054] S3. Reconstruct the low-resolution target image using the trained super-resolution image reconstruction model to obtain a super-resolution image corresponding to the low-resolution target image.

[0055] Optionally, in the above technical solution, each information fusion unit includes two information reorganization units; the information fusion unit is specifically used for: the output of the global information extraction unit and the output of the detail information extraction unit are processed by an information reorganization unit respectively, and then added as the output of the information fusion unit.

[0056] That is to say, the information fusion unit is specifically used to: output F of the global information extraction unit R and the output F of the detail information extraction unit C After being processed by an information reorganization unit, the reorganized output F is obtained. R1 and F C1 , and then add them together to get F CR As the output of the information fusion unit.

[0057] Take the i-th information fusion unit as an example to illustrate:

[0058] The output F of the i-th global information extraction unit R After being processed by the information reorganization unit, the feature F is obtained. R1 , the output F of the i-th detail information extraction unit C After being processed by the information reorganization unit, the feature F is obtained. C1 , by putting F R1 and F C1 Add them together and we get F CR is the output of the i-th information fusion unit, such as Figure 5 shown.

[0059] The data processing process of the information reorganization unit is as follows:

[0060] like Figure 5 As shown, the output F of the i-th global information extraction unit is R As the input of the information reorganization unit, the information reorganization unit will input F R Through three treatments, specifically:

[0061] 1) First treatment: F R After permutation, it is sent to the first pooling layer. Figure 5 In the figure, P represents the permutation process. The first pooling layer performs maximum pooling and average pooling on the permuted features, and then processes them through the eighth convolutional layer and the Sigmoid function. The result obtained by the processing is multiplied by the input of the first pooling layer, and then the result of the first processing is obtained through the permutation process.

[0062] 2) Second treatment: F R After the permutation processing, it is sent to the second pooling layer. The second pooling layer performs maximum pooling and average pooling on the features after the permutation processing, and then processes it through the ninth convolutional layer and the Sigmoid function. The processed result is multiplied by the input of the second pooling layer, and then the result of the second processing is obtained through the permutation processing.

[0063] 3) The third treatment: F R Directly sent to the third pooling layer, the third pooling layer for F R Perform maximum pooling and average pooling, then process through the tenth convolutional layer and the Sigmoid function, and multiply the result obtained by the processing with the input of the third pooling layer to obtain the result of the third processing;

[0064] Then add the results of the three processes and multiply them by W to get the output of the information reorganization unit. Among them, α3 is a coefficient, and the value of α3 is It can also be set according to actual conditions.

[0065] Optionally, in the above technical solution, both the low-resolution sample image and the low-resolution target image are remote sensing images.

[0066] In the process of training the super-resolution image reconstruction model, bicubic downsampling and random mixing degradation model are used to simulate LR images. Specifically:

[0067] Based on the characteristics of existing blind super-resolution methods and remote sensing sensors, the bicubic downsampling and random mixed degradation model includes multiple processing such as isotropic Gaussian blur, motion blur, scaling with different interpolation methods, additive Gaussian noise and JPEG compression noise on the LR image to generate a low-resolution sample image corresponding to the LR image for input into the super-resolution image reconstruction model. The execution order of isotropic Gaussian blur, motion blur, scaling with different interpolation methods, additive Gaussian noise and JPEG compression noise can be randomly determined.

[0068] During training, each HR image was randomly cropped into a patch of size 192 × 192 pixels, and the size of the corresponding LR image was set to 64 × 64 pixels. The batch size was set to 16, and the Adam optimizer was used for 200 epochs with an initial learning rate of 0.0001, which was halved at halfway point in the training.

[0069] The super-resolution image reconstruction model first passes a 3×3 first convolutional layer from the low-resolution sample image I LR Extract shallow features from Among them, H and W represent the height and width of the low-resolution sample image, C represents the number of channels, and F S This is the output of the first convolutional layer.

[0070] The deep features are obtained through the parallel global information extraction unit and the detail information extraction unit respectively, and the features of the two branches are fused through the information fusion unit. The deep feature obtained by the i-th global information extraction unit is F R,i , the depth feature obtained by the i-th detail information extraction unit is F C,i .

[0071] Use global residual connections to fuse shallow features and deep features, and finally upsample to obtain a super-resolution image I SR , the SR process of CNN-RWKVDeepSR (the process of fusing shallow features and deep features) can be expressed by mathematical formula:

[0072] I SR =Up(IF(F D,R ,F D,C )+F S )

[0073] Where Up(·) represents the upsampling function, IF(·) represents the information fusion operation of the information fusion group, and F D,R and F D,C Represents two deep features input to any information fusion unit.

[0074] During the training process, the loss function used is the L1 loss function, which is:

[0075]

[0076] The trained super-resolution image reconstruction model is used to reconstruct the low-resolution target image to obtain a super-resolution image corresponding to the low-resolution target image.

[0077] Among them, the low-resolution target image is a low-resolution image specified by the user. The low-resolution target image is simulated by bicubic downsampling and a random mixing degradation model. The simulation result is input into the trained super-resolution image reconstruction model to obtain a super-resolution image corresponding to the low-resolution target image.

[0078] In the above embodiments, although the steps are numbered S1, S2, etc., these are only specific embodiments given by the present invention. Those skilled in the art may adjust the execution order of S1, S2, etc. according to actual conditions, which is also within the protection scope of the present invention. It can be understood that in some embodiments, some or all of the above embodiments may be included.

[0079] like Figure 6 As shown, a super-resolution image reconstruction system 200 according to an embodiment of the present invention includes a model building module 201, a model training module 202 and a super-resolution image reconstruction module 203;

[0080] The model construction module 201 is used to: construct a super-resolution image reconstruction model, which includes n global information extraction units for extracting global context information, n detail information extraction units for extracting local information, n information fusion units for integrating local information and global context information, and a post-processing module; the inputs of the first global information extraction unit and the first detail information extraction unit are shallow features extracted from the low-resolution sample image, and the shallow features are extracted from the low-resolution sample image through the first convolution layer; the output of the i-th global information extraction unit is used as the input of the i+1-th global information extraction unit and the i-th information fusion unit respectively; the output of the i-th detail information extraction unit is used as the input of the i-th information fusion unit; the output of the i-th information fusion unit is used as the input of the i+1-th detail information extraction unit; the post-processing module processes the output of the n-th detail information extraction unit to obtain a super-resolution image corresponding to the low-resolution sample image, i and n are both positive integers, and 2≤i≤n-1;

[0081] The model training module 202 is used to: train the super-resolution image reconstruction model to obtain a trained super-resolution image reconstruction model;

[0082] The super-resolution image reconstruction module 203 is used to reconstruct the low-resolution target image using the trained super-resolution image reconstruction model to obtain a super-resolution image corresponding to the low-resolution target image.

[0083] Optionally, in the above technical solution, the post-processing module includes a second convolution layer, a third convolution layer, an upsampling layer and a fourth convolution layer;

[0084] The second convolutional layer is used to receive and process the output of the nth detail information extraction unit;

[0085] The output of the second convolutional layer is added to the shallow features extracted from the low-resolution sample image, and the addition result is processed by the third convolutional layer, the upsampling layer and the fourth convolutional layer in sequence to obtain a super-resolution image corresponding to the low-resolution target image.

[0086] Optionally, in the above technical solution, each information fusion unit includes two information reorganization units; the information fusion unit is specifically used for: the output of the global information extraction unit and the output of the detail information extraction unit are processed by an information reorganization unit respectively, and then added as the output of the information fusion unit.

[0087] Optionally, in the above technical solution, both the low-resolution sample image and the low-resolution target image are remote sensing images.

[0088] It should be noted that the beneficial effects of the super-resolution image reconstruction system 200 provided in the above embodiment are the same as the beneficial effects of the super-resolution image reconstruction method described above, which will not be described in detail here. In addition, when the system provided in the above embodiment realizes its functions, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the system can be divided into different functional modules according to actual conditions to complete all or part of the functions described above. In addition, the system and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be described in detail here.

[0089] Among them, the super-resolution image reconstruction system of the present invention can be a computer program (including program code) running in a computer device. For example, the super-resolution image reconstruction system of the present invention is an application software that can be used to execute the corresponding steps in the super-resolution image reconstruction method of the present invention.

[0090] In some embodiments, the super-resolution image reconstruction system of the present invention can be implemented by combining software and hardware. As an example, the super-resolution image reconstruction system of the present invention can be a processor in the form of a hardware decoding processor, which is programmed to execute the super-resolution image reconstruction method of the present invention. For example, the processor in the form of a hardware decoding processor can adopt one or more application specific integrated circuits (ASIC, Application Specific Integrated Circuit), DSP, programmable logic device (PLD, Programmable Logic Device), complex programmable logic device (CPLD, Complex Programmable Logic Device), field programmable gate array (FPGA, Field-Programmable Gate Array) or other electronic components.

[0091] The modules involved in the embodiments of the present invention may be implemented in software or hardware. The name of a module does not limit the module itself in some cases.

[0092] An electronic device according to an embodiment of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, any of the above-mentioned super-resolution image reconstruction methods is implemented. That is to say, an electronic device according to an embodiment of the present invention may include but is not limited to: a processor and a memory; the memory is used to store the computer program; the processor is used to execute the super-resolution image reconstruction method shown in any embodiment of the present invention by calling the computer program.

[0093] In an alternative embodiment, an electronic device is provided, such as Figure 7 As shown, Figure 7 The electronic device 4000 shown includes: a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, such as through a bus 4002. Optionally, the electronic device 4000 may also include a transceiver 4004, which may be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present invention.

[0094] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of the present invention. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0095] The bus 4002 may include a path to transmit information between the above components. The bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 In the figure, only one thick line is used to represent bus 4002, but this does not mean that there is only one bus or one type of bus.

[0096] The memory 4003 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to this.

[0097] The memory 4003 is used to store the application code (computer program) for executing the solution of the present invention, and the execution is controlled by the processor 4001. The processor 4001 is used to execute the application code stored in the memory 4003 to implement the content shown in the above method embodiment.

[0098] Among them, the electronic device can also be a terminal device, and the terminal device can be any device that can install applications, including at least one of a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a smart TV, and a smart car device.

[0099] It should be noted that Figure 7 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0100] A computer-readable storage medium according to an embodiment of the present invention stores a computer program, and when the computer program is executed by a processor, any of the above-mentioned super-resolution image reconstruction methods is implemented.

[0101] Optionally, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0102] In an exemplary embodiment, a computer program product or a computer program is also provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device performs any of the above-mentioned super-resolution image reconstruction methods.

[0103] Computer program code for performing the operations of the present invention may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0104] It should be understood that the flow charts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the module, the program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0105] The computer-readable storage medium provided by the embodiment of the present invention may be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, device, or component.

[0106] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0107] The above description is only a preferred embodiment of the present invention and an explanation of the technical principles used. Those skilled in the art should understand that the disclosure scope involved in the present invention is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present invention (but not limited to) to form a technical solution.

[0108] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects and represent the definition of a specific order or sequence. The order of use of similar objects can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than the order shown or described.

[0109] Those skilled in the art know that the present invention can be implemented as a system, method or computer program product. Therefore, the present invention can be specifically implemented in the following forms, namely: it can be complete hardware, it can be complete software (including firmware, resident software, microcode, etc.), or it can be a combination of hardware and software, which is generally referred to as "circuit", "module" or "system" herein. In addition, in some embodiments, the present invention can also be implemented in the form of a computer program product in one or more computer-readable media, and the computer-readable medium contains computer-readable program code.

[0110] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A super-resolution image reconstruction method, characterized in that: include: Constructing a super-resolution image reconstruction model, the super-resolution image reconstruction model includes n global information extraction units for extracting global context information, n detail information extraction units for extracting local information, n information fusion units for integrating local information and global context information, and a post-processing module, the input of the first global information extraction unit and the first detail information extraction unit is a shallow feature extracted from a low-resolution sample image, and the shallow feature is extracted from the low-resolution sample image through a first convolutional layer; the output of the i-th global information extraction unit is used as the input of the i+1-th global information extraction unit and the i-th information fusion unit respectively; The output of the i-th detail information extraction unit is used as the input of the i-th information fusion unit; the output of the i-th information fusion unit is used as the input of the i+1-th detail information extraction unit; the post-processing module processes the output of the n-th detail information extraction unit to obtain a super-resolution image corresponding to the low-resolution sample image, where i and n are both positive integers and 2≤i≤n-1; Training the super-resolution image reconstruction model to obtain a trained super-resolution image reconstruction model; The trained super-resolution image reconstruction model is used to reconstruct the low-resolution target image to obtain a super-resolution image corresponding to the low-resolution target image.

2. The method for reconstructing a super-resolution image according to claim 1, characterized in that: The post-processing module includes a second convolution layer, a third convolution layer, an upsampling layer and a fourth convolution layer; The second convolution layer is used to receive and process the output of the nth detail information extraction unit, add the output of the second convolution layer to the shallow features extracted from the low-resolution sample image, and process the addition result through the third convolution layer, the upsampling layer and the fourth convolution layer in sequence to obtain a super-resolution image corresponding to the low-resolution target image.

3. The method for reconstructing a super-resolution image according to claim 1, characterized in that: Each information fusion unit includes two information reorganization units; the information fusion unit is specifically used for: the output of the global information extraction unit and the output of the detail information extraction unit are processed by an information reorganization unit respectively, and then added as the output of the information fusion unit.

4. A super-resolution image reconstruction method according to claims 1 to 3, characterized in that: The low-resolution sample image and the low-resolution target image are both remote sensing images.

5. A super-resolution image reconstruction system, characterized in that: It includes a model building module, a model training module and a super-resolution image reconstruction module; The model construction module is used to: construct a super-resolution image reconstruction model, the super-resolution image reconstruction model includes n global information extraction units for extracting global context information, n detail information extraction units for extracting local information, n information fusion units for integrating local information and global context information, and a post-processing module, the input of the first global information extraction unit and the first detail information extraction unit is a shallow feature extracted from a low-resolution sample image, and the shallow feature is extracted from the low-resolution sample image through a first convolutional layer; the output of the i-th global information extraction unit is used as the input of the i+1-th global information extraction unit and the i-th information fusion unit respectively; The output of the i-th detail information extraction unit is used as the input of the i-th information fusion unit; the output of the i-th information fusion unit is used as the input of the i+1-th detail information extraction unit; the post-processing module processes the output of the n-th detail information extraction unit to obtain a super-resolution image corresponding to the low-resolution sample image, where i and n are both positive integers and 2≤i≤n-1; The model training module is used to: train the super-resolution image reconstruction model to obtain a trained super-resolution image reconstruction model; The super-resolution image reconstruction module is used to reconstruct a low-resolution target image using the trained super-resolution image reconstruction model to obtain a super-resolution image corresponding to the low-resolution target image.

6. A super-resolution image reconstruction system according to claim 5, characterized in that: The post-processing module includes a second convolution layer, a third convolution layer, an upsampling layer and a fourth convolution layer; The second convolution layer is used to receive and process the output of the nth detail information extraction unit, add the output of the second convolution layer to the shallow features extracted from the low-resolution sample image, and process the addition result through the third convolution layer, the upsampling layer and the fourth convolution layer in sequence to obtain a super-resolution image corresponding to the low-resolution target image.

7. The super-resolution image reconstruction system according to claim 5, characterized in that: Each information fusion unit includes two information reorganization units; the information fusion unit is specifically used for: the output of the global information extraction unit and the output of the detail information extraction unit are processed by an information reorganization unit respectively, and then added as the output of the information fusion unit.

8. A super-resolution image reconstruction system according to claims 5 to 7, characterized in that: The low-resolution sample image and the low-resolution target image are both remote sensing images.

9. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, a super-resolution image reconstruction method according to any one of claims 1 to 4 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for reconstructing a super-resolution image according to any one of claims 1 to 4 is implemented.

Images