Image super-resolution reconstruction method, electronic equipment and storage medium

Abstract

The invention relates to the field of image processing, and discloses an image super-resolution reconstruction method, electronic equipment and a computer readable storage medium, and the method comprises the steps: obtaining the size of a target image and a to-be-super-divided image, initializing an image super-divided model, and loading model parameters stored in a training stage into the model, extracting pixel point information and semantic information of the to-be-hyper-divided image to obtain a target shallow-layer feature map and a target deep-layer feature map, and performing residual connection on the two feature maps to obtain a target fusion feature map; according to the size of the target image, interpolation up-sampling processing is carried out on the to-be-super-divided image and the target fusion feature map, residual connection is carried out on two images obtained through interpolation up-sampling processing to obtain a target fusion image, and finally refining denoising is carried out on the target fusion image to obtain a target super-divided image.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing, and specifically to an image super-resolution reconstruction method, an electronic device, and a computer-readable storage medium. Background Art

[0002] Image super-resolution reconstruction refers to the technology of reconstructing low-resolution images into high-resolution images using image processing, thereby effectively improving the clarity of the image. It is of great significance in the field of intelligent monitoring. Limited by hardware conditions such as the power consumption of monitoring equipment, the resolution of videos or images obtained by electronic devices will be relatively low, which will lead to poor user experience and affect the user's use of monitoring equipment. When playing back videos for important tasks (such as criminal investigation installation, etc.), it will also affect the accuracy of people's judgment of important clues. In addition, low-resolution images cannot effectively guide downstream tasks (such as target detection and target recognition, etc.), affecting the accuracy of tasks. In order to solve the above problems, image super-resolution reconstruction technology is usually applied in the field of intelligent monitoring.

[0003] The existing image super-resolution reconstruction method mainly reconstructs the low-resolution image by inputting the low-resolution image into the trained image super-resolution model to obtain the required high-resolution image. Specifically, the ideas of the existing image super-resolution models are basically the same. First, a feature extraction module is composed of a convolution layer and a residual block to extract feature information on the low-resolution image to obtain a feature image. Then, a refinement module and an upsampling module (such as a sub-pixel convolution layer PixelShuffle and a traditional interpolation upsampling based on scale) are constructed through the convolution layer to interpolate and upsample the feature image through the upsampling module to achieve super-resolution reconstruction of the image. However, the interpolation upsampling module composed of the convolution layer has a complex structure, which makes the model large and difficult to deploy on electronic devices such as mobile phones and tablets. In addition, the interpolation upsampling module composed of the convolution layer itself does not have the characteristics of interpolating the image to any resolution. If you want to achieve super-resolution reconstruction of an image with any magnification, either use a large number of training images to train the image super-resolution model, which will result in a long training time for the model, or train multiple different image super-resolution models to process the image, which is troublesome to use the model. Summary of the invention

[0004] In view of the above problems, the embodiments of the present application provide an image super-resolution reconstruction method, an electronic device and a computer-readable storage medium, which are used to solve the problems in the prior art that the image super-resolution model has a complex structure, a large model size, and is not easy to deploy on electronic devices such as mobile phones and tablets.

[0005] According to one aspect of an embodiment of the present application, a method for image super-resolution reconstruction is provided, the method comprising: obtaining the size of a target image and an image to be super-resolved, the size of the image to be super-resolved being smaller than the size of the target image; initializing an image super-resolution model, and loading model parameters of the image super-resolution model saved in a training phase into the image super-resolution model; performing a first interpolation upsampling process on the image to be super-resolved according to the size of the target image to obtain a first target feature image; extracting pixel point information from the image to be super-resolved to obtain a target shallow feature map; extracting semantic information from the target shallow feature map to obtain a target deep feature map; performing a residual connection between the target shallow feature map and the target deep feature map to obtain a target fused feature map; performing a second interpolation upsampling process on the target fused feature map according to the size of the target image to obtain a second target feature image; performing a residual connection between the first target feature image and the second target feature image to obtain a target fused image; and performing refinement and denoising on the target fused image to obtain a target super-resolved image.

[0006] According to another aspect of an embodiment of the present application, an electronic device is provided, including a memory and a processor, the memory being used to store computer program code and transmit the computer program code to the processor; the processor being used to execute the above-mentioned image super-resolution reconstruction method according to instructions in the program code.

[0007] According to another aspect of an embodiment of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium is used to store a computer program code, which, when executed by a processor, implements the above-mentioned image super-resolution model training method, or the computer program, when executed by a processor, implements the above-mentioned image super-resolution reconstruction method.

[0008] The embodiment of the present application obtains the size of the target image and the image to be super-resolved, and sends the size of the target image and the image to be super-resolved into the image super-resolved model, so that the image super-resolved model performs super-resolution reconstruction on the image to be super-resolved according to the size of the target image to obtain the target super-resolved image. No matter how much the ratio is between the size of the target image and the size of the image to be super-resolved, the above-mentioned image super-resolution reconstruction method can achieve a good image super-resolution reconstruction effect, which well solves the problem that the image super-resolution reconstruction effect is poor as the size of the target image increases in the prior art, and a single model can achieve an image super-resolution effect of any ratio, without the need to install multiple models on the electronic device. In addition, the first upsampling module processing and the second upsampling module processing both use a sampling interpolation algorithm to implement the upsampling operation, and do not use a network module to implement the upsampling operation. On the one hand, the result of the image super-resolved model can be made simpler, which is conducive to deployment on the board end, mobile end and PC end, and the applicability of the image super-resolved model is stronger. On the other hand, the interpolation algorithm can be directly used to interpolate to any resolution. The characteristics of the image super-resolved effect of any ratio can be achieved without increasing the training time. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The accompanying drawings are only used to illustrate the embodiments and are not to be considered as limiting the present application. In addition, the same reference symbols are used to represent the same components throughout the accompanying drawings. In the accompanying drawings:

[0010] Figure 1 A diagram showing an application scenario of the image super-resolution reconstruction method provided by an embodiment of the present application;

[0011] Figure 2 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown;

[0012] Figure 3 A schematic diagram of a process flow of an image super-resolution reconstruction method provided in an embodiment of the present application is shown;

[0013] Figure 4 The image to be super-resolved and the target super-resolved image in the image super-resolution reconstruction method provided in the embodiment of the present application are shown;

[0014] Figure 5 A schematic diagram showing the flow of the training steps of the image super-resolution model provided in an embodiment of the present application is shown;

[0015] Figure 6 A schematic diagram of the structure of a training device provided in an embodiment of the present application is shown;

[0016] Figure 7 A schematic diagram showing a process of constructing a pair of high- and low-resolution images in the training step of the image super-resolution model provided in an embodiment of the present application is shown;

[0017] Figure 8 A schematic diagram of the process of image degradation processing in the training step of the image super-resolution model provided in an embodiment of the present application is shown;

[0018] Fig. 9 A flowchart of image degradation processing in the training step of the image super-resolution model provided in an embodiment of the present application is shown;

[0019] Fig.10 A schematic flow chart of a process for generating a super-resolution image in a training step of an image super-resolution model provided in an embodiment of the present application is shown;

[0020] Fig.11 A flowchart of a processing flow of an image super-resolution model in an image super-resolution reconstruction method provided in an embodiment of the present application is shown;

[0021] Fig.12 A schematic diagram of the process flow of optimization processing in the training step of the image super-resolution model provided in an embodiment of the present application is shown;

[0022] Fig.13 A flowchart of the calculation process of the verification index in the training step of the image super-resolution model provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0023] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein.

[0024] See also Figure 1 As shown, Figure 1 The application scenario diagram of the image super-resolution reconstruction method provided in the embodiment of the present application is shown. The image super-resolution reconstruction method in the embodiment of the present application can be applied to the first terminal device 2 and the second terminal device 3. When the camera device 1 can also display images, the image super-resolution reconstruction method in the embodiment of the present application can also be applied to the camera device 1. The camera device 1, the first terminal device 2 and the second terminal device 3 establish a communication connection through the network 4. The camera device 1 can be a monitoring device installed in an area to be monitored (such as a home, office, shopping mall, etc.), or it can be a camera, a video camera and other equipment. The first terminal device 2 and the second terminal device 3 can be a mobile phone, a tablet computer, a portable electronic device or other terminal electronic device with a display screen. The network 4 includes but is not limited to one or more of a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a 4G / 5G network, WIFI, Bluetooth (Bluetooth) and a point-to-point (P2P) communication network.

[0025] The hardware conditions of the camera device 1 itself and the transmission process of the image in the network 4 may cause the image acquired by the electronic device to be a low-resolution image. Specifically, the camera device 1 is used to shoot and record the shooting area that needs to be monitored. The camera lens, image sensor and other hardware on the camera device 1 will directly affect the resolution of the image. If the quality of the camera lens, image sensor and other hardware is relatively poor, the camera device 1 is limited by the hardware conditions and can only capture low-resolution images.

[0026] Secondly, the camera device 1, the first terminal device 2 and the second terminal device 3 need to realize image transmission through the network 4. For example, the camera device 1 sends the collected image to the first terminal device 2 or the second terminal device 3 through the network 4, and the first terminal device 2 and the second terminal device 3 also need to communicate with each other through the network 4. However, the image usually contains a large amount of pixel information. If it is directly transmitted as an original image, it will take up a lot of storage space and bandwidth. The higher the resolution of the image, the more pixel information it contains. In practical applications, due to the limitations of the bandwidth of the network 4, it is usually necessary to compress the image when transmitting the image, especially in an environment with slow or unstable network speed, and in application scenarios (such as video conferencing, remote monitoring, etc.) that require real-time transmission of images, often only the compressed image can be stably transmitted. However, when compressing the image, the resolution of the image may be reduced, resulting in the first terminal device 2 and the second terminal device 3 receiving the image as a low-resolution image.

[0027] Therefore, the image super-resolution reconstruction method in the embodiment of the present application can be applied to electronic devices 5 such as the camera device 1, the first terminal device 2, and the second terminal device 3, so that the electronic device 5 can perform super-resolution reconstruction on the acquired low-resolution image to obtain a high-resolution image, and display the high-resolution image with higher definition on the display screen of the electronic device 5. In the embodiment of the present application, see Figure 2 As shown, Figure 2 The structure diagram of the electronic device provided by the embodiment of the present application is shown as follows: Figure 2 As shown, the electronic device 5 may include: a processor (processor) 51 and a memory (memory) 52.

[0028] The memory 52 is used to store a computer program 53. The memory 52 may include a high-speed RAM memory, or may also include a non-volatile memory, such as at least one disk memory. The computer program 53 may include computer executable instructions.

[0029] The processor 51 is used to execute the computer program 53 to perform the embodiment of the image super-resolution reconstruction method described in the present application.

[0030] The electronic device 5 may include one or more processors 51, which may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the electronic device 5 may be processors of the same type, such as one or more CPUs, or may be processors of different types, such as one or more CPUs and one or more ASICs.

[0031] In the image super-resolution reconstruction method and electronic device 5 proposed in the present application, the size of the target image and the low-resolution image are first obtained, and then the low-resolution image is processed according to the size of the target image through a pre-trained image super-resolution model, and the final image obtained is the required high-resolution image.

[0032] Specifically, the image super-resolution reconstruction method provided in the embodiment of the present application is as follows: Figure 3 As shown, Figure 3 FIG. 5 is a flow chart of an image super-resolution reconstruction method provided in an embodiment of the present application, and the method is applied in an electronic device 5. Figure 3 As shown, the method comprises the following steps:

[0033] Step S100: obtaining the size of the target image and the image to be super-resolved, wherein the size of the image to be super-resolved is smaller than the size of the target image.

[0034] The image to be super-resolved is a low-resolution image acquired by the electronic device 5. The image to be super-resolved may be an image sent to the electronic device 5 by other devices through the network 4. When the image is transmitted through the network 4, it may be compressed due to the limitation of the bandwidth of the network 4. Therefore, when the image is displayed on the display screen of the electronic device 5, the image needs to be super-resolved. When the electronic device 5 itself has a camera lens, such as a mobile phone, a tablet, a camera device, etc., the image to be super-resolved may also be an image captured by the electronic device 5 through its own camera lens. The image may have a low resolution due to the limitation of hardware conditions such as the camera lens and the image sensor.

[0035] The size of the target image is the image size (i.e., image width and height) that the user expects to be displayed on the display screen of the electronic device 5. The size of the target image can be the expected image size set by the user. For example, the user can directly set the default minimum size for image display on the electronic device 5. When the size of the image to be super-resolved received by the electronic device 5 is smaller than the minimum size, the minimum size is used as the size of the target image. A button for image super-resolution reconstruction can also be set on the electronic device 5. When the user wants to increase the resolution of the image currently displayed by the electronic device 5, the size of the target image can be input by clicking the button for image super-resolution reconstruction, so that the image super-resolution model processes the image currently displayed by the electronic device 5 according to the size of the target image.

[0036] The size of the target image can also be automatically acquired by the electronic device 5. For example, the electronic device 5 can acquire the display resolution of the display screen as the size of the target image. No matter what the size of the image to be super-resolution is, the size of the image to be super-resolution is adjusted to the size of the target image through the image super-resolution model. When the electronic device 5 is playing a video, it can also automatically determine the size of the target image according to the clarity selected by the user (for example, 480P, 720P, 1080P, 4K, etc.) to perform super-resolution reconstruction of the image in the video through the image super-resolution model.

[0037] Step S110: Initialize the image super-resolution model, and load the model parameters saved in the training phase of the image super-resolution model into the image super-resolution model.

[0038] The model parameters of the image super-resolution model may be obtained by training through the following steps S200 to S700b.

[0039] Step S120: performing a first interpolation up-sampling process on the image to be super-resolved according to the size of the target image to obtain a first target feature image.

[0040] Among them, directly performing the first interpolation upsampling process on the original image to be super-resolved can retain more image details in the first target feature image and improve the fidelity of the image. The first interpolation upsampling process does not use network modules such as convolutional layers to implement upsampling operations, but uses an interpolation algorithm to upsample the image to be super-resolved. Specifically, interpolation algorithms such as nearest neighbor interpolation, bilinear interpolation, and bicubic interpolation can be used. According to the characteristic that the interpolation algorithm can interpolate to any resolution, no matter what the ratio between the size of the target image and the size corresponding to the image to be super-resolved is, the first target feature image with a size equal to the size of the target image can be obtained through interpolation upsampling.

[0041] Step S130: extract pixel information from the image to be super-resolved to obtain a target shallow feature map.

[0042] Among them, pixel information refers to information such as color and texture in the image, which can be extracted from low-resolution images through algorithms such as local binary patterns, Gabor filters, color moments, Fourier transforms, and Tamura texture features. In an embodiment of the present application, as an example, a 3×3 convolutional layer can be used to perform a convolution operation on the image to be super-resolved to extract pixel information in the image to be super-resolved. The receptive field of the 3×3 convolutional layer is small, and the pixel information is extracted directly from the original image, and the information on the original image is richer. Therefore, more pixel information can be extracted from the image to be super-resolved through the 3×3 convolutional layer, thereby retaining more image details in the shallow feature map.

[0043] Step S140: extracting semantic information from the target shallow feature map to obtain a target deep feature map.

[0044] Among them, semantic information refers to the content and meaning contained in the image that can be understood by humans or machines, which usually involves objects, scenes, activities, relationships, and attributes and concepts related to these elements in the image. Specifically, semantic information can be extracted from shallow feature maps through image segmentation algorithms, pattern recognition algorithms, convolutional neural networks, graph convolutional networks, generative adversarial networks, etc. In an embodiment of the present application, as an example, semantic information can be extracted from shallow feature maps through a deep feature extraction module composed of N Blocks submodules. Blocks can be any network blocks, such as Blocks of VGG (Visual Geometry Group), ResNet (Residual Neural Network), and RRDB (Residual in Residual Dense Block).

[0045] Step S150: Perform a residual connection on the target shallow feature map and the target deep feature map to obtain a target fused feature map.

[0046] Among them, the residual connection refers to directly connecting the target shallow feature map and the target deep feature map to form a new image (i.e., the target fused feature map). This connection method can learn the difference between the target shallow feature map and the target deep feature map, so as to retain more image details in the target fused feature map and improve the consistency between the shape in the target fused feature map and the shape in the image to be super-resolved. Furthermore, in order to retain the pixel information and semantic information in the image as much as possible in the target fused feature map, the specific process of the residual connection is shown in steps S151 and S152:

[0047] Step S151: respectively obtain the pixel value of each pixel point on the target shallow feature map and the pixel value of each pixel point on the target deep feature map.

[0048] Step S152: Determine the pixel value of each pixel on the target fusion feature map according to the pixel value of each pixel on the target shallow feature map and the pixel value of each pixel on the target deep feature map.

[0049] Among them, the pixel value of each pixel on the target shallow feature map and the pixel value of each pixel on the target deep feature map can be mathematically operated, and the numerical value obtained by the mathematical operation can be used as the pixel value of each pixel on the target fusion feature map, thereby generating the target fusion feature map. In an embodiment of the present application, as an example, the pixel value of each pixel on the target fusion feature map can be obtained by addition operation. Specifically, the target deep feature map is obtained by the target shallow feature map, and the sizes of the two will remain consistent, that is, the pixels on the target deep feature map and the pixels on the target shallow feature map can correspond one to one. When the residual connection is made, the pixel values ​​of the corresponding pixels on the target shallow feature map and the target deep feature map can be added, and the result of the addition operation can be used as the pixel value of the pixel at the corresponding position on the target fusion feature map, that is, the pixel value of the pixel point (x, y) on the target fusion feature map is obtained by adding the pixel value of the pixel point (x, y) on the target shallow feature map and the pixel value of the pixel point (x, y) on the target deep feature map.

[0050] Furthermore, different weights can be set for the pixel values ​​of each pixel on the target shallow feature map and the pixel values ​​of each pixel on the target deep feature map during addition operations. When the weight of the pixel value of each pixel on the target shallow feature map is greater than the weight of the pixel value of each pixel on the target deep feature map, the target fused feature map can focus on retaining pixel information. When the weight of the pixel value of each pixel on the target shallow feature map is less than the weight of the pixel value of each pixel on the target deep feature map, the target fused feature map can focus on retaining semantic information.

[0051] In step S151 to step S152, the pixel value of each pixel on the target fused feature image is directly determined by the pixel value of each pixel on the target shallow feature map and the target deep feature map, so that the generation process of the target fused feature map is simpler, and the target fused feature map can retain the image details on the target shallow feature map and the target deep feature map as much as possible, thereby making the shape of the target fused feature map basically consistent with the shape of the image to be super-resolved.

[0052] Step S160: performing a second interpolation upsampling process on the target fusion feature map according to the size of the target image to obtain a second target feature image.

[0053] The process of the second interpolation upsampling process is basically the same as the process of the first interpolation upsampling process. The second interpolation upsampling process does not use network models such as convolutional layers to implement upsampling operations, but uses an interpolation algorithm to upsample the super-resolution image. Specifically, interpolation algorithms such as nearest neighbor interpolation, bilinear interpolation, and bicubic interpolation can also be used. The interpolation algorithm used in the second interpolation upsampling process can be the same as the interpolation algorithm used in the first interpolation upsampling process, for example, both use bicubic interpolation, etc., or they can be different, for example, the first interpolation upsampling process uses bilinear interpolation, and the second interpolation upsampling process uses bicubic interpolation, etc.

[0054] Step S170: Perform residual connection on the first target feature image and the second target feature image to obtain a target fused image.

[0055] Among them, the process of residual connection between the first target feature image and the second target feature image is basically the same as the process of residual connection between the target shallow feature map and the target deep feature map, which will not be repeated here.

[0056] Step S180: performing refinement and denoising on the target fused image to obtain a target super-resolution image.

[0057] Among them, image noise refers to a random change in brightness or color information in an image, which is unnecessary or redundant interference information in the image data. The refinement module can perform convolution operations on the target fused image by setting convolution layers such as homogeneous filtering, median filtering, and Gaussian filtering to filter out the noise on the target fused image and improve the image quality of the target super-resolution image. Of course, network modules such as deep convolutional neural networks and convolutional neural network variants (for example, DnCNN, FastDVDnet, FFDNet) can also be set to perform feature refinement processing on the target fused image to improve the image quality of the target super-resolution image. In an embodiment of the present application, as an example, a network module can be composed of two 3x3 convolutional layers and an activation function to further denoise the first super-resolution image through the network module to achieve the purpose of refining the image details on the first super-resolution image.

[0058] In the embodiment of the present application, the relationship between the size of the target image and the size of the image to be super-resolved can be any ratio. Figure 4 As shown, Figure 4 The figure shows the image to be super-resolved and the target super-resolved image in the image super-resolution reconstruction method provided in the embodiment of the present application. Figure 4 The image (a) in the figure is the image to be super-resolved, and the size of the image to be super-resolved is 128x128. Figure 4The images (b), (c) and (d) in the figure can all be used as the target super-resolution images after the image (a) is super-resolved by the image super-resolution model. If the image (b) is the target super-resolved image, the size of the target image is 218x218, and the ratio between the size of the target image and the size of the image to be super-resolved is 1.7 times; if the image (c) is the target super-resolved image, the size of the target image is 294x294, and the ratio between the size of the target image and the size of the image to be super-resolved is 2.3 times; if the image (d) is the target super-resolved image, the size of the target image is 512x512, and the ratio between the size of the target image and the size of the image to be super-resolved is 4 times.

[0059] In the above embodiment, the size of the target image and the image to be super-resolved are obtained, and the size of the target image and the image to be super-resolved are sent to the image super-resolved model, so that the image super-resolved model performs super-resolution reconstruction on the image to be super-resolved according to the size of the target image to obtain the target super-resolved image. Figure 4 As shown, no matter what the ratio between the size of the target image and the size of the image to be super-resolved is, the above-mentioned image super-resolution reconstruction method can achieve a good image super-resolution reconstruction effect, which well solves the problem that the image super-resolution reconstruction effect becomes worse and worse as the size of the target image increases in the prior art, and a single model can achieve an image super-resolution effect of any ratio, without the need to install multiple models on the electronic device 5. In addition, the first upsampling module processing and the second upsampling module processing both use a sampling interpolation algorithm to implement the upsampling operation, instead of using a network module such as a convolutional layer to implement the upsampling operation. On the one hand, the result of the image super-resolution model can be made simpler, which is conducive to deployment on the board side, mobile side, PC side, etc., and the image super-resolution model is more applicable. On the other hand, the interpolation algorithm can be directly used to interpolate to any resolution, and the image super-resolution effect of any ratio can be achieved without increasing the training time.

[0060] In the embodiment of the present application, in order to be able to perform super-resolution processing on an image at any magnification, before using the image super-resolution model, it is necessary to train the image super-resolution model, such as Figure 5 As shown, Figure 5 FIG. 1 is a flow chart showing the training steps of the image super-resolution model provided in the embodiment of the present application. The training steps can be performed by the electronic device 5 or by another training device 6. Figure 6 As shown, Figure 6 A schematic diagram of the structure of a training device provided in an embodiment of the present application is shown. The training device 6 includes, but is not limited to: a processor 61 and a memory 62.

[0061] The memory 62 is used to store a computer program 63. The memory 62 may include a high-speed RAM memory, or may also include a non-volatile memory, such as at least one disk memory. The computer program 63 may include computer executable instructions.

[0062] The processor 61 is used to execute the computer program 63, and specifically can execute the specific steps of the training method of the image clarity detection model provided in this application.

[0063] The processor 61 may be a central processing unit (CPU), or a specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the training device 6 may be processors of the same type, such as one or more CPUs; or may be processors of different types, such as one or more CPUs and one or more ASICs.

[0064] Specifically, the training steps of the image super-resolution model include the following steps:

[0065] Step S200: constructing a high-resolution and low-resolution image pair, wherein the high-resolution and low-resolution image pair includes a high-resolution image and its corresponding low-resolution image.

[0066] Among them, the high-resolution image and its corresponding low-resolution image in the high-resolution and low-resolution image pair have the same content, that is, the objects, scenes, objects, activities and relationships in the images are the same, and only the number of pixels on the images is different. In order to ensure that the content in the low-resolution image is highly consistent with the content in the high-resolution image, the low-resolution image can be obtained by performing image degradation processing on the high-resolution image. Specifically, Figure 7 As shown, Figure 7 The flowchart of the process of constructing high-resolution and low-resolution image pairs in the training step of the image super-resolution model provided in the embodiment of the present application is shown, and the step S200 specifically includes the following:

[0067] Step S210: Acquire a first high-resolution image and a size corresponding to the first high-resolution image.

[0068] Among them, high-resolution images refer to images with very high pixel density, that is, the number of pixels contained in a unit area is very large. Such images can show more details and clearer visual effects, and can also provide more accurate and detailed information. The size of a high-resolution image refers to the pixel size of a high-resolution image, that is, the size of the image in digital form, which is defined by the width and height of the image (in pixels). For example, the size of an image with a resolution of 720P can be expressed as 1280x720, the size of an image with a resolution of 1080P can be expressed as 1920x1080, the size of an image with a resolution of 2K can be expressed as 2560x1440, and the size of an image with a resolution of 4K can be expressed as 3840x4320. The higher the resolution of the image, the larger its size.

[0069] Randomly extract B (B>1) first high-resolution images from the pre-constructed training data set to form the training images for the current training round. High-resolution images (such as images of grass, animals, and vehicles) can be collected from the Internet, or public image data sets (such as RealSR data sets) can be downloaded. High-resolution video data can also be collected from the camera device 1, and high-resolution images can be extracted from the high-resolution video data by extracting frames. The high-resolution images obtained by the above method are integrated to construct the training data set.

[0070] Step S220: performing arbitrary reduction processing on the sizes corresponding to the first high-resolution image to obtain at least one cropping size corresponding to the first high-resolution image.

[0071] Among them, any magnification can be an integer or a decimal. In the embodiment of the present application, as an example, when processing the size corresponding to the first high-resolution image, a value between 1 and 6 can be randomly generated by a random number generation function, and the size corresponding to the first high-resolution image can be reduced according to the value to obtain the crop size corresponding to the first high-resolution image. For example, assuming that the size corresponding to the first high-resolution image is 2048×2048, if the preset magnification is 2, then the crop size obtained after the size corresponding to the first resolution image is reduced is 1024×1024, and if the preset magnification is 1.6, then the crop size obtained after the size corresponding to the first resolution image is reduced is 1280×1280.

[0072] When reducing the sizes corresponding to multiple first high-resolution images of the same training round, a preset magnification may be randomly generated for each first high-resolution image, so that the sizes corresponding to each first high-resolution image are reduced by different preset magnifications, thereby making the cropping sizes corresponding to each first high-resolution image different. Of course, multiple preset magnifications may also be randomly generated for each first high-resolution image, so that the sizes corresponding to each first high-resolution image are reduced according to multiple preset magnifications, so that each first high-resolution image corresponds to multiple cropping sizes.

[0073] Step S230: cropping the first high-resolution image according to at least one cropping size corresponding to the first high-resolution image to obtain a second high-resolution image.

[0074] Among them, the cropping size is used to crop the first high-resolution image to the corresponding size, that is, the second high-resolution image is actually a portion of the image cropped from the first high-resolution image. Assuming that the corresponding size of the first high-resolution image is 2048x2048, if the cropping size is 1024x1024, the first high-resolution image is cropped to a 1024x1024 image, and the cropped image is used as the second high-resolution image. If the cropping size is 1280x1280. The first high-resolution image is cropped to a 1280x1280 image, and the cropped image is used as the second high-resolution image. The second high-resolution image can be an image cropped at any position on the first high-resolution image.

[0075] When each first high-resolution image corresponds to only one cropping size, and the cropping sizes corresponding to each first high-resolution image are different, each first high-resolution image can be cropped according to the cropping size, so that the second high-resolution image corresponds to the second high-resolution image one-to-one, and the sizes of the second high-resolution images corresponding to each first high-resolution image are different. When the first high-resolution image corresponds to multiple cropping sizes, the first high-resolution image is cropped according to the multiple cropping sizes to obtain multiple second high-resolution images, and the sizes of these second high-resolution images are different, and when cropping, the cropping can be performed at the same position on the first high-resolution image, or at different positions on the first high-resolution image.

[0076] Step S240: performing image degradation processing on the second high-resolution image according to a preset degraded image size to obtain a low-resolution image corresponding to the second high-resolution image.

[0077] Among them, if the magnifications of multiple second high-resolution images and their corresponding low-resolution images in the same training round are to be different, there can be two combinations, one is that the size of the second high-resolution images is the same and the size of the low-resolution images is different, and the other is that the size of the second high-resolution images is different and the size of the low-resolution images is the same. Since the larger the image, the longer the inference time of the image super-resolution model will be, for example, a low-resolution image with a resolution of 256ⅹ256 is compared with a low-resolution image with a resolution of 64ⅹ64, theoretically, it is necessary to increase the training time by 4 times. Therefore, if the size of the low-resolution images is different, the low-resolution image with a larger resolution requires more training time, which will increase the training time of the image super-resolution model. Therefore, in the embodiment of the present application, the size of the low-resolution images is fixed at a smaller size (for example, 64ⅹ64, 128ⅹ128, etc.), which can speed up the training speed of the model to a certain extent.

[0078] Therefore, in the embodiment of the present application, by performing degradation processing on each second high-resolution image according to the preset degraded image size, the size of the low-resolution images is made the same, that is, the size of the low-resolution images is equal to the preset degraded image size. The preset degraded image size can be 64×64, 128×128, etc., and can be any multiple of 8. Specifically, the corresponding low-resolution image can be obtained by blurring the second high-resolution image, adding noise pixels to the second high-resolution image, or interpolating and downsampling the second high-resolution image.

[0079] Step S250: determining the second high-resolution image and its corresponding low-resolution image as a high-low-resolution image pair.

[0080] By directly generating the corresponding low-resolution image from the second-resolution image through steps S210 to S250, it can be ensured that the content of the low-resolution image and the content of the high-resolution image are highly consistent in the high-resolution image pair, so that the content of the super-resolution image obtained after the low-resolution image is super-resolved can also be the same as the content of the high-resolution image. In addition, different cropping sizes are randomly generated for the first high-resolution image in each training round, so that the multiple second high-resolution images obtained according to the cropping size have different sizes; the second high-resolution image is subjected to image degradation processing according to the preset degraded image size, so that the size of the low-resolution image in the same training round is consistent, which not only makes the size of the second high-resolution image in the same training round have different magnification relationships with the size of the corresponding low-resolution image, but also the image super-resolution model can learn image features at different magnifications during the training process, so that the trained image super-resolution model can perform super-resolution reconstruction on the low-resolution image at any magnification, and can make full use of the computing power of the training device 6, effectively speeding up the training speed of the image super-resolution model.

[0081] Furthermore, in the training process of the image super-resolution model, in order to better simulate the degradation encountered by low-resolution images in the real world, such as Internet transmission, camera imaging system, camera blur, sensor noise, sharpening artifacts and JPEG compression, image degradation processing includes first-order degradation and second-order degradation, such as Figure 8 As shown, Figure 8 The flowchart of image degradation processing in the training step of the image super-resolution model provided in the embodiment of the present application is shown. Steps S241 to S243b can be used to replace the above step S240 to perform image degradation processing on the second high-resolution image to generate a low-resolution image corresponding to each second high-resolution image. The details are as follows:

[0082] Step S241: Obtaining the execution probability of second-order degradation in image degradation processing.

[0083] Among them, in order to better simulate the clarity of different low-resolution images, in the actual training process of the image super-resolution model, the second high-resolution image in each training round will perform first-order degradation, and the second-order degradation will perform second-order degradation on some of the second high-resolution images according to its execution probability. In the embodiment of the present application, as an example, assuming that the execution probability of the second-order degradation is 0.8, if there are 100 second high-resolution images in the current training round, and each second high-resolution image performs an image degradation process once, then it is necessary to perform 100 image degradation processes. When performing these 100 image degradation processes, according to the execution probability of the second-order degradation, only 80 image degradation processes will perform second-order degradation, and the other 20 image degradation processes will not perform second-order degradation, that is, only 80 second high-resolution images will generate corresponding low-resolution images through first-order degradation and second-order degradation, and the other 20 second high-resolution images will only generate corresponding low-resolution images through first-order degradation.

[0084] Step S242: determining whether to perform second-order degradation on the second high-resolution image according to the execution probability of the second-order degradation.

[0085] Specifically, the total number of times the image degradation processing needs to be performed can be determined first according to the number of second high-resolution images in the current training round, and then the number of times the second-order degradation is performed can be determined in combination with the execution probability of the second-order degradation. Finally, according to the number of second high-resolution images that have performed second-order degradation in the current training round, it can be determined whether the second high-resolution image currently undergoing image degradation processing performs second-order degradation.

[0086] Step S243a: If the second-order degradation is performed on the second high-resolution image, the first-order degradation in the image degradation processing is performed on the second high-resolution image according to the preset degraded image size, and the second-order degradation is performed on the image after the first-order degradation according to the preset degraded image size to obtain a low-resolution image corresponding to the second high-resolution image.

[0087] Step S243b: If the second-order degradation is not performed on the second high-resolution image, a first-order degradation is performed on the second high-resolution image according to a preset degraded image size to obtain a low-resolution image corresponding to the second high-resolution image.

[0088] Among them, the degradation types in the first-order degradation and the second-order degradation may include blur (generally, blur degradation modeling is achieved through convolution of a linear blur filter, such as an isotropic Gaussian filter, anisotropic Gaussian filter, and 2Dsinc filter, etc.), Resize (Resize refers to the image quality degradation caused by traditional interpolation up and down sampling, such as nearest neighbor interpolation, bilinear interpolation, and bicubic interpolation, etc.), noise (image noise caused by sensor noise, etc., such as additive noise and Poisson noise, etc.) and JPEG compression (JPEG compression is a commonly used digital image lossy compression technology in the process of image preservation and image transmission).

[0089] Specifically, Fig. 9 As shown, Fig. 9 A flowchart of the image degradation processing in the training step of the image super-resolution model provided in an embodiment of the present application is shown. When performing first-order degradation and second-order degradation, the image can be degraded in the order of blur, resize, noise, and JPEG compression. Of course, corresponding execution probabilities can also be set for these degradation types, so that each time the first-order degradation or second-order degradation is performed on the second high-resolution image, at least one degradation type is randomly selected for the second high-resolution image according to the execution probability corresponding to each degradation type for image degradation processing. For example, assuming that the execution probability corresponding to blur is 0.8, when the first-order degradation is performed on 100 second high-resolution images, only 80 second high-resolution images are blurred, and the other 20 second high-resolution images are not blurred. Degradation types such as resize, noise, and JPEG compression also use a similar method to process the second high-resolution image.

[0090] Furthermore, the execution probability can be set for the specific degradation mode in each degradation type, so as to randomly select a degradation mode from a plurality of specific degradation modes to realize the corresponding degradation type. For example, assuming that the degradation type is image blur, and the execution probability of the isotropic Gaussian filter is 0.3, the execution probability of the anisotropic Gaussian filter is 0.4, and the execution probability of the 2D sinc filter is 0.3, if there are 100 second high resolution images to be blurred in the same training round, the isotropic Gaussian filter can be used to blur 30 second high resolution images, the anisotropic Gaussian filter can be used to blur another 40 second high resolution images, and finally the 2D sinc filter can be used to blur the remaining 30 second resolution images. Degradation types such as resize, noise, and JPEG compression can also select specific degradation modes for the second high resolution images in a similar direction.

[0091] It should be noted that among the degradation types such as blur, Resize, noise, and JPEG compression, only Resize can adjust the size of the image. Therefore, when performing image degradation processing on the second high-resolution image, each second high-resolution image must be resized using Resize at least once to adjust the size of each low-resolution image to the preset degraded image size.

[0092] Specifically, when only the first-order degradation is performed on the second high-resolution image, or when Resize is not used for image degradation processing when performing the second-order degradation, when performing the first-order degradation, the second high-resolution image must be resized according to the preset degraded image size by Resize, so as to adjust the size of the corresponding low-resolution image to the preset degraded image size. When the second-resolution image will perform the second-order degradation, and when Resize is used for image adjustment when performing the second-order degradation, when performing the first-order degradation, you can choose not to use Resize for image degradation processing, or you can choose to use Resize for image adjustment, and when using Resize for image adjustment, you can process the image according to any size, and when performing the second-order degradation, you need to use Resize for image adjustment according to the preset degraded image size for the image after the first-order degradation.

[0093] By performing image degradation processing on the second high-resolution image through steps S241 to S243b, different degradation processes can be performed on different second high-resolution images in the same training round, which can better simulate the degradation conditions encountered by low-resolution images in the real world, and further improve the performance of the image super-resolution model in super-resolution reconstruction of low-resolution images under various degradation conditions.

[0094] Step S300: Send the size of the high-resolution image and the low-resolution image to the image super-resolution model to obtain a super-resolution image corresponding to the high-resolution and low-resolution image pair.

[0095] Among them, Fig.10 As shown, Fig.10 The flowchart of the super-resolution image generation process in the training step of the image super-resolution model provided in the embodiment of the present application is shown. The image super-resolution model can perform the following steps S310 to S370 according to the high-resolution size and the low-resolution image to generate the super-resolution image corresponding to the high-resolution and low-resolution image pair, that is, the steps S310 to S370 can be used to replace the above step S300, which specifically includes the following:

[0096] Step S310: performing a first interpolation up-sampling process on the low-resolution image according to a size corresponding to the high-resolution image, to obtain a first feature image corresponding to the low-resolution image.

[0097] Step S320: extract pixel information from the low-resolution image to obtain a shallow feature map.

[0098] Step S330: extracting semantic information from the shallow feature map to obtain a deep feature map.

[0099] Step S340: Perform residual connection on the shallow feature map and the deep feature map to obtain a fused feature map.

[0100] Step S350: performing a second interpolation upsampling process on the fused feature map according to the size corresponding to the high-resolution image to obtain a second feature image.

[0101] Step S360: Perform residual connection on the first feature image and the second feature image to obtain a fused image.

[0102] Step S370: performing refinement and denoising on the fused image to obtain a super-resolution image corresponding to the high-resolution and low-resolution image pair.

[0103] Among them, Fig.11 As shown, Fig.11 A flowchart of the processing flow of the image super-resolution model in the image super-resolution reconstruction method provided in an embodiment of the present application is shown. The image super-resolution model includes four major modules: shallow feature extraction, deep feature extraction, interpolation upsampling, and feature refinement. The shallow feature extraction can be composed of a 3×3 convolutional layer, which is used to extract pixel information on the image to be super-resolved, further retain more image details on the image to be super-resolved, and obtain the target shallow feature map.

[0104] Deep feature extraction can be composed of N Blocks submodules. Blocks can be any network block, such as the common VGG, ResNet and RRDB Blocks. Deep feature extraction is mainly used to extract the semantic information of the image to be super-resolved to obtain a deep feature map. In addition, the deep feature map and the shallow feature map are residually connected to obtain a fused feature map to further retain more image details on the low-resolution image. At this time, the shape of the fused feature map obtained can basically be consistent with the shape of the low-resolution image.

[0105] The image super-resolution model includes two interpolation upsampling operations. One of the interpolation upsampling operations is directly performed on the low-resolution image to obtain the first feature image. Its purpose is to further retain more image details and make the fidelity of the final super-resolution image better. The other interpolation upsampling operation is performed on the fused feature map to obtain the second feature image. Finally, the first feature image and the second feature image are residually connected to obtain the first fused image.

[0106] The feature refinement module may be composed of two 3x3 convolutional layers and an activation function. The feature refinement module traverses all pixels on the first fused image and performs convolution operation on the pixel value of each pixel to further denoise the first fused image, thereby achieving the purpose of refining image details and obtaining a super-resolution image to improve the image quality of the super-resolution image.

[0107] When training an image super-resolution model, the duration of the interpolation upsampling process is directly related to the size of the low-resolution image. No matter what the size of the target image is, as long as the size of the image to be super-resolved is the same, the duration required for the interpolation upsampling process will be basically the same. Therefore, through the interpolation upsampling method, super-resolution reconstruction of any resolution of the image to be super-resolved can be achieved without increasing the training time.

[0108] In addition, the first upsampling module processing and the second upsampling module processing are both implemented by the sampling interpolation algorithm. The interpolation algorithm has the characteristic of being able to interpolate to any resolution. Therefore, when training the image super-resolution model to perform arbitrary multiplication super-resolution on the image, it is not necessary to use a large number of training images to train the model like the interpolation upsampling processing composed of the training network module, which can effectively reduce the training time of the image super-resolution model. Secondly, the shallow feature map and the deep feature map, the first feature image and the second feature image are all connected through residuals to generate new images, which can save more image details in the new image. Finally, the fused image is refined and denoised through the refinement model to improve the clarity of the super-resolution image, thereby improving the consistency between the super-resolution image and the high-resolution image.

[0109] Step S400: determining a loss function according to the high-resolution image and its corresponding super-resolution image, and optimizing the image super-resolution model according to the loss function to obtain model parameters.

[0110] Among them, the loss function is a measure of the performance of the model, which is a measure of the difference between the predicted value (i.e., the first super-resolution image) and the true value (i.e., the second high-resolution image). During the model training process, the loss function is used to evaluate the prediction error of the model on the training data, thereby guiding the model to perform optimization processing. The loss function can be mean square error, cross entropy loss, or logarithmic loss, etc.

[0111] The image super-resolution model is optimized according to the loss function. The loss function can be processed by gradient descent method, Newton method, conjugate gradient method, or adaptive learning rate optimizer, and the image super-resolution model is optimized according to the processing result. Furthermore, in order to better optimize the image super-resolution model, the model parameters of the image super-resolution model can be updated by reverse gradient propagation of the chain rule. Fig.12 As shown, Fig.12The flowchart of the optimization process in the training step of the image super-resolution model provided in the embodiment of the present application is shown. Steps S410 to S430 can be used to replace the above-mentioned step S400, which specifically includes the following:

[0112] Step S410: According to the high-resolution image and its corresponding super-resolution image, the loss of the high-resolution image and its corresponding super-resolution image is calculated using a loss function.

[0113] Step S420: Calculate the gradient of the loss relative to the model parameters of the image super-resolution model using the loss function.

[0114] Step S430: Update the model parameters of the image super-resolution model by back-propagating gradients.

[0115] Among them, the loss function is derived according to the model parameters of the image super-resolution model to obtain the derivatives corresponding to the parameters of the image super-resolution model (i.e., the gradient of the loss function relative to the model parameters of the image super-resolution model), and then the model parameters of the image super-resolution model (e.g., the weights and biases of the network modules in the image super-resolution model) are updated using gradient descent or other optimization algorithms. Furthermore, the Adam optimizer can be used to update and optimize the model parameters of the image super-resolution model. The image super-resolution model is optimized by back-propagating the gradient, so that the image super-resolution model can learn the features of the input data (i.e., low-resolution images), handle complex problems, and improve the generalization ability through iterative training, thereby enhancing the performance of the image super-resolution model.

[0116] Furthermore, in order to enable the loss function to provide better guidance for the optimization processing of the image super-resolution model, the loss function can be determined according to the difference of different information on the image (for example, feature information, frequency domain information, etc.). Specifically, the loss function may include reconstruction loss and frequency domain loss, and steps S411 to S415 may be used to replace the above step S410 to obtain the loss function, and the loss function is used to calculate the loss of the high-resolution image and its corresponding super-resolution image, which specifically includes the following:

[0117] Step S411: Calculate the image error between the high-resolution image and its corresponding super-resolution image, and determine the reconstruction loss according to the image error.

[0118] Among them, the reconstruction loss is used to train the image super-resolution model to restore the feature information in the original image when super-resolution reconstruction is performed on the low-resolution image. The image error between the high-resolution image and its corresponding super-resolution image can be directly used as the reconstruction loss. The reconstruction loss can be mean square error, cross entropy loss, logarithmic loss, etc. In the embodiment of the present application, as an example, the reconstruction loss L1 can adopt L1 loss (also called L1 norm loss or absolute loss), and the specific formula is as follows:

[0119] L1=||I SR -I HR ||1 (1)

[0120] In formula (1), I SR represents the super-resolution image, I HR represents a high-resolution image, ||·||1 represents the L1 norm, and the reconstruction loss L1 is obtained by performing L1 norm operation on the super-resolution image and the high-resolution image.

[0121] Step S412: respectively obtaining frequency domain information of the high-resolution image and the frequency domain information of the corresponding super-resolution image.

[0122] Step S413: determining the frequency domain loss according to the frequency domain information of the high-resolution image and the frequency domain information of its corresponding super-resolution image.

[0123] Among them, the frequency domain loss is used to train the image super-resolution model to restore the high-frequency information in the image when super-resolution reconstruction is performed on the low-resolution image. The frequency domain information of the high-resolution image and the super-resolution image can be obtained through Fourier transform, frequency filtering, multi-scale analysis and other algorithms, and the error between the frequency domain information of the high-resolution image and the frequency domain information of the super-resolution image is used as the frequency domain loss. The frequency domain loss can be the frequency domain mean square error, frequency domain cross entropy loss, frequency domain logarithmic loss, etc. In the embodiment of the present application, as an example, the frequency domain loss L FFT FFT (Fast Fourier Transform) loss can be used, the specific formula is as follows:

[0124] L FFT =||F(I SR )-F(I HR )||1 (2)

[0125] In formula (2), F represents fast Fourier transform. First, fast Fourier transform is performed on the super-resolution image and the high-resolution image to obtain the frequency domain information in the image. Then, L1 norm operation is performed on the frequency domain information of the high-resolution image and the frequency domain information of the super-resolution image to obtain the frequency domain loss L FFT .

[0126] Step S414: determining a loss function according to the reconstruction loss and the frequency domain loss.

[0127] Among them, when determining the loss function, the reconstruction loss L1 and the frequency domain loss L FFT Set different weights. The specific formula is as follows:

[0128] L sum =λ1L1+λ2L FFT (3)

[0129] In formula (3), λ1 represents the weight of the reconstruction loss L1, and λ2 represents the frequency domain loss L FFT The weight of, when λ1>λ2, through the loss function L sum The trained image super-resolution model pays more attention to restoring the feature information of low-resolution images when super-resolution reconstruction is performed on low-resolution images. When λ1<λ2, the loss function L sum The trained image super-resolution model pays more attention to restoring the frequency domain information of low-resolution images when performing super-resolution reconstruction on low-resolution images. In practical applications, the reconstruction loss L1 and the frequency domain loss L2 can be calculated based on the actual super-resolution reconstruction requirements. FFT Set different weights.

[0130] Step S415: Calculate the loss of the high-resolution image and its corresponding super-resolution image according to the loss function.

[0131] Among them, the high-resolution image and its corresponding super-resolution image input are substituted into the relevant parameters of the loss function, for example, the loss function L sum The super-resolution image I SR and high resolution image I HR .

[0132] Steps S411 to S415 determine the loss function through reconstruction loss and frequency domain loss, so that the optimized image super-resolution model can better restore the feature information and frequency domain information in the low-resolution image, so as to improve the image super-resolution reconstruction effect of the image super-resolution model on the low-resolution image.

[0133] Step S500: Load the model parameters into the image super-resolution model, and calculate the verification index of the image super-resolution model.

[0134] Among them, when the model is iteratively trained, the more iterations, the better the model effect obtained by training. For the image super-resolution model, when the iterative training of the model reaches a certain number of times, the image super-resolution reconstruction effect of the image super-resolution model will reach convergence, that is, reach the best. In addition, after the image super-resolution model is optimized, the image super-resolution reconstruction effect of the image super-resolution model usually fluctuates to a certain extent, which leads to the image super-resolution model obtained in the last training not necessarily being the model with the best image super-resolution reconstruction effect. Therefore, in order to further ensure that the trained image super-resolution model is the model with the best image super-resolution reconstruction effect during the training process, the verification indicators corresponding to each image super-resolution model generated in the training stage can be calculated during the training process of the model.

[0135] The verification index is used to evaluate the image quality of the super-resolution image obtained by the image super-resolution model. Specifically, one or more of the indicators such as Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM), Naturalness Image Quality Evaluator (NIQE), Distance in Terms of Structural Similarity (DISTS), and Learned Perceptual Image Patch Similarity (LPIPS) can be used to obtain the model verification index. In the embodiment of the present application, as an example, the model verification index Score can be obtained by the following formula.

[0136]

[0137] Step S600: Determine whether the current training times of the image super-resolution model reaches the preset training times.

[0138] If yes, it means that the training of the image super-resolution model has been completed, and the trained image super-resolution model can be output, and the process jumps to step S700b;

[0139] If not, it means that the number of iterative training of the image super-resolution model is not enough, and the image super-resolution model needs to be further optimized. It is necessary to jump to step S700a to continue training the image super-resolution model.

[0140] Step S700a: If the current training times of the image super-resolution model do not reach the preset training times, steps S300 to S500 are executed repeatedly to obtain the optimal value of the verification index and its corresponding model parameter.

[0141] Step S700b: If the current training times of the image super-resolution model reaches the preset training times, the model parameters corresponding to the optimal values ​​of the verification indicators are saved as the final model parameters of the image super-resolution model.

[0142] Among them, all the image super-resolution models and their corresponding verification indicators generated during the training process can be recorded and retained, and after reaching the preset number of training times, the model parameters corresponding to the optimal value of the verification indicator (the optimal value can be the minimum value of all verification indicators, or the maximum value of all verification indicators, and a threshold can also be set, and the verification indicator closest to the threshold can be taken as the optimal value. The optimal value can be determined based on the actual formula used) are determined as the final model parameters of the image super-resolution model.

[0143] However, when the number of iterative training is too many, if all the verification indicators and model parameters in the training phase are saved, a large amount of memory space will be required. Therefore, in order to reduce the use of memory space, only the optimal values ​​of the verification indicators and their corresponding model parameters in all current training rounds can be stored in the training phase, and other verification indicators and model parameters are not stored. In an embodiment of the present application, as an example, an initial preset verification indicator can be set, and during the training process, the obtained verification indicator is compared with the preset verification indicator.

[0144] If the verification index is greater than or equal to the preset verification index, it means that the image super-resolution reconstruction effect of the optimized image super-resolution model is better than the image super-resolution reconstruction effect of the currently stored optimal image super-resolution model, that is, the optimized image super-resolution model is the model with the best image super-resolution reconstruction effect among all the image super-resolution models obtained by the current iterative training, and the optimal image super-resolution model currently stored needs to be updated.

[0145] The currently stored optimal image super-resolution model and the current preset verification index are updated. In the next verification process, the model verification index of the optimized image super-resolution model must be greater than or equal to the new preset verification index to become the optimal image super-resolution model. In addition, when the optimized image super-resolution model is updated to the optimal image super-resolution model, since the network module in the image super-resolution model usually takes up a large storage space, only the model parameters of the image super-resolution model (that is, the network weights of the model) can be stored. When the image super-resolution model needs to be called, the network module corresponding to the image super-resolution model is first constructed, and then the stored model parameters are loaded into the network module.

[0146] If the verification index is less than the preset verification index, it means that the image super-resolution reconstruction effect of the optimized image super-resolution model is not as good as the image super-resolution reconstruction effect of the currently stored optimal image super-resolution model, that is, the currently stored optimal image super-resolution model is the model with the best image super-resolution reconstruction effect among all image super-resolution models obtained by the current iterative training, and there is no need to update the currently stored optimal image super-resolution model.

[0147] When the current training times of the image super-resolution model reaches the preset training times, the model parameters of the currently stored optimal image super-resolution model are the final model parameters of the image super-resolution model, that is, the model parameters loaded into the initialized image super-resolution model when the image super-resolution reconstruction method is used to perform super-resolution reconstruction on the super-resolution image.

[0148] In the above embodiment, the image super-resolution model is trained by using high- and low-resolution images, so that the image super-resolution model can be continuously optimized through the loss function during the training process to improve the image super-resolution reconstruction effect of the image super-resolution model. In addition, by obtaining the verification index of the image super-resolution model, and saving the model parameters corresponding to the optimal value of the verification index as the final model parameters of the image super-resolution model, it is ensured that the model parameters finally saved are the model parameters corresponding to the model with the best image super-resolution reconstruction effect among all the image super-resolution models generated during the training process.

[0149] In order to further ensure that the trained image super-resolution model is the model with the best image super-resolution reconstruction effect during the training process, in some embodiments of the present application, such as Fig.13 As shown, Fig.13 The flowchart of the calculation process of the verification index in the training step of the image super-resolution model provided in the embodiment of the present application is shown. Steps S510 to S530 can be used to replace step S500, which specifically includes the following:

[0150] Step S510: Acquire a verification image pair, where the verification image pair includes a low-resolution verification image and its corresponding high-resolution verification image.

[0151] The verification image pair is used to verify the image super-resolution reconstruction effect of the image super-resolution model to determine the performance of the image super-resolution model. Before training the image super-resolution model, multiple high-resolution images can be extracted from the data set as high-resolution verification images, and then each high-resolution verification image is degraded according to at least one of the multiple image degradation sizes to obtain at least one low-resolution verification image corresponding to each high-resolution verification image. Finally, each low-resolution verification image and its corresponding high-resolution verification image are used as a verification image pair. You can also download a public data set (such as the Canon subset of the RealSR data set, which includes 50 sets of verification image pairs) as a verification data set.

[0152] Step S520: Obtain a super-resolution image corresponding to the verification image pair according to the size corresponding to the high-resolution verification image and the low-resolution verification image through the optimized image super-resolution model.

[0153] The size of the high-resolution verification image and the low-resolution verification image in the verification image pair are sent to the optimized image super-resolution model, so that the optimized image super-resolution model infers the low-resolution verification image and generates a corresponding second super-resolution image according to the size of the high-resolution verification image. The specific process of the optimized image super-resolution model generating the second super-resolution image is shown in the above steps S310 to S370.

[0154] Step S530: determining a model verification index according to the high-resolution verification image in each verification image pair and the second super-resolution image corresponding to the verification image pair.

[0155] Among them, the verification index is used to evaluate the image quality of the super-resolution image obtained by the image super-resolution model. Specifically, one or more of the indicators such as PSNR, SSIM, NIQE, DISTS, LPIPS, etc. can be used to obtain the model verification index.

[0156] In the above embodiment, the image super-resolution reconstruction effect of each image super-resolution model generated during the training process is verified by verifying image pairs. Specifically, the model verification index corresponding to each image super-resolution model is obtained, and the trained image super-resolution model is determined according to the model verification index, so that the image super-resolution model trained by the above-mentioned image super-resolution model training method embodiment must be the model with the best image super-resolution reconstruction effect among all the image super-resolution models generated during the training process.

[0157] An embodiment of the present application provides a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned image super-resolution reconstruction method embodiment is implemented.

[0158] An embodiment of the present application provides a computer program, which can be executed by a processor to implement the above-mentioned image super-resolution reconstruction method embodiment.

[0159] An embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the above-mentioned image super-resolution reconstruction method embodiment is implemented.

[0160] In several embodiments provided in the present application, if any function is implemented in the form of a software function module / unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, part or all of the technical solution of the present application can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, server or other electronic device) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (RandomAccess Memory, RAM), disk or optical disk and other media that can store computer program code.

[0161] The algorithm or display provided here are not inherently related to any specific computer, virtual system or other equipment. Various general systems can also be used together with the teaching based on this. According to the above description, it is obvious to construct the structure required for this type of system. In addition, the present application embodiment is not directed to any specific programming language yet. It should be understood that various programming languages ​​can be utilized to realize the content of the present application described here, and the above description of specific languages ​​is to disclose the best mode of implementation of the present application.

[0162] It should be noted that the above embodiments illustrate the present application rather than limit the present application, and that those skilled in the art may design alternative embodiments without departing from the scope of the appended claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present application may be implemented with the aid of hardware comprising a number of different elements and with the aid of a suitably programmed computer. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be understood as limiting the order of execution.

[0163] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for super-resolution image reconstruction, characterized in that: The method comprises: Acquire the size of the target image and the image to be super-resolved, wherein the size of the image to be super-resolved is smaller than the size of the target image; Initializing an image super-resolution model, and loading model parameters saved in the training phase of the image super-resolution model into the image super-resolution model; Performing a first interpolation up-sampling process on the image to be super-resolved according to the size of the target image to obtain a first target feature image; Extract pixel information from the image to be super-resolved to obtain a target shallow feature map; Extracting semantic information from the target shallow feature map to obtain a target deep feature map; Performing a residual connection between the target shallow feature map and the target deep feature map to obtain a target fused feature map; Performing a second interpolation upsampling process on the target fusion feature map according to the size of the target image to obtain a second target feature image; Performing residual connection on the first target feature image and the second target feature image to obtain a target fused image; and Refinement and denoising are performed on the target fused image to obtain a target super-resolution image.

2. The image super-resolution reconstruction method according to claim 1, characterized in that: The training steps of the image super-resolution model include: Step S200: constructing a high-resolution and low-resolution image pair, wherein the high-resolution and low-resolution image pair includes a high-resolution image and its corresponding low-resolution image; Step S300: sending the size of the high-resolution image and the low-resolution image to the image super-resolution model to obtain a super-resolution image corresponding to the high-resolution and low-resolution image pair; Step S400: determining a loss function according to the high-resolution image and its corresponding super-resolution image, and optimizing the image super-resolution model according to the loss function to obtain model parameters; Step S500: loading the model parameters into the image super-resolution model, and calculating the verification index of the image super-resolution model; Step S600: determining whether the current training times of the image super-resolution model reaches a preset training times; Step S700a: if the current training times of the image super-resolution model does not reach the preset training times, then looping through steps S300 to S500 to obtain the optimal value of the verification index and its corresponding model parameter; Step S700b: If the current training times of the image super-resolution model reaches the preset training times, the model parameters corresponding to the optimal values ​​of the verification indicators are saved as the final model parameters of the image super-resolution model.

3. The image super-resolution reconstruction method according to claim 2, characterized in that: The step S200 includes: Acquire a first high-resolution image and a size corresponding to the first high-resolution image; Performing arbitrary reduction processing on the sizes corresponding to the first high-resolution image respectively to obtain at least one cropping size corresponding to the first high-resolution image; Cropping the first high-resolution image according to at least one cropping size corresponding to the first high-resolution image to obtain a second high-resolution image; performing image degradation processing on the second high-resolution image according to a preset degraded image size to obtain a low-resolution image corresponding to the second high-resolution image; The second high-resolution image and its corresponding low-resolution image are determined as the high-low-resolution image pair.

4. The image super-resolution reconstruction method according to claim 3, characterized in that: The image degradation processing comprises the following steps: Obtaining an execution probability of second-order degradation in the image degradation processing; determining whether to perform the second-order degradation on the second high-resolution image according to the execution probability of the second-order degradation; If the second-order degradation is performed on the second high-resolution image, the first-order degradation in the image degradation processing is performed on the second high-resolution image according to the preset degraded image size, and the second-order degradation is performed on the image after the first-order degradation according to the preset degraded image size, so as to obtain a low-resolution image corresponding to the second high-resolution image; If the second-order degradation is not performed on the second high-resolution image, the first-order degradation is performed on the second high-resolution image according to the preset degraded image size to obtain a low-resolution image corresponding to the second high-resolution image.

5. The image super-resolution reconstruction method according to claim 2, characterized in that: The step S300 includes: Performing a first interpolation up-sampling process on the low-resolution image according to a size corresponding to the high-resolution image, to obtain a first feature image corresponding to the low-resolution image; Extract pixel information from the low-resolution image to obtain a shallow feature map; Extracting semantic information from the shallow feature map to obtain a deep feature map; Performing residual connection on the shallow feature map and the deep feature map to obtain a fused feature map; Performing a second interpolation up-sampling process on the fused feature map according to a size corresponding to the high-resolution image to obtain a second feature image; Performing residual connection on the first feature image and the second feature image to obtain a fused image; Refinement and denoising are performed on the fused image to obtain a super-resolution image corresponding to the high-resolution and low-resolution image pair.

6. The image super-resolution reconstruction method according to claim 2, characterized in that: The step S400 includes: According to the high-resolution image and its corresponding super-resolution image, using a loss function to calculate the loss of the high-resolution image and its corresponding super-resolution image; Calculating the gradient of the loss relative to the model parameters of the image super-resolution model using a loss function; The model parameters of the image super-resolution model are updated by back-propagating the gradient.

7. The image super-resolution reconstruction method according to claim 6, characterized in that: The using of a loss function to calculate the loss of the high-resolution image and its corresponding super-resolution image specifically includes: Calculating an image error between the high-resolution image and its corresponding super-resolution image, and determining a reconstruction loss according to the image error; Respectively acquiring frequency domain information of the high-resolution image and frequency domain information of its corresponding super-resolution image; Determining frequency domain loss according to frequency domain information of the high-resolution image and frequency domain information of its corresponding super-resolution image; Determine the loss function according to the reconstruction loss and the frequency domain loss; The loss of the high-resolution image and its corresponding super-resolution image is calculated according to the loss function.

8. The image super-resolution reconstruction method according to claim 2, characterized in that: The step S500 includes: Acquire a verification image pair, the verification image pair comprising a low-resolution verification image and its corresponding high-resolution verification image; Acquire a super-resolution image corresponding to the verification image pair according to the size corresponding to the high-resolution verification image and the low-resolution verification image through the optimized image super-resolution model; The verification index of the image super-resolution model is determined according to the high-resolution verification image in the verification image pair and the super-resolution image corresponding to the verification image pair.

9. An electronic device, comprising a memory and a processor, characterized in that: The memory is used to store computer program code and transmit the computer program code to the processor; The processor is used to execute the image super-resolution reconstruction method according to any one of claims 1 to 8 according to the instructions in the program code.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store computer program codes, and when the computer program codes are executed by a processor, the image super-resolution reconstruction method according to any one of claims 1 to 8 is implemented.

Images