Image super-resolution method, system and related device

By combining convolutional networks and residual networks with hidden image models, using global residual learning and sinusoidal period activation layer, the problem of high computational volume of deep neural networks in image super resolution is solved, and the rapid and efficient image reconstruction effect is achieved .

CN114022348BActive Publication Date: 2024-11-01INSPUR (BEIJING) ELECTRONICS INFORMATION IND CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111163138.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-09-30
Publication Date
2024-11-01
Estimated Expiration
2041-09-30

AI Technical Summary

Technical Problem

The existing deep neural network models have large parameters and high computational volume in image super-resolution tasks, resulting in serious demands for storage space and memory access bandwidth during deployment, making it difficult to achieve efficient image resolution improvement.

Method used

Image features are extracted using convolutional networks and residual networks, and processed through hidden image models. Global residual learning and sinusoidal period activation layer are used to realize high-resolution reconstruction of low-resolution images, simplifying the model structure to improve computational efficiency .

Benefits of technology

It realizes fast convergence and high-fidelity image reconstruction effects, and can generate super-resolution images of specified sizes within seconds on the GPU, reducing computing and storage requirements.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114022348B_ABST
    Figure CN114022348B_ABST
Patent Text Reader

Abstract

The present application provides a method for super-resolution of an image, including: obtaining a target image and a target image scale; performing low-multiple random downsampling on the target image to obtain a low-resolution image; using a convolutional network to process the low-resolution image to obtain shallow features, and using a residual dense layer to fuse all shallow features of the low-resolution image to obtain deep features; combining shallow features and deep features through global residual learning to obtain low-resolution global dense features; inputting the target image scale and the global dense features into a hidden image model, and outputting the corresponding super-resolution image. The present application uses a convolutional network and a residual network to represent and extract an image, and then uses a hidden image model to process it, which has a faster convergence speed and can convert the input image into a super-resolution image of a specified size. The present application also provides an image super-resolution system, a computer-readable storage medium, and an electronic device, which have the above-mentioned beneficial effects.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to an image super-resolution method, system and related devices. Background Art

[0002] Super Resolution (SR) is the process of restoring a high-resolution (HR) image from a given low-resolution (LR) image, and is a classic application of computer vision. SR refers to the reconstruction of a corresponding high-resolution image from an observed low-resolution image (in other words, improving the resolution) through software or hardware methods. It has important application value in monitoring equipment, satellite image remote sensing, digital high-definition, microscopic imaging, video coding communication, video restoration, and medical imaging. Today's neural networks are generally very deep and have many channels. For example, models such as VGG and ResNe have huge parameters. The inference process requires a lot of storage space, computing power, and high memory access bandwidth, which brings serious challenges to the deployment process. Summary of the invention

[0003] The purpose of this application is to provide an image super-resolution method, system, computer-readable storage medium and electronic device, which can output super-resolution images of any scale.

[0004] In order to solve the above technical problems, the present application provides an image super-resolution method, and the specific technical solution is as follows:

[0005] Obtain a target image and a target image scale;

[0006] Performing low-multiple random downsampling on the target image to obtain the low-resolution image;

[0007] Processing the low-resolution image using a convolutional network to obtain shallow features, and fusing all the shallow features of the low-resolution image using a residual dense layer to obtain deep features;

[0008] Combining the shallow features and the deep features through global residual learning to obtain the low-resolution global dense features;

[0009] The target image scale and the global dense features are input into a latent image model, and a corresponding super-resolution image is output.

[0010] Optionally, before inputting the latent image model according to the target image scale to obtain the corresponding super-resolution image, the method further includes:

[0011] The latent image model is obtained by performing feature training using the image data set.

[0012] Optionally, the performing feature training using the image data set to obtain the latent image model includes:

[0013] Obtain an image data set, randomly downsample a target image in the image data set to obtain low-resolution images at 2 times, 3 times, and 4 times, and group the low-resolution images and the high-resolution images in pairs as a training set;

[0014] The low-resolution image is subjected to a feature extraction module to obtain a low-resolution feature map of the same size as the input image;

[0015] Upsampling the low-resolution feature map and the corresponding coordinate information through a hidden image layer to obtain a super-resolution image;

[0016] An L1 loss is calculated for the super-resolution image and the target image, and the encoder and the latent image layer are reversely updated using the L1 loss to obtain the latent image model.

[0017] Optionally, inputting the target image scale and the global dense features into a latent image model, and outputting a corresponding super-resolution image comprises:

[0018] Inputting the target image scale and the global dense features into a latent image model;

[0019] Obtaining pixel information corresponding to the super-resolution according to the target image scale;

[0020] Mapping the global dense features containing continuous image domain coordinate information from the two-dimensional space to the RGB space using a sinusoidal periodic activation layer;

[0021] The global dense feature is calculated using a rectangle consisting of four latent features (upper left, upper right, lower left, and lower right) of the global dense feature and a loss function, and a corresponding super-resolution image is output.

[0022] Optionally, the sinusoidal periodic activation layer is composed of sinusoidal linear layers, each of the sinusoidal linear layers is composed of a linear layer and a positively selected function and includes preset fixed parameters, and the sinusoidal linear layers are connected in a sequence.

[0023] Optionally, if the target sinusoidal linear layer is not the last layer, the parameter output of the target linear layer in the target sinusoidal linear layer is filled with sampled values ​​from the first uniform distribution;

[0024] If the target sinusoidal linear layer is the last layer, the parameter output of the target linear layer in the target sinusoidal linear layer is filled with sampled values ​​from the second uniform distribution.

[0025] Optionally, the loss function is Among them, x i is a low-resolution image, yi is the corresponding target image, and G is the latent image model.

[0026] The present application also provides an image super-resolution system, comprising:

[0027] An acquisition module, used for acquiring a target image and a target image scale;

[0028] A downsampling module, used for performing random downsampling of the target image at a low multiple to obtain the low-resolution image;

[0029] A feature extraction module, used for processing the low-resolution image using a convolutional network to obtain shallow features, and fusing all the shallow features of the low-resolution image using a residual dense layer to obtain deep features;

[0030] A feature learning module, used for combining the shallow features and the deep features through global residual learning to obtain the low-resolution global dense features;

[0031] The image output module is used to input the target image scale and the global dense features into a latent image model and output a corresponding super-resolution image.

[0032] The present application also provides a computer-readable storage medium having a computer program stored thereon, and the computer program implements the steps of the above-mentioned method when executed by a processor.

[0033] The present application also provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps of the above-mentioned method when calling the computer program in the memory.

[0034] The present application provides an image super-resolution method, comprising: acquiring a target image and a target image scale; performing low-multiple random downsampling on the target image to obtain the low-resolution image; using a convolutional network to process the low-resolution image to obtain shallow features, and using a residual dense layer to fuse all the shallow features of the low-resolution image to obtain deep features; combining the shallow features and the deep features through global residual learning to obtain the low-resolution global dense features; inputting the target image scale and the global dense features into a latent image model, and outputting the corresponding super-resolution image.

[0035] This application uses convolutional networks and residual networks to represent and extract images, and then uses latent image models for processing. Compared with MLP+ReLU or other structures, the latent image module converges faster and often only takes a few seconds on the GPU to obtain high-fidelity image reconstruction effects. Therefore, this application can convert the input image into a super-resolution image of a specified size.

[0036] The present application also provides an image super-resolution system, a computer-readable storage medium, and an electronic device, which have the above-mentioned beneficial effects and are not described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0038] Figure 1 A flowchart of an image super-resolution method provided in an embodiment of the present application;

[0039] Figure 2 A schematic diagram of the structure used for feature extraction and the residual dense layer thereof provided in an embodiment of the present application;

[0040] Figure 3 A schematic diagram of the structure of an image super-resolution system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0042] Please refer to Figure 1 , Figure 1 A flowchart of a super-resolution method for an image provided in an embodiment of the present application, the specific scheme includes:

[0043] S101: Acquire a target image and a target image scale;

[0044] S102: performing low-multiple random downsampling on the target image to obtain the low-resolution image;

[0045] S103: Processing the low-resolution image using a convolutional network to obtain shallow features, and fusing all the shallow features of the low-resolution image using a residual dense layer to obtain deep features;

[0046] S104: combining the shallow features and the deep features through global residual learning to obtain the low-resolution global dense features;

[0047] S105: Input the target image scale and the global dense features into a latent image model, and output a corresponding super-resolution image.

[0048] In order to better describe the above process, the following first describes how to obtain the latent image model. This embodiment only requires that the latent image model be generated or obtained before step S105 is executed. Specifically, it is necessary to use the image data set for feature training to obtain the latent image model. The specific process is as follows:

[0049] The first step is to obtain an image data set, randomly downsample the target image in the image data set to obtain low-resolution images at 2 times, 3 times and 4 times, and group the low-resolution images and high-resolution images in pairs as training sets;

[0050] The second step is to obtain a low-resolution feature map of the same size as the input image through a feature extraction module on the low-resolution image;

[0051] Step 3: Up-sampling the low-resolution feature map and the corresponding coordinate information through a hidden image layer to obtain a super-resolution image;

[0052] Step 4: Calculate the L1 loss for the super-resolution image and the target image, and use the L1 loss to reversely update the encoder and the latent image layer to obtain the latent image model.

[0053] In the second step of image feature extraction, a low-resolution image is input to obtain the feature map of the low-resolution image. The low-resolution image is passed through the first convolutional network to obtain shallow features, and then a residual dense layer is gradually used to adaptively fuse all the hierarchical features of the low-resolution image. Finally, the shallow features and deep features of the original low-resolution image are combined through global residual learning to obtain global dense features. Figure 2 , Figure 2 A schematic diagram of the structure used for feature extraction and the residual dense layer therein provided in an embodiment of the present application.

[0054] In the third step, each consecutive image I i It can be expressed as a two-dimensional feature form M i ∈R H*W*D , then the super-resolution image can be expressed as s = f(z, [x, c]), where z is the image representation vector, x represents the continuous image domain coordinates, and c = [c h , c w ] indicates the width and height information of the pixel, s is the prediction signal (that is, the RGB value), and f(Z,.) can be represented as a continuous image, specifically a sinusoidal periodic activation layer, which maps the coordinates to the RGB space.

[0055] In M i The H*W feature vectors are evenly distributed in the two-dimensional space of the continuous image domain. in Indicates M i Middle distance x q The nearest hidden feature, then (00, 01, 10, 11) in t represent the distance from x q The nearest four hidden features: upper left, upper right, lower left, and lower right u * represents the coordinates corresponding to the hidden features, S t Represents x q and The area of ​​the rectangle formed between them is normalized.

[0056] The sinusoidal periodic activation layer used in the continuous image f(Z,.) is composed of sinusoidal linear layers. Each sinusoidal linear layer is composed of a linear layer and a sinusoidal function, which is then multiplied by a preset fixed parameter omega, and the formula is expressed as sin(omega*linear(x)).

[0057] The linear layer in the sinusoidal linear layer requires specific parameter initialization. If the sinusoidal linear layer is not the last layer, the parameter initialization of the linear layer needs to sample values ​​in a uniform distribution. If it is the last layer, the value is sampled from a uniform distribution. Where in_features is the input feature length and omega is a fixed parameter.

[0058] After the sinusoidal linear layers are constructed, they are connected in sequence, that is, the output of the previous layer is the input of the next layer, thus forming a sinusoidal periodic activation layer. The loss function of the above latent image model is specifically:

[0059]

[0060] Among them, x i is a low-resolution image, y i is the corresponding high-resolution original image, and G is the super-resolution model of the present invention. L1 loss is used to reversely update the encoder and the latent image layer, where the encoder refers to the encoder used in the encoding process.

[0061] In the process of super-resolution processing of an image, a target image and a target image scale are first obtained. The target image scale is usually set by a user, and S101 can receive the set image scale input by the user.

[0062] Thereafter, the target image is randomly downsampled to construct low-resolution images with low magnifications such as 2x, 3x, and 4x downsampling, and these pairs of low-resolution images and the original high-resolution image are used as training data sets, that is, the low-resolution image with 2x downsampling and the high-resolution image are grouped together, and so on. It should be noted that the low magnification in step S102 usually refers to no more than 10x.

[0063] Then, image blocks of size 48*48 are cropped from the training dataset, and these images are converted into pixel sample pairs, i.e., coordinate-image pairs, because the input of the latent image model in the following text is the coordinate and image feature map.

[0064] In S103, the low-resolution image is processed by a convolutional network to obtain shallow features, and all the shallow features of the low-resolution image are fused by a residual dense layer to obtain deep features, the purpose of which is to encode the low-resolution image into a feature map of the same size. The size of the input image is encoded in a grid form and stored in a 2D array. The innermost dimension stores the coordinates of the grid, and the storage method starts from the lower right corner, from bottom to top, and from left to right.

[0065] The low-resolution image feature map and the target image scale are input into the latent image model to generate a super-resolution image.

[0066] In S105, the following steps are specifically performed:

[0067] S1051: Inputting the target image scale and the global dense features into a latent image model;

[0068] S1052: Obtaining pixel information corresponding to the super-resolution according to the target image scale;

[0069] S1053: Mapping the global dense features containing continuous image domain coordinate information from the two-dimensional space to the RGB space using a sinusoidal periodic activation layer;

[0070] S1054: Calculate the global dense feature using the rectangle formed by the four latent features of the upper left, upper right, lower left and lower right of the global dense feature and the loss function, and output the corresponding super-resolution image.

[0071] The specific contents are similar to the operations in constructing the latent image model above, and will not be repeated here.

[0072] This application proposes a super-resolution method of arbitrary scale, which uses a single model to perform super-resolution of low-resolution images at arbitrary scales, and can have more flexible magnifications than fixed-scale super-resolution; compared with other models of arbitrary scale magnification, the use of SIREN network as an implicit function can better fit high-frequency information, so that the details of the image magnified by the model are better. By using convolutional networks and residual networks to represent and extract images, and then using latent image models for processing, the latent image module converges faster than MLP+ReLU or other structures, and often only takes a few seconds on the GPU to obtain high-fidelity image reconstruction effects. Therefore, this application can convert the input target image into a super-resolution image of a specified size.

[0073] The following is an introduction to an image super-resolution system provided in an embodiment of the present application. The super-resolution system described below and the image super-resolution method described above can be referenced to each other.

[0074] See also Figure 3 , Figure 3 This is a schematic diagram of the structure of an image super-resolution system provided in an embodiment of the present application. The present application also provides an image super-resolution system, including:

[0075] An acquisition module, used for acquiring a target image and a target image scale;

[0076] A downsampling module, used for randomly downsampling the target image at a low multiple to obtain the low-resolution image;

[0077] A feature extraction module, used for processing the low-resolution image using a convolutional network to obtain shallow features, and fusing all the shallow features of the low-resolution image using a residual dense layer to obtain deep features;

[0078] A feature learning module, used for combining the shallow features and the deep features through global residual learning to obtain the low-resolution global dense features;

[0079] The image output module is used to input the target image scale and the global dense features into a latent image model and output a corresponding super-resolution image.

[0080] Based on the above embodiment, as a more preferred embodiment, it may also include:

[0081] The training module is used to perform feature training using the image data set to obtain the latent image model.

[0082] Based on the above embodiment, as a more preferred embodiment, the training module includes:

[0083] A first acquisition unit is used to acquire an image data set, randomly downsample a target image in the image data set to obtain low-resolution images at 2 times, 3 times, and 4 times, and group the low-resolution images and the high-resolution images in pairs as a training set;

[0084] A second acquisition unit is used to obtain a low-resolution feature map with the same size as the input image through a feature extraction module for the low-resolution image;

[0085] A third acquisition unit is used to upsample the low-resolution feature map and the corresponding coordinate information through a latent image layer to obtain a super-resolution image;

[0086] The fourth acquisition unit is used to calculate the L1 loss for the super-resolution image and the target image, and use the L1 loss to reversely update the encoder and the latent image layer to obtain the latent image model.

[0087] Based on the above embodiment, as a more preferred embodiment, the image output module includes:

[0088] An input unit, used for inputting the target image scale and the global dense feature into a latent image model;

[0089] A fifth acquisition unit, configured to obtain pixel information corresponding to the super-resolution according to the target image scale;

[0090] A mapping unit, configured to map the global dense features including continuous image domain coordinate information from a two-dimensional space to an RGB space using a sinusoidal periodic activation layer;

[0091] The output unit is used to calculate the global dense feature using a rectangle formed by four latent features of the upper left, upper right, lower left and lower right of the global dense feature and a loss function, and output a corresponding super-resolution image.

[0092] Based on the above embodiments, as a more preferred embodiment, the sinusoidal periodic activation layer is composed of sinusoidal linear layers, each of the sinusoidal linear layers is composed of a linear layer and a positively selected function, and contains preset fixed parameters, and each of the sinusoidal linear layers is connected in a sequence.

[0093] Based on the above embodiment, as a more preferred embodiment, if the target sinusoidal linear layer is not the last layer, the parameter output of the target linear layer in the target sinusoidal linear layer is filled with sampled values ​​from the first uniform distribution;

[0094] If the target sinusoidal linear layer is the last layer, the parameter output of the target linear layer in the target sinusoidal linear layer is filled with sampled values ​​from the second uniform distribution.

[0095] Based on the above embodiment, as a more preferred embodiment, the loss function is Among them, x i is a low-resolution image, y i is the corresponding target image, and G is the latent image model.

[0096] The present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed, the steps provided in the above embodiment can be implemented. The storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0097] The present application also provides an electronic device, which may include a memory and a processor, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, the steps provided in the above embodiment may be implemented. Of course, the electronic device may also include various network interfaces, power supplies and other components.

[0098] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other. For the system provided in the embodiment, since it corresponds to the method provided in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0099] Specific examples are used herein to illustrate the principles and implementation methods of the present application, and the description of the above embodiments is only used to help understand the method and core ideas of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

[0100] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

Claims

1. A super-resolution method for an image, characterized in that: include: Obtain a target image and a target image scale; Performing random downsampling of the target image at a low multiple to obtain a low-resolution image; Processing the low-resolution image using a convolutional network to obtain shallow features, and fusing all the shallow features of the low-resolution image using a residual dense layer to obtain deep features; By combining the shallow features and the deep features through global residual learning, low-resolution global dense features are obtained; Inputting the target image scale and the global dense features into a latent image model, and outputting a corresponding super-resolution image; Inputting the target image scale and the global dense features into a latent image model, and outputting a corresponding super-resolution image comprises: Inputting the target image scale and the global dense features into a latent image model; Obtaining pixel information corresponding to the super-resolution according to the target image scale; Mapping the global dense features containing continuous image domain coordinate information from the two-dimensional space to the RGB space using a sinusoidal periodic activation layer; The global dense feature is calculated using a rectangle consisting of four latent features (upper left, upper right, lower left, and lower right) of the global dense feature and a loss function, and a corresponding super-resolution image is output.

2. The image super-resolution method according to claim 1, characterized in that: Before inputting the latent image model according to the target image scale to obtain the corresponding super-resolution image, the method further includes: The latent image model is obtained by performing feature training using the image data set.

3. The image super-resolution method according to claim 2, characterized in that: The method of performing feature training using the image data set to obtain the latent image model comprises: Obtain an image data set, randomly downsample a target image in the image data set to obtain low-resolution images at 2 times, 3 times, and 4 times, and group the low-resolution images and the high-resolution images in pairs as a training set; The low-resolution image is subjected to a feature extraction module to obtain a low-resolution feature map of the same size as the input image; Upsampling the low-resolution feature map and the corresponding coordinate information through a hidden image layer to obtain a super-resolution image; An L1 loss is calculated for the super-resolution image and the target image, and the encoder and the latent image layer are reversely updated using the L1 loss to obtain the latent image model.

4. The image super-resolution method according to claim 1, characterized in that: The sinusoidal periodic activation layer is composed of sinusoidal linear layers, each of which is composed of a linear layer and a positively selected function and contains preset fixed parameters, and each of the sinusoidal linear layers is connected in a sequence.

5. The image super-resolution method according to claim 1, characterized in that: If the target sinusoidal linear layer is not the last layer, the parameter output of the target linear layer in the target sinusoidal linear layer is filled with sampled values ​​from the first uniform distribution; If the target sinusoidal linear layer is the last layer, the parameter output of the target linear layer in the target sinusoidal linear layer is filled with sampled values ​​from the second uniform distribution.

6. The image super-resolution method according to claim 1, characterized in that: The loss function is Among them, x i is a low-resolution image, y i is the corresponding target image, and G is the latent image model.

7. A super-resolution system for an image, characterized in that: include: An acquisition module, used for acquiring a target image and a target image scale; A downsampling module, used for randomly downsampling the target image at a low multiple to obtain a low-resolution image; A feature extraction module, used for processing the low-resolution image using a convolutional network to obtain shallow features, and fusing all the shallow features of the low-resolution image using a residual dense layer to obtain deep features; A feature learning module, used for combining the shallow features and the deep features through global residual learning to obtain low-resolution global dense features; An image output module, used for inputting the target image scale and the global dense features into a latent image model, and outputting a corresponding super-resolution image; The image output module includes: An input unit, used for inputting the target image scale and the global dense feature into a latent image model; A fifth acquisition unit, configured to obtain pixel information corresponding to the super-resolution according to the target image scale; A mapping unit, configured to map the global dense features including continuous image domain coordinate information from a two-dimensional space to an RGB space using a sinusoidal periodic activation layer; The output unit is used to calculate the global dense feature using a rectangle formed by four latent features of the upper left, upper right, lower left and lower right of the global dense feature and a loss function, and output a corresponding super-resolution image.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the image super-resolution method according to any one of claims 1 to 6 are implemented.

9. An electronic device, characterized in that: The method comprises a memory and a processor, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, the steps of the image super-resolution method according to any one of claims 1 to 6 are implemented.