Image restoration method

Abstract

The invention discloses an image restoration method. The image restoration method comprises the following steps: acquiring a to-be-processed original image; sampling processing is carried out on the original image, feature information between adjacent areas in the original image is obtained through a deep network in the sampling process, pixel point amplification processing is carried out on the feature information, and the relative spatial position of each pixel point before and after amplification is not changed; performing inverse convolution processing on the feature information after pixel point amplification to obtain a target image, the target image comprising a plurality of pixel blocks; and deleting the pixel blocks of which the restoration degrees are less than a restoration degree threshold value in the obtained target image, and synthesizing the pixel blocks of which the restoration degrees are greater than or equal to the restoration degree threshold value to obtain a restored image.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to an image restoration method. Background Art

[0002] Cloud disk is a professional Internet storage tool and a product of Internet cloud technology. It provides information storage, reading, downloading and other services for enterprises and individuals through the Internet. It is safe, stable and has massive storage. It can provide users with storage backup, multi-terminal synchronization, online management, group sharing and other services for digital assets such as files, mobile phone pictures and contacts. Cloud disk can also provide picture beautification function for the convenience of users.

[0003] In the related art, the cloud disk picture beautification function is mainly to beautify pictures with portraits in the photos to enhance the beauty of the portraits. However, with the increasing number of images stored in the cloud disk by users, the quality of the uploaded images has also become uneven. For example, pictures taken earlier with low-pixel cameras will have problems such as blurry photos and low resolution. When users use the cloud disk to browse these pictures, they will have a bad viewing experience. Summary of the invention

[0004] The embodiment of the present application provides an image restoration method to solve the problem that image beautification in the related art cannot solve the problem of blurred photos and low resolution.

[0005] In a first aspect, an embodiment of the present application provides an image restoration method, comprising: Obtain the original image to be processed; The original image is sampled, and during the sampling process, feature information between adjacent regions in the original image is obtained using a deep network, and pixel amplification is performed on the feature information, wherein the relative spatial positions of the pixels before and after the amplification remain unchanged; Performing a deconvolution process on the feature information after pixel point amplification to obtain a target image, wherein the target image includes a plurality of pixel blocks; Pixel blocks with a restoration degree less than a restoration degree threshold in the obtained target image are deleted, and pixel blocks with a restoration degree greater than or equal to the restoration degree threshold are synthesized to obtain a restored image.

[0006] In a second aspect, an embodiment of the present application provides an image restoration device, comprising: An acquisition module, used for acquiring the original image to be processed; A sampling module is used to perform sampling processing on the original image, and in the sampling process, a deep network is used to obtain feature information between adjacent regions in the original image, and the feature information is subjected to pixel point amplification processing, wherein the relative spatial position of each pixel point before and after amplification remains unchanged; A processing module, used for performing a deconvolution process on the feature information after pixel point amplification to obtain a target image, wherein the target image includes a plurality of pixel blocks; The determination module is used to delete pixel blocks with a restoration degree less than a restoration degree threshold in the obtained target image, and synthesize pixel blocks with a restoration degree greater than or equal to the restoration degree threshold to obtain a restored image.

[0007] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method described in the first aspect.

[0008] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in the first aspect are implemented.

[0009] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions, which, when executed by a computer, implement the steps of the method described in the first aspect.

[0010] In an embodiment of the present application, the original image to be processed is first obtained, and then the original image is sampled. During the sampling process, a deep network is used to obtain feature information between adjacent areas in the original image, and the feature information is subjected to pixel point amplification processing, wherein the relative spatial position of each pixel before and after amplification remains unchanged, and then the feature information after pixel point amplification is subjected to inverse convolution processing to obtain a target image, wherein the target image includes multiple pixel blocks, and finally the pixel blocks in the obtained target image whose restoration degree is less than a restoration degree threshold are deleted, and the pixel blocks whose restoration degree is greater than or equal to the restoration degree threshold are synthesized to obtain a restored image. The embodiment of the present application obtains feature information between adjacent areas in the original image by sampling the original image, and performs pixel point amplification processing on the feature information, which can increase the receptive field of the original image, thereby beautifying the image while improving the image resolution. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 is a flow chart of an image restoration method provided by an embodiment of the present application; Figure 2 is a structural schematic diagram of an image restoration model provided in an embodiment of the present application; Figure 3 It is a structural diagram of the fusion coding module provided in an embodiment of the present application; Figure 4 is a schematic diagram of the structure of a high-density coding module provided in an embodiment of the present application; Figure 5 It is a structural schematic diagram of a pixel point amplification module provided in an embodiment of the present application; Figure 6 is a structural schematic diagram of a noise removal module provided in an embodiment of the present application; Figure 7 is a schematic diagram of the structure of a learning network provided in an embodiment of the present application; Figure 8 It is a schematic diagram of the structure of the discrimination network provided in the embodiment of the present application; Figure 9-10 is a schematic diagram of a user interface provided in an embodiment of the present application; Fig.11 is a schematic diagram of image comparison before and after using the image restoration method provided by the embodiment of the present application; Fig.12 is a schematic diagram of an image restoration device provided in an embodiment of the present application; Fig.13 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0012] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0013] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here. In addition, the "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally represents that the objects associated with each other are in an "or" relationship.

[0014] The following is combined with Figures 1 to 13 , an image restoration method provided by an embodiment of the present application is described in detail through specific embodiments and their application scenarios. In the related art, the beautification function in the cloud disk mainly performs beautification processing on pictures with portraits in the photos to improve the beauty of the portraits. However, with the continuous expansion of the scale of people using cloud disks, in order to further enrich the functions of cloud disks, further development is carried out for the beautification function in cloud disks. With the increasing number of images stored in cloud disks by users, the quality of the uploaded images has also become uneven. For example, pictures taken with low-pixel shooting equipment in the past will have problems of blurred photos and low resolution. When users use cloud disks to browse these pictures, it will bring a bad viewing experience to users. In order to solve the above problems, an embodiment of the present application provides an image restoration method, which increases the resolution of the image and improves the clarity of the image by increasing the receptive field of the original image.

[0015] like Figure 1 As shown, it is a flow chart of an image restoration method provided by an embodiment of the present application. Figure 1 As shown, the image repair method is applied to an electronic device, and the image repair method may include: the contents shown in S101 to S104.

[0016] In S101, an original image to be processed is obtained.

[0017] The original image may be a picture previously taken by the user using a low-pixel shooting device, and may be repaired using the image repair model provided in this embodiment.

[0018] In S102, the original image is sampled, and during the sampling process, a deep network is used to obtain feature information between adjacent regions in the original image, and the feature information is pixel-amplified, wherein the relative spatial positions of the pixels before and after the amplification remain unchanged.

[0019] Among them, the sampling process is an upsampling process, which uses a deep network to obtain more feature information between adjacent areas, and then performs pixel amplification processing to expand the receptive field without losing resolution and keeping the relative spatial position of the pixels unchanged.

[0020] In S103, the feature information after pixel amplification is subjected to inverse convolution processing to obtain a target image, wherein the target image includes a plurality of pixel blocks.

[0021] In this example, the image can be decoded through the inverse convolution operation to obtain the target image.

[0022] In S104, pixel blocks with a restoration degree less than a restoration degree threshold in the obtained target image are deleted, and pixel blocks with a restoration degree greater than or equal to the restoration degree threshold are synthesized to obtain a restored image.

[0023] In other words, the parts that do not achieve the ideal restoration effect can be removed to improve the overall clarity of the image.

[0024] In an embodiment of the present application, the original image to be processed is first obtained, and then the original image is sampled. During the sampling process, a deep network is used to obtain feature information between adjacent areas in the original image, and the feature information is subjected to pixel point amplification processing, wherein the relative spatial position of each pixel before and after amplification remains unchanged, and then the feature information after pixel point amplification is subjected to inverse convolution processing to obtain a target image, wherein the target image includes multiple pixel blocks, and finally the pixel blocks in the obtained target image whose restoration degree is less than a restoration degree threshold are deleted, and the pixel blocks whose restoration degree is greater than or equal to the restoration degree threshold are synthesized to obtain a restored image. The embodiment of the present application obtains feature information between adjacent areas in the original image by sampling the original image, and performs pixel point amplification processing on the feature information, which can increase the receptive field of the original image, thereby beautifying the image while improving the image resolution.

[0025] In a possible implementation of the present application, the original image is sampled, and during the sampling process, a deep network is used to obtain feature information between adjacent regions in the original image, and the feature information is pixel-amplified. This may include: sampling the original image, and during the sampling process, the original image is convolved through a first fusion coding module to obtain long-distance feature information of the original image, wherein the computational complexity of the long-distance feature information is lower than the computational complexity of the feature information in the original image; the long-distance feature information is processed through a shallow network convolution layer and a deep network convolution layer in a first high-density coding module to obtain feature information between multiple adjacent regions; the feature information between multiple adjacent regions is pixel-amplified through a first pixel density amplification module to obtain feature information with an increased receptive field.

[0026] Furthermore, after the first pixel density amplification module, a first noise removal module may be included.

[0027] That is, the modules in the upsampling process may include a first fusion encoding module, a first high-density encoding module, a first pixel density amplification module and a first noise removal module connected in sequence.

[0028] Among them, the first fusion coding module is used to obtain the long-distance feature information of the original image after convolution operation on the original image; the first high-density coding module is used to process the long-distance feature information output by the first fusion coding module through the shallow network convolution layer and the deep network convolution layer to obtain the feature information between multiple adjacent areas; the first pixel point density amplification module is used to perform pixel point amplification processing on the feature information between multiple adjacent areas output by the first high-density coding module to obtain the feature information with an increased receptive field; the first noise removal module is used to suppress and fill the noise features in the feature information with an increased receptive field output by the first pixel point density amplification module, and to enhance and fill the important features in the feature information with an increased receptive field to obtain enhanced feature information.

[0029] like Figure 2 The left half of the figure is the upsampling model. In terms of overall structure, the model follows the principle that in the upsampling process, the complexity of calculating features is first reduced through the fusion module encoding module, which is conducive to restoring texture details of the image to be repaired. At the same time, in the upsampling process, the high-density encoding module is used to process the features in the low-feature channels after processing, and the deep network is used to obtain more feature information between adjacent areas. Subsequently, the pixel density amplification module is used to perform pixel amplification processing to expand the receptive field without losing resolution and keep the relative spatial position of the pixels unchanged. The pixels are then input into the noise removal module. Under the input state of multiple feature channels, the noise features in the image are suppressed and filled, and the important features are strengthened and filled, so that the image is clear and more natural.

[0030] In a possible implementation of the present application, the first fusion encoding module includes multiple convolutional layers and average pooling layers.

[0031] Among them, the first fusion encoding module is used to: input the original image into two convolution layers to obtain the first image feature; input the first image feature into the average pooling layer and the maximum pooling module for convolution processing to obtain the second image feature and the third image feature; perform Hadamard product operation on the second image feature and the third image feature, and perform matrix product with the first image feature to obtain the fourth image feature; use residual connection to fuse the fourth image feature with the original image to obtain long-distance feature information.

[0032] In one example, if Figure 3 shown.

[0033] (1) The input original image first passes through two 3×3 convolutions to increase the encoder’s receptive field, which helps the encoder extract rich multi-scale detail information in the damaged image. The receptive field of the second 3×3 convolution in the two 3×3 convolution cascades is equivalent to a 5×5 convolution, which helps save model memory.

[0034] F1=Conv 3×3 (Conv 3×3 (F in )) (2) The image feature F1 after convolution is input to the average pooling layer for average pooling processing to obtain a smoother feature map, and the pooled content is input to the channel attention module to obtain the output F2.

[0035] F2=σ(δ(F fc (F ap (F1)))) Among them, F fc is a fully connected convolution operation, F ap is the average pooling operation, δ is the PReLU function, and σ is the Sigmoid activation function.

[0036] (3) At the same time, the image feature F1 after convolution is input into the maximum pooling operation module for maximum pooling operation. The content after maximum pooling is input into the 7×7 convolution for convolution processing, and finally the output F3 is obtained.

[0037] F3=σ((F fc (Max(F1))) Among them, Max is the operation of selecting the maximum value.

[0038] (4) Perform a Hadamard product operation on F2 and F3, and then perform a matrix multiplication operation with feature F1 to obtain the output weighted attention mechanism feature map of F4; F4=F1×(F2⊙F3) Where ⊙ is the Hadamard product.

[0039] After obtaining the weighted attention mechanism feature map F4, the feature F4 is fused with the input feature Fin using residual connection to obtain long-distance feature information. Finally, through a 4×4 convolution with a step size of 2, the outputs of the two fusion encoding modules are obtained, namely Fout11 and Fout12, where Fout11 is Figure 2 The output of the first fusion encoding module in , Fout12 is Figure 2 The output of the second fusion coding module in . In this embodiment, two fusion coding modules are used to better restore the texture details of the original image.

[0040] In a possible implementation of the present application, the first high-density coding module includes a high-density coding layer and a decoupling layer.

[0041] Among them, the high-density coding layer includes multiple interconnected convolution kernels with step sizes of 1 and 2 respectively, wherein the convolution kernel with a step size of 1 and the convolution kernel with a step size of 2 are connected through a gradient dissipation processing module; the decoupling layer includes multiple upsampling modules, and the convolution kernel with a step size of 1 in the high-density coding layer is input into the sampling module for upsampling processing, and the restored feature map is output, and the restored map includes feature information between multiple adjacent regions.

[0042] In this embodiment, when the image is feature extracted through the convolution layer of the high-density coding layer, the output of the shallow network convolution layer and the deep network convolution layer contain image information with different characteristics. When the image is feature extracted through the shallow network, since the overlapping area of ​​the receptive field of the convolution layer is very small, more detailed image texture information and more image details can be obtained. As the convolution layer continuously samples the image features, the receptive field of the convolution layer will gradually increase, and the overlapping area of ​​the receptive field will also increase accordingly. Therefore, the deep network can obtain more feature information between adjacent areas.

[0043] In one example, if Figure 4 shown.

[0044] (1) The input of the first high-density coding module is the output Fout12 of the second fusion coding module; (2) The features are upsampled through the convolution layer of the high-density coding layer. This layer consists of 12 convolution kernels with step sizes of 1 and 2 respectively. The convolution kernels with step sizes of 1 and 2 are connected using a gradient dissipation processing module to overcome the problem of gradient disappearance. (3) After the 3rd and 7th convolution kernels with a step size of 1, they are input into the upsampling module through the "gradient dissipation processing module" and then input into the convolution kernel with a step size of 1 for processing. The 9th convolution kernel with a step size of 1 is directly input into the upsampling module. In addition, the output of the entire high-density coding layer is also input into the upsampling module and then into the convolution kernel with a step size of 1 for processing; Among them, after the 3rd and 7th convolution kernels with a step size of 1, they are input into the upsampling module through the gradient dissipation processing module, so better features can be collected; The 9th convolution kernel with a step size of 1 is directly input to the upsampling module, which can trace back the input, making the result more accurate and able to converge quickly; Among them, the gradient dissipation processing module takes the data processed by the convolution kernel with a step size of 1 as input and inputs it into the module. First, the image edge is filtered by the zero padding module, and then it is input into the convolution module with a step size of 1. Finally, it is input into the BN layer for normalization. The processed data is combined with the data that has only been filtered by the zero padding module and then convolved with a step size of 1. After the outer product operation, the result processed by the gradient dissipation processing module is output.

[0045] (4) Finally, the decoupling layer aggregates three convolution kernels with a step size of 1 to upsample the image information output by the high-density coding layer according to the obtained input and restore the image.

[0046] Fout21 is Figure 2 The output of the first high-density encoding module in the upsampling model, Fout22 is Figure 2 The output of the second high-density encoding module in , that is, the high-density encoding module of the downsampling model part.

[0047] In a possible implementation of the present application, the first pixel density expansion module includes multiple feature extraction layers and multiple convolution layers.

[0048] Among them, the first pixel density amplification module is used to: input the feature map output by the high-density encoding module into multiple feature extraction layers for feature extraction to obtain a first feature map; input the first feature map into multiple convolution layers for convolution processing to obtain a second feature map; perform outer product operation on the second feature map and the first feature map to obtain a spatial weight feature map; sum and superimpose the spatial weight feature map and the feature map output by the high-density encoding module, and then process it through an activation function to obtain a feature map with an enlarged receptive field, and the pixel points of the feature map with the enlarged receptive field are more than the pixel points of the feature map output by the high-density encoding module.

[0049] In this embodiment, the pixel expansion module uses the output Fout21 of the high-density encoding module as the input of the module to perform pixel expansion processing to expand the receptive field without losing resolution and keep the relative spatial position of the pixels unchanged.

[0050] In this embodiment, Figure 2 Take the first pixel density amplification module in the upsampling model as an example to illustrate. Figure 5 shown.

[0051] (1) The output Fout21 of the high-density encoding module flows into the feature extraction layer composed of 3 different convolution and ReLU activation functions, and combined with the activation operation to obtain the feature map L1; Among them, the three convolutions in this layer are, from left to right: The first convolution operation is a convolution layer with a kernel of 1, a stride of 1, a padding of 0, and a dilation rate of 1, and a ReLU activation operation; The second convolution operation is a convolution layer with a kernel of 3, a stride of 1, a padding of 1, and a dilation rate of 1, and a ReLU activation operation; The second convolution operation is a convolution layer with a kernel of 1, a stride of 1, a padding of 0, and a dilation rate of 1, and a ReLU activation operation.

[0052] (2) Feature map L1 is subjected to four layers of convolution again to obtain feature map L2; Among them, the four convolutions in the 4-layer convolution operation are, from left to right: The first layer of convolution is a convolution with a kernel of 1, a stride of 1, a padding of 0, and a dilation rate of 1, and a convolution with a kernel of 3, a stride of 1, a padding of 1, and a dilation rate of 1; The second convolution layer is a convolution with a kernel of 1, a stride of 1, a padding of 0, and a dilation rate of 1, and a convolution with a kernel of 3, a stride of 1, a padding of 1, and a dilation rate of 2. The third convolution layer is a convolution with a kernel of 1, a stride of 1, a padding of 0, and a dilation rate of 1, and a convolution with a kernel of 3, a stride of 1, a padding of 1, and a dilation rate of 4. The fourth convolution layer is a convolution with a kernel of 1, a stride of 1, a padding of 0, and a dilation rate of 1, and a convolution with a kernel of 3, a stride of 1, a padding of 1, and a dilation rate of 8.

[0053] After this convolution operation, the number of image parameters is reduced while the field of view is expanded. The field of view features extracted by four layers of convolution are stacked to obtain a four-channel image.

[0054] In other words, the number of channels is doubled by performing a convolution with a convolution kernel of 1. (3) Perform outer product operation on the spatial weight feature map L1 and L2 to obtain the spatial weight feature map L3; (4) After the spatial weight feature map L3 is summed and superimposed with the input image, the outputs of the final four pixel point amplification modules are obtained through the LeakyReLU activation function, which are Fout31, Fout32, Fout33, and Fout34. Among them, Fout31 is Figure 2 The output of the first pixel density amplification module, Fout32, is Figure 2 The output of the second pixel density amplification module, Fout33, is Figure 2 The output of the third pixel density amplification module, Fout34, is Figure 2The output of the fourth pixel density amplification module in the embodiment. In this embodiment, multiple pixel density amplification modules are used to further expand the receptive field.

[0055] In a possible implementation of the present application, the first noise removal module includes a convolutional layer, a gating layer, a feature layer, an average pooling layer and a fully connected layer.

[0056] Among them, the first noise removal module is used to: input the feature map output by the first pixel density amplification module into the convolution layer for convolution processing to obtain a third feature map; input the third feature map into the gating layer and the feature layer for feature selection to obtain image channel features and spatial position features; perform product operation on the image channel features and the spatial position features to obtain a first output feature map; pass the output feature map through the average pooling layer and the fully connected layer, then through the activation function processing, and then through the fully connected layer and the activation function processing to obtain a second output feature map; perform outer product operation on the first output feature map and the second output feature map to obtain an enhanced feature image.

[0057] In this embodiment, the noise removal module mainly suppresses and fills the noise features in the image under the input state of multiple feature channels, and strengthens and fills the important features, so that the image is clear and more natural.

[0058] This embodiment uses Figure 2 Take the first noise removal module in the upsampling model as an example to illustrate. Figure 6 shown.

[0059] (1) The input is the pixel amplification module input Fout31, which first passes through a convolution layer with a convolution kernel of 2 to obtain the output Fz; Fz = Conv(Fout31) (2) Input into the gate layer kernel and feature layer respectively, perform dynamic feature selection, and obtain the output of image channel features and spatial position features; F 图像通道特征 =

[0060] F 空间位置特征 =

[0061] Among them, Wg and Wf are filters, Fz is the output after convolution, N is the number of feature layers, i is the i-th layer feature, and j is the j-th layer feature.

[0062] (3) Multiply the image channel features and spatial position features to obtain the corresponding output V; V= (F 图像通道特征)⊙σ(F 空间位置特征 ) in, is the exponential linear unit activation function, and σ is the sigmoid activation function.

[0063] (4) The output V of the image channel and spatial position features is input to the average pooling layer for average pooling, and then input to the fully connected layer for full connection operation. Then, ReLU is used for activation, and the activated features are input to the second fully connected layer, followed by Sigmoid activation to obtain the output of this branch; (5) The output of the image channel and spatial position features is outer-producted with the output in (3) to finally obtain the output of the noise removal module, where the outputs of the two noise removal modules are Fout41 and Fout42 respectively.

[0064] In a possible implementation of the present application, the downsampling model includes a second noise removal module, a second pixel density amplification module, a second high-density encoding module and a first fusion decoding module which are connected in sequence.

[0065] The first fusion decoding module is used to perform a deconvolution operation on the feature information output by the second high-density encoding module to obtain a decoded image of the enhanced feature image.

[0066] This embodiment implements image decoding by performing a deconvolution operation through a fusion decoding module.

[0067] It should be noted that the structures of the second noise removal module, the second pixel density amplification module, the second high-density encoding module, etc. are the same as the corresponding modules in the upsampling model, and will not be repeated in this embodiment, and the above embodiment shall prevail.

[0068] In a possible implementation of the present application, feature information after pixel amplification is subjected to deconvolution processing to obtain a target image, including: processing the feature information after pixel amplification through a shallow network convolution layer and a deep network convolution layer in a second high-density encoding module to obtain feature information between multiple adjacent regions; deconvolution processing is performed on the feature information between multiple adjacent regions through a main decoding module and multiple sub-decoding modules in a first fusion decoding module, and channel superposition processing is performed to obtain a target image, wherein the number of network layers in the main decoding module is the same as the number of network layers of the first fusion encoding module in the sampling process, and the multiple sub-decoding modules are used to decode features of different scales to the same size as the target features, and channel superposition is the channel superposition of features decoded by the main decoding module and features decoded by the multiple sub-decoding modules.

[0069] That is, the first fusion decoding module includes a main decoding module and multiple sub-decoding modules, and the number of network layers in the main decoding module is the same as the number of network layers of the first fusion encoding module in the upsampling model.

[0070] The first fusion decoding module is used to perform channel superposition processing on the features of different scales in the feature information output by the second high-density encoding module after deconvolution processing to obtain a decoded image of the enhanced feature image.

[0071] In one example, the fusion decoding module mainly includes a main decoding module and multiple sub-decoding modules, all of which are superpositions of deconvolution networks.

[0072] The number of network layers of the main decoding module is consistent with that of the fusion encoding module. Its main function is to restore the feature maps corresponding to each scale layer by layer. The number of decoding layers is L. For the main decoding module, the feature maps of each layer correspond to the encoding part, which are expressed as , where d is the number of gating layer kernel feature layers.

[0073] The first layer feature map of the main decoding module The formation depends only on the last layer in the encoder :

[0074] For the decoding layer features of the Lth layer in the middle of the main decoding module, the aggregation process is as follows: first, the sub-decoding module aggregates the features of different scales of the encoding part Decode to the target at the current scale The same size, the decoded feature fd ( ),…,fd( )with the feature fd decoded by the main decoding module( ) are aggregated, and the final aggregated target is obtained by comprehensively considering the main and secondary decoding modules. , expressed as: = ⊕fd( )... fd( )⊕ fd( )= ⊕fd( , ..., , ) Among them, ⊕ represents the channel dimension superposition operation, fd(·) represents the deconvolution operation on features of different scales, and e is the total number of feature layers, that is, the decoding process Among them, the outputs of the two fusion decoding modules are Fout51 and Fout52 respectively.

[0075] In order to further extract more effective low-level feature information in the backbone network and restore the real details and edge information of the image, this paper proposes a learning network. In this network, the information in the left branch of the high-density encoding module, pixel amplification module and noise removal module is extracted and learned, and then output to the corresponding right module. The gating module and feature distillation fusion module in the learning network are used to reduce the number of parameters while maintaining the network reconstruction accuracy. In view of the problem that the backbone network in this scheme is not efficient enough for channel segmentation, the feature distillation fusion module couples the information in the original input modules together.

[0076] In a possible implementation of the present application, the image restoration method may also include: inputting the feature information in the first high-density encoding module and the first pixel density amplification module in the sampling process into the learning network for processing, and then inputting it into the second pixel density amplification module and the second high-density encoding module in the inverse convolution processing process, wherein the learning network is used to condense and extract the input feature information, and input the learned feature information into subsequent modules.

[0077] In a possible implementation of the present application, the learning network includes at least one gating module and at least one feature distillation fusion module.

[0078] Among them, the gating module includes two convolutional layers, and the feature information in the first high-density encoding module, the first pixel density amplification module and the first noise removal module is respectively input into the two convolutional layers for convolution processing, and the processed feature information is respectively activated, and the feature information after activation is processed by outer product processing to obtain the gated feature; the feature distillation fusion module includes multiple convolutional layers with different convolutional kernels, and the feature information in the first high-density encoding module, the first pixel density amplification module and the first noise removal module is input into the convolutional layers of multiple different convolutional kernels of the feature distillation fusion module for convolution processing to obtain the feature distillation fusion feature; the gated feature and the feature distillation fusion feature are subjected to channel reduction processing through the convolutional layer to obtain the learned feature information.

[0079] In one example, if Figure 7 shown.

[0080] When the first high-density encoding module, the first pixel amplification module, and the first noise removal module are used as inputs, and the image features are input into the learning network, the image features flow into the gating module and the feature distillation fusion module respectively; the feature data flowing into the gating module undergoes two 3*3 convolutions, and is activated by LReLU and Sigmoid respectively, and then the outer product is processed; the feature data flowing into the feature distillation fusion module first undergoes a 3*3 convolution, and then the left branch uses three 1×1 convolutions for channel reduction, and the right branch uses four 3×3 convolutions to further process the refinement layer of the coarse features, the left branch The right branch extracts the main part of the deep features, the left branch uses 1×1 convolution to separate 25% of the channels and continues to pass them backwards, and the right branch uses 3×3 convolution to retain 75% of the channels and continues to extract detail features. The features processed by the left and right branches are summed and superimposed with the input of 3*3 convolution through the channel space attention mechanism; the feature data processed by the gating module and the feature distillation fusion module are channel-reduced using 1*1 convolution and continue to be input into the gating module and the feature distillation fusion module for stacked extraction; the process of (2) to (4) is repeated until the preset output result is met, and the output of the learning network is obtained.

[0081] After the integration of fusion coding module, high-density coding module, pixel amplification module, noise removal module and fusion decoding modules, the output image has restored the quality of the original image to a certain extent, denoised, removed the mask, etc. However, in the process of restoration and beautification, there will be a mixture of unsatisfactory restored images and images with good restoration effects.

[0082] In a possible implementation of the present application, after the gated features and feature distillation fusion features are subjected to channel reduction processing through a convolutional layer, the image restoration method may further include: inputting the gated features and feature distillation fusion features after channel reduction into a gated module and a feature distillation fusion module respectively for stacking processing.

[0083] In this embodiment, images that fail to be repaired are discarded, and at the same time, repair information that can be used for iteration is output to the learning network for subsequent model iterative optimization.

[0084] If there is content to be repaired with relatively less feature information in the damaged area of ​​the image, such as the color ring around the pigeon's eyes, the edge contour of the digital image, etc., this content is likely to cause the loss of the above feature information during the downsampling operation. In this embodiment, image features, semantic information and texture information of different scales are extracted and judged at the output layer to obtain an image that meets the restoration and beautification standards and can be output.

[0085] In a possible implementation of the present application, before deleting pixel blocks in the obtained target image whose restoration degree is less than a restoration degree threshold, the image restoration method may also include: inputting the target image into a discriminant network for evaluation, the discriminant network being used to extract features of the input image to obtain image features, semantic information, and texture information of different scales.

[0086] In a possible implementation of the present application, the discriminant network includes a first branch, a second branch, and a third branch, and the convolution kernel sizes and the number of layers of the first branch, the second branch, and the third branch are all different.

[0087] Among them, the discriminant network is used to: input the target image into the first branch, the second branch and the third branch for feature extraction to obtain features of the same size but different scales; fuse the features extracted by the first branch, the second branch and the third branch to obtain fused features; based on the fused features, obtain the restoration degrees of multiple local areas.

[0088] In one example, if Figure 8 shown.

[0089] The discriminator consists of three branches with different convolution kernel sizes and numbers of layers, including shallow branch 1, middle branch 2, and deep branch 3.

[0090] (1) Shallow branch 1: It consists of three convolutional layers with 7 convolution kernels, which is used to enhance the ability to extract geometric and texture information of the input image. (2) Middle layer branch 2: It consists of 4 convolutional layers with 5 convolution kernels, which enhances the ability to extract semantic information of the input image and improves the network's ability to extract information between adjacent regions of the receptive field; (3) Deep branch 3: It consists of 5 convolutional layers with 5 convolution kernels. This layer has more convolutional layers and fewer convolution kernels. While improving the stability of network training, it can ensure more comprehensive feature extraction of input images.

[0091] In the discrimination process, the discriminator extracts the features of the same size but different scales from the three branches. After fusion, the convolutional layer is used to judge the features in multiple local areas to play the role of discriminating the details of the image, thereby outputting an image of satisfactory quality.

[0092] In the overall structure of the embodiment of the present application, in the upsampling process, the complexity of calculating features is first reduced through the fusion model encoding module, which is beneficial to restore texture details of the image to be repaired. At the same time, in the upsampling process, the high-density encoding module is used to process the features in the low-feature channels after processing, and the deep network is used to obtain more feature information between adjacent areas. Subsequently, the pixel density amplification module is used to perform pixel amplification processing to expand the receptive field without losing resolution and keep the relative spatial position of the pixels unchanged. The pixels are then input into the noise removal module. Under the input state of multiple feature channels, the noise features in the image are suppressed and filled, and the important features are enhanced and filled, so that the image is clear and more natural.

[0093] The learning network is used to learn the outputs of the high-density encoding module, pixel amplification module, and noise removal module in the backbone network, and the content of the left branch is input into the right branch for learning to improve the output performance and accuracy of the right branch.

[0094] The final discriminator network adjusts the output results by distinguishing between high-quality and low-quality repaired images to output images with the best quality.

[0095] In one example, if Figure 9-10 As shown, the user clicks on the toolbox column in the cloud disk and clicks on the "Picture Beautification" function; then the user imports the image to be beautified and selects the beautification style; if the user chooses beautification and repair, the image repair method provided in this application is used to repair the image for the user.

[0096] In one example, the beautification and repair function is a new image processing function to be launched in the cloud disk product. The beautification and repair function can improve the resolution of the image and repair the image problems such as occlusion, white spots, dark spots, and image vignetting caused by lens stains that appeared in the early photos, and finally improve the image refinement and quality. Fig.11 As shown, the left side is the original image and the right side is the restored image. The image is clearer and the colors are brighter.

[0097] like Fig.12 As shown in FIG. 1 , a schematic diagram of an image restoration device provided by the present application is shown. Fig.12 As shown, the image restoration device can be applied to electronic devices, and the image restoration device can include: an acquisition module 1201 , a sampling module 1202 , a processing module 1203 and a determination module 1204 .

[0098] Among them, the acquisition module 1201 is used to acquire the original image to be processed; the sampling module 1202 is used to sample the original image, and in the sampling process, the deep network is used to obtain the feature information between adjacent areas in the original image, and the feature information is subjected to pixel point amplification processing, wherein the relative spatial position of each pixel point before and after amplification remains unchanged; the processing module 1203 is used to perform deconvolution processing on the feature information after pixel point amplification to obtain a target image, wherein the target image includes multiple pixel blocks; the determination module 1204 is used to delete the pixel blocks with a restoration degree less than a restoration degree threshold in the obtained target image, and synthesize the pixel blocks with a restoration degree greater than or equal to the restoration degree threshold to obtain a restored image.

[0099] In an embodiment of the present application, first, the acquisition module 1201 acquires the original image to be processed, and then the sampling module 1202 performs sampling processing on the original image. During the sampling process, the deep network is used to obtain the feature information between adjacent areas in the original image, and the feature information is subjected to pixel point amplification processing, wherein the relative spatial position of each pixel point before and after amplification remains unchanged, and the processing module 1203 then performs deconvolution processing on the feature information after pixel point amplification to obtain a target image, wherein the target image includes multiple pixel blocks, and finally the determination module 1204 deletes the pixel blocks whose restoration degree is less than the restoration degree threshold in the obtained target image, and synthesizes the pixel blocks whose restoration degree is greater than or equal to the restoration degree threshold to obtain a restored image. The embodiment of the present application obtains the feature information between adjacent areas in the original image by sampling the original image, and performs pixel point amplification processing on the feature information, which can increase the receptive field of the original image, thereby beautifying the image while improving the image resolution.

[0100] In a possible implementation of the present application, the sampling module 1202 is used to perform sampling processing on the original image. During the sampling process, the original image is convolved by the first fusion coding module to obtain long-distance feature information of the original image, and the computational complexity of the long-distance feature information is lower than the computational complexity of the feature information in the original image; the long-distance feature information is processed by the shallow network convolution layer and the deep network convolution layer in the first high-density coding module to obtain feature information between multiple adjacent regions; the feature information between multiple adjacent regions is pixel-amplified by the first pixel density amplification module to obtain feature information with an enlarged receptive field.

[0101] In a possible embodiment of the present application, the first fusion coding module includes multiple convolution layers and average pooling layers, wherein the first fusion coding module is used to: input the original image into two convolution layers to obtain a first image feature; input the first image feature into the average pooling layer and the maximum pooling module respectively for convolution processing to obtain a second image feature and a third image feature; perform a Hadamard product operation on the second image feature and the third image feature, and perform a matrix product with the first image feature to obtain a fourth image feature; use a residual connection to fuse the fourth image feature with the original image to obtain long-distance feature information.

[0102] In a possible embodiment of the present application, the first high-density coding module includes a high-density coding layer and a decoupling layer, the high-density coding layer includes a plurality of interconnected convolution kernels with step sizes of 1 and 2, respectively, wherein the convolution kernel with a step size of 1 and the convolution kernel with a step size of 2 are connected through a gradient dissipation processing module; the decoupling layer includes a plurality of upsampling modules, the convolution kernel with a step size of 1 in the high-density coding layer is input into the sampling module for upsampling processing, and a restored feature map is output, and the restored map includes feature information between multiple adjacent regions.

[0103] In a possible embodiment of the present application, the first pixel density amplification module includes multiple feature extraction layers and multiple convolution layers, wherein the first pixel density amplification module is used to: input the feature map output by the high-density encoding module into multiple feature extraction layers for feature extraction to obtain a first feature map; input the first feature map into multiple convolution layers for convolution processing to obtain a second feature map; perform outer product operation on the second feature map and the first feature map to obtain a spatial weight feature map; sum and superimpose the spatial weight feature map and the feature map output by the high-density encoding module, and then process them through an activation function to obtain a feature map with an enlarged receptive field, wherein the pixel points of the feature map with the enlarged receptive field are more than the pixel points of the feature map output by the high-density encoding module.

[0104] In a possible implementation of the present application, the processing module 1203 is used to process the feature information after pixel amplification through the shallow network convolution layer and the deep network convolution layer in the second high-density encoding module to obtain feature information between multiple adjacent areas; the feature information between multiple adjacent areas is deconvolved through the main decoding module and multiple sub-decoding modules in the first fusion decoding module, and is processed through channel superposition to obtain a target image, wherein the number of network layers in the main decoding module is the same as the number of network layers of the first fusion encoding module in the sampling process, and the multiple sub-decoding modules are used to decode features of different scales to the same size as the target features, and the channel superposition is the channel superposition of the features decoded by the main decoding module and the features decoded by the multiple sub-decoding modules.

[0105] In a possible implementation of the present application, the sampling module 1202 is used to: input the feature information in the first high-density encoding module and the first pixel density amplification module in the sampling process into the learning network for processing, and then input it into the second pixel density amplification module and the second high-density encoding module in the deconvolution processing process, wherein the learning network is used to condense and extract the input feature information, and input the learned feature information into subsequent modules.

[0106] In a possible embodiment of the present application, the learning network includes at least one gating module and at least one feature distillation fusion module, the gating module includes two convolutional layers, the feature information in the first high-density coding module, the first pixel density amplification module and the first noise removal module are respectively input into the two convolutional layers for convolution processing, the processed feature information are respectively activated, and the activated feature information is processed by outer product to obtain the gated feature; the feature distillation fusion module includes multiple convolutional layers with different convolutional kernels, the feature information in the first high-density coding module, the first pixel density amplification module and the first noise removal module are input into the convolutional layers of multiple different convolutional kernels of the feature distillation fusion module for convolution processing to obtain the feature distillation fusion feature; the gated feature and the feature distillation fusion feature are subjected to channel reduction processing through the convolutional layer to obtain the learned feature information.

[0107] In a possible implementation manner of the present application, the image restoration device may further include a judgment module.

[0108] Among them, the evaluation module is used to input the target image into the discriminant network for evaluation. The discriminant network is used to extract features of the input image to obtain image features, semantic information and texture information of different scales.

[0109] In a possible implementation of the present application, the discriminant network includes a first branch, a second branch, and a third branch, and the convolution kernel sizes and the number of layers of the first branch, the second branch, and the third branch are all different.

[0110] Among them, the evaluation module is used to: input the target image into the first branch, the second branch and the third branch for feature extraction to obtain features of the same size but different scales; fuse the features extracted by the first branch, the second branch and the third branch to obtain fused features; based on the fused features, obtain the restoration degree of multiple local areas.

[0111] The image restoration device provided in the embodiment of the present application can achieve Figures 1 to 11 The various processes implemented by the method embodiment achieve the same technical effect and will not be described again here to avoid repetition.

[0112] like Fig.13As shown, an embodiment of the present application also provides an electronic device 1300, including a processor 1301, a memory 1302, and a program or instruction stored in the memory 1302 and executable on the processor 1301. When the program or instruction is executed by the processor 1301, each process of the above-mentioned image restoration method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0113] Optionally, the embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, each process of the above-mentioned image restoration method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0114] Optionally, an embodiment of the present application also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the various processes of the above-mentioned image restoration method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not repeated here.

[0115] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0116] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0117] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An image restoration method, characterized in that: include: Obtain the original image to be processed; The original image is sampled, and during the sampling process, feature information between adjacent regions in the original image is obtained using a deep network, and pixel amplification is performed on the feature information, wherein the relative spatial positions of the pixels before and after the amplification remain unchanged; Performing a deconvolution process on the feature information after pixel point amplification to obtain a target image, wherein the target image includes a plurality of pixel blocks; Pixel blocks with a restoration degree less than a restoration degree threshold in the obtained target image are deleted, and pixel blocks with a restoration degree greater than or equal to the restoration degree threshold are synthesized to obtain a restored image.

2. The method according to claim 1, characterized in that The sampling process of the original image, obtaining feature information between adjacent regions in the original image by using a deep network during the sampling process, and performing pixel point amplification process on the feature information, includes: Sampling the original image, during the sampling process, the original image is subjected to a convolution operation of a first fusion coding module to obtain long-distance feature information of the original image, wherein the computational complexity of the long-distance feature information is lower than the computational complexity of the feature information in the original image; Processing the long-distance feature information through a shallow network convolution layer and a deep network convolution layer in a first high-density encoding module to obtain feature information between multiple adjacent regions; The feature information between the plurality of adjacent regions is subjected to pixel point amplification processing by a first pixel point density amplification module to obtain feature information with an increased receptive field.

3. The method according to claim 2, characterized in that The first fusion coding module includes a plurality of convolutional layers and an average pooling layer, wherein the first fusion coding module is used for: Inputting the original image into two convolutional layers to obtain a first image feature; Inputting the first image feature into the average pooling layer and the maximum pooling module respectively for convolution processing to obtain the second image feature and the third image feature; Performing a Hadamard product operation on the second image feature and the third image feature, and performing a matrix product operation on the first image feature to obtain a fourth image feature; The fourth image feature is fused with the original image by using residual connection to obtain the long-distance feature information.

4. The method according to claim 2, characterized in that: The first high-density coding module includes a high-density coding layer and a decoupling layer, The high-density coding layer includes a plurality of interconnected convolution kernels with step sizes of 1 and 2, respectively, wherein the convolution kernel with a step size of 1 is connected to the convolution kernel with a step size of 2 through a gradient dissipation processing module; The decoupling layer includes multiple upsampling modules, and the convolution kernel with a step size of 1 in the high-density coding layer is input into the sampling module for upsampling processing, and a restored feature map is output, which includes feature information between multiple adjacent areas.

5. The method according to claim 2, characterized in that: The first pixel density amplification module includes a plurality of feature extraction layers and a plurality of convolution layers, wherein the first pixel density amplification module is used to: Inputting the feature map output by the high-density encoding module into the multiple feature extraction layers for feature extraction to obtain a first feature map; Inputting the first feature map into the multiple convolutional layers for convolution processing to obtain a second feature map; Performing an outer product operation on the second feature map and the first feature map to obtain a spatial weight feature map; The spatial weight feature map and the feature map output by the high-density encoding module are summed and superimposed, and then processed by an activation function to obtain a feature map with an increased receptive field. The pixel points of the feature map with an increased receptive field are more than the pixel points of the feature map output by the high-density encoding module.

6. The method according to claim 1, characterized in that The step of performing a deconvolution process on the feature information after pixel amplification to obtain a target image includes: The feature information after pixel amplification is processed by the shallow network convolution layer and the deep network convolution layer in the second high-density encoding module to obtain feature information between multiple adjacent areas; The feature information between the multiple adjacent areas is deconvolved through a main decoding module and multiple sub-decoding modules in a first fusion decoding module, and is processed through channel superposition to obtain a target image, wherein the number of network layers in the main decoding module is the same as the number of network layers of the first fusion encoding module in the sampling process, the multiple sub-decoding modules are used to decode features of different scales to the same size as the target features, and the channel superposition is the channel superposition of the features decoded by the main decoding module and the features decoded by the multiple sub-decoding modules.

7. The method according to claim 6, characterized in that The method further comprises: The feature information in the first high-density encoding module and the first pixel density amplification module in the sampling process is input into the learning network for processing, and then input into the second pixel density amplification module and the second high-density encoding module in the deconvolution processing process, wherein the learning network is used to condense and extract the input feature information, and input the learned feature information into subsequent modules.

8. The method according to claim 7, characterized in that The learning network includes at least one gating module and at least one feature distillation fusion module. The gating module includes two convolutional layers, the feature information in the first high-density encoding module, the first pixel density amplification module and the first noise removal module are respectively input into the two convolutional layers for convolution processing, the processed feature information are respectively activated, and the activated feature information is processed by outer product processing to obtain the gating feature; The feature distillation fusion module includes a plurality of convolution layers with different convolution kernels, and the feature information in the first high-density encoding module, the first pixel density amplification module and the first noise removal module is input into the convolution layers with a plurality of different convolution kernels of the feature distillation fusion module for convolution processing to obtain feature distillation fusion features; The gated features and the feature distillation fusion features are subjected to channel reduction processing through a convolutional layer to obtain learned feature information.

9. The method according to claim 1, characterized in that: Before deleting the pixel blocks in the obtained target image whose restoration degree is less than the restoration degree threshold, the method further includes: The target image is input into a discriminant network for evaluation. The discriminant network is used to extract features of the input image to obtain image features, semantic information, and texture information of different scales.

10. The method according to claim 9, characterized in that The discriminant network includes a first branch, a second branch and a third branch, the convolution kernel size and the number of layers of the first branch, the second branch and the third branch are all different, and the discriminant network is used for: Inputting the target image into the first branch, the second branch and the third branch to extract features, and obtaining features of the same size but different scales; Fusing the features extracted by the first branch, the second branch, and the third branch to obtain a fused feature; Based on the fusion features, restoration degrees of multiple local areas are obtained.

Images