Image restoration method, device, computer equipment and computer readable storage medium

Abstract

The present application relates to an image restoration method, device, computer equipment and computer-readable storage medium. The method comprises: performing low-frequency frequency division enhancement on the image to be restored to obtain a low-frequency enhanced image; processing the image to be restored and the low-frequency enhanced image based on a preset Gaussian blur kernel to obtain multi-scale low-frequency component information; extracting high-frequency components from the image to be restored and the low-frequency enhanced image based on the multi-scale low-frequency component information to obtain multi-scale high-frequency component information; and performing high-frequency restoration on the multi-scale high-frequency component information to obtain a restored image. By adopting the present application, it is possible to improve the restoration effect of the restored image.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image restoration method, apparatus, computer equipment, and computer-readable storage medium. Background Art

[0002] Adverse factors such as noise, compression loss and aging will reduce the image quality. In order to restore the image to its original state, the image needs to be repaired.

[0003] Currently, one way to repair an image is to input the image to be repaired into a trained deep learning model for processing, and the deep learning model outputs a repaired image that is close to the original state.

[0004] However, the current deep learning model adopts an end-to-end joint optimization learning strategy of low-frequency and high-frequency signals, which leads to mutual interference between low-frequency signals and high-frequency signals in the process of image repair, resulting in poor repair effect of the repaired image. Summary of the invention

[0005] Based on this, it is necessary to provide an image restoration method, apparatus, computer equipment, computer-readable storage medium and computer program product to address the above technical problems, which can improve the restoration effect of the restored image.

[0006] In a first aspect, the present application provides an image restoration method, comprising:

[0007] The image to be repaired is subjected to low-frequency division enhancement to obtain a low-frequency enhanced image;

[0008] Processing the image to be repaired and the low-frequency enhanced image based on a preset Gaussian blur kernel to obtain multi-scale low-frequency component information;

[0009] Based on the multi-scale low-frequency component information, high-frequency components are extracted from the image to be repaired and the low-frequency enhanced image to obtain multi-scale high-frequency component information;

[0010] High-frequency restoration is performed on the multi-scale high-frequency component information to obtain a repaired image.

[0011] In a second aspect, the present application provides an image restoration device, comprising:

[0012] An image low-frequency enhancement module is used to perform low-frequency division enhancement on the image to be repaired to obtain a low-frequency enhanced image;

[0013] A low-frequency component generation module, used for processing the image to be repaired and the low-frequency enhanced image based on a preset Gaussian blur kernel to obtain multi-scale low-frequency component information;

[0014] A high-frequency component generation module, configured to extract high-frequency components from the image to be restored and the low-frequency enhanced image based on the multi-scale low-frequency component information to obtain multi-scale high-frequency component information;

[0015] The high-frequency component recovery module performs high-frequency recovery on the multi-scale high-frequency component information to obtain a restored image.

[0016] In a third aspect, the present application provides a computer device, the computer device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the steps in the above method when executing the computer program.

[0017] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the steps in the above method when executed by a processor.

[0018] In a fifth aspect, the present application provides a computer program product, which includes a computer program, and the computer program implements the steps in the above method when executed by a processor.

[0019] The above-mentioned image restoration method, device, computer equipment, computer-readable storage medium and computer program product perform low-frequency division enhancement on the image to be restored to obtain a low-frequency enhanced image; process the image to be restored and the low-frequency enhanced image based on a preset Gaussian blur kernel to obtain multi-scale low-frequency component information; extract high-frequency components from the image to be restored and the low-frequency enhanced image based on the multi-scale low-frequency component information to obtain multi-scale high-frequency component information; perform high-frequency restoration on the multi-scale high-frequency component information to obtain a restored image; in the process of image restoration by the above-mentioned method, multi-scale low-frequency component information is first obtained, and then multi-scale high-frequency component information is further calculated based on the multi-scale low-frequency component information, that is, when performing image restoration, multi-scale low-frequency component information and multi-scale high-frequency component information are generated in a logical order of succession, rather than using an end-to-end joint optimization learning strategy of low-frequency and high-frequency signals as is generally used in the prior art. The multi-scale low-frequency component information and the multi-scale high-frequency component information are generated successively, so the problem of mutual interference between low-frequency signals and high-frequency signals can be avoided, thereby improving the restoration effect of the restored image. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 An application environment diagram of an image restoration method for multi-scale low-frequency component information provided in an embodiment of the present application;

[0021] Figure 2 A schematic diagram of a process of an image restoration method provided in an embodiment of the present application;

[0022] Figure 3A schematic diagram of the structure of a low-frequency crossover enhancer provided in an embodiment of the present application;

[0023] Figure 4 A structural block diagram of an image restoration device provided in an embodiment of the present application;

[0024] Figure 5 An internal structure diagram of a computer device provided in an embodiment of the present application;

[0025] Figure 6 An internal structure diagram of another computer device provided in an embodiment of the present application;

[0026] Figure 7 An internal structure diagram of a computer-readable storage medium provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0028] The image restoration method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through a communication network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. Among them, the terminal 102 can be but is not limited to various personal computers, laptops, smart phones, tablets, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be implemented with an independent server or a server cluster consisting of multiple servers.

[0029] like Figure 2 As shown, the embodiment of the present application provides an image restoration method, which is applied to Figure 1 The terminal 102 or the server 104 in the example is used for explanation. It is understandable that the computer device may include at least one of the terminal and the server. The method comprises the following steps:

[0030] S110 , performing low-frequency division enhancement on the image to be repaired to obtain a low-frequency enhanced image.

[0031] Among them, the image to be repaired is an image that needs to be repaired by the image repair method described in this embodiment. The image to be repaired may be an image that is affected by adverse factors such as noise, compression loss and aging, resulting in poor image quality; low-frequency frequency division enhancement is an image processing technology used to smooth the image and remove high-frequency noise from the image, thereby improving the image quality; exemplarily, low-frequency frequency division enhancement of the image can be achieved by a corresponding low-frequency frequency division enhancer; the low-frequency enhanced image is the image to be repaired after the low-frequency frequency division enhancement is completed.

[0032] Specifically, an image to be repaired is acquired to obtain an image to be repaired, and then a low-frequency division enhancement process is performed on the image to be repaired to obtain a low-frequency enhanced image.

[0033] S120 , processing the image to be restored and the low-frequency enhanced image based on a preset Gaussian blur kernel to obtain multi-scale low-frequency component information.

[0034] The Gaussian blur kernel is a convolution kernel based on a Gaussian function, which is used to blur an image. The Gaussian blur kernel replaces each pixel with the weighted average of its neighboring pixels by applying a Gaussian function to each pixel of the image, thereby achieving the effect of blurring the image. The image to be repaired can be processed by a preset Gaussian blur kernel to obtain a blurred image pyramid structure, and the blurred image pyramid structure is divided into multiple layers of blurred images. Exemplarily, the first layer of the blurred image pyramid structure is the image to be repaired. The image to be repaired is blurred by a Gaussian blur kernel to obtain a blurred image, and then the blurred image is downsampled to a lower layer blurred image. Then, the lower layer blurred image is blurred and downsampled to obtain a lower layer blurred image. By analogy, a blurred image pyramid structure with multiple layers of blurred images is obtained. In the process of calculating each layer of the blurred image of the blurred image pyramid structure by the Gaussian blur kernel, the low-frequency component corresponding to each layer of the blurred image can also be obtained. The low-frequency components of each layer of the blurred image of the blurred image pyramid structure can be integrated to obtain multi-scale low-frequency components corresponding to the image to be repaired, wherein the number of multi-scale scales is also the number of layers of the blurred image pyramid structure. Similarly, by processing the low-frequency enhanced image through Gaussian blur kernel, the multi-scale low-frequency components corresponding to the low-frequency enhanced image can also be obtained. The multi-scale low-frequency component information is a collection of the multi-scale low-frequency components corresponding to the image to be repaired and the multi-scale low-frequency components corresponding to the low-frequency enhanced image.

[0035] Specifically, the image to be repaired is processed based on a preset Gaussian blur kernel to obtain a multi-scale low-frequency component corresponding to the image to be repaired, and the low-frequency enhanced image is processed based on a preset Gaussian blur kernel to obtain a multi-scale low-frequency component corresponding to the low-frequency enhanced image; the multi-scale low-frequency component corresponding to the image to be repaired and the multi-scale low-frequency component corresponding to the low-frequency enhanced image are combined to obtain multi-scale low-frequency component information.

[0036] S130. Extract high-frequency components from the image to be restored and the low-frequency enhanced image based on the multi-scale low-frequency component information to obtain multi-scale high-frequency component information.

[0037] Among them, the high-frequency component extraction is to calculate the first difference information between the image to be repaired and the multi-scale low-frequency component corresponding to the image to be repaired, so as to obtain the multi-scale high-frequency component corresponding to the image to be repaired, and calculate the second difference information between the low-frequency enhanced image and the multi-scale low-frequency component corresponding to the low-frequency enhanced image, so as to obtain the multi-scale high-frequency component corresponding to the low-frequency enhanced image; the multi-scale high-frequency component information is a collection of the multi-scale high-frequency component corresponding to the image to be repaired and the multi-scale high-frequency component corresponding to the low-frequency enhanced image.

[0038] Specifically, the first difference information between the image to be repaired and the multi-scale low-frequency component corresponding to the image to be repaired in the multi-scale low-frequency component information is calculated to obtain the multi-scale high-frequency component corresponding to the image to be repaired; the second difference information between the low-frequency enhanced image and the multi-scale low-frequency component corresponding to the low-frequency enhanced image in the multi-scale low-frequency component information is also calculated to obtain the multi-scale high-frequency component corresponding to the low-frequency enhanced image; the multi-scale high-frequency component corresponding to the image to be repaired and the multi-scale high-frequency component corresponding to the low-frequency enhanced image are combined to obtain the multi-scale high-frequency component information.

[0039] S140, performing high-frequency restoration on the multi-scale high-frequency component information to obtain a restored image.

[0040] Among them, high-frequency recovery is to uniformly perform high-frequency recovery on the multi-scale high-frequency components corresponding to the image to be repaired and the multi-scale high-frequency components corresponding to the low-frequency enhanced image in the multi-scale high-frequency component information, so as to deblur the image to be repaired and the low-frequency enhanced image, thereby realizing the restoration of the image to be repaired; the repaired image is the repaired image to be repaired; exemplarily, a high-frequency restorer can be used to perform high-frequency recovery on the multi-scale high-frequency component information to obtain the repaired image.

[0041] Specifically, high-frequency restoration is performed on the multi-scale high-frequency components corresponding to the image to be repaired and the multi-scale high-frequency components corresponding to the low-frequency enhanced image in the multi-scale high-frequency component information to obtain a repaired image.

[0042] It can be seen that in the embodiment of the present application, a low-frequency enhanced image is obtained by performing low-frequency division enhancement on the image to be repaired; multi-scale low-frequency component information is obtained by processing the image to be repaired and the low-frequency enhanced image based on a preset Gaussian blur kernel; high-frequency components are extracted from the image to be repaired and the low-frequency enhanced image based on the multi-scale low-frequency component information to obtain multi-scale high-frequency component information; high-frequency restoration is performed on the multi-scale high-frequency component information to obtain a repaired image; in the process of image repair by the above method, multi-scale low-frequency component information is first obtained, and then multi-scale high-frequency component information is further calculated based on the multi-scale low-frequency component information, that is, when performing image repair, multi-scale low-frequency component information and multi-scale high-frequency component information are generated in a logical order of sequence, rather than using an end-to-end joint optimization learning strategy of low-frequency and high-frequency signals as is generally used in the prior art. The multi-scale low-frequency component information and the multi-scale high-frequency component information are generated successively, so the problem of mutual interference between low-frequency signals and high-frequency signals can be avoided, thereby improving the repair effect of the repaired image.

[0043] In some embodiments, the image to be restored and the low-frequency enhanced image are processed based on a preset Gaussian blur kernel to obtain multi-scale low-frequency component information, including:

[0044] S121 . Processing the image to be restored based on a preset Gaussian blur kernel to obtain a first multi-scale low-frequency component.

[0045] In the process of calculating each layer of blurred image of the blurred image pyramid structure corresponding to the image to be repaired by the Gaussian blur kernel, the low-frequency component corresponding to each layer of the blurred image can also be obtained, that is, the multi-scale low-frequency component corresponding to the image to be repaired is obtained, and the first multi-scale low-frequency component is the multi-scale low-frequency component corresponding to the image to be repaired; the multi-scale low-frequency component has a corresponding multi-scale low-frequency component calculation formula, and the multi-scale low-frequency component calculation formula is as follows:

[0046]

[0047] in, (x, y) is the low-frequency component of the image to be repaired at one scale (the low-frequency component corresponding to the blurred image of one layer of the blurred image pyramid structure corresponding to the image to be repaired), (x, y) is the coordinate of the pixel in the image to be repaired, g is the Gaussian blur kernel, I is the image to be repaired, is the number of layers of the blurred image pyramid structure at different scales of the image to be repaired, The blurred image is obtained by downsampling the blurred image of the previous layer in the blurred image pyramid structure corresponding to the image to be repaired by 2 times.

[0048] Specifically, the low-frequency components of the blurred image at the corresponding scale of the first layer of the blurred image pyramid structure are first calculated according to the preset Gaussian blur kernel g and the image to be restored I; then the low-frequency components of the blurred image at the corresponding scale of the second layer are continued to be calculated according to the Gaussian blur kernel g and the low-frequency components of the blurred image at the corresponding scale of the first layer, and so on, until the low-frequency components of the blurred image at the corresponding scale of the last layer of the blurred image pyramid structure are calculated, and the low-frequency components of the blurred images at the corresponding scales of each layer of the blurred image pyramid structure corresponding to the image to be restored I are gathered to obtain the first multi-scale low-frequency components.

[0049] S122: Process the low-frequency enhanced image based on a preset Gaussian blur kernel to obtain a second multi-scale low-frequency component.

[0050] In the process of calculating each layer of blurred image of the blurred image pyramid structure corresponding to the low-frequency enhanced image by the Gaussian blur kernel, the low-frequency component corresponding to each layer of the blurred image can also be obtained, that is, the multi-scale low-frequency component corresponding to the low-frequency enhanced image is obtained, and the second multi-scale low-frequency component is the multi-scale low-frequency component corresponding to the low-frequency enhanced image; the multi-scale low-frequency component has a corresponding multi-scale low-frequency component calculation formula, and the multi-scale low-frequency component calculation formula is as follows:

[0051]

[0052] in, (x, y) is the low-frequency component of the low-frequency enhanced image at one scale (the low-frequency component corresponding to the blurred image of one layer of the blurred image pyramid structure corresponding to the low-frequency enhanced image), (x, y) is the coordinate of the pixel in the low-frequency enhanced image, g is the Gaussian blur kernel, To enhance the low frequency image, is the number of layers of blurred images at different scales of the low-frequency enhanced image in the blurred image pyramid structure, The blurred image is obtained by downsampling the blurred image of the previous layer in the blurred image pyramid structure corresponding to the low-frequency enhanced image by 2 times.

[0053] Specifically, according to the preset Gaussian blur kernel g and the low-frequency enhanced image First, calculate the low-frequency components of the blurred image at the corresponding scale of the first layer of the blurred image pyramid structure; then continue to calculate the low-frequency components of the blurred image at the corresponding scale of the second layer according to the Gaussian blur kernel g and the low-frequency components of the blurred image at the corresponding scale of the first layer, and so on, until the low-frequency components of the blurred image at the corresponding scale of the last layer of the blurred image pyramid structure are calculated, and the collection is combined with the low-frequency enhanced image The low-frequency components at corresponding scales of the blurred images of each layer of the blurred image pyramid structure corresponding to the blurred image obtain the second multi-scale low-frequency components.

[0054] S123. Obtain the multi-scale low-frequency component information based on the first multi-scale low-frequency component and the second multi-scale low-frequency component.

[0055] The multi-scale low-frequency component information includes a first multi-scale low-frequency component and a second multi-scale low-frequency component.

[0056] It can be seen that in this embodiment, by processing the image to be repaired and the low-frequency enhanced image respectively through the Gaussian blur kernel, multi-scale low-frequency components corresponding to the image to be repaired and the low-frequency enhanced image can be obtained, which helps to reduce the noise of the image to be repaired and the low-frequency enhanced image, and achieve image smoothing of the image to be repaired and the low-frequency enhanced image.

[0057] In some embodiments, extracting high-frequency components from the image to be restored and the low-frequency enhanced image based on the multi-scale low-frequency component information to obtain multi-scale high-frequency component information includes:

[0058] S131. Extract high-frequency components of the image to be restored based on the first multi-scale low-frequency components to obtain first multi-scale high-frequency components.

[0059] Among them, the calculation formula for extracting high-frequency components of the image to be repaired is as follows:

[0060]

[0061] in, is the high-frequency component of the image to be repaired at one scale (the high-frequency component corresponding to the blurred image of one layer of the blurred image pyramid structure corresponding to the image to be repaired), I is the image to be repaired, is the low-frequency component of the image to be repaired at one scale, is the number of layers of the blurred image pyramid structure at different scales of the image to be repaired, is a blurred image obtained by downsampling the blurred image of the previous layer in the blurred image pyramid structure by a factor of 2, and K is the layer number of the last blurred image of the blurred image pyramid structure.

[0062] Specifically, the low-frequency components of the image to be repaired at each scale are obtained through the above embodiment. , and then through the above The calculation formula is: first calculate the low-frequency component of the image to be repaired I and the blurred image of the first layer of the blurred image pyramid structure at the corresponding scale The difference between them is used to obtain the high-frequency component of the image to be repaired at the first scale. ; Then calculate the difference between the low-frequency component of the blurred image at the corresponding scale of the first layer of the blurred image pyramid structure and the low-frequency component of the blurred image at the corresponding scale of the second layer of the blurred image pyramid structure to obtain the high-frequency component of the second scale of the image to be repaired , and so on, until the high-frequency component of the image to be repaired at the K-1th scale is calculated Finally, according to The calculation formula is used to calculate the high-frequency component of the image to be repaired at the Kth scale. ; Comprehensively obtain the first multi-scale high-frequency component of the image to be repaired at each scale , =( , … ).

[0063] S132: Extract high-frequency components from the low-frequency enhanced image based on the second multi-scale low-frequency components to obtain second multi-scale high-frequency components.

[0064] Among them, the calculation formula for extracting high-frequency components of the image to be repaired is as follows:

[0065]

[0066] in, is the high-frequency component of the low-frequency enhanced image at one scale (the high-frequency component corresponding to the blurred image of one layer of the blurred image pyramid structure corresponding to the low-frequency enhanced image), To enhance the low frequency image, It is the low-frequency component of the low-frequency enhanced image at one scale. is the number of layers of blurred images at different scales of the low-frequency enhanced image in the blurred image pyramid structure, is a blurred image obtained by downsampling the blurred image of the previous layer in the blurred image pyramid structure by a factor of 2, and K is the layer number of the last blurred image of the blurred image pyramid structure.

[0067] Specifically, the low-frequency components of the low-frequency enhanced image at each scale are obtained through the above embodiment. , and then through the above The calculation formula is to first calculate the low-frequency enhanced image The low-frequency component at the scale corresponding to the blurred image in the first layer of the blurred image pyramid structure The difference between them is used to obtain the high-frequency component of the low-frequency enhanced image at the first scale. ; Then calculate the difference between the low-frequency component of the blurred image at the corresponding scale of the first layer of the blurred image pyramid structure and the low-frequency component of the blurred image at the corresponding scale of the second layer of the blurred image pyramid structure to obtain the high-frequency component of the second scale of the low-frequency enhanced image , and so on, until the high-frequency component of the low-frequency enhanced image at the K-1th scale is calculated Finally, according to The calculation formula is used to calculate the high-frequency component of the low-frequency enhanced image at the Kth scale. ; The high-frequency components of each scale of the low-frequency enhanced image are integrated to obtain the second multi-scale high-frequency component , =( , … ).

[0068] S133: Obtain the multi-scale high-frequency component information based on the first multi-scale high-frequency component and the second multi-scale high-frequency component.

[0069] The multi-scale high-frequency component information includes the first multi-scale high-frequency component and the second multi-scale high frequency component .

[0070] It can be seen that in this embodiment, by extracting high-frequency components of the image to be restored and the low-frequency enhanced image, the details and edge information of the image to be restored and the low-frequency enhanced image can be obtained.

[0071] In some embodiments, performing high-frequency restoration on the multi-scale high-frequency component information to obtain a repaired image includes:

[0072] S141. Stack the multi-scale high-frequency component information to obtain comprehensive high-frequency component information.

[0073] Wherein, stacking is to stack the first multi-scale high-frequency component in the multi-scale high-frequency component information and the second multi-scale high frequency component The high-frequency component information is integrated to form a whole data. and the second multi-scale high frequency component After stacking, the overall data is formed; illustratively, the comprehensive high-frequency component information is:

[0074]

[0075] Specifically, the first multi-scale high-frequency component in the stacked multi-scale high-frequency component information and the second multi-scale high frequency component Get comprehensive high-frequency component information.

[0076] S142: Processing the comprehensive high-frequency component information based on the trained high-frequency restorer to obtain Laplace components.

[0077] Among them, the comprehensive high-frequency component information includes high-frequency components, the trained high-frequency restorer is used to restore the high-frequency components of the comprehensive high-frequency component information and output K Laplace components ( , … ).

[0078] Specifically, the high frequency restorer processing by completing the training includes After processing, the high-frequency restorer outputs K Laplace components ( , … ).

[0079] The training data for training the high-frequency restorer in the initial state includes a set of corresponding sample images and identification images; the sample images are analogous to the images to be restored, and the identification images are analogous to the expected restored images.

[0080] The steps of training the high frequency restorer in the initial state to obtain the trained high frequency restorer include:

[0081] The sample image and the identification image are respectively subjected to the above steps S110-S140 to obtain K Laplace components corresponding to the sample image ( , … ) and K high-frequency components ( , … ), and the K Laplacian components corresponding to the logo image ( , … );

[0082] Get and , and based on and Establishing low frequency consistency loss ,in, ;

[0083] Based on the K Laplace components corresponding to the sample image ( , … ) and the K Laplace components corresponding to the logo image ( , … ) Establish the frequency division reconstruction loss ;

[0084] Reconstruction loss based on frequency division , low frequency consistency loss And the preset coefficient α, establish the overall loss ,in, ;

[0085] It can be seen from the above formula that the high-frequency restorer is applicable to any image restoration model. It only needs to simply adjust the number of channels of the input layer and the output layer (3->2K), so it has good scalability.

[0086] Based on overall loss The high-frequency restorer in the initial state is trained to obtain a trained high-frequency restorer; the trained high-frequency restorer is subsequently used to execute step S142.

[0087] S143, adding the Laplace components to obtain a restored image.

[0088] Among them, K Laplace components are output by adding high-frequency restorer , … The restored image can be obtained.

[0089] Specifically, the sum of the high frequency restorer outputs K Laplace components , … Get the restored image.

[0090] It can be seen that in this embodiment, by processing the comprehensive high-frequency component information through the trained high-frequency restorer, that is, processing the high-frequency components of the image to be restored and the low-frequency enhanced image, the details of the restored image can be obtained, that is, the high-frequency restorer outputs K Laplace components, thereby improving the clarity and image quality of the restored image.

[0091] In some embodiments, performing low-frequency division enhancement on the image to be repaired to obtain a low-frequency enhanced image includes:

[0092] S111, processing the image to be repaired based on a low-frequency division enhancer to obtain local features and global features.

[0093] The trained low-frequency division enhancer is used to process the image to be repaired, thereby obtaining local features and global features of the image to be repaired.

[0094] Specifically, the image to be repaired is input into the trained low-frequency frequency division enhancer for processing. After the processing is completed, the trained low-frequency frequency division enhancer outputs the local features of the image to be repaired. With global features .

[0095] S112: Obtain the low-frequency enhanced image based on the local features, the global features, and the image to be restored.

[0096] Among them, local features , global features , image to be repaired I and low-frequency enhanced image There is a corresponding calculation formula, as follows:

[0097]

[0098] Specifically, the local features , global features And the image to be repaired I is substituted into the low-frequency enhanced image The low-frequency enhanced image is obtained by calculating the formula .

[0099] It can be seen that in this embodiment, the local features of the image I to be repaired are first calculated With global features , and then according to the repair image I, local features With global features Calculate the low-frequency enhanced image corresponding to the image to be repaired I In this way, the low-frequency division enhancement of the image to be repaired I can be achieved. By performing the low-frequency division enhancement on the image to be repaired I, the high-frequency noise in the image to be repaired I can be removed, and the image smoothing of the image to be repaired I can be achieved.

[0100] In some embodiments, the image to be repaired is processed based on the low-frequency division enhancer to obtain local features and global features, including:

[0101] S1111. Perform a first convolution process on the image to be repaired based on the trained low-frequency division enhancer to obtain local features.

[0102] Among them, refer to Figure 3 The trained low-frequency frequency division enhancer includes a convolution module (a) and a self-attention module (b). The convolution module (a) is composed of 10 convolution layers Conv connected in series. The convolution module (a) is used to process the image to be repaired. The first convolution processing, that is, the convolution module (a), processes the image to be repaired. The convolution module (a) obtains the local features after processing the image to be repaired. .

[0103] Specifically, the convolution module (a) in the trained low-frequency frequency division enhancer is used to process the image to be repaired to obtain local features. .

[0104] The training data for training the low-frequency crossover enhancer in the initial state includes a training image and a label image, the training image is analogous to the image to be repaired, and the label image is analogous to the desired repaired image; by performing the above steps S111-S112 on the training image, a low-frequency enhanced image corresponding to the training image can be obtained. Similarly, by performing the above steps S111-S112 on the label image, a low-frequency enhanced image corresponding to the label image can be obtained. In this embodiment, the step of training the low-frequency crossover enhancer in the initial state includes:

[0105] Calculate the low-frequency enhanced image corresponding to the training image Low frequency enhanced image corresponding to the label image ;

[0106] Low-frequency enhanced images corresponding to training images Low frequency enhanced image corresponding to the label image Establish supervised learning loss loss, where ;

[0107] The low-frequency division enhancer in the initial state is trained based on the supervised learning loss loss to obtain a trained low-frequency division enhancer; the trained low-frequency division enhancer is subsequently used to execute the above steps S111-S112.

[0108] S1112: performing a second convolution process on the image to be repaired based on the trained low-frequency division enhancer to obtain image features.

[0109] Among them, continue to refer to Figure 3 , the self-attention module (b) includes a single convolution layer Conv, the output end of the single convolution layer Conv is connected to the input end of the self-attention layer Self-Attention, and the output end of the self-attention layer Self-Attention is connected to the input end of the fully connected layer MLP; the single convolution layer Conv is used to process the image to be repaired I, and the second convolution processing is also the single convolution layer Conv to process the image to be repaired I. The single convolution layer Conv processes the image to be repaired I to obtain image features .

[0110] Specifically, the image feature is obtained by processing the image to be repaired I through the single convolution layer Conv in the trained low-frequency frequency division enhancer .

[0111] S1113. Perform self-attention aggregation on the image features to obtain an aggregation result.

[0112] Among them, continue to refer to Figure 3 , the self-attention layer is used to analyze image features For processing, self-attention aggregation, that is, the self-attention layer Self-Attention on image features For processing, the self-attention layer Self-Attention is used to process image features Processing can get aggregate results .

[0113] Specifically, the self-attention layer in the trained low-frequency frequency division enhancer is used to analyze the image features. Processing to obtain aggregated results .

[0114] S1114. Perform global information extraction on the aggregation result to obtain global features.

[0115] Among them, continue to refer to Figure 3 , the fully connected layer MLP is used to aggregate the results Processing, global information extraction, that is, the fully connected layer MLP aggregates the results Processing, the fully connected layer MLP aggregates the results Processing can obtain global features .

[0116] Specifically, the aggregation results are obtained by completing the fully connected layer MLP in the trained low-frequency frequency division enhancer. Processing to obtain global features .

[0117] It should be noted that the above , that is, the calculation logic of the low-frequency crossover enhancer, refer to Figure 3 , the low-frequency crossover enhancer obtains the global feature , the image to be repaired I and local features , further, calculate the local features With global features The product of the image to be repaired I is used to obtain the low-frequency enhanced image .

[0118] It can be seen that in this embodiment, by obtaining the global features of the image to be repaired It is easy to understand the overall texture and color information of the image to be repaired, and it is also necessary to follow up based on the global features Calculate the low-frequency enhanced image of the image to be repaired I .

[0119] In some embodiments, performing self-attention aggregation on the image features to obtain an aggregation result includes:

[0120] S11131. Resize the image features to obtain reshaped features.

[0121] Among them, the self-attention layer Self-Attention can first globally pool the image features to obtain the pooled features , and then the pooling features Resize to get the reshape feature ; Sizing includes global pooling and resizing.

[0122] Specifically, the self-attention layer first performs global pooling on the image features to obtain the pooled features. , and then the pooling features Resize to get the reshape feature , thereby achieving the size adjustment of image features.

[0123] S11132. Project the reshaped features to obtain projection features.

[0124] Among them, the reshaping characteristics There are three preset projection matrices: , , ,; The projection matrix is ​​used to reshape the features Perform a product operation to obtain the corresponding projection features.

[0125] Specifically, calculate the reshaping features and The product of gets the first projection feature q, ; Calculate reshape features and The product of gets the second projection feature k, ; Calculate reshape features and The product of gets the third projection feature v, .

[0126] S11133. Perform self-similarity calculation on the projection feature to obtain self-similarity.

[0127] The self-similarity calculation is to calculate the initial self-similarity s based on the first projection feature q and the second projection feature k, where: ; Then the initial self-similarity s is normalized to obtain the target self-similarity ,in, .

[0128] Specifically, substitute the first projection feature q and the second projection feature k into Calculate the initial self-similarity s, and then substitute s into The target self-similarity is calculated in .

[0129] S11134. Obtain an aggregation result based on the self-similarity and the projection features.

[0130] Among them, the aggregation result has a corresponding calculation formula as follows: ,in, is the target self-similarity, and v is the third projection feature mentioned above.

[0131] Specifically, the target self-similarity Substitute the third projection feature v into Calculate the aggregate result .

[0132] It can be seen that in this embodiment, by calculating the aggregation result It is easy to understand how the self-attention layer aggregates the global features of the repaired image, and the aggregation results This facilitates the subsequent calculation of the low-frequency enhanced image.

[0133] It should be understood that, although the steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indications of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear description in this article, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0134] Based on the same inventive concept, the embodiment of the present application also provides an image restoration device. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more image restoration device embodiments provided below can refer to the limitations on the image restoration method above, and will not be repeated here.

[0135] like Figure 4 As shown, the embodiment of the present application provides an image restoration device 400, comprising:

[0136] An image low-frequency enhancement module 402 is used to perform low-frequency division enhancement on the image to be repaired to obtain a low-frequency enhanced image;

[0137] A low-frequency component generation module 404 is used to process the image to be repaired and the low-frequency enhanced image based on a preset Gaussian blur kernel to obtain multi-scale low-frequency component information;

[0138] A high-frequency component generation module 406 is used to extract high-frequency components from the image to be restored and the low-frequency enhanced image based on the multi-scale low-frequency component information to obtain multi-scale high-frequency component information;

[0139] The high-frequency component recovery module 408 is used to perform high-frequency recovery on the multi-scale high-frequency component information to obtain a restored image.

[0140] In some embodiments, the low-frequency component generation module 404 is specifically used to: process the image to be repaired based on a preset Gaussian blur kernel to obtain a first multi-scale low-frequency component; process the low-frequency enhanced image based on a preset Gaussian blur kernel to obtain a second multi-scale low-frequency component; and obtain the multi-scale low-frequency component information based on the first multi-scale low-frequency component and the second multi-scale low-frequency component.

[0141] In some embodiments, the high-frequency component generation module 406 is specifically used to: extract the high-frequency component of the image to be repaired based on the first multi-scale low-frequency component to obtain a first multi-scale high-frequency component; extract the high-frequency component of the low-frequency enhanced image based on the second multi-scale low-frequency component to obtain a second multi-scale high-frequency component; and obtain the multi-scale high-frequency component information based on the first multi-scale high-frequency component and the second multi-scale high-frequency component.

[0142] In some embodiments, the high-frequency component recovery module 408 is specifically used to: stack the multi-scale high-frequency component information to obtain comprehensive high-frequency component information; process the comprehensive high-frequency component information based on the trained high-frequency restorer to obtain Laplace components; and add the Laplace components to obtain a restored image.

[0143] In some embodiments, the image low-frequency enhancement module 402 is specifically used to: process the image to be repaired based on a low-frequency frequency division enhancer to obtain local features and global features; and obtain the low-frequency enhanced image based on the local features, the global features and the image to be repaired.

[0144] In some embodiments, the image low-frequency enhancement module 402 is specifically used to: perform a first convolution process on the image to be repaired based on a trained low-frequency division frequency enhancer to obtain local features; perform a second convolution process on the image to be repaired based on a trained low-frequency division frequency enhancer to obtain image features; perform self-attention aggregation on the image features to obtain aggregation results; perform global information extraction on the aggregation results to obtain global features; and obtain the low-frequency enhanced image based on the local features, the global features, and the image to be repaired.

[0145] In some embodiments, the image low-frequency enhancement module 402 is specifically used to: perform a first convolution process on the image to be repaired based on a trained low-frequency frequency division enhancer to obtain local features; perform a second convolution process on the image to be repaired based on the trained low-frequency frequency division enhancer to obtain image features; resize the image features to obtain reshaped features; project the reshaped features to obtain projection features; perform self-similarity calculation on the projection features to obtain self-similarity; obtain an aggregation result based on the self-similarity and the projection features; extract global information on the aggregation result to obtain global features; and obtain the low-frequency enhanced image based on the local features, the global features, and the image to be repaired.

[0146] Each module in the above-mentioned image restoration device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module above.

[0147] In some embodiments, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the image to be repaired and the repaired image. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the steps in the above-mentioned image repair method are implemented.

[0148] In some embodiments, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 6As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be realized through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, the steps in the above-mentioned image restoration method are implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen; the input device of the computer device can be a touch layer covering the display screen, or a key, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0149] Those skilled in the art will understand that Figure 5 or Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0150] In some embodiments, a computer device is provided. The computer device includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, the steps in the above method embodiments are implemented.

[0151] In some embodiments, Figure 7 The figure shows an internal structure diagram of a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0152] In some embodiments, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0153] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0154] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0155] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0156] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An image restoration method, characterized in that: include: Perform low-frequency division enhancement on the image to be repaired to obtain a low-frequency enhanced image; Processing the image to be repaired and the low-frequency enhanced image based on a preset Gaussian blur kernel to obtain multi-scale low-frequency component information; Based on the multi-scale low-frequency component information, high-frequency components are extracted from the image to be repaired and the low-frequency enhanced image to obtain multi-scale high-frequency component information; Performing high-frequency restoration on the multi-scale high-frequency component information to obtain a repaired image; The step of processing the image to be repaired and the low-frequency enhanced image based on a preset Gaussian blur kernel to obtain multi-scale low-frequency component information includes: Processing the image to be restored based on a preset Gaussian blur kernel to obtain a first multi-scale low-frequency component; Processing the low-frequency enhanced image based on a preset Gaussian blur kernel to obtain a second multi-scale low-frequency component; Obtaining the multi-scale low-frequency component information based on the first multi-scale low-frequency component and the second multi-scale low-frequency component; The step of extracting high-frequency components from the image to be restored and the low-frequency enhanced image based on the multi-scale low-frequency component information to obtain multi-scale high-frequency component information includes: Extracting high-frequency components from the image to be restored based on the first multi-scale low-frequency components to obtain first multi-scale high-frequency components; Extracting high-frequency components from the low-frequency enhanced image based on the second multi-scale low-frequency components to obtain second multi-scale high-frequency components; The multi-scale high-frequency component information is obtained based on the first multi-scale high-frequency component and the second multi-scale high-frequency component.

2. The method according to claim 1, characterized in that The step of performing high-frequency restoration on the multi-scale high-frequency component information to obtain a repaired image includes: Stacking the multi-scale high-frequency component information to obtain comprehensive high-frequency component information; Processing the comprehensive high-frequency component information based on the trained high-frequency restorer to obtain the Laplace component; The Laplace components are added to obtain a restored image.

3. The method according to claim 1, characterized in that The step of performing low-frequency division enhancement on the image to be repaired to obtain a low-frequency enhanced image comprises: Processing the image to be repaired based on a low-frequency division enhancer to obtain local features and global features; The low-frequency enhanced image is obtained based on the local features, the global features and the image to be repaired.

4. The method according to claim 3, characterized in that The step of processing the image to be repaired based on the low-frequency division enhancer to obtain local features and global features includes: Based on the trained low-frequency frequency division enhancer, a first convolution process is performed on the image to be repaired to obtain a local feature; Performing a second convolution process on the image to be repaired based on the trained low-frequency division enhancer to obtain image features; Performing self-attention aggregation on the image features to obtain an aggregation result; Perform global information extraction on the aggregation result to obtain global features.

5. The method according to claim 4, characterized in that The performing self-attention aggregation on the image features to obtain an aggregation result includes: Resizing the image features to obtain reshaped features; Projecting the reshaped feature to obtain a projection feature; Performing self-similarity calculation on the projection feature to obtain self-similarity; An aggregation result is obtained based on the self-similarity and the projection feature.

6. An image restoration device, characterized in that: include: An image low-frequency enhancement module is used to perform low-frequency division enhancement on the image to be repaired to obtain a low-frequency enhanced image; A low-frequency component generation module, used for processing the image to be repaired and the low-frequency enhanced image based on a preset Gaussian blur kernel to obtain multi-scale low-frequency component information; A high-frequency component generation module, configured to extract high-frequency components from the image to be restored and the low-frequency enhanced image based on the multi-scale low-frequency component information to obtain multi-scale high-frequency component information; A high-frequency component recovery module performs high-frequency recovery on the multi-scale high-frequency component information to obtain a restored image; The step of processing the image to be repaired and the low-frequency enhanced image based on a preset Gaussian blur kernel to obtain multi-scale low-frequency component information includes: Processing the image to be restored based on a preset Gaussian blur kernel to obtain a first multi-scale low-frequency component; Processing the low-frequency enhanced image based on a preset Gaussian blur kernel to obtain a second multi-scale low-frequency component; Obtaining the multi-scale low-frequency component information based on the first multi-scale low-frequency component and the second multi-scale low-frequency component; The step of extracting high-frequency components from the image to be restored and the low-frequency enhanced image based on the multi-scale low-frequency component information to obtain multi-scale high-frequency component information includes: Extracting high-frequency components from the image to be restored based on the first multi-scale low-frequency components to obtain first multi-scale high-frequency components; Extracting high-frequency components from the low-frequency enhanced image based on the second multi-scale low-frequency components to obtain second multi-scale high-frequency components; The multi-scale high-frequency component information is obtained based on the first multi-scale high-frequency component and the second multi-scale high-frequency component.

7. A computer device, comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images