Face image super-resolution restoration method, storage medium and electronic device

Abstract

The application provides a face image super-resolution recovery method, a storage medium and an electronic device, and the method comprises the following steps: receiving a face image to be processed; performing first processing on the face image through a double attention module to obtain a first structure information image of the face image; performing second processing on the first structure information through a face key point activation module to activate key point information on the face image and obtain a second structure information image; identifying the difference between the frequency domain style of the second structure information image and a real image in a frequency domain space through a frequency domain perception decomposition discriminator, and obtaining corresponding confidence information based on the different frequency domain styles. The above scheme can finely recover the key positions of the face, further improve the structure information on the face image, and make the recovered face better in terms of sensory effect.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, in particular to a face image super-resolution recovery method, a storage medium and an electronic device. BACKGROUND

[0002] Among all images, face images are the most researched and most widely applied. However, due to the limitations of imaging systems and imaging conditions, the resolution of many face images is low, so it is necessary to recover the face super-resolution. Face super-resolution recovery is a specific field of image recovery in the field of computer vision, and it is necessary to pay attention to the essential difference between face and other images, that is, the face has a specific structure.

[0003] At present, the face super-resolution recovery methods mainly include a method based on a convolutional neural network and a method based on a generative adversarial network. The method based on the convolutional neural network encodes and decodes the global or local high-dimensional features of the face image through the convolutional network, and the face image generated by this method is too smooth and lacks structural information, which is poor in sensory effect. The method based on the generative adversarial network recovers a high-resolution face image by using the powerful image generation capability of the generative adversarial network. This method is prone to chessboard effect and generates artifacts, resulting in unnatural or abnormal texture in some areas. SUMMARY

[0004] Therefore, it is necessary to provide a face image super-resolution recovery technical scheme to solve the problems of lack of structural information and texture abnormality in the recovered super-resolution face image in the prior art.

[0005] To achieve the above-mentioned purpose, in a first aspect, the present application provides a face image super-resolution recovery method, which comprises the following steps:

[0006] S1: receiving a face image to be processed;

[0007] S2: performing first processing on the face image through a double attention module to obtain a first structural information image of the face image;

[0008] S3: performing second processing on the first structural information through a face key point activation module to activate key point information on the face image, and obtaining a second structural information image;

[0009] S4: identifying the difference between the second structural information image and a real image in the frequency domain style in the frequency domain space through a frequency domain perception decomposition discriminator, and obtaining corresponding confidence information based on the different frequency domain styles.

[0010] In some embodiments, the human face image to be processed is a low-resolution image, and before step S2, step S2 further comprises S11:

[0011] The low-resolution image is subjected to bicubic interpolation processing to convert it into a high-resolution image.

[0012] In some embodiments, the dual attention module comprises an encoder, a feature processing layer, and a decoder in the order of image processing; the encoder comprises a first preset number of dual attention units, the feature processing layer comprises a second preset number of dual attention units, and the decoder comprises a third preset number of dual attention units.

[0013] In some embodiments, the dual attention unit comprises a channel attention unit and a spatial attention unit.

[0014] The channel attention unit is configured to receive a first value, perform average pooling and maximum pooling operations on the first value, add the features obtained by the average pooling and the maximum pooling pixel by pixel, and obtain first channel attention information after passing through an activation function.

[0015] The spatial attention unit is configured to receive the first value and perform sampling and residual operations thereon to obtain second channel attention information.

[0016] In some embodiments, the first processing comprises:

[0017] The channel attention information and the second channel attention information are multiplied pixel by pixel with the human face image.

[0018] In some embodiments, the spatial attention unit adopts a funnel model.

[0019] In some embodiments, step S3 comprises:

[0020] Two funnel models are used to predict key point information heat maps.

[0021] A key point heat map prediction model is used to generate corresponding human face image key point information ground truth.

[0022] A maximum pooling operation is used to compress the key point information heat map to one dimension in the channel dimension, so that the obtained one-dimensional channel feature map can highlight the key point information position, and a second structure information image is obtained.

[0023] In some embodiments, the frequency domain perception decomposition discriminator comprises a frequency domain perception decomposition module and a convolutional network calculation module.

[0024] Step S4 comprises:

[0025] input the second structure information image into the frequency domain perception decomposition module, separate the low frequency information and the high frequency information on the image in different matrix positions through discrete cosine transform, and convert the frequency domain back to the image domain through inverse discrete cosine transform;

[0026] input the image domain into the convolution network calculation module, and output corresponding confidence information.

[0027] In a second aspect, the present application further provides a storage medium, wherein the storage medium stores a computer program, and the computer program is executed to realize the method in the first aspect of the present application.

[0028] In a third aspect, the present application further provides a server, comprising:

[0029] a storage medium, which is the storage medium in the second aspect of the present application;

[0030] a processor, which is electrically connected with the storage medium, and is used to execute the computer program stored in the storage medium to realize the method in the first aspect of the present application.

[0031] Different from the prior art, the present application has the following characteristics:

[0032] The present application provides a face image super-resolution recovery method, a storage medium and an electronic device, the method comprising: receiving a face image to be processed; performing first processing on the face image through a double attention module to obtain a first structure information image of the face image; performing second processing on the first structure information through a face key point activation module to activate key point information on the face image, and obtaining a second structure information image; identifying the difference between the second structure information image and a real image in the frequency domain style in the frequency domain space through a frequency domain perception decomposition discriminator, and obtaining corresponding confidence information based on the different frequency domain styles. The above-mentioned scheme can finely recover the key positions of the face, further improve the structure information on the face image, and make the recovered face better in the sense of the human senses. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 a flowchart of the face image super-resolution recovery method according to the first embodiment of the present application;

[0034] Figure 2 a flowchart of the face image super-resolution recovery method according to the second embodiment of the present application;

[0035] Figure 3 a flowchart of the face image super-resolution recovery method according to the third embodiment of the present application;

[0036] Figure 4A flow chart of a face image super-resolution recovery method according to a fourth embodiment of the present application;

[0037] Figure 5 A flow chart of a face image super-resolution recovery method according to a fifth embodiment of the present application;

[0038] Figure 6 A schematic diagram of an electronic device according to an embodiment of the present application;

[0039] Figure 7 A schematic diagram of the overall network structure of a computer program according to an embodiment of the present application;

[0040] Figure 8 A schematic diagram of the structure of a dual attention unit according to an embodiment of the present application;

[0041] Figure 9 A schematic diagram of the working principle of a channel attention unit and a spatial attention unit according to an embodiment of the present application;

[0042] Figure 10 A schematic diagram of a face key point activation module according to an embodiment of the present application;

[0043] Figure 11 A schematic diagram of a frequency domain perception decomposition discriminator according to an embodiment of the present application;

[0044] Figure 12 A comparison chart of a face image super-resolution recovery method according to an embodiment of the present application and other existing methods based on sensory evaluation indicators. DETAILED DESCRIPTION

[0045] To explain the possible application scenarios, technical principles, specific implementation schemes, and the purposes and effects that can be achieved of the present application in detail, the following will be described in detail in combination with the specific embodiments listed and the accompanying drawings. The embodiments described in this paper are only used to more clearly illustrate the technical solutions of the present application, and therefore only serve as examples, and cannot limit the protection scope of the present application.

[0046] In this paper, the term "embodiment" means that the specific features, structures or characteristics described in combination with the embodiment can be included in at least one embodiment of the present application. The term "embodiment" appearing at various places in the specification does not necessarily refer to the same embodiment, and does not particularly limit its independence or association with other embodiments. In principle, in the present application, as long as there is no technical contradiction or conflict, each technical feature mentioned in each embodiment can be combined in any way to form a corresponding implementable technical solution.

[0047] Unless otherwise defined, the meanings of technical terms used in the present application are the same as commonly understood by one of ordinary skill in the art to which the present application belongs; the use of related terms herein is only for the purpose of describing specific embodiments of the present application and is not intended to limit the present application.

[0048] In the description of the present application, the phrase "and / or" is a description of a logical relationship between objects, which means that there can be three relationships, for example, A and / or B, which means that there are three cases: A exists, B exists, and A and B exist at the same time. In addition, the character " / " herein generally represents that the associated objects before and after are a "or" logical relationship.

[0049] In the present application, phrases such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual quantity, primary and secondary, or order relationship between the entities or operations.

[0050] In the present application, without more limitation, the use of "includes", "contains", "has" or other similar expressions in a sentence is intended to cover non-exclusive inclusion, and these expressions do not exclude the presence of additional elements in the process, method or product including the elements, so that the process, method or product including a series of elements can not only include those defined elements, but also include other elements not explicitly listed, or also include elements inherent to such process, method or product.

[0051] As the same understanding as in the "Guidelines for Examination", in the present application, the expressions "greater than", "less than", "exceed" and the like are understood as not including the number; the expressions "above", "below", "within" and the like are understood as including the number. In addition, in the description of the embodiments of the present application, the meaning of "multiple" is more than two (including two), and similar expressions related to "multiple" are also understood in this way, for example, "multiple groups", "multiple times" and the like, unless otherwise explicitly limited.

[0052] Please refer to Figure 1 The present application provides a face image super-resolution recovery method, which comprises the following steps:

[0053] S1: receiving a face image to be processed;

[0054] S2: performing first processing on the face image through a double attention module to obtain a first structure information image of the face image;

[0055] S3: performing second processing on the first structure information through a face key point activation module to activate key point information on the face image, and obtaining a second structure information image;

[0056] S4: The frequency domain sensing decomposition discriminator identifies the differences in frequency domain patterns between the second structural information image and the real image in the frequency domain space, and obtains the corresponding confidence information based on the different frequency domain patterns.

[0057] The above scheme can accurately restore the key positions of the face, further enhance the structural information on the face image, and make the sensory effect of face restoration better.

[0058] In some embodiments, such as Figure 2 As shown, the face image to be processed is a low-resolution image. Before step S2, step S11 is included: performing bicubic interpolation on the low-resolution image to convert it into a high-resolution image. Low-resolution images have too few pixels, making it difficult for the model to learn effective semantic and structural information. Bicubic interpolation converts the low-resolution face image into a high-resolution face image as input to the network model. Bicubic interpolation uses the gray values ​​of 16 points surrounding the sampling point for cubic interpolation, considering not only the gray value influence of the four directly adjacent points but also the influence of the gray value change rate between neighboring points, thus achieving a magnification effect closer to a high-resolution image.

[0059] In some embodiments, the dual attention module includes an encoder, a feature processing layer, and a decoder in the image processing order; the encoder includes a first preset number of dual attention units, the feature processing layer includes a second preset number of dual attention units, and the decoder includes a third preset number of dual attention units. Preferably, the first and third preset numbers are 3, and the second preset number is 8. The dual attention module enables the network to pay more attention to locations such as the corners of the mouth, the corners of the eyes, and the facial contours, which play a crucial role in evaluating the quality of super-resolution faces.

[0060] like Figure 3 As shown, the dual attention unit includes a channel attention unit and a spatial attention unit. The method includes:

[0061] First, in step S301, the channel attention unit receives the first value, performs average pooling and max pooling operations on the first value, adds the features obtained by average pooling and max pooling pixel by pixel, and obtains the first channel attention information after processing by the activation function.

[0062] Then, in step S302, the spatial attention unit receives the first value and performs sampling and residual operations on it to obtain the second channel attention information.

[0063] Then, in step S303, the first processing includes: multiplying the face image pixel by pixel based on the channel attention information and the second channel attention information.

[0064] Preferably, the spatial attention unit adopts a hourglass model.

[0065] As shown in Figure 8 , the double attention unit adopts a structure in which channel attention and spatial attention are connected in series. Channel attention is used to enable the network to learn the importance of each facial feature channel, so as to reduce the computational load of the network and enhance the expression of the channel features. Spatial attention is used to enable the network to learn which part of the facial feature map is the most important, so as to enhance the sensitivity of the network to structural information.

[0066] As shown in Figure 9 , the input of the channel attention is f, which is subjected to average pooling and maximum pooling operations respectively. The average pooling is used to integrate spatial information, and the maximum pooling is used to emphasize the difference of the features. Then a multi-layer perceptron is input to learn the relevant features, and finally the features obtained by the average pooling and the maximum pooling are added pixel by pixel, and the channel attention ac is obtained through the sigmoid activation function. The input f of the spatial attention is the output of the channel attention module, and a hourglass model is used to process multi-scale information to capture spatial information at each scale. After successive downsampling, upsampling and residual operations through the hourglass model, the spatial attention as is finally obtained. Finally, the channel attention and the spatial attention are multiplied pixel by pixel on the facial image to enhance the structural information.

[0067] As shown in Figure 4 , in some embodiments, step S3 comprises:

[0068] Firstly, step S401 adopts two hourglass models to predict key point information heat maps;

[0069] Then, step S402 adopts a key point heat map prediction model to generate the true value of the corresponding facial image key point information;

[0070] Then, step S403 uses a maximum pooling operation to compress the key point information heat map to one dimension in the channel dimension, so that the one-dimensional channel feature map obtained can highlight the key point information position, and a second structural information image is obtained. The facial key point activation module considers that the facial key point prediction task and the facial super-resolution task can promote each other, and adopts a priori parallel structure to assist the recovery of facial super-resolution.

[0071] As shown in Figure 10 , in order to restore more accurate facial priori, the present application first inputs the intermediate features obtained from the double attention module into the facial key point activation module. In the branch of predicting the facial key point heat map, a deeper network structure is adopted, for example, as shown in Figure 10The two funnel models shown are used to predict the key point heatmap MSR. At the same time, the key point heatmap prediction model Super-fan is used to generate the corresponding face key point true value MHR as supervision. The loss function Lheatmap adopts the mean square error loss. In order to be able to utilize the face key point prior information, while generating the face key point heatmap, the maximum pooling operation is used to compress the key point heatmap to 1 dimension in the channel dimension, so that the obtained one-dimensional channel feature map emphasizes the key point position. Finally, it is multiplied with the feature map in the face super-resolution process pixel by pixel to achieve the purpose of activating the key point. In the training process, the loss function Lpixel of the face super-resolution process adopts L1Loss.

[0072] As Figure 5 shown, in certain embodiments, the frequency domain perception decomposition discriminator includes a frequency domain perception decomposition module and a convolutional network calculation module; step S4 includes:

[0073] Firstly, step S501 inputs the second structure information image into the frequency domain perception decomposition module, separates the low-frequency information and the high-frequency information on the image in different matrix positions through discrete cosine transformation, and then converts the frequency domain back to the image domain through inverse discrete cosine transformation;

[0074] Then, step S502 inputs the image domain into the convolutional network calculation module, and outputs the corresponding confidence information.

[0075] As Figure 11 shown, the present application learns from deep face anti-forgery technology, introduces frequency domain information in RGB space, lets the discriminator learn how to find the hidden forgery style (each frequency band information composition) in the frequency domain through frequency domain decomposition, and enhances the perception ability of the discriminator to the frequency domain information. The discriminator is divided into two parts: a frequency domain perception decomposition module and a convolutional neural network Xception.

[0076] The frequency domain perception decomposition module is composed of 4 basic filters fbase and 4 learnable filters flearn, corresponding to low, medium, high and full frequency domain segments. The low frequency domain segment corresponds to the first 1 / 16 frequency domain, the medium frequency domain segment corresponds to the first 1 / 16 to 1 / 8 frequency domain, and the high frequency domain segment corresponds to the remaining frequency domain. The basic filter is set to 1 at the corresponding element position of the received frequency, and is set to 0 at the rejected frequency. The 4 learnable filters are designed the same, with a value range of -1 to 1, and their element values at each frequency domain represent whether to adopt a certain frequency.

[0077] The specific implementation process of the discriminator is as follows: first, input the super-resolution face and the high-resolution face into the frequency domain perception decomposition module, and through discrete cosine transform D (Discrete Cosine Transform), the low-frequency information of the image is arranged in the upper left corner of the matrix, and the high-frequency signal is located in the lower right corner, so that the corresponding frequency domain can be selected according to the element position. Then, four frequency band matrices are obtained, and then the frequency domain is converted back to the image domain through inverse discrete cosine transform. Finally, input into the classic convolutional network Xception, and output the corresponding confidence.

[0078] In a second aspect, the present application also provides a storage medium, wherein the storage medium stores a computer program, and the computer program is executed to realize the method of the first aspect of the present application.

[0079] In a third aspect, as shown in Figure 6 The present application also provides an electronic device 10, comprising a storage medium 102 and a processor 101, the storage medium 102 is the storage medium of the second aspect; the processor 101 is electrically connected with the storage medium 102, and is used for executing the computer program stored in the storage medium to realize the method of the first aspect.

[0080] Preferably, the electronic device can be a computer device, including but not limited to: personal computer, server, general-purpose computer, special-purpose computer, network device, embedded device, programmable device, smart mobile terminal, smart home device, wearable smart device, vehicle-mounted smart device, etc. The storage medium includes but is not limited to: RAM, ROM, magnetic disk, magnetic tape, optical disc, flash memory, U disk, mobile hard disk, memory card, memory stick, network server storage, network cloud storage, etc. The processor includes but is not limited to CPU (central processing unit), GPU (graphics processor), MCU (microprocessor), etc.

[0081] The present application adopts the generator-discriminator structure, wherein the generator comprises a face double attention module and a face key point activation module, and the discriminator is a frequency domain perception decomposition discriminator designed for the face super-resolution recovery task. The model structure diagram of the present application is shown in Figure 7 .

[0082] As shown in Figure 7 From left to right, they are face double attention module, face key point activation module and discriminator based on frequency domain perception decomposition. The whole network structure adopts the generator-discriminator structure, first, the face double attention module extracts the low-resolution image I LRThe basic feature structure of the face image, such as the approximate proportion of the five organs and the stereoscopic effect, is extracted. Secondly, the face key point activation module is used to refine the face structure information, such as further accurately positioning the five organs. Finally, the frequency domain perception decomposition discriminator is used to excavate the forged frequency domain style in the generated image, so as to improve the discrimination ability of the discriminator on the generated face I SR and the real face I HR , and promote the generator to generate more reasonable and natural texture.

[0083] The present application has the following advantages:

[0084] Firstly, the present application adopts a face super-resolution method based on attention mechanism and frequency domain perception. This method improves both the generator part and the discriminator part, and uses the key point semantic information of the face to guide the super-resolution process, so that the restored face has more stereoscopic structure and more reasonable and clear texture details.

[0085] Secondly, the present application designs a double attention module combining channel attention and spatial attention, and improves the spatial attention for the face. This can reduce the amount of calculation while restoring more stereoscopic structure information, so that the restored face has better sensory effect.

[0086] Thirdly, the present application designs a face key point activation module to fully utilize the semantic information of the face key points and guide the super-resolution process. This can finely restore the key positions of the face and further improve the structure information.

[0087] Fourthly, the present application designs a frequency domain perception decomposition discriminator to discriminate the generated image in the frequency domain space, providing a new angle to distinguish the authenticity of the face image. This can restore reasonable frequency domain style, reduce the occurrence of distortion and artifacts, and enhance texture details.

[0088] The experimental results of the model designed by the present application are shown in Figure 12 The model designed by the present application surpasses the existing face super-resolution method in the sensory-based effect on the face dataset CelebA, with an improvement of 17% in Frechet inception distance (FID) and 11% in Learned Perceptual Image Patch Similarity (LPIPS).

[0089] Although the above-mentioned embodiments have been described, those skilled in the art can make further changes and modifications to the embodiments once they know the basic inventive concept, and therefore the above-mentioned embodiments are only examples of the present application, and are not intended to limit the patent protection scope of the present application, and any equivalent structure or equivalent process transformation made by using the content of the present application specification and drawings, or directly or indirectly applied to other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for super-resolution recovery of a face image, characterized in that, The method comprises the following steps: S1: receiving a face image to be processed; S2: performing first processing on the face image through a double attention module to obtain a first structure information image of the face image; S3: performing second processing on the first structure information through a face key point activation module to activate key point information on the face image, and obtaining a second structure information image; S4: identifying the difference in frequency domain style between the second structure information image and a real image in a frequency domain space through a frequency domain perception decomposition discriminator, and obtaining corresponding confidence information based on the different frequency domain styles; The double attention module comprises an encoder, a feature processing layer, and a decoder in the order of image processing; the encoder comprises a first preset number of double attention units, the feature processing layer comprises a second preset number of double attention units, and the decoder comprises a third preset number of double attention units; The double attention unit comprises a channel attention unit and a spatial attention unit; The channel attention unit is configured to receive a first value, perform average pooling and maximum pooling operations on the first value, add the features obtained through the average pooling and the maximum pooling pixel by pixel, and obtain first channel attention information through an activation function; The spatial attention unit is configured to receive the first value and perform sampling and residual operations thereon to obtain second channel attention information; The first processing comprises: multiplying the face image pixel by pixel according to the channel attention information and the second channel attention information; Step S3 comprises: using two funnel models to predict a key point information heat map; using a key point heat map prediction model to generate a corresponding face image key point information true value; using a maximum pooling operation to compress the key point information heat map to one dimension in the channel dimension, so that the obtained one-dimensional channel feature map can highlight the key point information position, and a second structure information image is obtained; The frequency domain perception decomposition discriminator comprises a frequency domain perception decomposition module and a convolutional network calculation module; Step S4 comprises: inputting the second structure information image into the frequency domain perception decomposition module, separating the low-frequency information and the high-frequency information on the image in different matrix positions through a discrete cosine transform, and converting the frequency domain back to the image domain through an inverse discrete cosine transform; inputting the image domain into the convolutional network calculation module to output corresponding confidence information. 2.The method of claim 1, wherein, The face image to be processed is a low-resolution image, and before step S2, there is further S1-1: performing a bicubic interpolation on the low-resolution image to convert it into a high-resolution image. 3.The method of claim 1, wherein The spatial attention unit uses a funnel model.

4. A storage medium, characterized by The storage medium stores a computer program, and the computer program is executed to implement the method of any one of claims 1 to 3.

5. An electronic device, comprising: It comprises: a storage medium, as claimed in claim 4; a processor, electrically connected to the storage medium, configured to execute the computer program stored in the storage medium to implement the method of any one of claims 1 to 3.

Images