A generative learning method for restoring blurred face images

Abstract

The present invention belongs to the field of image processing technology, and specifically is a generative learning method for restoring blurred face images. The method includes the following steps: a step of receiving a blurred face image as an input of a feature extraction network, a step of strictly screening the extracted hierarchical features through a feature screening network, and a step of inputting the screened hierarchical features into a backbone network; wherein the feature extraction step performs feature extraction through a feature extraction network; the feature screening step screens the input hierarchical features through a double-layer self-attention layer of the feature screening network; the progressive adversarial generation step is to add the screened hierarchical features to the output signal of the generator trunk of the generator according to the "mirror principle" and progressively generate the target image. The advantage of the present invention is that after strictly screening the extracted hierarchical features, the corresponding level of the input generation network is input, and the global supervision module of the backbone network is designed to realize local and global supervision in the network adversarial process.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and in particular relates to a generative learning method for restoring blurred face images. Background Art

[0002] Face images have many applications, such as face recognition, face tracking, face swapping, and facial emotion detection (Jin et al., 2018). However, faces captured in the real world are often affected by the imaging process. During sensor exposure, due to camera shake or unintentional movement, the relative position of the utilized camera / scene system moves, which can introduce blur into the resulting face image. In such cases, we need to recover a sharp image from the blurred image. In most cases, the relative motion, i.e., the blur kernel, is unknown. Moreover, only a few images, such as a single frame, are captured. This poses the problem of blindly deblurring images using a single frame.

[0003] In an ideal digital imaging system, a bin (usually considered a point) in a scene corresponds to a pixel on the image screen. When the imaging system is applied to an image that is out of focus or has large aberrations or intense motion, the pixel is expanded to a finite area on the image plane. This causes the images formed by adjacent bins in the scene to overlap, resulting in image degradation. In this process, each pixel in the image is obtained by superimposing the images of the corresponding bin and its adjacent bins at that pixel location. If the imaging system is linear and translation-invariant, the degraded imaging result can be described by the following equation:

[0004] y=x*k+n (1)

[0005] Where y is the degraded image; x corresponds to the ideal sharp image; k is the point spread function (PSF), defined as the imaging system's response to a point source or object; n is the noise; an asterisk (*) indicates a convolution operation; and a plus sign (+) indicates an addition operation. The above equation describes how each pixel in the degraded image can be considered a weighted sum of the corresponding pixel in the ideal image and its neighboring pixels.

[0006] The goal of single-image blind deblurring is to recover a sharp image from a single blurred input. Without any prior or additional information, this problem is inherently difficult to solve because there are infinitely many pairs of latent images and blur kernels that can equally well explain a given blurred image (Jin et al., 2018). Traditional methods formulate the blurring process as a convolution operation between the latent sharp image and the blur kernel and solve this problem based on optimization methods. Optimization methods such as the maximum a posteriori (MAP) method have been proposed to address this problem. Due to the uncertainty nature of this problem, natural image priors such as sparsity (Krishnan et al., 2011), L0 gradient (Xu et al., 2013), patch priors (Sun et al., 2013), manifold priors (Ni et al., 2011), low-rank priors (Ren et al., 2016), and dark channel priors (Pan et al., 2016) have been introduced to constrain the solution space. These priors have been introduced to constrain the solution space, resulting in a large number of effective deblurring methods. Recently, due to the rapid development of deep learning, methods based on convolutional neural networks (CNNs) have become mainstream. (Shen et al., 2020) proposed a deep multi-scale CNN that deblurs images using global semantic priors and local structural constraints. The local structural loss adaptively adjusts the weights based on the size of each facial component. (Rajeev et al., 2020) proposed a mechanism called UMSN that independently learns category-specific features and combines them to deblur the image. Using a new confidence calculation method, it reconstructs each class in the deblurred image to balance the importance of semantic classes in training.

[0007] Deep learning-based face deblurring methods have made considerable progress, but challenges remain. Qualitatively, mainstream models perform poorly for significantly blurred portraits, resulting in distorted faces and blurred, blended contours. Implementation is complex and requires prior knowledge, such as a facial segmentation map. Summary of the Invention

[0008] The purpose of the present invention is to provide a generative learning method for restoring blurred facial images to overcome the problems of poor restoration effect for relatively blurred portraits in the prior art and relatively complex models.

[0009] To achieve the above objectives, the present invention adopts a technical solution: a generative learning method for restoring blurred face images, comprising the following steps:

[0010] Step 1: Inputting a blurred face image: receiving a blurred face image, inputting it into a hierarchical feature extraction network for feature extraction, and inputting it into the feature extraction network;

[0011] Step 2: Hierarchical feature screening step: The extracted hierarchical features are strictly screened through the feature screening network. The feature screening network is a double-layer self-attention layer that screens the input hierarchical features.

[0012] Step 3: Input the filtered hierarchical features into the backbone network, and according to the “mirror principle”, input them into the generator of the generative network and add them to the output signal of the generator backbone.

[0013] Step 4: Generate the target image progressively: During the generation process, a two-layer efficient self-attention layer is added as the global supervision module of the backbone network before the last two resolution blocks of the generator and the first two resolution blocks of the discriminator in the backbone network, and the target resolution size is progressively generated according to the resolution size.

[0014] Furthermore, in the above step 1, the convolution layer performs a convolution operation on the received input data, then performs layer normalization calculation on the result obtained by the convolution operation, and finally inputs it into the nonlinear excitation layer for calculation; the maximum pooling layer divides the received input data into 2*2 local neighborhoods, and selects the maximum value to pool the image.

[0015] Furthermore, in step 2 above, the double-layer self-attention layer adopts a double-layer efficient attention structure. By learning three matrices of Query, Key, and Value that are the same size as the input, the connection between the pixels between Query and Key is used to generate an attention matrix heat map, and then the dot product method is used on the Value matrix to allocate attention during the feature learning process.

[0016] Furthermore, in the above step three, the "mirror principle" means that the structure of the feature extraction network and the structure of the generator in the backbone network are symmetrical to each other, that is, the feature maps obtained at the corresponding levels are of the same size; the hierarchical features output by the feature extraction network and the output signal of the generator backbone are added as point additions between matrices.

[0017] Compared with the prior art, the advantages of the present invention are:

[0018] 1. This invention carefully constructs a symmetrical relationship between the pre-trained feature extraction module and the generator, presenting a "mirror" structure. The advantage is that the size of the extracted hierarchical features is consistent with the corresponding hierarchical features generated in the generator network, and can be directly fused without additional operations, while avoiding unnecessary operations that damage the original feature information.

[0019] 2. The feature correction module provided by this invention consists of a dual-layer efficient self-attention algorithm. Compared to the costly self-attention module of traditional Transform designs, this algorithm saves video memory during training without sacrificing performance. Efficient attention optimizes the order of operations in attention formulas based on the consistency of mathematical reasoning, thereby reducing the size of intermediate matrices during calculations. The dual-layer structural design considers the balance between overall model resource consumption and performance.

[0020] 3. The present invention extracts features according to the "mirror principle" and inputs them into the generator of the progressive generative adversarial network. The output signals are then fused together using an "addition" method, which helps to achieve strong control of features at the pixel level. In the solution of the present invention, the features of the last few resolution levels are discarded, and only the features of the first few resolution levels are used as prompt feature information for the generator. These are then added to the output features of the corresponding layers in the generative trunk, achieving feature fusion and better enhancing the model's learning ability.

[0021] 4. The present invention designs powerful local and global supervision modules for the task of face deblurring, aiming to learn and generate fuzzy face features in a refined manner. The local supervision module grasps the filtered fuzzy face features of each convolutional layer and symmetrical input in the generator to realize the learning and positioning of the local domain of features. The global supervision module is composed of a two-layer efficient attention module. The special thing about it is that the insertion position is before the last two resolution blocks of the unsupervised generator model and before the first two resolution blocks of the discriminator model. By correlating the feature points with each other, the corresponding attention map is obtained, which prompts the model to focus on the area, namely the face area, and realizes the global feature correlation and supervision generation function of the features in the model training. It is worth noting that the global supervision module in the discriminator is symmetrical with the generator in terms of framework, which realizes global supervision in the network confrontation process. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a schematic diagram of the backbone network structure proposed by the present invention;

[0023] Figure 2 This is a schematic diagram of the steps for feature extraction of blurred face images proposed by the present invention;

[0024] Figure 3 This is a schematic diagram of the steps for feature screening of blurred facial images proposed by the present invention;

[0025] Figure 4 This is a schematic diagram of the steps for progressively generating blurred face images proposed by the present invention. DETAILED DESCRIPTION

[0026] The specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0027] This patent proposes a generative learning method for restoring blurred face images. The method includes the following steps, as shown in the following diagram: Figure 1 As shown:

[0028] Step 1: Inputting blurred face images: Receive the blurred face images and pre-process them before inputting them into the network, and then input them into the feature extraction network. The hierarchical feature extraction network includes five layers of feature extraction modules, each of which includes two convolutional layers and one maximum pooling layer. Figure 2 As shown, the specific process includes: the operation of the convolution layer is to first perform a convolution operation on the received input data, and then perform layer normalization calculation on the result obtained by the convolution operation, and finally input it to the nonlinear excitation layer for calculation: according to the input blurred face image, after passing through the five resolution layer feature extraction modules of the hierarchical feature extraction network from large to small, five sets of feature information are obtained, and then the sixth set of feature information is obtained by downsampling using the maximum pooling scheme according to the feature information map of the last layer. For the feature extraction module of each layer, the input passes through two layers of convolution and one layer of maximum pooling. The convolution layer performs a convolution operation on the input data, trains and learns the feature space distribution of the input, and performs layer normalization on the result after convolution to achieve standardization of the feature space distribution, and then inputs it to the nonlinear excitation layer to calculate the feature output of the current layer. The nonlinear excitation layer uses a linear rectification function (LeakyReLU) to complete the operation.

[0029] In this step, the blurred face image is used for feature extraction by the hierarchical feature extraction network. The received blurred face image is a degraded image, and the final result obtained is the hierarchical feature information of the degraded image.

[0030] Step 2: Hierarchical feature screening step: The extracted hierarchical features are strictly screened through the feature screening network. The feature screening network is a double-layer self-attention layer, the purpose of which is to screen the input hierarchical features. Its structural diagram is shown in the figure below. Figure 3 shown.

[0031] For the six groups of feature maps extracted from the hierarchical feature extraction network, a two-layer efficient self-attention module is used as a feature screening module to filter the degradation information of the feature input with strong degradation information and output a relatively clean feature map. In this embodiment, the features of the last two resolution layers are discarded, and only the features of the first four resolution levels are used as the prompt feature information of the generator. For the first four resolution level feature inputs, a feature screening module of the corresponding level is designed to filter the potential degradation information of the input features. The specific implementation of efficient attention is a self-attention module consisting of pre-layer normalization and a linear attention layer, plus a forward feedback layer consisting of pre-layer normalization and two convolutional layers.

[0032]

[0033] The linear attention layer in the self-attention module is a key component of efficient attention. It first convolutionally learns the query, key, and value matrices, then multiplies the transpose of the key by the value to obtain a smaller intermediate matrix, which is then dot-multiplied with the query matrix to achieve point-to-point relationship learning of the feature graph.

[0034] In this step, the received face blurred image hierarchical features are hierarchical features containing a large amount of degradation information, and the final result obtained is the filtered optimized hierarchical features corresponding to the hierarchical features containing the degradation information.

[0035] Step 3: Input the filtered hierarchical features into the backbone network, and according to the "mirror principle", input the corresponding generator of the generation network, and add and fuse them with the output signal of the generator backbone.

[0036] The "mirror image principle" refers to the symmetry between the structure of the feature extraction network and the structure of the generator in the backbone network. That is, the feature maps generated at corresponding layers are of the same size. The hierarchical features output by the feature extraction network and the output signal of the generator backbone are added as a point-by-point addition between matrices.

[0037] Step 4: Generate the target image using a progressive approach: During the generation process, a two-layer efficient self-attention layer is added to the backbone network before the last two resolution blocks of the generator and the first two resolution blocks of the discriminator as a global supervision module, participating in the model generation adversarial process. By leveraging the correlation between feature magnitudes generated by the self-attention layer during training, the importance of point-to-point relationships is learned, enabling global supervision during the generation of high-resolution layers, thereby improving the overall quality of the generated image.

[0038] The generative network uses a progressive generation method to generate images to the target resolution. The specific method is that the generative network generates images layer by layer according to the size of the resolution, starting from low resolution, and then improving the resolution to learn to generate images at higher resolutions, step by step. At the same time, a symmetrical discriminator network is designed to act as a "referee". By scoring the generated images generated by the generative network and the real images, it promotes training with the generator and finally reaches the Nash equilibrium. The schematic diagram of the progressive generation process is shown below. Figure 4 shown.

[0039] Finally, it should be noted that the above implementation methods are only used to illustrate the technical solutions of the embodiments of the present invention and are not limiting. Although the embodiments of the present invention are described in detail with reference to the above preferred implementation methods, ordinary technicians in this field should understand that the technical solutions of the embodiments of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A generative learning method for restoring blurred face images, characterized by: The following steps are included Step 1: Inputting a blurred face image: receiving a blurred face image and inputting it into a hierarchical feature extraction network for feature extraction; Step 2: Hierarchical feature screening step: The extracted hierarchical features are screened through a feature screening network. The feature screening network is a double-layer self-attention layer that screens the input hierarchical features. Step 3: Input the filtered hierarchical features into the backbone network, and according to the "mirror principle", input them into the generator of the generative network, and add and fuse them with the output signal of the generator backbone. Step 4: Generate the target image progressively: In the process of generating the target image, a two-layer efficient self-attention layer is added as the global supervision module of the backbone network before the last two resolution blocks of the generator and the first two resolution blocks of the discriminator in the backbone network, and an image of the target resolution size is progressively generated according to the resolution size; In step 1, the hierarchical feature extraction network includes five layers of feature extraction modules, each of which includes two convolutional layers and one maximum pooling layer. The convolutional layer performs a convolution operation on the received input data, then performs layer normalization calculation on the result obtained by the convolution operation, and finally inputs it into the nonlinear excitation layer for calculation; the maximum pooling layer divides the received input data into 2*2 local neighborhoods for judgment, selects the maximum value to pool the image; In step 2, the two-layer self-attention layer adopts a two-layer efficient attention structure, first convolutionally learning the query, key, and value matrices, then multiplying the transpose of the key with the value to obtain a smaller intermediate matrix, which is then dot-multiplied with the query matrix to achieve learning of the point-to-point relationship between the feature graph; In step three, the "mirror principle" refers to the symmetry between the hierarchical feature extraction network structure and the structure of the generator in the backbone network; the hierarchical features output by the hierarchical feature extraction network and the output signal of the generator backbone are added as point additions between matrices.

Images