A face super-resolution method based on multi-scale attention residual and equivariant mapping

Abstract

The present invention discloses a human face super-resolution method based on multi-scale attention residual and equivariant mapping, firstly, the shallow features of the low-resolution human face side image are extracted through the convolution layer; then, the shallow features are input into the feature extraction subnet, and the deep features are obtained through n multi-scale attention residual modules; the deep features obtained are further input into the residual equivariant mapping module, and the deep features are combined with the residual features of the fused yaw coefficient in the deep representation feature space, and the feature vector of the side face is transformed into the same feature vector space as the front face; finally, a high-resolution human face front image is obtained through the reconstruction module. The present invention is suitable for human face recognition, and no longer overly relies on a large number of front and side face data pairs. For the input low-resolution side face image, a front face image with high definition and more realistic texture can be reconstructed.

Description

Technical Field

[0001] The present invention belongs to the field of image processing and computer vision, and relates to a super-resolution reconstruction method of a face image, and in particular to a face super-resolution method based on multi-scale attention residual and equivariant mapping. Background Art

[0002] Face recognition has achieved extraordinary success due to the emergence of deep learning. However, many existing face recognition models still perform relatively poorly when processing profile images compared to frontal images. In practical applications, the resolution of the obtained face images is relatively low, and even if they are converted to frontal images, the image quality is still relatively poor.

[0003] In order to solve the above problems, the present invention proposes a face super-resolution method based on multi-scale attention residual and equivariant mapping to achieve super-resolution reconstruction of low-resolution side face images into high-resolution frontal face images. Summary of the invention

[0004] The present invention is used to solve the problem that the current image super-resolution reconstruction method can only simply reconstruct the image super-resolution into a high-resolution image, but cannot realize the angle conversion of the image from the front face to the side face. In order to solve the above problem, the present invention provides a face super-resolution method based on multi-scale attention residual and equivariant mapping. This method uses convolution kernels of different scales and a method of increasing attention to extract image features, and uses a residual equivariant mapping module to process and reconstruct features. The feature vector of the side face image can be transformed into the same feature vector space as the front face image, and finally a high-resolution front face image is obtained after passing through the reconstruction module. Specifically, the following steps are included:

[0005] 1) Take the low-resolution face profile image I LR As the input of the network, I LR Perform convolution operation to extract shallow features M0;

[0006] 2) The shallow feature M0 is used as the input of the feature extraction subnet composed of n multi-scale attention residual modules, and then the features output by each module in the subnet are fused through the feature fusion layer to obtain the deep feature M c ;

[0007] 3) The fused deep features M c Through a fully connected layer, we get feature M d ;

[0008] 4) The feature M d Input to the residual equivariant mapping module, the feature M obtained after two fully connected layers gAt the same time, the angle estimation module extracts 21 key points of the input low-resolution side face image to obtain the yaw coefficient Y of the face angle, and the yaw coefficient Y and M g Multiply and finally add the input feature M d , and obtain the final transformed feature M f ;

[0009] 5) The transformed feature vector M f Input the reconstruction module to obtain the final reconstructed high-resolution frontal face image I SR .

[0010] As a further preferred embodiment, the feature extraction subnet described in step 2) is specifically:

[0011] 2.1) The feature extraction subnet consists of n multi-scale attention residual modules, each of which consists of two parts: a multi-scale residual module and an attention module.

[0012] For the i-th multi-scale attention residual module, its input feature is the output M of the previous multi-scale attention residual module i-1 , the output feature is M i ;

[0013] The multi-scale residual module described in 2.1.1) is composed of two 3*3 convolutions, two 5*5 convolutions and one 1*1 convolution. The input features are respectively passed through the 3*3 convolution kernel and the 5*5 convolution kernel to generate features F1 and P1. The output results of these two convolution kernels are concatted together as the input of the next 3*3 convolution kernel and the 5*5 convolution kernel, and the generated features are F2 and P2. Then F2 and P2 are concatted together as the input of the 1*1 convolution kernel, and the generated feature is S1. Convolution kernels of different sizes extract different features, so if two convolution kernels are used in the same network, different features can be obtained in one network, thereby achieving the requirement of higher image resolution. Because two concat operations are performed, the number of channels of the feature map becomes four times the original number. In order to realize the residual operation, a 1*1 convolution kernel is added at the end to compress the number of channels of the feature map.

[0014] 2.1.2) The attention module described in the above method is mainly to adaptively scale channel-level features by using second-order feature statistics for more discriminative representation. First, the covariance matrix of the input features is calculated, and then the covariance matrix is ​​decomposed into eigenvalues, and finally the covariance is normalized and converted into the 1 / 2 power of the eigenvalue. In this way, the normalized covariance matrix a is obtained, and this normalized covariance matrix represents the correlation between different channel features. Finally, each layer of the normalized covariance matrix is ​​subjected to global average pooling to obtain the channel descriptor of each layer, which is specifically expressed as follows:

[0015] Z=H GCP (a)

[0016] Among them, H GCP () indicates the global average pooling operation.

[0017] After obtaining the channel descriptor of each layer, the final weight W is obtained through the gating function, which is specifically expressed as follows:

[0018] W=f(W U ·r(W D ·Z))

[0019] Among them, W U , W D is the weight of the convolutional layer, f and r represent the Sigmoid function and Relu function respectively.

[0020] Finally, the input feature S1 is multiplied by the weight W to obtain the final weighted feature S2 = W*S1.

[0021] 2.1.3) Finally, the output feature S2 of the attention module and the input M of the i-th multi-scale attention residual module i-1 Add together to get the final output feature M i =M i-1 +S2.

[0022] 2.2) Finally, the outputs of all multi-scale attention residual modules are fused, as shown below:

[0023] M c =concat(M0,M1.....M n )

[0024] Among them, concat() represents the feature fusion operation, M i ,i=1,2,...,n represents the features output by each multi-scale attention residual module.

[0025] As a further preferred embodiment, step 3) is specifically as follows:

[0026] 3.1) The purpose of adding a fully connected layer between the feature extraction subnet and the residual equivariant mapping module is to make the feature extraction subnet more suitable for the subsequent face transformation task.

[0027] As a further preferred embodiment, the residual equivariant mapping module described in step 4) is specifically:

[0028] 4.1) First, the input feature M of the residual equivariant mapping module d The feature M is obtained through two fully connected layers g ;

[0029] 4.2) The yaw coefficient Y is obtained through the angle estimation module. First, 21 key points of the input low-resolution side face image are extracted, and then the EPnP algorithm is used to estimate the initial solution to fit the model to obtain the deflection angle. Finally, the sigmoid function is used to map the angle to [0,1] to obtain the yaw coefficient Y. The yaw coefficient Y and the feature M are combined. g Multiply them and add the multiplication result to M d Add, and we get the transformed feature M f .

[0030] As a further preferred embodiment, the reconstruction module in step 5) is specifically represented as follows:

[0031] 5.1) The reconstruction module consists of two convolutional layers and one sub-pixel convolutional layer;

[0032] First, a convolution layer with a convolution kernel of 3 is used to process the feature M f Rearrange and get feature M k :

[0033] M k =Conv 3*3 (M f )

[0034] Among them, Conv 3*3 () indicates a convolution operation with a convolution kernel of 3;

[0035] 5.2) Then the rearranged features M k Input into a sub-pixel convolution layer, enlarge it to the corresponding scale, and obtain the reconstructed image I:

[0036] I=H px (M k )

[0037] Among them, H px () represents sub-pixel convolution operation;

[0038] 5.3) Finally, a convolutional layer is used to obtain the final reconstructed high-resolution frontal face image ISR .

[0039] The present invention provides a face super-resolution method based on multi-scale attention residual and equivariant mapping. First, shallow features of low-resolution side face images are extracted through convolutional layers; then, the shallow features are input into a feature extraction subnet, and deep features are obtained through multiple multi-scale attention residual modules; the deep features obtained are further input into a residual equivariant mapping module, and the deep features are combined with the residual features of the fused yaw coefficient in the deep representation feature space, and the feature vector of the side face is transformed into the same feature vector space as the front face; finally, a high-resolution front face image is obtained through a reconstruction module. The present invention solves the problem of being unable to fully utilize low-resolution image features and the problem of poor front face image effects due to insufficient side face samples. The present invention is suitable for face recognition, and no longer overly relies on a large number of front face and side face data pairs. For the input low-resolution side face image, a front face image with high definition and more realistic texture can be reconstructed.

[0040] Beneficial Effects

[0041] Firstly, the present invention adopts a set of multi-scale attention residual modules to extract the features of low-resolution side face images, which can make full use of the detail information of face images; secondly, the residual equivariant mapping module is used to transform the feature vector of the side face into the same feature vector space as that of the front face at the feature level; finally, a high-definition frontal face image is generated based on the reconstruction module. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 Flowchart of the face super-resolution method based on multi-scale attention residual and equivariant mapping of the present invention;

[0043] Figure 2 The network structure diagram of the face super-resolution method based on multi-scale attention residual and equivariant mapping in the present invention;

[0044] Figure 3 The structural diagram of the multi-scale attention residual module designed by the present invention;

[0045] Figure 4 The structural diagram of the attention module designed by the present invention; DETAILED DESCRIPTION

[0046] The present invention provides a face super-resolution method based on multi-scale attention residual and equivariant mapping, firstly extracting shallow features of low-resolution side face images through convolutional layers; then, inputting shallow features into feature extraction subnet, passing through multiple multi-scale attention residual modules, obtaining deep features; further inputting the obtained deep features into residual equivariant mapping module, combining deep features with residual features of fused yaw coefficients in deep representation feature space, transforming feature vectors of side faces into feature vector space same as that of front faces; finally, passing through reconstruction module, obtaining high-resolution frontal face images. The present invention is applicable to face recognition, no longer overly relying on a large number of frontal and side face data pairs, and can reconstruct frontal face images with high definition and more realistic texture for input low-resolution side face images.

[0047] The present invention is described in more detail below with reference to specific examples and drawings.

[0048] like Figure 1 As shown, the present invention comprises the following steps:

[0049] 1) Take the low-resolution face profile image I LR As the input of the network, I LR Perform convolution operation to extract shallow features M0;

[0050] M0=Conv(I LR )

[0051] Among them, Conv() represents the convolution operation.

[0052] 2) The shallow feature M0 is used as the input of the feature extraction subnet composed of multiple multi-scale attention residual modules, and then the features output by each multi-scale attention residual module in the feature extraction subnet are fused through the feature fusion layer to obtain the deep feature M c ;

[0053] 2.1) The feature extraction subnet consists of n multi-scale attention residual modules, each of which consists of two parts: a multi-scale residual module and an attention module.

[0054] For the i-th multi-scale attention residual module, its input feature is the output M of the previous multi-scale attention residual module i-1 , the output feature is M i ;

[0055] The multi-scale residual module described in 2.1.1) is composed of two 3*3 convolutions, two 5*5 convolutions and one 1*1 convolution. The input features are respectively passed through the 3*3 convolution kernel and the 5*5 convolution kernel to generate features F1 and P1. The output results of these two convolution kernels are concatted together as the input of the next 3*3 convolution kernel and the 5*5 convolution kernel, and the generated features are F2 and P2. Then F2 and P2 are concatted together as the input of the 1*1 convolution kernel, and the generated feature is S1.

[0056] 2.1.2) The attention module mainly uses the second-order feature statistics to adaptively rescale the channel-level features for more discriminative representation. First, before performing the global average pooling operation, the normalized covariance matrix of the input features is calculated. The specific operation is as follows:

[0057] First, the covariance matrix of the input features is calculated, then the covariance matrix is ​​decomposed into eigenvalues, and finally the covariance is normalized and converted into the 1 / 2 power of the eigenvalue. In this way, the normalized covariance matrix a is obtained, which represents the correlation between different channel features.

[0058] Then, each layer of the normalized covariance matrix is ​​subjected to global average pooling to obtain the channel descriptor of each layer, which is specifically expressed as follows:

[0059] Z=H GCP (a)

[0060] Among them, H GCP () indicates the global average pooling operation. After obtaining the channel descriptor of each layer, the final weight W is obtained through the gating function, which is specifically expressed as follows:

[0061] W=f(W U ·r(W D ·Z))

[0062] Among them, W U , W D is the weight of the convolutional layer, f and r represent the Sigmoid function and Relu function respectively. Finally, the input feature S1 is multiplied by the weight W to obtain the final weighted feature S2 = W*S1.

[0063] 2.1.3) Finally, the output feature S2 of the attention module and the input M of the i-th multi-scale attention residual module i-1 Add together to get the final output feature M i =M i-1 +S2.

[0064] 2.2) Finally, the outputs of all multi-scale attention residual modules are fused, as shown below:

[0065] M c =concat(M0,M1.....M n )

[0066] Among them, concat() represents the feature fusion operation, M i ,i=1,2......n represents the features output by n multi-scale attention residual modules.

[0067] 3) The fused deep features M c Through a fully connected layer, we get feature M d , specifically expressed as follows:

[0068] M d = linear(M c )

[0069] Among them, represents the linear() fully connected layer operation;

[0070] 4) The feature M d Input to the residual equivariant mapping module, the feature M obtained after two fully connected layers g Then, the angle estimation module extracts 21 key points of the input low-resolution side face image to obtain the yaw coefficient Y of the face angle. The yaw coefficient Y and M g Multiply and finally add the input feature M d , and obtain the final transformed feature M f :

[0071] M f =Y*M g +M d

[0072] 4.1) The input feature M of the residual equivariant mapping module d The feature M is obtained through two fully connected layers g ;

[0073] 4.2) The yaw coefficient Y is obtained through the angle estimation module. First, 21 key points of the low-resolution face profile image are extracted, and then the EPnP algorithm is used to estimate the initial solution to fit the model to obtain the deflection angle. Finally, the sigmoid function is used to map the angle to [0,1] to obtain the yaw coefficient Y.

[0074] 4.3) The yaw coefficient Y and the characteristic M g Multiply them, and then add the multiplication result to M d Add, and we get the transformed eigenvector M f :

[0075] M f =Y*M g +M d

[0076] 5) The transformed feature vector M f Input the reconstruction module to obtain the final reconstructed high-resolution frontal face image I SR .

[0077] 5.1) First, use a convolution layer with a convolution kernel of 3 to process the processed feature M f Rearrange and get feature M k :

[0078] M k =Conv 3*3 (M f )

[0079] Among them, Conv 3*3 () indicates a convolution operation with a convolution kernel of 3;

[0080] 5.2) Then the rearranged features M k Input into a sub-pixel convolution layer, enlarge it to the corresponding scale, and obtain the reconstructed image I:

[0081] I=H px (M k )

[0082] Among them, H px () represents sub-pixel convolution operation;

[0083] 5.3) Finally, a convolutional layer is used to obtain the final reconstructed high-resolution frontal face image I SR .

[0084] The present invention has a wide range of applications in the field of computer vision, such as face recognition and comparison, face door opening, face retrieval, screening of criminal suspects, and full database search of missing persons, etc. The present invention is described in detail below with reference to the accompanying drawings.

[0085] 1) Take the low-resolution face profile image I LR As the input of the network, I LR Perform convolution operation to extract shallow features M0;

[0086] 2) The shallow feature M0 is used as the input of the feature extraction subnet composed of multiple multi-scale attention residual modules, and then the features output by each multi-scale attention residual module in the feature extraction subnet are fused through the feature fusion layer to obtain the deep feature M c ;

[0087] 3) The fused deep features M c Through a fully connected layer, we get feature M d ;

[0088] 4) The feature M d Input to the residual equivariant mapping module, the feature M obtained after two fully connected layers g Then, the angle estimation module extracts 21 key points of the input low-resolution side face image to obtain the yaw coefficient Y of the face angle. The yaw coefficient Y and M g Multiply and finally add the input feature M d , and obtain the final transformed feature M f ;

[0089] 5) The transformed feature vector M f Input the reconstruction module to obtain the final reconstructed high-resolution frontal face image I SR .

[0090] The method was implemented using the Tensorflow deep learning framework and Pycharm programming tool under NVIDIA GeForce GTX1080 Ti and Ubuntu16.04 64-bit operating system.

[0091] The present invention provides a face super-resolution method based on multi-scale attention residual and equivariant mapping. The present invention is applicable to face recognition and no longer overly relies on a large number of frontal face and side face data pairs. For the input low-resolution side face image, a frontal face image with high definition and more realistic texture can be reconstructed.

Claims

1. A face super-resolution method based on multi-scale attention residual and equivariant mapping, characterized in that: The following steps are involved: Step 1) Take the low-resolution face profile image I LR As the input of the network, I LR Perform convolution operation to extract shallow features M0; Step 2) Take the shallow feature M0 as the input of the feature extraction subnet composed of n multi-scale attention residual modules, and then fuse the features output by each multi-scale attention residual module in the feature extraction subnet to obtain the deep feature M c ; Step 3) The fused feature M c Through a fully connected layer, we get feature M d ; Step 4) The feature M d Input to the residual equivariant mapping module, the feature M obtained after two fully connected layers g ; The angle estimation module extracts 21 key points of the input low-resolution side face image, obtains the yaw coefficient Y of the face angle, and converts the yaw coefficient Y and M g Multiply and finally add the input feature M d , and obtain the final transformed feature M f : M f =Y*M g +M d Step 5) Transform the feature vector M f Input the reconstruction module to obtain the final reconstructed high-resolution frontal face image I SR ; The feature extraction subnetwork described in step 2 is composed of n multi-scale attention residual modules, where the input of the i-th multi-scale attention residual module is the output M of the previous multi-scale attention residual module. i-1 , the output feature is M i ; Each multi-scale attention residual module consists of two parts: a multi-scale residual module and an attention module; 2.1) The multi-scale residual module is composed of two 3*3 convolutions, two 5*5 convolutions and one 1*1 convolution. The input features are respectively passed through the 3*3 convolution kernel and the 5*5 convolution kernel. The output results of these two convolution kernels are concatenated as the input of the next 3*3 convolution kernel and the 5*5 convolution kernel, and then their output results are concatenated as the input of the 1*1 convolution kernel. In order to realize the residual operation, a 1*1 convolution kernel is added at the end to compress the number of channels of the feature map. 2.2) The attention module described in the embodiment mainly uses the second-order feature statistics to adaptively scale the channel-level features by making a more discriminative representation; first, the covariance matrix of the input features is calculated, and then the covariance matrix is ​​decomposed by eigenvalues, and finally the covariance is normalized and converted to the 1 / 2 power of the eigenvalue; in this way, the normalized covariance matrix a is obtained, and this normalized covariance matrix represents the correlation of different channel features; finally, each layer of the normalized covariance matrix is ​​subjected to a global average pooling operation to obtain the channel descriptor of each layer, which is specifically expressed as follows: Z=H GCP (a) Among them, H GCP () indicates the global average pooling operation; After obtaining the channel descriptor of each layer, the final weight W is obtained through the gating function, which is specifically expressed as follows: W=f(W U ·r(W D ·Z)) Among them, W U , W D is the weight of the convolutional layer, f and r represent the Sigmoid function and Relu function respectively; Finally, the input feature S1 is multiplied by the weight W to obtain the final weighted feature S2 = W*S1; 2.3) Finally, the output of the attention module and the input M of the i-th multi-scale attention residual module i-1 Add together to get the final output feature M i .

2. The face super-resolution method based on multi-scale attention residual and equivariant mapping according to claim 1, characterized in that: Feature M g The fusion is specifically expressed as follows: M g =concat(M0,M1.....M n ) Among them, concat() represents the feature fusion operation, M i ,i=1,2,...,n represents the features output by n multi-scale attention residual modules.

3. The face super-resolution method based on multi-scale attention residual and equivariant mapping according to claim 1, characterized in that: The purpose of adding a fully connected layer between the feature extraction subnet and the residual equivariant mapping module in step 3 is to make the features extracted by the feature extraction subnet more suitable for the subsequent face angle conversion task.

4. The face super-resolution method based on multi-scale attention residual and equivariant mapping according to claim 1, characterized in that: The residual equivariant mapping module described in step 4 is specifically: 4.1) First, the input feature M of the residual equivariant mapping module d The feature M is obtained through two fully connected layers g ; 4.2) The yaw coefficient Y is obtained through the angle estimation module. First, 21 key points of the input low-resolution side face image are extracted, and then the EPnP algorithm is used to estimate the initial solution to fit the model to obtain the deflection angle. Finally, the sigmoid function is used to map the angle to [0,1] to obtain the yaw coefficient Y. The yaw coefficient Y and the feature M are combined. g Multiply them and add the multiplication result to M d Add, and we get the transformed feature M f .

5. The face super-resolution method based on multi-scale attention residual and equivariant mapping according to claim 1, characterized in that: The reconstruction module described in step 5 is specifically represented as follows: The reconstruction module consists of two convolutional layers and one sub-pixel convolutional layer; 5.1) First, use a convolution layer with a convolution kernel of 3 to process the processed feature M f Rearrange and get feature M c : M c =Conv3*3(M f ) Among them, Conv 3*3 () indicates a convolution operation with a convolution kernel of 3; 5.2) Then the rearranged features M f Input into a sub-pixel convolution layer, enlarge it to the corresponding scale, and obtain the reconstructed image I: I=H px (M c ) Among them, H px () represents sub-pixel convolution operation; 5.3) Finally, after a convolutional layer, the final reconstructed high-resolution frontal face image I is obtained. SR .

Images