Robustness feature learning method for low-quality deep face image

Abstract

The invention provides a feature learning method for a low-quality depth face image, and the method comprises the following steps: 1, constructing a low-quality depth face comparison image pair; 2, carrying out comparison image pair pre-training; 3, inputting the comparison image pair into an image encoder for feature extraction; 4, a Connuous Normalization Flow module is introduced, and feature denoising is carried out on the obtained object; and 5, designing an image pair-oriented contrast loss function to carry out overall training. And 6, a test stage: inputting a to-be-tested face image into the trained network to obtain identity prediction of the network model on the tested face image. The robust feature learning method provided by the invention can effectively solve the problems that the data reliability is low and the feature extraction is easily influenced by factors such as expressions, postures, shielding and the like in the recognition process of a low-quality deep face image. In addition, the image pair-oriented contrast loss function provided by the invention has the advantages of low complexity and high real-time performance.

Description

Technical Field

[0001] The present application belongs to the technical field of image processing and pattern recognition, and more specifically relates to a robust feature learning method for low-quality depth face images. Background Art

[0002] In the early research on deep face image recognition, researchers mainly relied on high-quality deep face data. It is relatively difficult to obtain high-quality deep face data, which has brought limitations to research. It was not until the emergence of low-cost 3D scanners such as Kinect v2 and RealSense that the quality of low-quality deep face data obtained was relatively improved compared to the low-cost 3D scanner Kinectv1, which promoted the progress of related research. The public release of the Lock3DFace dataset has enabled researchers to focus on using these low-quality face data to improve recognition accuracy.

[0003] In recent years, many studies have used deep learning methods to improve the accuracy of low-quality deep face image recognition and achieved good results. For example, some researchers proposed a lightweight and efficient deep method and designed a data processing process to effectively improve the recognition accuracy. However, the convolutional neural network design in this method does not consider the influence of noise in low-quality deep face data; in addition, some researchers established an RGB-D database containing high-quality and low-quality deep face data and used high-quality data to guide the model training of low-quality data, and achieved good results on this dataset. However, due to the lack of a dataset that covers both high-quality and low-quality data, the generalization ability of the proposed method cannot be verified; some researchers proposed a method of joint soft threshold denoising and video data fusion for low-quality deep face image recognition. This method has a time-consuming process for processing multiple frames of video data, resulting in a decrease in real-time performance. In addition, a joint gradient loss function is proposed in this method, which verifies its effectiveness in improving the recognition accuracy of low-quality deep faces. However, the effect of this method in dealing with non-stationary noise needs to be further optimized; some researchers have proposed a Point CNN, which has achieved a high recognition accuracy in low-quality three-dimensional face image recognition. However, this method has high time cost and poor real-time performance.

[0004] Current methods for improving the recognition accuracy of low-quality deep face images mainly use single data to train convolutional neural networks, which have problems such as high risk of overfitting and sensitivity to adversarial attacks. Summary of the invention

[0005] 1. Technical issues to be resolved

[0006] In response to the shortcomings of the prior art, this application proposes a robust feature learning method for low-quality deep face images, which effectively enhances the ability to extract robust features from a single depth data, thereby improving the accuracy of deep face image recognition in any situation including expression, occlusion, posture, etc. in the face image, and proposes a contrast loss function for the constructed contrast image pairs, which reduces the computational complexity of the loss function and effectively improves the efficiency of model training, thereby improving the real-time performance of the method.

[0007] (II) Technical solution

[0008] To achieve the above objectives, this application provides the following technical solutions:

[0009] 1. A robust feature learning method for low-quality deep face images, characterized in that it includes:

[0010] First, construct low-quality deep face images into contrast image pairs, each of which has a corresponding identity label.

[0011] Second, pre-training is carried out for constructing contrast image pairs. The pre-training goal is: for images of the same identity, maximize the similarity between samples; for images of different identities, minimize the similarity between their respective samples to improve recognition accuracy.

[0012] 3. Input the contrast image pairs and corresponding identity labels into the image encoder for feature extraction.

[0013] 4. During the feature extraction process, the Continuous Normalizing Flow module is introduced to perform feature denoising.

[0014] 5. After feature denoising, a contrastive loss function for image pairs is used by minimizing the KL divergence between the distribution of image pair similarity scores and the distribution of true label matches.

[0015] 6. Input the face image to be tested into the trained network model and load the model parameters saved in the training phase to recognize the face image to be tested.

[0016] 2. The robust feature learning method for low-quality deep face images according to claim 1 is characterized in that: the step of constructing a comparison image pair in step 1 includes: first, defining the neutral reference image in the comparison image pair as an image that can stably extract facial features and better reflect facial features and identity information, and the diversity variation image refers to an image that lacks key parts or is affected by factors such as posture, expression, occlusion, and illumination, resulting in inaccurate representation of some information. Then, the neutral reference image and the diversity variation image of the same identity label are constructed into an image pair.

[0017] 3. According to claim 1, the robust feature learning method for low-quality deep face images is characterized in that: the two image encoders in steps 2 and 3 are composed of two identical five-layer convolutional neural network models, and the network models share parameters. The image encoders extract neutral reference image features and diversity change image features respectively. The image features extracted in step 2 are linearly projected after being combined in pairs to obtain features of the same dimension and L2 normalized. The model predicts the similarity between N possible image-image pairs based on the generated image-image feature combination.

[0018] 4. The robust feature learning method for low-quality deep face images according to claim 1, characterized in that: after the image encoder extracts the image features in step 3, a learnable parameter mean μ i and variance Gaussian distribution Approximate the true posterior distribution q(z|x i ), the posterior distribution is called Gaussian posterior distribution, denoted by p(z|x i ), z~p(z|x i ).z represents the potential feature with noise, x i represents the observed 3D face. To make the training process differentiable and to obtain the Gaussian posterior distribution p(z|x i ) to obtain the maximum possible sample, using the reparameterization technique, the calculation formula is ε∈N(0,I).

[0019] 5. The robust feature learning method for low-quality deep face images according to claim 1, characterized in that: the Continuous Normalizing Flow module in step 4 converts the Gaussian distribution into a flexible distribution representation by parameterizing CNF, such as p(z′|x i ). After obtaining the flexible posterior distribution, the maximum possible sample z′ is selected from it, which represents the clean feature and reduces the negative impact of noisy features on face recognition effect.

[0020] 6. The robust feature learning method for low-quality deep face images according to claim 1, characterized in that: the contrastive image pair loss function (CIPL) in step 5 calculates the contrastive loss between the flexible posterior distributions of the contrastive image pairs. The contrastive loss function embeds the similarity distribution of N image-image pairs into the KL divergence to associate the feature representations of the contrastive image pairs. For N image-image pairs, the global representation f of each neutral reference image is i v Based on this, we construct a set of image-image representation pairs In the formula, Represents the global representation of the diverse variation image, y i,j is the true matching label assigned to each identity during the data preprocessing stage.

[0021] 7. The robust feature learning method for low-quality deep face images according to claim 6, characterized in that: sim(x,y)=x T y / ||x||||y|| represents the cosine similarity of x and y after l2 normalization. The Softmax function is used to calculate the probability of image-image pairs. The calculation formula is as follows:

[0022]

[0023] Where k represents the number of global representations of the diversity variation image, τ is the temperature coefficient, and the matching probability q i,j is the number of image pairs in a mini-batch. i v and The cosine similarity score between i v and The ratio of the sum of the cosine similarity scores between .

[0024] 8. The robust feature learning method for low-quality deep face images according to claim 6 is characterized in that: in the image contrast loss function, the contrast loss calculation formula from the neutral reference image to the diversity change image is as follows:

[0025]

[0026] In the formula, δ represents a small value to avoid the denominator being zero. represents the true matching probability.

[0027] Contrastive loss L from diverse variation images to neutral reference images d2n By exchanging and f i v and The overall contrast loss function of the image-image pair is calculated as: L CIPL =L n2d +L d2n

[0028] 9. The robust feature learning method for low-quality deep face images according to claim 1, characterized in that: in step 5, during model training, the gradient information of the contrast loss function is used to continuously update the model parameters to better fit the training data. When calling the contrast loss function, only three parameters need to be passed in, which are expressed as follows:

[0029] Compare lass =CIPL(w diveraity , w nu , target)

[0030] In the formula, w divarsity and w nu They represent the feature representations of diverse variation image data and neutral reference image data respectively, which are obtained by continuous normalizing flow processing, and target represents the true identity label.

[0031] The present application also provides a method for constructing a comparison image pair, characterized by a standard for dividing a neutral reference image and a diversity variation image, and a manner of constructing an image pair from a neutral reference expression image and a diversity variation image.

[0032] The present application also provides a contrast loss function for constructing image pairs, characterized by minimizing the KL divergence between the image pair similarity score distribution and the true identity label matching distribution.

[0033] (III) Beneficial effects

[0034] Compared with the prior art, this application provides a robust feature learning method for low-quality deep face images, which has the following beneficial effects:

[0035] 1. The present application discloses a robust feature learning method for low-quality deep face images. The provided contrast image pair construction and contrast image pair learning method can improve the recognition accuracy of low-quality deep face images during the recognition process due to low data reliability and feature extraction being easily affected by factors such as expression, posture, and occlusion. It effectively avoids the problems of high overfitting risk and sensitivity to adversarial attacks in previous methods of training convolutional neural networks with single data.

[0036] 2. The present application provides a robust feature learning method for low-quality deep face images, and provides a contrast loss function that can simplify the loss calculation process and improve the training efficiency of the model, thereby improving the real-time performance of the method to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is an overall framework diagram of a robust feature learning method for low-quality depth face images according to a specific implementation method of the present application.

[0038] Figure 2 A schematic diagram of constructing a comparison image pair proposed in this application.

[0039] Figure 3Schematic diagram of pre-training for contrasting image pairs.

[0040] Figure 4 This is an overall diagram of model training.

[0041] Figure 5 Schematic diagram of loss calculation for comparison images. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0043] The present application provides a robust feature learning method for low-quality deep face images. By defining the division criteria of neutral reference images and diverse variation images and a method for constructing contrast image pairs, the previous mode of using single data for model training is changed without increasing the sample size of the data set. Diversified data is used for model training, thereby enhancing the robust feature learning ability from a single deep data. This method ensures fairness compared with other methods to a certain extent, and the improvement in recognition accuracy further reflects the robust feature learning ability of this method.

[0044] In the present application, the construction of contrast image pairs according to the definition refers to the contrast image pairs consisting of neutral reference images and diverse variation images. The present application classifies images that can stably extract facial features and better reflect facial features and identity information as neutral reference images. Such images are essentially neutral expression images, which can usually accurately display the key features of the face; images that lack key parts or that do not accurately express some information due to factors such as posture, expression, occlusion, and lighting are classified as diverse variation images, namely, deep facial images such as expression, posture, occlusion, and time.

[0045] This application is based on the face identity tag. Identity tags are conventional in the field, for example, different categories can be used to mark faces of different identities; the identity tags described in this application are used to guide feature extraction. Identity tags can have multiple attributes, for example, they can contain more and richer soft biological attributes, semantic attributes, etc. More and richer identity tags can improve the accuracy of the method to a certain extent, but they will also increase the computational complexity of the method. Therefore, after comprehensive consideration, this method uses simple identity tags and uses digital numbers to identify different identity information.

[0046] In some embodiments, the step of constructing the low-quality depth face image into a comparison image pair in step 1 includes: extracting a neutral expression image of an identity sample as a neutral reference image, and the remaining images are divided into image subsets such as neutral expression, expression, posture, occlusion, and time, that is, diversity variation images. The neutral reference image is paired with each image of the diversity variation image subset, and a corresponding identity label is assigned to it, and the number of comparison image pairs covered by each identity sample is the same as the number of depth images it contains.

[0047] In this application, the expression image set is composed of deep face images of expressions such as happiness, surprise, sadness, fear, anger, shyness, etc.; the posture image set is composed of images of front, side, head up, head down, etc.; the occlusion image set is composed of images of glasses occlusion, hand occlusion, hair occlusion, mask occlusion, etc.;

[0048] In some embodiments, the pre-training of the contrast image pairs in step 2 represents the neutral reference image features and the diversity variation image features extracted by the image encoder according to the classification of the image subsets as follows: (neutral baseline image features), (expression features), (Neutral expression features), (occlusion feature), (Posture Features), (Time feature). The values ​​of superscripts i, j, k, m, n, and t all start from 0. For the same identity, a neutral reference image feature is combined with N diverse change image features in pairs. Subsequently, these two features are linearly projected to obtain features of the same dimension and L2 normalized.

[0049] In the present application, after the image feature combination is obtained in step 2, the image-image feature combination is used The model predicts the similarity between N possible image-image pairs based on the combination of image features. The similarity prediction is achieved by calculating the cosine similarity between the neutral reference image features and the diversity variation image features. For images of the same identity, the training goal is to maximize the similarity of these N samples; for images of different identities, the training goal is to minimize the similarity between each sample to improve the recognition accuracy.

[0050] In some embodiments, the overall model architecture includes a Continuous Normalizing Flow module for feature denoising. It is a method for probability density estimation and generative models. The core idea is to map a simple prior distribution to a complex posterior distribution through a continuous and differentiable transformation, thereby getting rid of the limitations of a fixed distribution form. The basis of this transformation is a parameterized ordinary differential equation (ODE), and the transformation and modeling of the data distribution are achieved by solving the ODE. CNF defines a continuous mapping h as an ordinary differential equation with an initial value of z(t0)=z0 In this process, the calculation of the logarithmic density log p(z(t1)) at time t1 can be obtained by integration, as follows:

[0051]

[0052] In the formula, Tr represents the trace operator, represents the derivative with respect to h. The output and gradient of CNF can be calculated by a black box solver.

[0053] In some embodiments, the loss function of the contrast image pair is calculated as follows:

[0054] L CIPL =L n2d +L d2n

[0055] L n2d represents the contrast loss from the neutral reference image to the diverse variation image, L d2n represents the contrastive loss from the diverse variation image to the neutral reference image.

[0056] The present application can improve the recognition accuracy of images with diverse changes through the above-mentioned robust feature learning method for low-quality deep face images.

[0057] Figure 1 This is a general framework diagram of a robust feature learning method for low-quality deep face images in a specific implementation of the present application, including the following steps:

[0058] First, data preprocessing is performed. The first frame is extracted from the first neutral expression video of each identity as the neutral reference image. Subsequently, six frames are extracted at the same time interval and data enhancement is performed. The enhancement methods include out-of-plane rotation, posture generation, shape jitter, and shape scaling. All frames in the remaining videos (such as posture, expression, occlusion, etc.) and the enhanced neutral expression frames are all used as diversity variation images. Next, the neutral reference image is paired with the diversity variation image to form a contrast image pair, such as Figure 2shown.

[0059] After the contrast image pairs are constructed, contrast pre-training is performed, which is inspired by the idea of ​​contrastive learning. Using neutral faces as supervisory signals for contrastive learning can semantically align all faces with neutral faces, thereby obtaining more stable and robust facial features. The experimental results of the robust feature learning method for low-quality deep face images proposed in this application verify the effectiveness of the proposed contrast image pair construction and contrastive learning strategy in low-quality deep face image recognition tasks; secondly, it helps to prove the transferability of contrastive learning ideas in such face image recognition tasks.

[0060] In the model training phase, the constructed contrast image pairs are sent to the image encoder to complete feature extraction. After that, the extracted neutral reference image features and diversity change image features are re-parameterized to obtain the Gaussian posterior distribution of the neutral expression image and the diversity change image. The posterior distribution is processed by the Continuous Normalizing Flow module to obtain a relatively clean feature. This process removes some noise features and alleviates the problem of low accuracy of deep face image recognition caused by the distribution of noise features. After feature denoising, w diversity and w nu , which represent the feature representation of the diversity change image data and the neutral reference image data respectively, and target represents the true identity label. These three parameters are fed into the contrast loss function, and the gradient information of the loss function is used to continuously update the model parameters to better fit the training data.

[0061] The contrastive loss function embeds the similarity distribution of N image-image pairs into the KL divergence to associate the feature representations of contrasting image pairs. For a given N image-image pair, the global representation f i v Based on this, we construct a set of image-image representation pairs in Represents the global representation of the diverse variation image, y i，j is the true matching label assigned to each identity during the data preprocessing stage.

[0062] Define sim(x,y)=x T y / ||x||||y|| represents the cosine similarity of x and y after l2 normalization. Then, the following Softmax function is used to calculate the probability of image-image matching pairs:

[0063]

[0064] Where k represents the number of global representations of the diversity change image; τ is the temperature coefficient. By adjusting this coefficient, the scale of the probability distribution in the loss function can be changed, thereby affecting the training effect and convergence of the loss function. i,j It can be understood that in a small batch of image pairs, f i v and The cosine similarity score between i v and Next, for small batches of image pairs, the contrast loss from the neutral reference image (NU) to the diversity image (Diversity) is calculated using the following formula:

[0065]

[0066] In the formula, δ represents a small value to avoid the denominator being zero. represents the true match probability. Correspondingly, the contrast loss L from the diverse variation image to the neutral reference image is d2n By exchanging the Softmax function and L n2d f i v and The calculation formula of the two-way contrast loss is as follows:

[0067] L CIPL =L n2d +L d2n

[0068] L n2d represents the contrast loss from the neutral reference image to the diverse variation image, L d2n represents the contrastive loss from the diverse variation image to the neutral reference image.

[0069] Finally, the face image to be tested is input into the trained network and the model parameters saved during training are loaded to recognize the test face image.

[0070] This application describes a robust feature learning method for low-quality deep face images. In the training phase, firstly, a comparison image pair is constructed and an identity label is assigned; then, the face image and the identity label are input into the deep network model for training until convergence. In the testing phase, given any face image, the trained deep learning model is directly input and the model parameters saved during training are loaded to obtain the predicted identity information.

[0071] Although the embodiments of the present application have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A robust feature learning method for low-quality deep face images, characterized in that: It includes: First, construct low-quality deep face images into contrast image pairs, each of which has a corresponding identity label.

2. Carry out pre-training for constructing contrast image pairs. The pre-training goal is: for images of the same identity, maximize the similarity between samples; for images of different identities, minimize the similarity between their respective samples to improve recognition accuracy.

3. Input the contrast image pairs and corresponding identity labels into the image encoder for feature extraction.

4. During the feature extraction process, the Continuous Normalizing Flow module is introduced to perform feature denoising.

5. After feature denoising, a contrastive loss function for image pairs is used by minimizing the KL divergence between the distribution of image pair similarity scores and the distribution of true label matches.

6. Input the face image to be tested into the trained network model and load the model parameters saved in the training phase to recognize the face image to be tested.

2. The robust feature learning method for low-quality deep face images according to claim 1, characterized in that: The step of constructing a comparison image pair in step 1 includes: first, defining the neutral reference image in the comparison image pair as an image that can stably extract facial features and better reflect facial features and identity information, and the diversity variation image refers to an image that lacks key parts or is affected by factors such as posture, expression, occlusion, and lighting, resulting in inaccurate representation of some information. Then, constructing an image pair with the neutral reference image and the diversity variation image of the same identity label.

3. The robust feature learning method for low-quality deep face images according to claim 1, characterized in that: The two image encoders in steps 2 and 3 are composed of two identical five-layer convolutional neural network models, and parameters are shared between the network models. The image encoders extract neutral reference image features and diversity change image features respectively. The image features extracted in step 2 are linearly projected after being combined in pairs to obtain features of the same dimension and L2 normalized. The model predicts the similarity between N possible image-image pairs based on the generated image-image feature combination.

4. The robust feature learning method for low-quality deep face images according to claim 1, characterized in that: After the image encoder extracts the image features in step 3, the image encoder uses a learnable parameter mean μ i and variance Gaussian distribution Approximate the true posterior distribution q(z|x i ), the posterior distribution is called Gaussian posterior distribution, denoted by p(z|x i ), z~p(z|x i ).z represents the potential feature with noise, x i represents the observed 3D face. To make the training process differentiable and to obtain the Gaussian posterior distribution p(z|x i ) to obtain the maximum possible sample, using the reparameterization technique, the calculation formula is 5. The robust feature learning method for low-quality deep face images according to claim 1, characterized in that: The Continuous Normalizing Flow module in step 4 converts the Gaussian distribution into a flexible distribution representation by parameterizing CNF, such as p(z′|x i ). After obtaining the flexible posterior distribution, the maximum possible sample z′ is selected from it, which represents the clean feature and reduces the negative impact of noisy features on face recognition effect.

6. The robust feature learning method for low-quality deep face images according to claim 1, characterized in that: The contrastive image pair loss function (CIPL) in step 5 calculates the contrastive loss between the flexible posterior distributions of the contrastive image pairs. The contrastive loss function embeds the similarity distribution of N image-image pairs into the KL divergence to associate the feature representations of the contrastive image pairs. For N image-image pairs, the global representation f of each neutral reference image is i v Based on this, we construct a set of image-image representation pairs In the formula, Represents the global representation of the diverse variation image, y i,j is the true matching label assigned to each identity during the data preprocessing stage.

7. The robust feature learning method for low-quality deep face images according to claim 6, characterized in that: Define sim(x,y)=x T y / ||x||||y|| represents the cosine similarity of x and y after i2 normalization. The Softmax function is used to calculate the probability of image-image pairs. The calculation formula is as follows: Where k represents the number of global representations of the diversity variation image, τ is the temperature coefficient, and the matching probability q i,j is the number of image pairs in a mini-batch. i v and The cosine similarity score between i v and The ratio of the sum of the cosine similarity scores between .

8. The robust feature learning method for low-quality deep face images according to claim 6, characterized in that: In the image contrast loss function, the contrast loss calculation formula from the neutral reference image to the diversity change image is as follows: In the formula, δ represents a small value to avoid the denominator being zero. represents the true matching probability. Contrastive loss L from diverse variation images to neutral reference images d2n By exchanging and f i v and The overall contrast loss function of the image-image pair is calculated as: L CIPL =L n2d +L d2n 9. The robust feature learning method for low-quality deep face images according to claim 1, characterized in that: In step 5, during model training, the gradient information of the contrast loss function is used to continuously update the model parameters to better fit the training data. When calling the contrast loss function, only three parameters need to be passed in, which are expressed as follows: Compare loss =CIPL(w diversity ,w nu ,target) In the formula, w diversity and w nu They represent the feature representations of diverse variation image data and neutral reference image data respectively, which are obtained by continuous normalizing flow processing, and target represents the true identity label.

Images