Three-dimensional video face recognition method
By converting the three -dimensional face into a multi -angle two -dimensional face projection image, and using the fusion time of the mask self -supervision and learning method and the space vitamin face information, the accurate face recognition of the lack of multi -angle and dimension integration in the existing technology is solved. The problem achieves higher face recognition accuracy.
Patent Information
- Application Number
- CN202411584585.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-07
- Publication Date
- 2025-01-28
AI Technical Summary
The prior art lacks an accurate face recognition method that can achieve multi-angle and combines time-dimensional and space-dimensional facial information in actual video recognition scenarios.
By converting the three -dimensional face into a multi -angle two -dimensional face projection image, and using the mask self -supervision and learning method to combine the loss function, the convergence time support space vitamin information is used to achieve face recognition.
It improves the accuracy in the actual video face recognition scene, can effectively identify multi -angle faces, and accurately integrate human face information in time and space dimension.
Smart Images
Figure CN119380391A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of face recognition, and in particular to a method for three-dimensional video face recognition. Background Art
[0002] In recent years, with the rapid development of deep neural network learning technology, face recognition technology has made great progress. This is due to the continuous upgrading of advanced neural network architecture and the unremitting efforts of scientific researchers in feature extraction theory.
[0003] The existing face recognition technology generally includes four steps, namely: 1. Obtain relevant face images and related data; 2. Face image preprocessing; 3. Face feature extraction; 4. Face comparison and recognition. At present, the common face recognition methods are mainly based on static images, with relatively mature technology and high recognition accuracy, but the shortcomings in practical applications are also obvious. The performance of face recognition algorithms based on static images is very dependent on the quality of the input image. Fast and accurate recognition requires that the input image is clear, the lighting is uniform, the posture of the person cannot be greatly deflected, and the key parts of the face cannot be blocked. However, in practical applications, especially in the most common video scenes, the objects or targets to be recognized are usually imperceptible, and the time and space of their appearance in the video are somewhat random, so it is difficult to ensure that the captured images fully meet the technical requirements of face recognition based on static images. In addition, the commonly used facial feature extraction methods, such as principal component analysis (PCA algorithm) and local binary pattern algorithm (LBP algorithm), also have great limitations. They are usually effective for feature extraction of frontal faces, but fail for feature extraction of non-frontal faces. The effectiveness of the facial feature extraction model directly affects the classification speed and recognition performance. Therefore, more effective solutions are needed to achieve accurate recognition of non-frontal faces from multiple angles.
[0004] The current problem is that in actual video recognition scenarios, there is a lack of an accurate face recognition method that can achieve multi-angle and simultaneously combine temporal and spatial facial information. Summary of the invention
[0005] The purpose of the present invention is to overcome the lack of accurate face recognition from multiple angles and combining temporal and spatial dimensions in the prior art, and to provide a three-dimensional method for recognizing faces in videos.
[0006] In a first aspect, the present invention provides a three-dimensional method for recognizing a face in a video, characterized in that it comprises the following steps:
[0007] S1. Preprocess the real face and video face in the form of point cloud + texture map to obtain two-dimensional face projection images at multiple angles as intermediate information, and extract the features of the multi-angle face projection images;
[0008] S2. Using a mask self-supervised learning method in combination with a loss function, based on the multi-angle face projection image features, the temporal dimension face information and spatial dimension face information of the real face information and the video face information are integrated to achieve face recognition.
[0009] Preferably, the preprocessing in step S1 specifically includes converting the multi-angle three-dimensional face into the multi-angle two-dimensional face projection image through point cloud data, and performing facial feature point monitoring and facial posture estimation on the two-dimensional face projection image.
[0010] Further preferably, the three-dimensional face is converted into point cloud data by representing the three-dimensional face geometric features through point cloud, and on this basis, the texture features are fitted on the surface patch, and projected at preset angle intervals to generate a set of two-dimensional face projection images, that is:
[0011] 3D→2D:P=f {Φ,Ω} (V)
[0012] Where: P is the combination of the two-dimensional face projection images, {Φ, Ω} is the combination angle of the two-dimensional face projection images, V is the three-dimensional face model, and a set of deflection angles is used to determine the posture of the three-dimensional face model V in space, which is composed of the roll angle θ, the yaw angle ω and the pitch angle composition;
[0013] Further preferably, the roll angle can be fixed to θ=0°, the yaw angle range is: ω∈[-90°,90°], and the sampling interval is ω step ; The pitch angle range is: The sampling interval is
[0014] Preferably, the facial feature point monitoring and facial posture estimation are specifically performed by aligning the multi-angle two-dimensional facial projection image to a unified coordinate system and cropping it to a unified size, and inputting the cropped multi-angle two-dimensional facial projection image into a pre-trained two-dimensional portrait feature extraction teacher model to obtain the two-dimensional feature X and posture feature p corresponding to the multi-angle two-dimensional facial projection image.
[0015] Preferably, the step S2 is to construct a neural network using the mask self-supervised learning method as a spatial dimension face information fusion module and a temporal dimension face information fusion module to uniformly identify the three-dimensional face information and the video face information, specifically including two steps of pre-training based on three-dimensional data and fine-tuning the model in combination with the three-dimensional data and the video data. The specific steps are as follows:
[0016] S21, performing the pre-training based on the three-dimensional data to realize that the mask self-supervised feature extraction network can, under the condition of inputting different numbers and different angles of two-dimensional portrait features, have substantially consistent feature representations for the same face and substantially different feature representations for different faces, the specific steps are as follows:
[0017] Step A, prepare a data set {V} consisting of a large amount of high-precision three-dimensional face data to form the three-dimensional face model V, and perform preprocessing to obtain a spatial dimension associated face set {P {Φ,Ω}}, that is, for each of the three-dimensional face models V, there is a set of projected two-dimensional face images {P} generated by the combination of {Φ, Ω} angles;
[0018] Step B: Through random sampling, randomly select a set of projection images of 5% to 10% of the 3D face angles each time Extracting the two-dimensional feature X and the posture feature p of each projected two-dimensional face image P;
[0019] Step C: Input the two-dimensional feature X and the posture feature p into the mask self-supervised feature extraction network to extract a unified identity feature representation U. For the unified identity feature representation, a loss function is constructed using spatiotemporal consistency supervision to ensure that the features are consistent under different partial angle information.
[0020] S22, fine-tuning the model by combining the three-dimensional data and the video data, so that the time dimension face information fusion module and the space dimension face information fusion module can respectively fuse the space dimension face information and the time dimension face information, thereby effectively extracting the face information and achieving a better recognition effect, the specific steps are as follows:
[0021] Step A, preprocessing the three-dimensional face model V and the video face data S to obtain a spatial dimension associated face set {P {Φ,Ω}} and the time dimension associated face set {P S};
[0022] Step B: Through random sampling, randomly select a set of projection images of 5% to 10% of the 3D face angles each time And some facial images that appear in the video
[0023] Step C, input the pre-trained mask self-supervised feature extraction network based on three-dimensional data, extract the unified identity feature representation U, and construct a loss function for the unified identity feature representation using the spatiotemporal consistency supervision to achieve a closer unified feature representation of the same person appearing in the three-dimensional face model and the video face data than the unified feature representation of different people.
[0024] Further preferably, the pre-training based on three-dimensional data specifically adopts a knowledge distillation method, and specifically uses the following loss function:
[0025]
[0026] Where: N represents the data size in each data batch, D represents the dimension of the two-dimensional portrait feature X, represents the jth dimension of the portrait feature extracted from the i-th training two-dimensional face projection image under the processing of the teacher model, represents the previous layer feature map of the portrait features extracted from the i-th training two-dimensional face projection image under the processing of the teacher model, p i represents the posture feature of the i-th training two-dimensional face projection image, cat(,) represents the feature splicing operation, F(Z) j represents a mask feature extraction module, which inputs the feature map obtained by the feature splicing operation and the posture feature of the two-dimensional face projection image, and outputs the unified identity feature representation U with a dimension of D, wherein the subscript j represents the j-th dimension of the identity feature representation U.
[0027] Further preferably, a metric learning method is specifically adopted in the fine tuning of the model by combining the three-dimensional data and the video data, and the following loss function is specifically used:
[0028]
[0029] Where: N represents the data size in each data batch, m is the distance between the regularized positive sample pairs and the negative sample pairs, represents the portrait feature map extracted from the two-dimensional face projection image of the i-th group of training under the processing of the feature extraction model, where k = a, n, p represent anchor samples, negative samples and positive samples respectively, and p ik represents the posture features of the i-th group of training the two-dimensional face projection image, cat(,) represents the feature splicing operation, F(Z) represents the mask feature extraction module, inputs the feature map obtained by the feature splicing operation and the posture features of the two-dimensional face projection image, and outputs a unified identity feature representation U, cos(A,B) represents the cosine similarity of the feature vectors A and B.
[0030] Compared with the prior art, the present invention has the following beneficial effects:
[0031] The present invention provides a three-dimensional method for video face recognition, which obtains a multi-angle two-dimensional face projection image as intermediate information in the form of point cloud + texture map, and extracts the features of the multi-angle two-dimensional face projection image; based on the obtained multi-angle two-dimensional face projection image features, mask self-supervised learning is used as a framework, and the time dimension face information and the space dimension face information are integrated in combination with the loss function to realize face recognition. Compared with the prior art, the present invention is based on the multi-angle two-dimensional face projection image features, combined with mask self-supervised learning and loss function, to realize multi-angle recognition and recognition of fused time dimension and space dimension face information in video face recognition, thereby improving the accuracy in actual video face recognition scenarios, as shown in the experimental results in the embodiments for details. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] Figure 1 The figure is an overall framework diagram of the three-dimensional video face recognition method in an embodiment of the present invention.
[0033] Figure 2 This is a typical experimental data in an embodiment of the present invention.
[0034] Figure 3 Graph showing the experimental results of the three-dimensional video face recognition method in an embodiment of the present invention. DETAILED DESCRIPTION
[0035] The present invention is further described in detail below in conjunction with test examples and specific implementation methods. However, this should not be understood as the scope of the above subject matter of the present invention being limited to the following embodiments, and all technologies realized based on the content of the present invention belong to the scope of the present invention.
[0036] Example 1
[0037] The purpose of the present invention is to overcome the lack of accurate face recognition from multiple angles and combining temporal and spatial dimensions in the prior art, and to provide a three-dimensional method for recognizing faces in videos.
[0038] In this experiment, we use the representative LightCNN29 as the 2D face feature extraction network, the MAE Transformer network to construct the mask feature extraction module, and HopeNet as the posture feature extraction network. In training, we set the batch size to 256 and the initial batch learning rate to 1e -4 , set the weight decay to 0.05, use AdamW as the optimizer, and set the momentum to 0.9.
[0039] The present invention uses real scene data to conduct experiments, which mainly include four parts. The first part is the video surveillance captured image data, including 306 different identity IDs, totaling 9361 face images; the second part is the three-dimensional registration data, including 306 high-precision three-dimensional face data with the same identity ID set as the first part of the data; the third part is the three-dimensional training data, including 5000 high-precision three-dimensional face data with different identity IDs from the first two parts of data, which is only used for model training; the fourth part is the three-dimensional interference registration data, including 3000 high-precision three-dimensional face data with different identity IDs from the first three parts of data. Typical experimental data such as Figure 2 shown.
[0040] A three-dimensional video face recognition method, the overall framework is shown in the figure Figure 1 As shown, the specific steps include:
[0041] S1, preprocessing real faces and video faces in the form of point cloud + texture map;
[0042] S2. Use the mask self-supervised learning method to uniformly identify real face information and video face information.
[0043] The preprocessing in step S1 specifically includes converting the three-dimensional face into a two-dimensional face projection image through point cloud data, and performing facial feature point monitoring and facial posture estimation on the two-dimensional face projection image.
[0044] The three-dimensional face is converted into point cloud data in step S1 specifically by representing the three-dimensional face geometric features through point cloud, and on this basis, the texture features are fitted on the surface patch, and projected at preset angle intervals to generate a set of two-dimensional face projection images, that is:
[0045] 3D→2D:P=f {Φ,Ω} (V)
[0046] Where: P is the combination of the two-dimensional face projection images, {Φ, Ω} is the combination angle of the two-dimensional face projection images, V is the three-dimensional face model, and a set of deflection angles is used to determine the posture of the three-dimensional face model V in space, which is composed of the roll angle θ, the yaw angle ω and the pitch angle Composition; the roll angle can be fixed to θ = 0°, the yaw angle range is: ω∈[-90°,90°], and the sampling interval is ω step ; The pitch angle range is: The sampling interval is
[0047] The facial feature point monitoring and facial posture estimation in step S1 specifically include aligning the two-dimensional facial projection image to a unified coordinate system and cropping it to a unified size, inputting the cropped two-dimensional facial projection image into a pre-trained two-dimensional portrait feature extraction teacher model, and obtaining the two-dimensional feature X and posture feature p corresponding to the two-dimensional facial projection image.
[0048] The step S2 is to construct a neural network using the mask self-supervised learning method as a spatial dimension face information fusion module and a temporal dimension face information fusion module to uniformly identify the three-dimensional face information and the video face information, specifically including two steps of pre-training based on three-dimensional data and fine-tuning the model by combining the three-dimensional data and the video data. The specific steps are as follows:
[0049] S21, performing the pre-training based on the three-dimensional data to realize that the mask self-supervised feature extraction network can, under the condition of inputting different numbers and different angles of two-dimensional portrait features, have substantially consistent feature representations for the same face and substantially different feature representations for different faces, the specific steps are as follows:
[0050] Step A, prepare a data set {V} consisting of a large amount of high-precision three-dimensional face data to form the three-dimensional face model V, and perform preprocessing to obtain a spatial dimension associated face set {P {Φ,Ω}}, that is, for each of the three-dimensional face models V, there is a set of projected two-dimensional face images {P} generated by the combination of {Φ, Ω} angles;
[0051] Step B: Through random sampling, randomly select a set of projection images of 5% to 10% of the 3D face angles each time Extracting the two-dimensional feature X and the posture feature p of each projected two-dimensional face image P;
[0052] Step C: Input the two-dimensional feature X and the posture feature p into the mask self-supervised feature extraction network to extract a unified identity feature representation U. For the unified identity feature representation, a loss function is constructed using spatiotemporal consistency supervision to ensure that the features are consistent under different partial angle information.
[0053] S22, fine-tuning the model by combining the three-dimensional data and the video data, so that the time dimension face information fusion module and the space dimension face information fusion module can respectively fuse the space dimension face information and the time dimension face information, thereby effectively extracting the face information and achieving a better recognition effect, the specific steps are as follows:
[0054] Step A, preprocessing the three-dimensional face model V and the video face data S to obtain a spatial dimension associated face set {P{Φ,Ω}} and the time dimension associated face set {P S};
[0055] Step B: Through random sampling, randomly select a set of projection images of 5% to 10% of the 3D face angles each time And some facial images that appear in the video
[0056] Step C, input the pre-trained mask self-supervised feature extraction network based on three-dimensional data, extract the unified identity feature representation U, and construct a loss function for the unified identity feature representation using the spatiotemporal consistency supervision to achieve a closer unified feature representation of the same person appearing in the three-dimensional face model and the video face data than the unified feature representation of different people.
[0057] The pre-training based on three-dimensional data in step S21 specifically adopts a knowledge distillation method, and specifically uses the following loss function:
[0058]
[0059] Where: N represents the data size in each data batch, D represents the dimension of the two-dimensional portrait feature X, represents the jth dimension of the portrait feature extracted from the i-th training two-dimensional face projection image under the processing of the teacher model, represents the previous layer feature map of the portrait features extracted from the i-th training two-dimensional face projection image under the processing of the teacher model, p i represents the posture feature of the i-th training two-dimensional face projection image, cat(,) represents the feature splicing operation, F(Z) j represents a mask feature extraction module, which inputs the feature map obtained by the feature splicing operation and the posture feature of the two-dimensional face projection image, and outputs the unified identity feature representation U with a dimension of D, wherein the subscript j represents the j-th dimension of the identity feature representation U.
[0060] In the step S22, a metric learning method is specifically used in fine-tuning the model by combining the three-dimensional data and the video data, and the following loss function is specifically used:
[0061]
[0062] Where: N represents the data size in each data batch, m is the distance between the regularized positive sample pairs and the negative sample pairs, represents the portrait feature map extracted from the two-dimensional face projection image of the i-th group of training under the processing of the feature extraction model, where k = a, n, p represent anchor samples, negative samples and positive samples respectively, and p ik represents the posture features of the i-th group of training the two-dimensional face projection image, cat(,) represents the feature splicing operation, F(Z) represents the mask feature extraction module, inputs the feature map obtained by the feature splicing operation and the posture features of the two-dimensional face projection image, and outputs a unified identity feature representation U, cos(A,B) represents the cosine similarity of the feature vectors A and B.
[0063] This embodiment uses the Rank-1 identification accuracy and Top-5 identification accuracy of 1 to N recognition as evaluation indicators. That is to say, it is necessary to evaluate the overall proportion of the matching scores of the corresponding correct IDs that rank first or top five among the 3306 matching scores after comparing the random combination data of each identity ID in the first part of the data (constructed similarly as the video face set data) with the total 3306 registration data in the second and fourth parts.
[0064] This embodiment compares the recognition performance of three methods under four experimental settings. The three methods are: 1. Baseline method, that is, directly using the aforementioned LightCNN29 to extract image features for comparison, finding the closest three-dimensional face projection for each frame in the video face, and calculating the mean of the comparison score; 2. The recognition method described in this embodiment combined with the video single frame method, that is, using the recognition method described in this embodiment to extract unified features from the projection images of the three-dimensional face at each angle, and then extract features from the face of each frame in the video separately, and then use the mean of the comparison score to perform information fusion of the video face; 3. The recognition method described in this embodiment, that is, using the recognition method described in this embodiment to extract unified features from the projection images of the three-dimensional face at each angle, and similarly, multiple frames of the video face are input into the mask self-supervised feature extraction network together to obtain the information fused video face feature representation, and then compare. Four experimental settings refer to comparing the performance of the above methods under four different video frame numbers (each 1 frame as a set, each 3 frames as a set, each 5 frames as a set, and each 10 frames as a set).
[0065] Depend on Figure 3It can be seen that the information fusion unified facial feature representation method designed by the present invention can achieve better recognition results than the baseline method. Specifically, by comparing the "baseline method" and "the present invention + single video frame", it can be seen that the latter is consistently better than the former. The difference between the two schemes lies in the feature representation for single-frame images, which shows that the present invention has better feature extraction capabilities for single-frame images and better information fusion for three-dimensional faces; and by comparing "the present invention + single video frame" and "the present invention", the difference between the two schemes lies in how to fuse the information of multiple frames in the video. It can be seen that there is no fusion problem under the experimental setting of "every 1 frame", and the performance of the two is consistent. In the latter three multi-frame experimental settings, the latter is consistently better than the former, indicating that the present invention can better realize the fusion and unified representation of facial information in the video, thereby achieving better recognition performance.
Claims
1. A three-dimensional method for video face recognition, characterized in that: The following steps are involved: S1. Preprocess the real face and video face in the form of point cloud + texture map to obtain two-dimensional face projection images at multiple angles as intermediate information, and extract the features of multi-angle face projection images; S2. Using a mask self-supervised learning method in combination with a loss function, based on the multi-angle face projection image features, the real face information and the video face information are fused to achieve face recognition, wherein the fused information includes temporal face information and spatial face information.
2. A three-dimensional video face recognition method according to claim 1, characterized in that: The preprocessing in step S1 specifically includes converting the multi-angle three-dimensional face into a multi-angle two-dimensional face projection image through point cloud data, and performing facial feature point monitoring and facial posture estimation on the multi-angle two-dimensional face projection image.
3. A three-dimensional video face recognition method according to claim 2, characterized in that: Specifically, the three-dimensional face is transformed into point cloud data by representing the three-dimensional face geometric features through point cloud, and on this basis, the texture features are fitted on the surface patch, and a group of two-dimensional face projection images are generated by projection at preset angle intervals, that is: 3D→2D:P=f {Φ,Ω} (V) Where: P is the combination of the two-dimensional face projection images, {Φ, Ω} is the combination angle of the two-dimensional face projection images, V is the three-dimensional face model, and a set of deflection angles is used to determine the posture of the three-dimensional face model V in space, which is composed of the roll angle θ, the yaw angle ω and the pitch angle composition.
4. A three-dimensional video face recognition method according to claim 3, characterized in that: The roll angle can be fixed to θ=0°, the yaw angle range is: ω∈[-90°,90°], and the sampling interval is ω step ; The pitch angle range is: The sampling interval is 5. The three-dimensional video face recognition method according to claim 2, characterized in that: The facial feature point monitoring and facial posture estimation specifically include aligning the multi-angle two-dimensional facial projection image to a unified coordinate system and cropping it to a unified size, inputting the cropped multi-angle two-dimensional facial projection image into a pre-trained two-dimensional portrait feature extraction teacher model, and obtaining the two-dimensional feature X and posture feature p corresponding to the multi-angle two-dimensional facial projection image.
6. The three-dimensional video face recognition method according to claim 1, characterized in that: The step S2 is to construct a neural network using the mask self-supervised learning method, use the neural network to fuse the spatial dimension face information and the temporal dimension face information, and perform feature extraction and face recognition on the real face information and the video face information, specifically including two steps: pre-training based on three-dimensional data and fine-tuning the model in combination with three-dimensional data and video data. The specific steps are as follows: S21, performing the pre-training based on the three-dimensional data to achieve that the mask self-supervised feature extraction network can, under the condition of inputting different numbers and different angles of two-dimensional portrait features, have substantially consistent feature representations for the same face and substantially different feature representations for different faces; S22. Fine-tune the model by combining the three-dimensional data and the video data, so that the time dimension face information fusion module and the space dimension face information fusion module can respectively fuse the space dimension face information and the time dimension face information, thereby realizing effective extraction of face information.
7. A three-dimensional video face recognition method according to claim 6, characterized in that: The step S21 is based on pre-training of three-dimensional data, and the specific steps are as follows: Step A, prepare a data set {V} composed of high-precision three-dimensional face data to form the three-dimensional face model V, and perform preprocessing to obtain a spatial dimension associated face set {P {Φ,Ω} }, that is, for each of the three-dimensional face models V, there is a set of projected two-dimensional face images {P} generated by the combination of {Φ, Ω} angles; Step B: Through random sampling, randomly select a set of projection images of 5% to 10% of the 3D face angles each time Extracting the two-dimensional feature X and the posture feature p of each projected two-dimensional face image P; Step C: Input the two-dimensional feature X and the posture feature p into the mask self-supervised feature extraction network to extract a unified identity feature representation U. For the unified identity feature representation, a loss function is constructed using spatiotemporal consistency supervision. Ensure that the features are consistent with different partial angle information.
8. The three-dimensional video face recognition method according to claim 6, characterized in that: The step S22 combines the three-dimensional data and the video data to fine-tune the model, and the specific steps are as follows: Step A, preprocessing the real face model V and the video face data S to obtain a spatial dimension associated face set {P {Φ,Ω} } and the time dimension associated face set {P S }; Step B: Through random sampling, randomly select a set of projection images of 5% to 10% of the 3D face angles each time And some facial images that appear in the video Step C, input the pre-trained mask self-supervised feature extraction network based on three-dimensional data, extract the unified identity feature representation U, and construct a loss function for the unified identity feature representation using the spatiotemporal consistency supervision to achieve a closer unified feature representation of the same person appearing in the real face model and the video face data than the unified feature representation of different people.
9. A three-dimensional method for video face recognition, characterized in that: The invention comprises at least one processor and a memory which is communicatively connected to the at least one processor; the memory stores instructions which can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a three-dimensional video face recognition method according to any one of claims 1 to 8.