Face pose estimation method and device based on large-scale representation information

Abstract

The invention provides a face pose estimation method and device based on large-scale representation information, and belongs to the technical field of computer vision, and the method comprises the steps: extracting a to-be-detected face feature of a to-be-detected image, carrying out the feature similarity calculation of the to-be-detected face feature and each front face feature in a preset front face feature set, and obtaining a face pose estimation result; obtaining a first similarity feature vector; performing increment processing on the first similarity feature vector to obtain a second similarity feature vector; inputting the second similarity feature vector into a pre-trained random forest classifier, and determining a face pose of the to-be-detected image by using the random forest classifier; wherein the random forest classifier is trained according to the second similarity feature vector sample and the sample label corresponding to the second similarity feature vector sample. The invention provides an efficient and accurate human face pose estimation method, and especially under the condition of limited computing resources, relatively high pose estimation accuracy can still be kept.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method and device for estimating human face posture based on large-scale representation information. Background Art

[0002] In the field of computer vision, face detection and recognition is an important research direction, and face pose estimation is a key task in this direction. Face pose estimation usually refers to determining the angle position of the face in the image relative to the camera, for example, judging whether the face is facing the camera or sideways. This technology is of great significance to face recognition systems, human-computer interaction systems, monitoring systems and other fields.

[0003] Traditional face pose estimation methods may rely on geometric features or manually designed features, which have lower accuracy in complex backgrounds or when the face is partially occluded. In recent years, with the development of deep learning, deep learning-based methods have gradually become mainstream, which can automatically learn feature representations that are helpful for pose estimation from large amounts of data.

[0004] However, deep learning models often require a large amount of labeled data to train, and the model complexity is high, which may lead to an increase in the demand for computing resources. When computing resources are limited, how to accurately estimate facial posture remains a difficulty. Summary of the invention

[0005] The present invention provides a face posture estimation method and device based on large-scale representation information, which are used to solve the defects in the prior art and are suitable for various application scenarios, especially when computing resources are limited, and can still maintain a high posture estimation accuracy.

[0006] In a first aspect, the present invention provides a method for estimating facial posture of the image to be tested based on large-scale characterization information, comprising: extracting facial features to be tested of the image to be tested, calculating feature similarity between the facial features to be tested and each frontal facial feature in a pre-set frontal facial feature set to obtain a first similarity feature vector; performing incremental processing on the first similarity feature vector to obtain a second similarity feature vector; inputting the second similarity feature vector into a pre-trained random forest classifier, and determining the facial posture of the image to be tested using the random forest classifier; wherein the random forest classifier is trained based on the second similarity feature vector samples and the sample labels corresponding to the second similarity feature vector samples, and the sample labels are used to characterize whether the facial posture is a frontal facial posture.

[0007] According to the face posture estimation method based on large-scale representation information provided by the present invention, the step of obtaining the frontal face feature set includes: extracting image features of each frontal face image from the frontal face image data set; clustering the image features of all frontal face images to obtain multiple categories of image features; performing feature fusion on each category of image features to obtain corresponding frontal face features to construct a frontal face feature set.

[0008] According to the face posture estimation method based on large-scale representation information provided by the present invention, the first similarity feature vector is incrementally processed to obtain the second similarity feature vector, including: determining the element with the largest value in the first similarity feature vector; adding an element that is the same as the element with the largest value on the first dimension of the first similarity feature vector to construct the second similarity feature vector.

[0009] According to the face posture estimation method based on large-scale representation information provided by the present invention, the random forest classifier includes multiple sub-classifiers, and the face posture of the image to be tested is determined by using the random forest classifier, including: when the classification results of more than half of the sub-classifiers are that the face posture is a frontal face posture, determining that the face posture of the image to be tested is a frontal face posture.

[0010] The method for estimating facial posture based on large-scale representation information provided by the present invention also includes: when the facial posture is determined to be a frontal posture, performing facial recognition using the facial features to be measured.

[0011] According to the face posture estimation method based on large-scale characterization information provided by the present invention, the face features of the image to be tested are extracted, including: preprocessing the image to be tested; inputting the preprocessed image to be tested into the ArcFace face recognition model to extract the face features of the image to be tested.

[0012] In a second aspect, the present invention further provides a face posture estimation device based on large-scale representation information, comprising:

[0013] A similarity feature vector extraction module is used to extract the face feature of the image to be tested, and perform feature similarity calculation between the face feature to be tested and each frontal face feature in a preset frontal face feature set to obtain a first similarity feature vector;

[0014] An incremental processing module, used for performing incremental processing on the first similarity feature vector to obtain a second similarity feature vector;

[0015] A face pose estimation module, used for inputting the second similarity feature vector into a pre-trained random forest classifier, and using the random forest classifier to determine the face pose of the image to be tested;

[0016] The random forest classifier is trained based on the second similarity feature vector samples and the sample labels corresponding to the second similarity feature vector samples, and the sample labels are used to indicate whether the facial posture is a frontal posture.

[0017] In a third aspect, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the method for estimating facial posture based on large-scale representation information as described above are implemented.

[0018] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-described methods for estimating facial posture based on large-scale representation information.

[0019] The face posture estimation method and device based on large-scale representation information provided by the present invention can improve the accuracy of posture estimation by calculating the similarity between the facial features to be measured and the preset frontal facial feature set, and using a random forest classifier for classification; compared with complex deep learning models, the present invention adopts a traditional machine learning algorithm such as random forest, which usually has a low computational cost and can reduce the demand for computing resources while ensuring a certain accuracy.

[0020] The technical solution of the present invention enables the classifier to adapt to different scenarios by training samples of various postures, thereby improving the robustness in practical applications.

[0021] In summary, the present invention provides an efficient and accurate method for estimating facial pose, which is suitable for various application scenarios, especially when computing resources are limited, and can still maintain a high accuracy of pose estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0023] Figure 1 It is a flowchart of a method for estimating face posture based on large-scale representation information provided by the present invention;

[0024] Figure 2 It is a flowchart of the method for obtaining a frontal face feature set provided by the present invention;

[0025] Figure 3It is a structural schematic diagram of a face posture estimation device based on large-scale representation information provided by the present invention;

[0026] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] It should be noted that, in the description of the embodiments of the present invention, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "include one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances.

[0029] The terms "first", "second", etc. in this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more.

[0030] Combine the following Figure 1-Figure 4 The present invention describes a method and device for estimating facial posture based on large-scale representation information provided by an embodiment of the present invention.

[0031] Figure 1 FIG. 1 is a flow chart of a face posture estimation method based on large-scale representation information provided by the present invention. Figure 1 As shown, including but not limited to the following steps:

[0032] Step 101: extracting the facial features of the image to be tested, and calculating the feature similarity between the facial features of the image to be tested and each frontal face feature in a preset frontal face feature set to obtain a first similarity feature vector.

[0033] The present invention can use a pre-trained feature extraction model (such as a convolutional neural network CNN, a deep feature extraction network, etc.) to extract facial features from the image to be tested.

[0034] As an optional embodiment, the present invention extracts the facial features of the image to be tested, including:

[0035] (1) Preprocess the image to be tested.

[0036] Preprocessing operations include but are not limited to image scaling, normalization, noise removal, and cropping.

[0037] In the preprocessing stage, face detection can be performed on the image to be tested, and the facial features area can be cropped according to the detection frame, so that the extracted features can be focused on the facial features.

[0038] (2) The preprocessed image to be tested is input into the ArcFace face recognition model to extract the facial features of the image to be tested.

[0039] The ArcFace algorithm introduces an objective function called "Angular Margin Loss", which enables ArcFace to better distinguish different facial features when facing different postures.

[0040] Therefore, the invention uses Arcface and Resnet100 models to unify the preprocessed images into a size of 112×112 and then extract facial features.

[0041] Optionally, the present invention may use cosine similarity to characterize feature similarity.

[0042] Among them, the first similarity feature vector F c It can be expressed as:

[0043] F c ={C k (F x ,f k )|i=1,2,3…,K}

[0044] F c represents the first similarity feature vector, (F x ,f k ) represents the face feature F to be tested x The similarity with the k-th frontal face feature.

[0045] Step 102: perform incremental processing on the first similarity feature vector to obtain a second similarity feature vector.

[0046] The incremental processing may be to further optimize the similarity feature vector to make it more suitable for classification. Optional methods include standardization, normalization, or some form of transformation of the similarity value to highlight important features or smooth noise.

[0047] As an optional embodiment, the present invention obtains a second similarity feature vector by performing incremental processing on the first similarity feature vector, including:

[0048] (1) determining the element with the largest value in the first similarity feature vector;

[0049] (2) Add an element with the same value as the element with the largest value to the first dimension of the first similarity feature vector to construct the second similarity feature vector F t .

[0050] F t =[F cmax ,F c ];

[0051] Among them, F cmax Indicates F c The element with the largest value in .

[0052] The present invention is through F c Incremental processing is performed to improve F c The weight of the maximum feature similarity in is used to improve the classification accuracy of the subsequent random forest classifier.

[0053] Step 103: input the second similarity feature vector into a pre-trained random forest classifier, and use the random forest classifier to determine the facial posture of the image to be tested.

[0054] The random forest classifier is trained based on the second similarity feature vector samples and the sample labels corresponding to the second similarity feature vector samples, and the sample labels are used to indicate whether the facial posture is a frontal posture.

[0055] The following is a brief description of the training and classification process of the random forest classifier.

[0056] (1) Data preparation:

[0057] First, prepare a training data set, which should include a feature vector and a corresponding label. In this technical solution, the feature vector is a second similarity feature vector, and the label is whether the face posture is a frontal face posture.

[0058] (2) Training data sampling:

[0059] Samples are extracted from the training data set with replacement to generate multiple sub-training sets. The size of each sub-training set can be set according to requirements.

[0060] (3) Decision tree construction:

[0061] Use each sub-training set to train a decision tree (i.e., sub-classifier) ​​independently. In the process of building a decision tree, instead of using all features, a subset of features is selected each time. This subset of features is randomly selected from all available features, which can increase the diversity between individual decision trees.

[0062] (4) Selection of split nodes:

[0063] For each internal node, the best split feature and split point are selected. This best split is usually based on some criteria, such as Gini Impurity or Information Gain.

[0064] (5) Tree growth:

[0065] The decision tree keeps growing until some stopping condition is met, such as reaching the maximum depth, the minimum number of leaf nodes, etc. Sometimes the decision tree is grown without pruning, which can get stronger learning ability, but may also increase the risk of overfitting.

[0066] (6) Integrated prediction:

[0067] Each tree makes a prediction for the test sample. For classification problems, majority voting is used to decide the final classification result.

[0068] (7) Integrated training results:

[0069] When all the trees are trained, the Random Forest classifier is ready to make predictions. For new input data, each tree will give a prediction, and then the final prediction is made based on the majority rule or the average rule.

[0070] Specifically, suppose that for image x, the classification result of the i-th subclassifier is y i (y i 1 indicates a positive face category, y i 0 indicates a non-frontal face category), then the final binary classification result of the random forest classifier can be calculated by the following formula:

[0071] y_final=1, if

[0072] y_final=0, if

[0073] Based on the contents in the above embodiments, as an optional embodiment, the face posture estimation method based on large-scale representation information provided by the present invention further includes: when the face posture is determined to be a frontal face posture, face recognition is performed using the face features to be measured.

[0074] For example, the present invention can be applied to elevator monitoring scenarios in residential communities. When the facial posture is determined to be a positive posture, facial recognition can be performed to help the community achieve intelligent security and ensure the safety of residents' daily lives.

[0075] Figure 2 FIG. 1 is a flow chart of a method for obtaining a frontal face feature set provided by the present invention. Figure 2 As shown, the steps include but are not limited to:

[0076] Step 201: extracting image features of each frontal face image from a frontal face image dataset.

[0077] Optionally, the present invention collects about 600,000 frontal face images of Asians to construct a frontal face image dataset, which summarizes Asian frontal face categories so that any Asian frontal face image can show a high similarity with one of the 600,000 frontal face categories.

[0078] Step 202: clustering the image features of all frontal face images to obtain multiple categories of image features.

[0079] The present invention can utilize k-means clustering to process a frontal face image data set, and through clustering, a plurality of frontal face images with similar features can be classified into the same category.

[0080] Step 203: Perform feature fusion on each type of image features to obtain corresponding frontal face features to construct a frontal face feature set.

[0081] The present invention can average the image features of the same type of frontal face images to determine the frontal face features. The present invention forms the corresponding frontal face features by averaging similar features, which greatly reduces the amount of calculation and improves the efficiency of feature matching when performing feature matching.

[0082] In summary, the present invention provides a method for improving the accuracy of posture estimation by calculating the similarity between the facial features to be tested and the preset frontal facial feature set, and using a random forest classifier for classification; compared with complex deep learning models, the present invention adopts a traditional machine learning algorithm such as a random forest, which usually has a low computational cost and can reduce the demand for computing resources while ensuring a certain accuracy. The technical solution of the present invention trains samples of multiple postures so that the classifier can adapt to different scenarios, thereby improving the robustness in practical applications.

[0083] Figure 3is a schematic diagram of the structure of a face posture estimation device based on large-scale representation information provided by the present invention, such as Figure 3 As shown, the device comprises:

[0084] The similarity feature vector extraction module 310 is used to extract the face feature of the image to be tested, and perform feature similarity calculation between the face feature to be tested and each frontal face feature in the preset frontal face feature set to obtain a first similarity feature vector;

[0085] An incremental processing module 320, configured to perform incremental processing on the first similarity feature vector to obtain a second similarity feature vector;

[0086] A face pose estimation module 330, configured to input the second similarity feature vector into a pre-trained random forest classifier, and determine the face pose of the image to be tested by using the random forest classifier;

[0087] The random forest classifier is trained based on the second similarity feature vector samples and the sample labels corresponding to the second similarity feature vector samples, and the sample labels are used to indicate whether the facial posture is a frontal posture.

[0088] It should be noted that the face pose estimation device based on large-scale representation information provided in the embodiment of the present invention can execute the face pose estimation method based on large-scale representation information described in any of the above embodiments during specific operation, which will not be elaborated in this embodiment.

[0089] Figure 4 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute a face posture estimation method based on large-scale representation information, the method comprising: extracting the face features to be tested of the image to be tested, performing feature similarity calculation on the face features to be tested and each frontal face feature in a pre-set frontal face feature set to obtain a first similarity feature vector; performing incremental processing on the first similarity feature vector to obtain a second similarity feature vector; inputting the second similarity feature vector into a pre-trained random forest classifier, and determining the face posture of the image to be tested using the random forest classifier; wherein the random forest classifier is trained based on the second similarity feature vector sample and the sample label corresponding to the second similarity feature vector sample, and the sample label is used to characterize whether the face posture is a frontal face posture.

[0090] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the face posture estimation method based on large-scale characterization information provided by the above-mentioned embodiments, the method including: extracting the face features to be tested of the image to be tested, performing feature similarity calculation on the face features to be tested and each frontal face feature in a pre-set frontal face feature set to obtain a first similarity feature vector; performing incremental processing on the first similarity feature vector to obtain a second similarity feature vector; inputting the second similarity feature vector into a pre-trained random forest classifier, and using the random forest classifier to determine the face posture of the image to be tested; wherein the random forest classifier is trained based on the second similarity feature vector sample and the sample label corresponding to the second similarity feature vector sample, and the sample label is used to characterize whether the face posture is a frontal face posture.

[0091] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the face posture estimation method based on large-scale characterization information provided in the above-mentioned embodiments, the method comprising: extracting the face features to be tested of the image to be tested, performing feature similarity calculation on the face features to be tested and each frontal face feature in a pre-set frontal face feature set to obtain a first similarity feature vector; performing incremental processing on the first similarity feature vector to obtain a second similarity feature vector; inputting the second similarity feature vector into a pre-trained random forest classifier, and determining the face posture of the image to be tested using the random forest classifier; wherein the random forest classifier is trained based on the second similarity feature vector sample and the sample labels corresponding to the second similarity feature vector samples, and the sample labels are used to characterize whether the face posture is a frontal face posture.

[0092] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0093] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A face pose estimation method based on large-scale representation information, characterized in that: include: Extracting the face feature of the image to be tested, and calculating the feature similarity between the face feature to be tested and each frontal face feature in a preset frontal face feature set to obtain a first similarity feature vector; Performing incremental processing on the first similarity feature vector to obtain a second similarity feature vector; Inputting the second similarity feature vector into a pre-trained random forest classifier, and using the random forest classifier to determine the face posture of the image to be tested; The random forest classifier is trained based on the second similarity feature vector samples and the sample labels corresponding to the second similarity feature vector samples, and the sample labels are used to indicate whether the facial posture is a frontal posture.

2. The method for estimating facial posture based on large-scale representation information according to claim 1, characterized in that: The step of acquiring the frontal face feature set comprises: Extracting image features of each frontal face image from the frontal face image dataset; Cluster the image features of all frontal face images to obtain multi-class image features; Perform feature fusion on each type of image features to obtain the corresponding frontal face features to construct a frontal face feature set.

3. The method for estimating face pose based on large-scale representation information according to claim 1, characterized in that: Incrementally processing the first similarity feature vector to obtain a second similarity feature vector includes: Determine the element with the largest value in the first similarity feature vector; An element that is the same as the element with the largest value is added to the first dimension of the first similarity feature vector to construct a second similarity feature vector.

4. The method for estimating face pose based on large-scale representation information according to claim 1, characterized in that: The random forest classifier includes a plurality of sub-classifiers, and the random forest classifier is used to determine the face posture of the image to be tested, including: When the classification results of more than half of the sub-classifiers indicate that the face posture is a frontal face posture, it is determined that the face posture of the image to be tested is a frontal face posture.

5. The method for estimating face pose based on large-scale representation information according to claim 1, characterized in that: Also includes: When it is determined that the face posture is a frontal face posture, face recognition is performed using the face features to be tested.

6. The method for estimating face pose based on large-scale representation information according to claim 1, characterized in that: Extract the facial features of the image to be tested, including: Preprocess the image to be tested; The preprocessed image to be tested is input into the ArcFace face recognition model to extract the facial features of the image to be tested.

7. The method for estimating face pose based on large-scale representation information according to claim 1, characterized in that: The feature similarity is characterized by using cosine similarity.

8. A face pose estimation device based on large-scale representation information, characterized in that: include: A similarity feature vector extraction module is used to extract the face feature of the image to be tested, and perform feature similarity calculation between the face feature to be tested and each frontal face feature in a preset frontal face feature set to obtain a first similarity feature vector; An incremental processing module, used for performing incremental processing on the first similarity feature vector to obtain a second similarity feature vector; A face pose estimation module, used for inputting the second similarity feature vector into a pre-trained random forest classifier, and using the random forest classifier to determine the face pose of the image to be tested; The random forest classifier is trained based on the second similarity feature vector samples and the sample labels corresponding to the second similarity feature vector samples, and the sample labels are used to indicate whether the facial posture is a frontal posture.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method for estimating facial posture based on large-scale representation information as described in any one of claims 1 to 7 are implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for estimating facial posture based on large-scale representation information as described in any one of claims 1 to 7 are implemented.

Images