Face mask binding method based on real-time face 3D key point detection

Abstract

The invention discloses a face mask binding method based on real-time face 3D key point detection, and the method comprises the following steps: S1, constructing a deep learning model which is a software or hardware computing system based on an artificial neural network algorithm, and comprises one or more trained or untrained artificial neural networks; s2, acquiring a face image through a camera; s3, extracting feature information of the key points of the face image, wherein the feature information refers to related information which is included in the key points of the face image and is required for three-dimensional coordinate measurement of the key points of the face image; s4, inputting the feature information of the key points of the face image into the deep learning model; s5, the deep learning model calculates three-dimensional coordinates of key points of the face image; and S6, constructing a face grid model according to the three-dimensional coordinates, and generating a finished face mask for real-time face 3D key point detection. According to the invention, the problem that the three-dimensional key point coordinates of the human face are difficult to accurately obtain in real time can be solved.

Description

Technical Field

[0001] The present invention relates to the field of face recognition, and in particular to a face mask binding method based on real-time face 3D key point detection. Background Art

[0002] Face key points refer to 5 key points such as both eyes, nose, and two mouth corners, or 28 key points including more contour points, or even 64 or 128 facial key points. The quality of face key points to a certain extent represents the quality of the face. For example, if the key points are occluded or blurred, then the face cannot be effectively recognized. Selecting face images with high-quality key points for face recognition in the early stage can not only improve the accuracy of face recognition, but also improve the efficiency of face recognition.

[0003] Face detection feature extraction is usually based on methods such as geometric features, deep learning, and PCA. These methods have been very mature in recent years and are used in many fields of life, achieving great achievements.

[0004] For the three-dimensional mask binding of the face, it is necessary to obtain the three-dimensional coordinates of the main key points of the face, which is relatively difficult to handle for traditional methods, especially to obtain the three-dimensional key point coordinates of the face in real time and accurately. Summary of the Invention

[0005] The purpose of the present invention is to propose a face mask binding method based on real-time face 3D key point detection in order to solve the problem of difficult to obtain the three-dimensional key point coordinates of the face in real time and accurately.

[0006] In order to achieve the above purpose, on the one hand, the present invention adopts the following technical solution: A face mask binding method based on real-time face 3D key point detection, comprising the following steps:

[0007] S1: Construct a deep learning model, where the deep learning model is a software or hardware computing system based on an artificial neural network algorithm, including one or more trained or untrained artificial neural networks;

[0008] S2: Obtain a face image through a camera;

[0009] S3: Extract the feature information of the face image key points, where the feature information refers to the relevant information required for measuring the three-dimensional coordinates of the face image key points included in the face image key points;

[0010] S4: Input the feature information of the face image key points into the deep learning model;

[0011] S5: The deep learning model calculates the three-dimensional coordinates of the face image key points;

[0012] S6: Construct a facial mesh model based on three-dimensional coordinates, and use model processing software to adjust the facial mesh model to generate a face mask for real-time face 3D key point detection after trimming.

[0013] As a further description of the above technical solution:

[0014] The face image obtained in step S2 also needs to be preprocessed to obtain a preprocessed face image, and the preprocessed face image is used to extract the feature information of key points.

[0015] As a further description of the above technical solution:

[0016] The preprocessing includes image scaling and image filling.

[0017] As a further description of the above technical solution:

[0018] The steps for obtaining the face image in step S2 include:

[0019] S21: Obtain a two-dimensional image and a depth image of the face;

[0020] S22: Register the depth image with a reference face 3D texture image to obtain the pose information of the face;

[0021] S23: Project the reference face 3D texture image according to the pose information to project a reference face two-dimensional image;

[0022] S24: Compare the similarity between the target face two-dimensional image and the reference face two-dimensional image;

[0023] S25: If the similarity exceeds the preset value, the obtained face image is a real face; otherwise, the obtained face image is not a real face, and a new face image needs to be obtained.

[0024] As a further description of the above technical solution:

[0025] The steps for obtaining the reference face 3D texture image include:

[0026] S221: Obtain a depth image sequence and a two-dimensional image sequence containing the reference face;

[0027] S222: Calculate the reference face 3D texture image.

[0028] As a further description of the above technical solution:

[0029] Step S1 includes:

[0030] S11: Obtain multiple face images;

[0031] S12: Label each face image to mark the pose state of each face image, and obtain a training image dataset;

[0032] S13: Use the training image dataset to train a pre-established neural network model to obtain the artificial neural network.

[0033] As a further description of the above technical solution:

[0034] The steps for extracting key points in step S3 include:

[0035] S31: Input the face image into a preset model set, and calculate the positions and first confidence levels of the face key points;

[0036] S32: When the first confidence level is greater than or equal to a preset threshold, obtain a second target image corresponding to the position of the face key point;

[0037] S33: Input the second target image into the preset model set, calculate a second confidence level, and determine the accuracy of the position of the first face key point according to the second confidence level;

[0038] S34: When the second confidence level is greater than or equal to a preset accuracy, determine the position of the face key point as the final position of the face key point of the face image.

[0039] As a further description of the above technical solution:

[0040] The preset model set is used to represent a set of relationship models between face images and the positions and confidence levels of face key points.

[0041] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are as follows: A deep learning model is constructed. The deep learning model is a software or hardware computing system based on an artificial neural network algorithm, which includes one or more trained or untrained artificial neural networks; a face image is obtained through a camera; the feature information of the face image key points is extracted. The feature information refers to the relevant information required for measuring the three-dimensional coordinates of the face image key points included in the face image key points; the feature information of the face image key points is input into the deep learning model; the deep learning model calculates the three-dimensional coordinates of the face image key points; a facial mesh model is constructed according to the three-dimensional coordinates, and the facial mesh model is adjusted by model processing software to generate a trimmed face mask for real-time face 3D key point detection, so as to achieve high-precision facial key point positioning, quickly detect multiple key points of the face, and generate real-time positioning of the three-dimensional space coordinates of the corresponding positions of the face in real time, ensuring the perfect fit of the face mask to the face. Description of the Drawings

[0042] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for use in the embodiments. It should be understood that the following drawings only show some embodiments of the present invention, and thus should not be regarded as limiting the scope. For those of ordinary skill in the art, without creative efforts, other related drawings can also be obtained based on these drawings.

[0043] Figure 1 It is a flowchart of a face mask binding method based on real-time face 3D key point detection.

[0044] Figure 2 It is a schematic structural diagram of the FKD neural network structure. Specific embodiments

[0045] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0046] Please refer to Figure 1 , the present invention provides a technical solution: a face mask binding method based on real-time face 3D key point detection, including the following steps:

[0047] S1: Construct a deep learning model, which is a computing system of software or hardware based on an artificial neural network algorithm, including one or more trained or untrained artificial neural networks;

[0048] S2: Obtain a face image through a camera;

[0049] S3: Extract the feature information of the face image key points, where the feature information refers to the relevant information required for measuring the three-dimensional coordinates of the face image key points included in the face image key points;

[0050] S4: Input the feature information of the face image key points into the deep learning model;

[0051] S5: The deep learning model calculates the three-dimensional coordinates of the face image key points;

[0052] S6: Construct a facial mesh model according to the three-dimensional coordinates, and use model processing software to adjust the facial mesh model to generate a trimmed face mask for real-time face 3D key point detection.

[0053] The face image obtained in step S2 also needs to be preprocessed to obtain a preprocessed face image, and the preprocessed face image is used to extract the feature information of key points. The preprocessing includes image scaling and image filling. The face image is filled to 1:1 and scaled to 256×256, and data augmentation operations including random image rotation, random image flipping, random color transformation, and random noise enhancement are performed.

[0054] In one implementation, the COC0 training set is used. The optimized image is gradually pooled and reduced using 128*128 grayscale images, with a minimum of 8*8 size tensors, and trained at different scales, combined with a post-processing algorithm. Real-time detection can be achieved on desktop computers and mobile phones, ensuring the real-time performance and accuracy of the detection model.

[0055] The steps for obtaining the face image in step S2 include:

[0056] S21: Obtain the two-dimensional image and depth image of the face;

[0057] S22: Register the depth image with the reference face 3D texture image to obtain the pose information of the face;

[0058] S23: Project the reference face 3D texture image according to the pose information to obtain a reference face two-dimensional image;

[0059] S24: Compare the similarity between the target face two-dimensional image and the reference face two-dimensional image;

[0060] S25: If the similarity exceeds the preset value, the obtained face image is a real face; otherwise, the obtained face image is not a real face, and a new face image needs to be obtained. Combining the live detection based on the depth image and the two-dimensional image can avoid the misrecognition of false faces and improve the acquisition quality of the image.

[0061] The steps for obtaining the reference face 3D texture image include:

[0062] S221: Obtain a depth image sequence and a two-dimensional image sequence containing the reference face;

[0063] S222: Calculate the reference face 3D texture image.

[0064] Step S1 includes:

[0065] S11: Obtain multiple face images;

[0066] S12: Annotate each face image to annotate the pose state of each face image, obtaining a training image data set;

[0067] S13: Train the pre-established Yolov8 neural network model using the training image dataset to obtain the artificial neural network.

[0068] The neural network model is the Yolov8 neural network model. The Yolov8 neural network model has a fast inference speed, high accuracy, and is easy to train and adjust. Selecting the Yolov8 neural network model as the recognition model can perform pose state recognition quickly and accurately.

[0069] In one implementation, the neural network model is the FKD neural network structure, such as Figure 2 .

[0070] The steps for extracting key points in step S3 include:

[0071] S31: Input the face image into the preset model set to calculate the positions and first confidence levels of the face key points;

[0072] S32: When the first confidence level is greater than or equal to the preset threshold, obtain the second target image corresponding to the positions of the face key points;

[0073] S33: Input the second target image into the preset model set to calculate the second confidence level, and determine the accuracy of the positions of the first face key points according to the second confidence level;

[0074] S34: When the second confidence level is greater than or equal to the preset accuracy, determine the positions of the face key points as the final face key point positions of the face image. The preset model set is used to represent the set of relationship models between face images and the positions and confidence levels of face key points. When using the preset model set for face key point localization, the accuracy of the face key point localization can be determined simultaneously. The accuracy of face key point tracking can be judged while reducing the time and space occupied by subsequent re-evaluation of accuracy, thereby improving the processing speed of face key point tracking.

[0075] Working principle: Build a deep learning model, which is a type of computing system of software or hardware based on artificial neural network algorithms, including one or more trained or untrained artificial neural networks; obtain a face image through a camera; extract the feature information of the key points of the face image, where the feature information refers to the relevant information required for measuring the three-dimensional coordinates of the key points of the face image contained in the key points of the face image; input the feature information of the key points of the face image into the deep learning model; the deep learning model calculates the three-dimensional coordinates of the key points of the face image; construct a facial mesh model based on the three-dimensional coordinates, and use model processing software to adjust the facial mesh model to generate a face mask for real-time face 3D key point detection after trimming, so as to achieve highly accurate facial key point positioning, quickly detect multiple key points of the face, and generate real-time positioning of the three-dimensional space coordinates of the corresponding positions of the face in real time, ensuring the complete fitting of the face mask to the face.

[0076] As described above, it is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention, according to the technical solution and inventive concept of the present invention, making equivalent substitutions or changes should be covered within the protection scope of the present invention.

Claims

1. A face mask binding method based on real-time face 3D key point detection, characterized in that: The following steps are involved: S1: constructing a deep learning model, wherein the deep learning model is a type of software or hardware computing system based on an artificial neural network algorithm, which includes one or more trained or untrained artificial neural networks; S2: Acquire a face image through a camera; S3: extracting feature information of key points of the face image, where the feature information refers to the relevant information contained in the key points of the face image required for measuring the three-dimensional coordinates of the key points of the face image; S4: Inputting feature information of key points of the face image into the deep learning model; S5: The deep learning model calculates the three-dimensional coordinates of key points of the face image; S6: construct a facial mesh model according to the three-dimensional coordinates, adjust the facial mesh model using model processing software, and generate a trimmed face mask for real-time face 3D key point detection.

2. A face mask binding method based on real-time face 3D key point detection according to claim 1, characterized in that: The facial image obtained in step S2 needs to be preprocessed to obtain a preprocessed facial image, and then the preprocessed facial image is used to extract feature information of key points.

3. The face mask binding method based on real-time face 3D key point detection according to claim 2 is characterized in that: The preprocessing includes image scaling and image filling.

4. The face mask binding method based on real-time face 3D key point detection according to claim 1, characterized in that: The steps of acquiring the face image in step S2 include: S21: Acquire a two-dimensional image and a depth image of the face; S22: registering the depth image with the reference face 3D texture image to obtain face posture information; S23: Projecting the reference face 3D texture image into a reference face 2D image according to the posture information; S24: performing a similarity comparison between the target face two-dimensional image and the reference face two-dimensional image; S25: If the similarity exceeds a preset value, the acquired face image is a real face; otherwise, the acquired face image is not a real face and a new face image needs to be acquired.

5. The face mask binding method based on real-time face 3D key point detection according to claim 4, characterized in that: The reference face 3D texture image acquisition step comprises: S221: Acquire a depth image sequence and a two-dimensional image sequence including a reference face; S222: Calculate the reference face 3D texture image.

6. The face mask binding method based on real-time face 3D key point detection according to claim 1, characterized in that: Step S1 includes: S11: Acquire multiple face images; S12: labeling each face image to label the posture state of each face image, and obtaining a training image data set; S13: Using the training image data set to train the pre-established neural network model to obtain the artificial neural network.

7. The face mask binding method based on real-time face 3D key point detection according to claim 1, characterized in that: The key point extraction step in step S3 includes: S31: inputting the face image into a preset model set, and calculating the position and first confidence of the face key points; S32: when the first confidence level is greater than or equal to a preset threshold, obtaining a second target image corresponding to the position of the facial key point; S33: inputting the second target image into the preset model set, calculating a second confidence level, and determining the accuracy of the position of the first facial key point according to the second confidence level; S34: When the second confidence level is greater than or equal to a preset accuracy, determining the position of the facial key point as the final position of the facial key point of the facial image.

8. The face mask binding method based on real-time face 3D key point detection according to claim 7, characterized in that: The preset model set is a set of relationship models for characterizing the positions and confidence levels of facial images and facial key points.

Images