A method, device and storage medium for three-dimensional facial key point detection

Abstract

This invention discloses a method, apparatus, and storage medium for 3D facial keypoint detection. The 3D facial keypoint detection method includes: acquiring original input data to characterize 3D face shape and texture data; uniformly sampling the original input data using a point cloud mesh to generate first sampled data; filling the first sampled data to generate first filled data; using a graph convolutional network to output a probability distribution graph model of 3D facial keypoints; and predicting the positions of 3D facial keypoints based on the probability distribution graph model. This invention proposes a transformation from the 3D facial keypoint probability graph model to keypoint positions, which can be adapted to the training of graph convolutional networks, improving the robustness and accuracy of the training results. Furthermore, this invention proposes a standardized representation method for 3D face geometry and texture data, which allows the training network to adapt to different types of data, thus possessing cross-dataset generalization ability.

Description

Technical Field

[0001] This invention relates to the field of face detection, and in particular to a method, apparatus and storage medium for three-dimensional face key point detection. Background Technology

[0002] Existing methods for 3D facial landmark detection are mainly divided into traditional methods and deep learning methods.

[0003] Traditional methods primarily utilize conventional machine learning techniques to detect key points on 3D faces, often employing traditional geometric features, 3D face prior models, or dense registration of 3D faces for key point localization. For example, Segundo et al. proposed a 3D face key point detection algorithm that combines 3D face surface curvature and depth confidence curves; Gilani et al. proposed a method that uses adaptive geometric functions for dense registration of 3D faces and further localizes facial key points. Researchers proposed a method for facial landmark detection using scale-invariant features (SIFT) and grid functions. Traditional methods, by constructing geometrically meaningful artificial features, possess a certain degree of robustness. However, due to a lack of effective data utilization and learning, their accuracy is significantly limited, making it difficult to apply to diverse 3D facial data and adapt to a wide range of application scenarios.

[0004] Deep learning methods employ advanced deep learning techniques to locate key points on 3D faces using an adaptive data-driven approach. These methods can be further subdivided into convolutional neural network (CNN)-based methods and graph convolutional network (GCN)-based methods. CNN-based methods typically transform the 3D shape into an image mesh format, such as multi-view depth maps or geometric maps, then use CNNs for 2D key point detection, and finally convert it back to 3D key points. These methods often rely on subjective preprocessing methods and suffer from systematic errors in the 2D-to-3D reprojection process. Research on graph convolutional network-based methods is still in its early stages. Wang et al. proposed an adaptive regression method for 3D facial key point detection using graph convolutional networks; however, it only supports geometric feature extraction and does not incorporate texture features; it only supports geometric data sampled at a fixed resolution, thus limiting its generalization ability.

[0005] Due to the special characteristics of 3D face point cloud data, existing 3D face key point detection methods mainly have the following shortcomings: learning the mapping from 3D face to key point positions using graph convolutional networks is an ill-conditioned optimization problem, and the results obtained by directly regressing 3D coordinates lack robustness.

[0006] Furthermore, 3D face data can be divided into textured and non-textured data. Existing methods cannot process different types of data simultaneously. This results in existing methods being accurate on existing single datasets, but lacking generalization performance across data. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention provides a three-dimensional facial key point detection method. This method proposes a transformation from a probability graph to key point locations to adapt to the training of graph convolutional networks, thus overcoming the lack of robustness in existing technologies.

[0008] To achieve the above objectives, the present invention provides a method for detecting three-dimensional facial key points, comprising the following steps:

[0009] Acquire raw input data, which is used to characterize the three-dimensional face shape and texture data;

[0010] The original input data is uniformly sampled from a point cloud grid to generate the first sampled data;

[0011] Fill the first sampled data to generate the first filled data;

[0012] A graph convolutional network is used to output a probability distribution graph model of 3D facial key points on the first filled data;

[0013] Based on the aforementioned probability distribution graph model, the locations of key points on a 3D face are predicted.

[0014] Preferably, the original input data includes:

[0015] The first format data is used to represent shape data, which includes x, y, and z channel data;

[0016] The second format data is used to represent shape and black-and-white texture data, wherein the black-and-white texture data includes v channel data;

[0017] The third format data is used to represent shape and color texture data, which includes r, g, and b three-channel color texture data.

[0018] Preferably, the point cloud grid is uniformly sampled as follows:

[0019] N′∝A f / r 2 ,

[0020] Among them, A f N represents the surface area of ​​the input 3D face point cloud mesh, N′ represents the number of sampling points, and r represents the sampling interval radius.

[0021] Preferably, the first sampling data includes: first format sampling data, second format sampling data, and third format sampling data, which are used to represent the data generated after uniformly sampling the first, second, and third format data using the point cloud grid.

[0022] Preferably, the first filling data is obtained using an N×6 six-channel normalized data representation method, wherein the N×6 six-channel normalized data representation method includes:

[0023] When the first format sampling data is input, the x, y, z channel data are sequentially filled into the first, second, and third channels of the N×6 six-channel system, and the fourth, fifth, and sixth channels of the N×6 six-channel system are filled with -1.

[0024] When the second format sampling data is input, the x, y, z, v channel data are sequentially filled into the first, second, third, and fourth channels of the N×6 six-channel system, and the fifth and sixth channels are filled with -1;

[0025] When the third format sampling data is input, the six channels x, y, z, r, g, b are sequentially filled into the N×6 channels.

[0026] Preferably, the probability map is in the form of a probability map centered on the key point location and following a Gaussian distribution.

[0027] Preferably, predicting the location of 3D facial key points based on the probability distribution graph model includes the following steps:

[0028] Obtain the target sampling point set;

[0029] The target sampling point set is processed based on the soft threshold maximization method, and the location of the three-dimensional facial key points is output.

[0030] Preferably, obtaining the target sampling point set includes the following steps:

[0031] Step 1: Select the first center point q corresponding to the maximum value of the heat map, and construct the initial set of sampling points V′={q};

[0032] Step 2: Select multiple points closest to the center point, and denote them as the first set V;

[0033] Step 3: Select the second center point q′ corresponding to the maximum value of the heat map in the first set V, and update the second center point q′ to the initial sampling point set to obtain the updated sampling point set V′∪{q′};

[0034] Step 4: Remove the second center point q′ and all points in the first set V whose angle with the vector qq′ formed by connecting the first center point q and the second center point q′ is less than a preset angle;

[0035] Step 5: Repeat steps 3 and 4 until all points in the first set V are removed, and use the final set of sampling points as the target set of sampling points.

[0036] Preferably, the soft threshold maximization method is expressed as follows:

[0037]

[0038] Where H represents the heat map, S max The coordinates of the target key points are represented by α, which represents the coefficient of the exponent, and h is the value of h. i v′ represents the value of the i-th coordinate in the heatmap. i This represents the coordinates of the i-th 3D vertex.

[0039] On the other hand, the present invention provides a three-dimensional facial key point detection device, employing any of the above-mentioned three-dimensional facial detection methods, including:

[0040] The data acquisition module is used to acquire raw input data, including 3D face shape and texture data;

[0041] The data sampling module is used to uniformly sample the original input data using a point cloud grid to generate the first sampled data;

[0042] The data filling module is used to fill the first sampled data and generate the first filled data;

[0043] The probabilistic graphical model generation module is used to output a probabilistic graphical model of the distribution of three-dimensional facial key points using a graph convolutional network on the first padding data.

[0044] The 3D facial landmark location output module is used to predict the location of 3D facial landmarks based on the probability distribution map model.

[0045] As can be seen from the above solutions, the advantages of the present invention are:

[0046] This invention proposes a transformation from a 3D facial key point probabilistic graph model to key point locations to adapt to the training of graph convolutional networks, thereby improving the robustness and accuracy of the training results.

[0047] Furthermore, this invention proposes a standardized representation method for 3D face geometry and texture data to adapt to the training process of graph convolutional networks, enabling the trained network to adapt to different types of data and thus have the ability to generalize across datasets. Attached Figure Description

[0048] Figure 1 This is a flowchart of the three-dimensional face key point detection method of the present invention;

[0049] Figure 2 for Figure 1 Display image;

[0050] Figure 3 A schematic diagram of the N×6 six-channel normalized data representation method;

[0051] Figure 4 Flowchart for predicting the location of 3D facial landmarks based on a probability distribution graphical model;

[0052] Figure 5 Flowchart for obtaining the target sampling set;

[0053] Figure 6 for Figure 5 Display image;

[0054] Figure 7 This is a diagram illustrating the effect of 3D facial key point detection in this invention.

[0055] Figure 8 This is a schematic diagram of the structure of the three-dimensional facial key point detection device of the present invention;

[0056] Among them, the attached figures are labeled

[0057] 1-3D facial landmark detection device;

[0058] 10-Data Acquisition Module;

[0059] 11-Data sampling module;

[0060] 12-Data Filling Module;

[0061] 13-Probabilistic graphical model generation module;

[0062] 14-3D facial key point location output module;

[0063] Steps S10~S14, S140~S141, S1401~S1405. Detailed Implementation

[0064] In their research on 3D facial landmark detection, the inventors discovered that 3D data differs from 2D images in that its size is proportional to the size of a human face in the real world. Furthermore, the latest 3D facial data typically includes texture information. To address these technical challenges in 3D facial landmark detection, this invention proposes a standardized 3D facial data representation method. This method simultaneously encodes the geometric and shape information of the 3D face into a standardized data format, which is then input into a graph convolutional network for training. Simultaneously, to address the robustness issue of 3D facial landmark detection, this invention proposes a method for converting probabilistic graphs into landmark locations based on the probabilistic graphical representation of landmark positions using Gaussian functions, thus adapting to the training of graph convolutional networks.

[0065] To make the above features and effects of the present invention clearer and easier to understand, specific embodiments are described below, and detailed descriptions are provided in conjunction with the accompanying drawings.

[0066] Figure 1 This is a flowchart of a three-dimensional face key point detection method according to an embodiment of the present invention. Figure 2 for Figure 1 The method includes the following steps: (The image is shown in the original text.)

[0067] S10: Obtain raw input data to characterize 3D face shape and texture data;

[0068] S11: Uniformly sample the point cloud grid from the original input data to generate the first sampled data;

[0069] S12: Fill in the first sampled data and generate the first filled data;

[0070] S13: Use a graph convolutional network to output a graph model of the distribution probability of 3D facial key points on the first padding data;

[0071] S14: Predict the location of 3D facial key points based on a probability distribution graph model.

[0072] This method uses a training framework based on graph convolutional networks. By sampling and padding the original input data, it adaptively regresses the probability distribution map of 3D facial key points, and finally transforms the probability map into the precise location of 3D facial key points.

[0073] In step S10:

[0074] The original input data includes:

[0075] The first format data is used to represent shape data, which includes x, y, and z channel data;

[0076] The second format data is used to represent shape and black-and-white texture data, which includes v single-channel data;

[0077] The third format data is used to represent shape and color texture data, which includes three-channel color texture data: r, g, and b.

[0078] In step S11:

[0079] Uniform sampling of point cloud meshes is expressed by the following formula:

[0080] N′∝A f / r 2 ,

[0081] Among them, A f N represents the surface area of ​​the input 3D face point cloud mesh, N′ represents the number of sampling points, and r represents the sampling interval radius.

[0082] The first sampled data includes: first format sampled data, second format sampled data, and third format sampled data, which are used to represent the data generated after uniformly sampling the first, second, and third format data into a point cloud grid, respectively.

[0083] The goal of this stage is to sample the original input data online, while incorporating data augmentation techniques such as random rotation, scaling, and shifting to provide sufficient shape and texture variations. Unlike 2D face images, 3D face data is proportional to the size of a face in the real world. This is an important property, which this stage leverages to sample points on the surface of all original input data. This differs from existing point cloud resampling preprocessing methods and improves the generalization ability of the trained model.

[0084] In step S12:

[0085] like Figure 3 As shown, the first padding data is obtained using an N×6 six-channel normalized data representation method, which includes:

[0086] When the first format sampling data is input, the x, y, z channel data are sequentially filled into the first, second, and third channels of the N×6 six-channel system, and -1 is filled into the fourth, fifth, and sixth channels of the N×6 six-channel system.

[0087] When the second format sampling data is input, the x, y, z, and v channels are filled into the first, second, third, and fourth channels of the N×6 six-channel array in sequence, and -1 is filled into the fifth and sixth channels.

[0088] When the third format sampling data is input, the six channels x, y, z, r, g, b are sequentially filled into the N×6 six channels.

[0089] Specifically, this invention defines the data representation during model training as data uniformly resampled from a point cloud mesh. Assuming the number of points uniformly resampled from a face point cloud mesh is N′, this data (number of sampling points N′) can be filled in two dimensions to obtain an N×6 normalized data representation (N is an integer). This can be divided into three cases:

[0090] ① For shape data that does not contain texture information, after resampling, N′×3 (including the three channels of point cloud position x, y, z) of data are filled with content of value -1;

[0091] ② For shape data containing black and white texture information, after resampling, N′×4 data is filled with content of value -1 (including the three channels of point cloud position x, y, z and the one channel of color v);

[0092] ③ For shape data containing color texture information, after resampling, N′×6 data (including the three channels of point cloud position x, y, z and the three channels of color r, g, b) are filled with content of value -1;

[0093] This invention employs an N×6 normalized data representation method, the technical effect of which is that 3D face data of different formats, including textureless and textured data, can be represented as a uniform data shape, thus facilitating its input into graph convolutional networks for training.

[0094] In step S13, the probability map is a probability map centered on the key point location and following a Gaussian distribution.

[0095] In step S14: as Figure 4 As shown, it also includes the following steps:

[0096] S140: Obtain the target sampling point set;

[0097] S141: Based on the soft threshold maximization method, process the target sampling point set and output the location of the three-dimensional facial key points.

[0098] Among them, such as Figure 5 and Figure 6 As shown, step S140 further includes the following steps:

[0099] Step S1401: Select the first center point q corresponding to the maximum value of the heat map, and construct the initial sampling point set V′={q};

[0100] Step S1402: Select multiple points closest to the center point, and denote them as the first set V;

[0101] In this embodiment, 30 points closest to the center point are selected and denoted as the first set V.

[0102] Step S1403: Select the second center point q′ corresponding to the maximum value of the heat map in the first set V, and update the second center point q′ to the initial sampling point set to obtain the updated sampling point set V′∪{q′};

[0103] Step S1404: Remove the second center point q′ and all points in the first set V whose angle with the vector qq′ formed by connecting the first center point q and the second center point q′ is less than a preset angle;

[0104] In this embodiment, the preset angle is π / 3.

[0105] Step S1405: Repeat steps S1403 and S1404 until all points in the first set V are removed, i.e. The final set of sampling points is used as the target set of sampling points.

[0106] The sampling method used in step S140 to obtain the target sampling point set can also be called the directional equalization sampling method. This method can compensate for the systematic errors caused by the uneven spatial distribution of point cloud data.

[0107] In step S141:

[0108] After obtaining the target sampling point set V′, its position and probability map values ​​can be processed by soft-max (soft thresholding) to obtain accurate 3D face key point location detection results.

[0109] The soft threshold maximization method is expressed as follows:

[0110]

[0111] Where H represents the heat map, S max The coordinates of the target key points are represented by α, which represents the coefficient of the exponent, and h is the value of h. i v′ represents the value of the i-th coordinate in the heatmap. i This represents the coordinates of the i-th 3D vertex.

[0112] The present invention provides a method for converting probabilistic maps into 3D facial key point locations. The technical effect of this method is that it takes into account the non-uniform distribution of point cloud data in space, compensates for directional deviations, and improves the robustness and accuracy of the training model.

[0113] Figure 7 The image shown is a result of three-dimensional facial key point detection obtained using the method of the present invention. Figure 7 The reference and detection locations are marked with red and blue dots, respectively. Figure 7As can be seen from the examples, the method of the present invention can be applied to data with different postures, noise, and missing regions, and has the advantages of high detection accuracy and good robustness.

[0114] The following are apparatus embodiments corresponding to the above method embodiments. This embodiment can be implemented in conjunction with the above embodiments. The relevant technical details mentioned in the above embodiments remain valid in this embodiment, and will not be repeated here to reduce repetition. Correspondingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiments.

[0115] Figure 8 The image shown is a three-dimensional face key point detection device 1 (hereinafter referred to as device 1) according to an embodiment of the present invention. Device 1 employs the above-described... Figure 1 The method for detecting any three-dimensional facial key points shown includes: a data acquisition module 10, a data sampling module 11, a data filling module 12, a probabilistic graphical model generation module 13, and a three-dimensional facial key point position output module 14.

[0116] Specifically, the data acquisition module 10 acquires raw input data including 3D face shape and texture data. The data sampling module 11 uniformly samples the raw input data using a point cloud mesh to generate first sampled data. The data filling module 12 fills the first sampled data to generate first filled data. The probabilistic graphical model generation module 13 uses a graph convolutional network to output a probability distribution graphical model of 3D facial key points on the first filled data. The 3D facial key point location output module 14 predicts the location of 3D facial key points based on the probability distribution graphical model.

[0117] In another embodiment of the present invention, a storage medium is further included for storing execution. Figure 1 The computer processing program for the method shown.

[0118] In summary, this invention, based on graph convolutional networks as the carrier for 3D facial keypoint detection, firstly proposes a standardized representation method for 3D facial geometry and texture data to adapt to the training process of graph convolutional networks, enabling the trained network to adapt to different types of data and thus possessing cross-dataset generalization ability; secondly, based on the probabilistic graph representation method of 3D facial keypoint locations, this invention proposes a transformation method from probabilistic graphs to keypoint locations to adapt to the training of graph convolutional networks and improve the robustness and accuracy of training results.

[0119] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims, and all of these forms fall within the scope of protection of the present invention.

Claims

1. A method for detecting key points of a three-dimensional face, characterized in that, include: Acquire raw input data, which is used to characterize the three-dimensional face shape and texture data; The original input data is uniformly sampled from a point cloud grid to generate the first sampled data; Fill the first sampled data to generate the first filled data; A graph convolutional network is used to output a probability distribution graph model of 3D facial key points on the first filled data; Based on the aforementioned probability distribution graph model, a directionally balanced sampling method trained with an adaptive graph convolutional network is used to predict the location of 3D facial key points.

2. The three-dimensional facial key point detection method according to claim 1, characterized in that, The original input data includes: First format data, used to represent shape data, the shape data including x, y, z channel data; The second format data is used to represent shape and black-and-white texture data, wherein the black-and-white texture data includes v channel data; The third format data is used to represent shape and color texture data, wherein the color texture data includes three-channel color texture data of r, g, and b.

3. The three-dimensional facial key point detection method according to claim 1 or 2, characterized in that, The uniform sampling of the point cloud grid is expressed by the following formula: ， in, This represents the surface area of ​​the input 3D face point cloud mesh. This represents the number of sampling points, and r represents the sampling interval radius.

4. The three-dimensional facial key point detection method according to claim 3, characterized in that, The first sampling data includes: first format sampling data, second format sampling data, and third format sampling data, which are used to represent the data generated after uniformly sampling the first, second, and third format data using the point cloud grid.

5. The three-dimensional facial key point detection method according to claim 4, characterized in that, The first filling data uses The six-channel normalized data representation method is used to obtain the data. Six-channel normalized data representation methods include: When the first format sampling data is input, the x, y, z channel data are sequentially filled into... The first, second, and third channels of the six channels, the For the fourth, fifth, and sixth channels of a six-channel system, fill in -1; When the second format sampling data is input, the x, y, z, v channel data are sequentially filled into the... The first, second, third, and fourth channels of the six-channel system, and the fifth and sixth channels are filled with -1; When the third format sampling data is input, the x, y, z, r, g, b channel data are sequentially filled into the... In the six channels.

6. The three-dimensional facial key point detection method according to claim 1, characterized in that, The probability graph is a Gaussian distribution centered on the key point location.

7. The three-dimensional facial key point detection method according to claim 1, 2, 4, 5 or 6, characterized in that, The prediction of 3D facial landmark locations based on the probability distribution graph model includes the following steps: Obtain the target sampling point set; The target sampling point set is processed based on the soft threshold maximization method, and the location of the three-dimensional facial key points is output.

8. The three-dimensional facial key point detection method according to claim 7, characterized in that, The process of obtaining the target sampling point set includes the following steps: Step 1: Select the first center point corresponding to the maximum value of the heatmap. Construct the initial set of sampling points. ; Step 2: Select multiple points closest to the center point, and denote them as the first set. ; Step 3: In the first set Select the second center point corresponding to the maximum value of the heat map. and the second center point Update the initial set of sampling points to obtain the updated set of sampling points. ; Step 4: In the first set Remove the second center point and all connections to the first center point With the second center point The resulting vector Points whose included angle is less than a preset angle; Step 5: Repeat steps 3 and 4 until the first set is reached. All points are removed, and the final set of sampling points is used as the target set of sampling points.

9. The three-dimensional facial key point detection method according to claim 7, characterized in that, The soft threshold maximization method is expressed as follows: ， in, Represents a heat map. Indicates the coordinates of the target's key points. The coefficient representing the exponent. This represents the value of the i-th coordinate in the heatmap. This represents the coordinates of the i-th 3D vertex.

10. A three-dimensional face key point detection device, employing any one of the three-dimensional face detection methods described in claims 1 to 9, comprising: The data acquisition module is used to acquire raw input data, including 3D face shape and texture data; The data sampling module is used to uniformly sample the original input data using a point cloud grid to generate the first sampled data; The data filling module is used to fill the first sampled data and generate the first filled data; The probabilistic graphical model generation module is used to output a probabilistic graphical model of the distribution of three-dimensional facial key points using a graph convolutional network on the first padding data. The 3D facial landmark location output module is used to predict the location of 3D facial landmarks based on the probability distribution graph model and using a directionally balanced sampling method trained by an adaptive graph convolutional network.

11. A storage medium for storing a program that executes any one of the three-dimensional face key point detection methods as described in claims 1 to 9.

Images