A method and system for collecting 3D key points of human face based on multiple sensors

Abstract

The present invention proposes a multi-sensor based 3D key point acquisition method and system for a face, which belongs to the field of face recognition, including: S1, constructing a deep neural network and a recurrent neural network model, and training the two network models respectively to obtain a key point acquisition model and a key point mapping model; S2, arranging multiple image acquisition devices, including at least two RGB cameras and one RGBD camera, the image acquisition devices are configured around the face to be acquired, and the camera external parameters of the multiple image acquisition devices are calibrated respectively through a calibration plate; S3, using multiple image acquisition devices to acquire a multi-view image data sequence of the face to be acquired; S4, deploying a key point acquisition model on a local control device, inputting the multi-view image data sequence into the key point acquisition model, determining the first frame image data, and using the key point mapping model to determine the predicted key point position of the face to be acquired based on the first frame image data, using the key point mapping model to form a dense face grid through the predicted key point position of the face to be acquired, and calculating the head posture of the face to be acquired. This application uses multiple image acquisition devices and deep neural network models to achieve high-precision and high-efficiency facial 3D key point and head posture data acquisition, improving the accuracy and robustness of data acquisition while reducing the difficulty and cost of acquisition.

Description

Technical Field

[0001] The present invention relates to the field of face recognition technology, and in particular to a multi-sensor based method and system for acquiring 3D key points of a face. Background Art

[0002] In recent years, with the continued rapid development of China's economy and society, face-related AI functions and applications such as facial recognition, expression analysis, and augmented reality have shown rapid development. The core foundation of these applications is accurate 3D facial key point positioning and head pose estimation.

[0003] However, existing technologies use 2D image-based automatic detection algorithms to automatically detect key points from a single 2D image, but their accuracy is often insufficient in scenarios with complex lighting and wide-angle poses. Structured light and Time-of-Flight (collectively referred to as RGB-D cameras) can only capture sparse 3D facial data, while the dense 3D point data captured by radar has a high cost and high accuracy. Furthermore, head pose can also be directly acquired using devices such as IMUs, but this accuracy is also linked to price, resulting in high costs while achieving high accuracy.

[0004] Therefore, finding a method that can not only improve the accuracy of face acquisition, thereby improving the accuracy of head posture acquisition, but also reduce the acquisition cost is a technical problem that needs to be solved urgently by those skilled in the art. Summary of the Invention

[0005] In view of this, the present invention proposes a multi-sensor based method and system for collecting 3D key points of the face, which can use multiple image acquisition devices and deep neural network models to achieve high-precision and high-efficiency collection of 3D key points of the face and head posture data, improve the accuracy and robustness of data collection, and reduce the difficulty and cost of collection.

[0006] The technical solution of the present invention is achieved as follows:

[0007] In a first aspect, the present invention provides a method for collecting 3D key points of a face based on multiple sensors, comprising the following steps:

[0008] S1. Build a deep neural network model and a bidirectional recurrent neural network model, and train the two network models separately to obtain a key point acquisition model and a key point mapping model;

[0009] S2. Arrange multiple image acquisition devices, including at least two RGB cameras and one RGBD camera, and configure the image acquisition devices around the face to be captured. Calibrate the camera external parameters of the multiple image acquisition devices using a calibration plate.

[0010] S3, using multiple image acquisition devices to acquire a multi-view image data sequence of the face to be acquired;

[0011] S4. Deploy a key point acquisition model on the local control device, input the multi-view image data sequence into the key point acquisition model, determine the first frame image data, and use the key point mapping model to determine the predicted key point position of the face to be collected based on the first frame image data, and calculate the head posture of the face to be collected through the predicted key point position of the face to be collected.

[0012] Based on the above technical solution, preferably, step S1 specifically includes:

[0013] S11. Build a deep neural network model; wherein the deep neural network model includes a backbone network and a decoupled prediction head, wherein cross-layer links are provided between each stage layer in the backbone network;

[0014] S12. Construct a bidirectional recurrent neural network; wherein the bidirectional recurrent neural network includes two hidden layers and one fully connected layer;

[0015] S13. Use the training data to train the deep neural network model and the bidirectional recurrent neural network to obtain a key point acquisition model and a key point mapping model.

[0016] On the basis of the above technical solution, preferably, the decoupled prediction head includes a key point prediction branch and a head posture prediction branch.

[0017] On the basis of the above technical solution, preferably, the orthogonal projection calculation formula for determining the key point positions of the face to be collected based on the first frame image data is as follows:

[0018]

[0019] Among them, x represents the horizontal coordinate of the 3D key point, y represents the vertical coordinate of the 3D key point, z represents the depth information provided by the RGBD camera, M represents the head pose matrix, K represents the camera intrinsic parameter matrix, s represents the scale factor, u represents the horizontal coordinate of the 2D key point, and v represents the vertical coordinate of the 2D key point.

[0020] More preferably, step S4 specifically includes:

[0021] S41. Deploy a key point acquisition model on a local control device, use multiple image acquisition devices to obtain multi-view image data of the face, and input it into a deep neural network model to obtain the predicted 2D key points and head posture of each camera;

[0022] S42, calculating 3D key points of the face according to the depth value provided by the RGBD camera and the 2D key points predicted based on the RGBD camera in S41 according to orthogonal projection;

[0023] S43, using the 3D key points and head posture based on the RGBD camera calculated in S42, according to the orthogonal projection formula, combined with the head posture and internal and external parameters of the remaining cameras, the 3D key points are back-projected onto the images of each camera, and compared with the predicted values based on the remaining cameras in S41 to verify the error;

[0024] S44, repeat S41-43 until the error is less than the set threshold;

[0025] S45, using a bidirectional recurrent neural network to map the 3D key points calculated in S42 to obtain 392 key points to form a dense face network;

[0026] S46, using the 3D key points based on the RGBD camera and the inverse matrix of the head posture, calculating the 3D face template in the standard posture;

[0027] S47, repeating steps S41-S46 until all true value data are acquired, and bidirectionally verifying the validity of the true value based on orthogonal projection, and deleting data with large errors;

[0028] S48. Use the filtered true value data to iteratively train the deep neural network model and the bidirectional recurrent neural network until the accuracy of each network model reaches the preset requirements. The trained network model will be the acquisition model used for the next acquisition.

[0029] Based on the above technical solution, preferably, the calculation process of predicting 2D key points and head posture is as follows:

[0030] The training data is sequentially input into multiple stage layers of the backbone network for data processing to obtain the first feature;

[0031] The first feature is processed using the key point prediction branch and the head posture prediction branch to obtain the initial key point coordinates, the confidence of the initial key point and the rotation matrix of the initial head posture;

[0032] The initial key point coordinates, the initial key point confidences, and the rotation matrix of the initial head pose are normalized to obtain predicted 2D key points and head poses. The predicted key points include 68 key point coordinates and key point confidences.

[0033] On the basis of the above technical solution, preferably, the use of a bidirectional recurrent neural network to map the predicted 2D key points specifically includes:

[0034] Predict the 2D key points and perform normalization to obtain 68 normalized face coordinates;

[0035] Flatten the normalized 3D coordinates of the 68 face coordinates to form a 204-dimensional one-dimensional vector;

[0036] The 204-dimensional one-dimensional vector is sequentially input into two hidden layers and one fully connected layer of the bidirectional recurrent neural network to obtain a 1176-dimensional vector;

[0037] The 1176-dimensional vector is converted into 3D key points, resulting in 392 key points.

[0038] In a second aspect, the present invention provides a multi-sensor based 3D facial key point acquisition system, which adopts the above-mentioned 3D facial key point acquisition method, including:

[0039] A camera installation and calibration module is configured to arrange multiple image acquisition devices, including at least two RGB cameras and one RGBD camera, around the face to be captured, and calibrate the camera external parameters of the multiple image acquisition devices using a calibration plate;

[0040] A model training module is configured to construct a deep neural network model and a recurrent neural network model, and train the two network models separately to obtain a key point acquisition model and a key point mapping model;

[0041] A camera installation and calibration module is configured to arrange multiple image acquisition devices, including at least two RGB cameras and one RGBD camera, around the face to be captured, and calibrate the camera external parameters of the multiple image acquisition devices using a calibration plate;

[0042] A data acquisition module, which uses multiple image acquisition devices to acquire a multi-view image data sequence of the face to be captured;

[0043] A key point calculation module is configured to input a multi-view image data sequence into a key point acquisition model, determine the first frame image data, and use a key point mapping model to determine the key point positions of the face to be acquired based on the first frame image data;

[0044] The head posture calculation module is configured to calculate the head posture of the face to be collected based on the key point positions of the face to be collected.

[0045] In a third aspect, the present invention provides a computer-readable storage medium storing computer instructions, which enable the computer to implement the method for collecting 3D key points of a face as described above.

[0046] In a fourth aspect, the present invention provides an electronic device comprising at least one processor, a memory, a communication interface and a bus; wherein,

[0047] The processor, memory, and communication interface communicate with each other via the bus;

[0048] The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to implement the above-mentioned method for collecting 3D key points of a face.

[0049] The method for collecting 3D key points of a face according to the present invention has the following advantages over the prior art:

[0050] (1) By using multiple image acquisition devices and deep neural network models, high-precision and high-efficiency facial 3D key points and head posture data acquisition is achieved, which improves the accuracy and robustness of data acquisition while reducing the difficulty and cost of acquisition;

[0051] (2) By combining the backbone network, decoupled prediction head, and bidirectional recurrent neural network into a deep neural network model, the feature extraction capability of the key point acquisition model is improved, and the decoupling of key point positioning and head posture estimation is achieved, which greatly improves the accuracy and generalization capability of the key point acquisition model;

[0052] (3) The accuracy of the model is improved by completing the iteration through the true value collection, thereby capturing the true value of the data with higher accuracy. The model is continued to iterate through the data with higher accuracy, which can form a data closed loop and improve the robustness and accuracy of the 3D face key point matching. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0054] Figure 1 This is a flow chart of the method for collecting 3D key points of a face according to the present invention;

[0055] Figure 2 A schematic diagram of the arrangement of image acquisition equipment for the method for acquiring 3D key points of a face according to the present invention;

[0056] Figure 3 Schematic diagram of cross-layer links of the method for collecting 3D key points of a face according to the present invention;

[0057] Figure 4 Schematic diagram of the decoupled prediction head of the 3D facial key point acquisition method of the present invention. DETAILED DESCRIPTION

[0058] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0059] like Figure 1 As shown, the present invention provides a method for collecting 3D key points of a face based on multiple sensors, comprising the following steps:

[0060] S1. Arrange multiple image acquisition devices, including at least two RGB cameras and one RGBD camera. The image acquisition devices are configured around the face to be captured, and the camera external parameters of the multiple image acquisition devices are calibrated using a calibration plate.

[0061] like Figure 2 As shown, in one embodiment of the present application, the two RGB cameras are respectively placed at 30 degrees to the left and right sides of the face to be captured, and the RGBD camera is placed between the two RGB cameras and in front of the face to be captured.

[0062] By combining two RGB cameras and one RGBD camera, multi-angle and multi-modal data acquisition is achieved, which improves the accuracy and completeness of 3D key point positioning. By fusing images from different sensors, the limitations of a single sensor are overcome, providing more comprehensive and stable 3D facial data.

[0063] S1. Build a deep neural network model and train the deep neural network model to obtain a key point acquisition model;

[0064] Specifically, step S1 includes:

[0065] S11, constructing a deep neural network model; wherein the deep neural network model includes a backbone network and a decoupled prediction head, wherein each stage layer in the backbone network is provided with a cross-layer link; Figure 3 As shown;

[0066] S12. Construct a bidirectional recurrent neural network; wherein the bidirectional recurrent neural network includes two hidden layers and one fully connected layer;

[0067] S13. Use multiple image acquisition devices to obtain multi-view image data sequences of human faces as training data, and divide the training data into multiple groups. Select one group of training data and input it into the deep neural network model and the bidirectional recurrent neural network network to obtain the key point acquisition model and the key point mapping model.

[0068] In one embodiment of the present application, the preset requirements can be set according to actual conditions, and the present application does not limit this.

[0069] In one embodiment of the present application, the deep neural network model is based on MobileNet V3 as the backbone network.

[0070] In one embodiment of the present application, the loss function of the deep neural network model is mean square error.

[0071] In one embodiment of the present application, the training of the recurrent neural network model requires specifying the number of sequences, and the number of sequences used is 3. It uses every 3 key points to form a small triangle to capture the feature information in the area and form a denser key point.

[0072] like Figure 4 As shown, in one embodiment of the present application, the decoupled prediction head includes a key point prediction branch and a head posture prediction branch.

[0073] In one embodiment of the present application, the orthogonal projection calculation formula for determining the key point positions of the face to be captured based on the first frame image data is as follows:

[0074]

[0075] Among them, x represents the horizontal coordinate of the 3D key point, y represents the vertical coordinate of the 3D key point, z represents the depth information provided by the RGBD camera, M represents the head pose matrix, K represents the camera intrinsic parameter matrix, s represents the scale factor, u represents the horizontal coordinate of the 2D key point, and v represents the vertical coordinate of the 2D key point.

[0076] By combining the backbone network, decoupled prediction head and bidirectional recurrent neural network into a deep neural network model, the feature extraction capability of the key point acquisition model is improved, and the decoupling of key point positioning and head posture estimation is achieved, which greatly improves the accuracy and generalization capability of the key point acquisition model.

[0077] In one embodiment of the present application, a bidirectional recurrent neural network is used to map the predicted key points, and 1176 key points can be obtained, further improving the accuracy of the key point acquisition model.

[0078] In one embodiment of the present application, the calculation process of the true value data is as follows:

[0079] The calculated value is further compared with the predicted value, and the data with large errors are screened out, and the final true value is GT. n ={(L n |P n ),(M n |H n )}; Among them, P nRepresents the 2D key point coordinates, H n represents the head posture, u represents the 2D image x coordinate, v represents the 2D image y coordinate, and K represents the camera intrinsic parameter matrix.

[0080] S3, using multiple image acquisition devices to acquire a multi-view image data sequence of the face to be acquired;

[0081] S4. Deploy a key point acquisition model on the local control device, input the multi-view image data sequence into the key point acquisition model, determine the first frame image data, and use the key point mapping model to determine the predicted key point position of the face to be collected based on the first frame image data, and calculate the head posture of the face to be collected through the predicted key point position of the face to be collected.

[0082] Specifically, step S4 includes:

[0083] S41. Deploy a key point acquisition model on a local control device, use multiple image acquisition devices to obtain multi-view image data of the face, and input it into a deep neural network model to obtain the predicted 2D key points and head posture of each camera;

[0084] S42, calculating 3D key points of the face according to the depth value provided by the RGBD camera and the 2D key points predicted based on the RGBD camera in S41 according to orthogonal projection;

[0085] S43, using the 3D key points and head posture based on the RGBD camera calculated in S42, according to the orthogonal projection formula, combined with the head posture and internal and external parameters of the remaining cameras, the 3D key points are back-projected onto the images of each camera, and compared with the predicted values based on the remaining cameras in S41 to verify the error;

[0086] S44, repeat S41-43 until the error is less than the set threshold;

[0087] S45, using a bidirectional recurrent neural network to map the 3D key points calculated in S42 to obtain 392 key points to form a dense face network;

[0088] S46, using the 3D key points based on the RGBD camera and the inverse matrix of the head posture, calculating the 3D face template in the standard posture;

[0089] S47, repeating steps S41-S46 until all true value data are acquired, and bidirectionally verifying the validity of the true value based on orthogonal projection, and deleting data with large errors;

[0090] S48. Use the filtered true value data to iteratively train the deep neural network model and the bidirectional recurrent neural network until the accuracy of each network model reaches the preset requirements. The trained network model will be the acquisition model used for the next acquisition.

[0091] Furthermore, the calculation process of predicting key points and head pose is as follows:

[0092] The training data is sequentially input into multiple stage layers of the backbone network for data processing to obtain the first feature;

[0093] The first feature is processed using the key point prediction branch and the head posture prediction branch to obtain the initial key point coordinates, the confidence of the initial key point and the rotation matrix of the initial head posture;

[0094] The initial key point coordinates, the initial key point confidences, and the rotation matrix of the initial head pose are normalized to obtain predicted 2D key points and head poses. The predicted key points include 68 key point coordinates and key point confidences.

[0095] In one embodiment of the present application, a bidirectional recurrent neural network is used to map the predicted 2D key points, specifically including:

[0096] Normalize the predicted 2D key points to obtain 68 normalized face coordinates;

[0097] Flatten the normalized 2D coordinates of the 68 face coordinates to form a 204-dimensional one-dimensional vector;

[0098] The 204-dimensional one-dimensional vector is sequentially input into two hidden layers and one fully connected layer of the bidirectional recurrent neural network to obtain a 1176-dimensional vector;

[0099] The 1176-dimensional vector is converted into 3D key points, resulting in 392 key points.

[0100] In one embodiment of the present application, the loss function of the bidirectional recurrent neural network is the mean square error.

[0101] In one embodiment of the present application, a multi-view image data series is input into a key point acquisition model to obtain a predicted output, and a quality assessment is performed on the predicted output to determine the first frame image data;

[0102] Calculate the coordinates of 3D key points based on the predicted output of the first frame image data and the camera parameters of the image acquisition device;

[0103] Use geometric algorithms to verify whether the head posture is consistent:

[0104] If they are inconsistent, repeat the above steps until the head posture of the multi-view image data is consistent;

[0105] The predicted output is mapped through a bidirectional recurrent neural network to generate a dense face network and calculate the 3D coordinates of the face in standard poses;

[0106] The collected face moves according to the collection action. The face at each moment will be matched with the standard face through the key point collection model output to obtain the true value of 2D and 3D key points and head posture.

[0107] It is understandable that if the captured face changes, the multi-view image data series of the captured face after the change will be re-input into the key point acquisition model.

[0108] Specifically, a specific embodiment is used to illustrate:

[0109] Using two RGB cameras (RGB A and RGB C ) and an RGBD camera (RGBD B ) obtain three images, namely I A , I B and I C ;

[0110] Will I A , I B and I C Input into the deep neural network model to obtain the model prediction value {O A ,O B ,O C |O n =(L n ,M n )}, where O A Represents RGB A The predicted output, O B Represents RGBD B The predicted output, O C Represents RGB C The predicted output, L n Represents the predicted key point coordinates, M n represents the predicted head pose matrix;

[0111] Statistics B The confidence level of each point is set to a threshold of 0.8. If there are more than 98 points below the threshold, repeat this step and use that moment as the first frame.

[0112] According to O B Key point branch and head posture branch, combined with RBGD B Internal reference and RBGD B Depth information D B , calculate the 3D coordinates, the calculation formula is as follows:

[0113]

[0114] Among them, x BIndicates the horizontal coordinate of the key point, y B Indicates the vertical coordinate of the key point, z B Represents depth information, provided by the RGBD camera, M B -1 Represents the head posture matrix M B The inverse matrix of T B represents the translation matrix, R B Represents the head pose matrix M B The rotation matrix part, K B -1 Represents the camera intrinsic parameter matrix K B The inverse matrix of s represents the scale factor, L B Indicates the confidence of the key point.

[0115] According to {O A ,O C}The 3D point matching obtained by the key point branch uses the PnP algorithm to calculate the head posture {H A ,H C}, and with {O A ,O C} in {M A ,M C} for comparison. If it is greater than the set threshold, the next frame will repeat the operations 1 and 2 until the mth frame meets the conditions. A Indicates the RGB calculated using the PnP algorithm A The head pose corresponding to the camera, H C Indicates the RGB calculated using the PnP algorithm C The head pose corresponding to the camera;

[0116] {O A ,O B ,O C} is mapped to 392 key points using a bidirectional recurrent neural network to form a dense face network Mesh = {(x i ,y i ,z i )|i=(0,1,2,…,391)};where, x i Indicates the x-coordinate and y-coordinate of the key point i Indicates the y coordinate of the key point, z i Indicates the z coordinate of the key point;

[0117] At the mth frame, according to the moment O B The head posture value in the standard posture is used to calculate the 3D face points. The calculation formula is:

[0118]

[0119] Among them, T B represents the translation matrix, x0 represents the x-coordinate of the 3D face point in the standard pose, y0 represents the y-coordinate of the 3D face point in the standard pose, and z0 represents the z-coordinate of the 3D face point in the standard pose;

[0120] At the {t|t>m}th frame, using the formula Bring in the corresponding predicted value of each camera (L n ,M n ), and calculate the corresponding 2D key points and head posture (P n ,H n ).

[0121] By combining the PnP algorithm, multi-frame data comparison and error screening, high-precision true value data is automatically generated, which improves the quality and consistency of the data and significantly reduces the cost and time of constructing large-scale data sets.

[0122] This application achieves high-precision and high-efficiency facial 3D key point and head posture data collection by using multiple image acquisition devices and deep neural network models, improving the accuracy and robustness of data collection while reducing the difficulty and cost of collection.

[0123] The present invention provides a multi-sensor based 3D facial key point acquisition system, which adopts the above-mentioned 3D facial key point acquisition method, including:

[0124] A model training module is configured to construct a deep neural network model and a recurrent neural network model, and train the two network models separately to obtain a key point acquisition model and a key point mapping model;

[0125] A camera installation and calibration module is configured to arrange multiple image acquisition devices, including at least two RGB cameras and one RGBD camera, around the face to be captured, and calibrate the camera external parameters of the multiple image acquisition devices using a calibration plate;

[0126] A data acquisition module, which uses multiple image acquisition devices to acquire a multi-view image data sequence of the face to be captured;

[0127] A key point calculation module is configured to input a multi-view image data sequence into a key point acquisition model, determine the first frame image data, and use a key point mapping model to determine the key point positions of the face to be acquired based on the first frame image data;

[0128] The head posture calculation module is configured to calculate the head posture of the face to be collected based on the key point positions of the face to be collected.

[0129] The present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions enable the computer to implement the above-mentioned method for collecting 3D key points of a face.

[0130] The present invention provides an electronic device comprising at least one processor, a memory, a communication interface, and a bus; wherein the processor, memory, and communication interface communicate with each other via the bus; the memory stores program instructions executable by the processor, and the processor calls the program instructions to implement the above-described method for collecting 3D key points of a face.

[0131] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multi-sensor based method for collecting 3D key points of a face, characterized in that: The following steps are involved: S1. Construct a deep neural network model and a recurrent neural network model, and train the deep neural network model and the recurrent neural network model respectively to obtain a key point acquisition model and a key point mapping model; S2. Arrange multiple image acquisition devices, including at least two RGB cameras and one RGBD camera. The image acquisition devices are configured around the face to be captured, and the camera external parameters of the multiple image acquisition devices are calibrated using a calibration plate; the two RGB cameras are placed 30 degrees to the left and right sides of the face to be captured, respectively, and the RGBD camera is placed between the two RGB cameras and in front of the face to be captured; S3, using multiple image acquisition devices to acquire a multi-view image data sequence of the face to be acquired; S4. Deploy a key point acquisition model on a local control device, input the multi-view image data sequence into the key point acquisition model, determine the first frame image data, and use the key point mapping model to determine the predicted key point positions of the face to be collected based on the first frame image data. Calculate the head posture of the face to be collected based on the predicted key point positions of the face to be collected; Step S4 specifically includes: S41. Deploy a key point acquisition model on a local control device, use multiple image acquisition devices to obtain multi-view image data of the face, and input it into a deep neural network model to obtain the predicted 2D key points and head posture of each camera; S42, calculating 3D key points of the face according to the depth value provided by the RGBD camera and the 2D key points predicted based on the RGBD camera in S41 according to orthogonal projection; S43, using the 3D key points and head posture based on the RGBD camera calculated in S42, according to the orthogonal projection formula, combined with the head posture and internal and external parameters of the remaining cameras, the 3D key points are back-projected onto the images of each camera, and compared with the predicted values based on the remaining cameras in S41 to verify the error; S44, repeat S41-43 until the error is less than the set threshold; S45, using a bidirectional recurrent neural network to map the 3D key points calculated in S42 to obtain 392 key points to form a dense face network; S46, using the 3D key points based on the RGBD camera and the inverse matrix of the head posture, calculating the 3D face template in the standard posture; S47, repeating steps S41-S46 until all true value data are acquired, and bidirectionally verifying the validity of the true value based on orthogonal projection, and deleting data with large errors; S48. Use the filtered true value data to iteratively train the deep neural network model and the bidirectional recurrent neural network until the accuracy of each network model reaches the preset requirements. The trained network model will be the acquisition model used for the next acquisition.

2. The multi-sensor based 3D facial key point acquisition method according to claim 1, characterized in that: S1 specifically includes: S11. Build a deep neural network model; wherein the deep neural network model includes a backbone network and a decoupled prediction head, wherein cross-layer links are provided between each stage layer in the backbone network; S12. Construct a bidirectional recurrent neural network; wherein the bidirectional recurrent neural network includes two hidden layers and one fully connected layer; S13. Use the training data to train the deep neural network model and the bidirectional recurrent neural network to obtain a key point acquisition model and a key point mapping model.

3. The multi-sensor based 3D facial key point acquisition method according to claim 2, characterized in that: The decoupled prediction head includes a key point prediction branch and a head pose prediction branch.

4. The multi-sensor based 3D facial key point acquisition method according to claim 1, wherein: The orthogonal projection calculation formula for determining the key point positions of the face to be collected based on the first frame image data is as follows: Among them, x represents the horizontal coordinate of the 3D key point, y represents the vertical coordinate of the 3D key point, z represents the depth information provided by the RGBD camera, M represents the head pose matrix, K represents the camera intrinsic parameter matrix, s represents the scale factor, u represents the horizontal coordinate of the 2D key point, and v represents the vertical coordinate of the 2D key point.

5. The multi-sensor based 3D facial key point acquisition method according to claim 4, characterized in that: The calculation process of predicting 2D key points and head posture is as follows: The training data is sequentially input into multiple stage layers of the backbone network for data processing to obtain the first feature; The first feature is processed using the key point prediction branch and the head posture prediction branch to obtain the initial key point coordinates, the confidence of the initial key point and the rotation matrix of the initial head posture; The initial key point coordinates, the initial key point confidences, and the rotation matrix of the initial head pose are normalized to obtain predicted 2D key points and head poses. The predicted key points include 68 key point coordinates and key point confidences.

6. The multi-sensor based 3D facial key point acquisition method according to claim 5, characterized in that: The use of a bidirectional recurrent neural network to map the predicted 2D key points specifically includes: Predict the 2D key points and perform normalization to obtain 68 normalized face coordinates; Flatten the normalized 3D coordinates of the 68 face coordinates to form a 204-dimensional one-dimensional vector; The 204-dimensional one-dimensional vector is sequentially input into two hidden layers and one fully connected layer of the bidirectional recurrent neural network to obtain a 1176-dimensional vector; The 1176-dimensional vector is converted into 3D key points, and 392 three-dimensional key points are obtained.

7. A multi-sensor based 3D facial key point acquisition system, characterized in that: The method for collecting 3D facial key points according to any one of claims 1 to 6 is adopted, comprising: A model training module is configured to construct a deep neural network model and a recurrent neural network model, and train the two network models separately to obtain a key point acquisition model and a key point mapping model; A camera installation and calibration module is configured to arrange multiple image acquisition devices, including at least two RGB cameras and one RGBD camera, around the face to be captured, and calibrate the camera external parameters of the multiple image acquisition devices using a calibration plate; A data acquisition module, which uses multiple image acquisition devices to acquire a multi-view image data sequence of the face to be captured; A key point calculation module is configured to input a multi-view image data sequence into a key point acquisition model, determine the first frame image data, and use a key point mapping model to determine the key point positions of the face to be acquired based on the first frame image data; The head posture calculation module is configured to calculate the head posture of the face to be collected based on the key point positions of the face to be collected.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions enable the computer to implement the method for collecting 3D key points of a face as described in any one of claims 1 to 6.

9. An electronic device, characterized in that: including at least one processor, a memory, a communication interface and a bus; wherein, The processor, memory, and communication interface communicate with each other via the bus; The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to implement the method for collecting 3D key points of a face according to any one of claims 1 to 6.

Images