Face 3D key point acquisition method and system based on multiple sensors

Abstract

The invention provides a face 3D key point acquisition method and system based on multiple sensors, and belongs to the field of face recognition, and the method comprises the steps: S1, constructing a deep neural network model and a recurrent neural network model, and training the two network models respectively to obtain a key point acquisition model and a key point mapping model; s2, arranging a plurality of image acquisition devices including at least two RGB cameras and one RGBD camera, configuring the image acquisition devices around a to-be-acquired face, and respectively calibrating camera external parameters of the plurality of image acquisition devices through a calibration board; s3, acquiring a multi-view image data sequence of the to-be-acquired face by using a plurality of image acquisition devices; and S4, deploying a key point acquisition model in local control equipment, inputting the multi-view image data sequence into the key point acquisition model, determining first frame image data, and determining predicted key point positions of the face to be acquired according to the first frame image data by using a key point mapping model. And forming a dense face grid by using the key point mapping model through the predicted key point position of the to-be-collected face, and calculating the head posture of the to-be-collected face. According to the method and the device, the plurality of image acquisition devices and the deep neural network model are used, high-precision and high-efficiency human face 3D key point and head posture data acquisition is realized, the accuracy and the robustness of data acquisition are improved, and meanwhile, the acquisition difficulty and the acquisition cost are reduced.

Description

Technical Field

[0001] The present invention relates to the field of face recognition technology, and in particular to a method and system for acquiring 3D key points of a face based on multiple sensors. Background Art

[0002] In recent years, with the sustained and rapid development of China's economy and society, face recognition, expression analysis, augmented reality and other face-related AI functions and applications have shown a rapid development trend. The core foundation of these applications is accurate facial 3D key point positioning and head posture estimation.

[0003] However, in the existing technology, the automatic detection algorithm based on 2D images automatically detects key points from a single 2D image, but the accuracy is often insufficient in scenes such as complex lighting and large-angle postures; based on structured light, ToF, etc. (collectively referred to as RGB-D cameras), only sparse 3D face data can be obtained, while the accuracy of dense 3D point data captured by radar is proportional to the price and is very time-consuming. In addition, the acquisition of head posture can also be directly obtained through devices such as IMU, and the accuracy is also linked to the price. While obtaining high accuracy, the cost is high.

[0004] Therefore, finding a method that can both improve the accuracy of face acquisition and thus improve the accuracy of head posture acquisition and reduce the acquisition cost is a technical problem that needs to be solved urgently by those skilled in the art. Summary of the invention

[0005] In view of this, the present invention proposes a multi-sensor based 3D facial key point acquisition method and system, which can use multiple image acquisition devices and deep neural network models to achieve high-precision and high-efficiency facial 3D key point and head posture data acquisition, improve the accuracy and robustness of data acquisition, and reduce the acquisition difficulty and cost.

[0006] The technical solution of the present invention is achieved in this way:

[0007] In a first aspect, the present invention provides a method for collecting 3D key points of a face based on multiple sensors, comprising the following steps:

[0008] S1. Construct a deep neural network and a bidirectional recurrent neural network model, and train the two network models respectively to obtain a key point acquisition model and a key point mapping model;

[0009] S2. Arrange multiple image acquisition devices, including at least two RGB cameras and one RGBD camera, wherein the image acquisition devices are configured around the face to be acquired, and calibrate the camera external parameters of the multiple image acquisition devices respectively through a calibration plate;

[0010] S3, using multiple image acquisition devices to acquire a multi-view image data sequence of the face to be acquired;

[0011] S4. Deploy a key point acquisition model on the local control device, input the multi-view image data sequence into the key point acquisition model, determine the first frame image data, and use the key point mapping model to determine the predicted key point positions of the face to be collected based on the first frame image data, and calculate the head posture of the face to be collected through the predicted key point positions of the face to be collected.

[0012] On the basis of the above technical solution, preferably, step S1 specifically includes:

[0013] S11, constructing a deep neural network model; wherein the deep neural network model includes a backbone network and a decoupled prediction head, wherein cross-layer links are provided between each stage layer in the backbone network;

[0014] S12, constructing a bidirectional recurrent neural network; wherein the bidirectional recurrent neural network includes two hidden layers and one fully connected layer;

[0015] S13. Use the training data to train the deep neural network model and the bidirectional recurrent neural network to obtain a key point acquisition model and a key point mapping model.

[0016] On the basis of the above technical solution, preferably, the decoupled prediction head includes a key point prediction branch and a head posture prediction branch.

[0017] On the basis of the above technical solution, preferably, the orthogonal projection calculation formula for determining the key point positions of the face to be collected according to the first frame image data is as follows:

[0018]

[0019] Among them, x represents the horizontal coordinate of the 3D key point, y represents the vertical coordinate of the 3D key point, z represents the depth information provided by the RGBD camera, M represents the head posture matrix, K represents the camera intrinsic parameter matrix, s represents the scale factor, u represents the horizontal coordinate of the 2D key point, and v represents the vertical coordinate of the 2D key point.

[0020] More preferably, step S4 specifically includes:

[0021] S41, deploying a key point acquisition model on a local control device, using multiple image acquisition devices to obtain multi-view image data of the face and inputting it into a deep neural network model to obtain predicted 2D key points and head postures of each camera;

[0022] S42, calculating the 3D key points of the face according to the depth value provided by the RGBD camera and the 2D key points predicted by the RGBD camera in S41 according to the orthogonal projection;

[0023] S43, using the 3D key points and head posture based on the RGBD camera calculated in S42, according to the orthogonal projection formula, combined with the head posture and internal and external parameters of the remaining cameras, the 3D key points are back-projected onto the images of each camera, and compared and screened with the predicted values ​​based on the remaining cameras in S41 to verify the error;

[0024] S44, repeat S41-43 until the error is less than the set threshold;

[0025] S45, using a bidirectional recurrent neural network to map the 3D key points calculated in S42 to obtain 392 key points to form a dense face network;

[0026] S46, using the 3D key points based on the RGBD camera and the inverse matrix of the head posture, calculating the 3D face template in the standard posture;

[0027] S47, repeating steps S41-S46 until all true value data are acquired after the acquisition, and bidirectionally verifying the validity of the true value according to the orthogonal projection, and deleting the data with large errors;

[0028] S48. Use the filtered true value data to iteratively train the deep neural network model and the bidirectional recurrent neural network until the accuracy of each network model reaches the preset requirements. Then, each network model after training will be the acquisition model used for the next acquisition.

[0029] Based on the above technical solution, preferably, the calculation process of predicting 2D key points and head posture is as follows:

[0030] The training data is sequentially input into multiple stage layers in the backbone network for data processing to obtain the first feature;

[0031] The first feature is processed using the key point prediction branch and the head posture prediction branch to obtain the initial key point coordinates, the confidence of the initial key point and the rotation matrix of the initial head posture;

[0032] The initial key point coordinates, the initial key point confidences, and the rotation matrix of the initial head posture are normalized to obtain predicted 2D key points and head postures, wherein the predicted key points include 68 key point coordinates and key point confidences.

[0033] On the basis of the above technical solution, preferably, the use of a bidirectional recurrent neural network to map the predicted 2D key points specifically includes:

[0034] Predict the 2D key points and normalize them to get 68 normalized face coordinates;

[0035] Flatten the normalized 3D coordinates of the 68 face coordinates to form a 204-dimensional 1D vector;

[0036] The 204-dimensional one-dimensional vector is sequentially input into two hidden layers and one fully connected layer of the bidirectional recurrent neural network to obtain a 1176-dimensional vector;

[0037] The 1176-dimensional vector is converted into 3D key points, resulting in 392 key points.

[0038] In a second aspect, the present invention provides a multi-sensor based 3D face key point acquisition system, which adopts the above-mentioned 3D face key point acquisition method, including:

[0039] A camera installation calibration module is configured to arrange multiple image acquisition devices, including at least two RGB cameras and one RGBD camera, wherein the image acquisition devices are arranged around the face to be acquired, and the camera external parameters of the multiple image acquisition devices are calibrated respectively through a calibration board;

[0040] A model training module is configured to construct a deep neural network model and a recurrent neural network model, and train the two network models respectively to obtain a key point acquisition model and a key point mapping model;

[0041] A camera installation calibration module is configured to arrange multiple image acquisition devices, including at least two RGB cameras and one RGBD camera, wherein the image acquisition devices are arranged around the face to be acquired, and the camera external parameters of the multiple image acquisition devices are calibrated respectively through a calibration board;

[0042] A data acquisition module, using multiple image acquisition devices to acquire a multi-view image data sequence of a face to be acquired;

[0043] A key point calculation module is configured to input a multi-view image data sequence into a key point acquisition model, determine the first frame image data, and use a key point mapping model to determine the key point position of the face to be acquired based on the first frame image data;

[0044] The head posture calculation module is configured to calculate the head posture of the face to be collected based on the key point positions of the face to be collected.

[0045] In a third aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions enable the computer to implement the method for collecting 3D key points of a face as described above.

[0046] In a fourth aspect, the present invention provides an electronic device, comprising at least one processor, a memory, a communication interface and a bus; wherein:

[0047] The processor, memory, and communication interface communicate with each other via the bus;

[0048] The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to implement the method for collecting 3D key points of a face as described above.

[0049] Compared with the prior art, the method for collecting 3D key points of a face of the present invention has the following beneficial effects:

[0050] (1) By using multiple image acquisition devices and deep neural network models, high-precision and high-efficiency facial 3D key points and head posture data acquisition is achieved, which improves the accuracy and robustness of data acquisition while reducing the difficulty and cost of acquisition;

[0051] (2) By combining the backbone network, decoupled prediction head and bidirectional recurrent neural network into a deep neural network model, the feature extraction capability of the key point acquisition model is improved, and the decoupling of key point positioning and head posture estimation is achieved, which greatly improves the accuracy and generalization ability of the key point acquisition model.

[0052] (3) The accuracy of the model can be improved by completing model iteration through true value collection, thereby capturing higher-precision data true values. The model can be further iterated through higher-precision data to form a data closed loop, thereby improving the robustness and accuracy of 3D face key point matching. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0054] Figure 1 This is a flow chart of the method for collecting 3D key points of a face according to the present invention;

[0055] Figure 2 A schematic diagram of the arrangement of image acquisition equipment for the method for acquiring 3D key points of a face according to the present invention;

[0056] Figure 3 It is a cross-layer link schematic diagram of the face 3D key point acquisition method of the present invention;

[0057] Figure 4 Schematic diagram of the decoupled prediction head of the face 3D key point acquisition method of the present invention. DETAILED DESCRIPTION

[0058] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] like Figure 1 As shown, the present invention provides a method for collecting 3D key points of a face based on multiple sensors, comprising the following steps:

[0060] S1. Arrange multiple image acquisition devices, including at least two RGB cameras and one RGBD camera. The image acquisition devices are configured around the face to be acquired, and the camera external parameters of the multiple image acquisition devices are calibrated respectively through a calibration board.

[0061] like Figure 2 As shown, in one embodiment of the present application, the two RGB cameras are respectively placed at 30 degrees on the left and right sides of the face to be captured, and the RGBD camera is placed between the two RGB cameras and in front of the face to be captured.

[0062] By combining two RGB cameras and one RGBD camera, multi-angle and multi-modal data collection is achieved, the accuracy and completeness of 3D key point positioning are improved, and by fusing images from different sensors, the limitations of a single sensor are overcome to provide more comprehensive and stable facial 3D data.

[0063] S1. Build a deep neural network model and train the deep neural network model to obtain a key point acquisition model;

[0064] Specifically, step S1 specifically includes:

[0065] S11, constructing a deep neural network model; wherein the deep neural network model includes a backbone network and a decoupled prediction head, wherein each stage layer in the backbone network is provided with a cross-layer link; Figure 3 As shown;

[0066] S12, constructing a bidirectional recurrent neural network; wherein the bidirectional recurrent neural network includes two hidden layers and one fully connected layer;

[0067] S13. Use multiple image acquisition devices to obtain a multi-view image data sequence of a human face as training data, and divide the training data into multiple groups. Select any one group of training data and input it into a deep neural network model and a bidirectional recurrent neural network to obtain a key point acquisition model and a key point mapping model.

[0068] In one embodiment of the present application, the preset requirements can be set according to actual conditions, and the present application does not limit this.

[0069] In one embodiment of the present application, the deep neural network model is based on MobileNet V3 as the backbone network.

[0070] In one embodiment of the present application, the loss function of the deep neural network model is mean square error.

[0071] In one embodiment of the present application, the training of the recurrent neural network model requires the specification of a sequence number, and the number of sequences used is 3. It uses every 3 key points to form a small triangle to capture the feature information in the area and form a denser key point.

[0072] like Figure 4 As shown, in one embodiment of the present application, the decoupled prediction head includes a key point prediction branch and a head posture prediction branch.

[0073] In one embodiment of the present application, the orthogonal projection calculation formula for determining the key point positions of the face to be collected based on the first frame image data is as follows:

[0074]

[0075] Among them, x represents the horizontal coordinate of the 3D key point, y represents the vertical coordinate of the 3D key point, z represents the depth information provided by the RGBD camera, M represents the head posture matrix, K represents the camera intrinsic parameter matrix, s represents the scale factor, u represents the horizontal coordinate of the 2D key point, and v represents the vertical coordinate of the 2D key point.

[0076] By combining the backbone network, decoupled prediction head and bidirectional recurrent neural network into a deep neural network model, the feature extraction capability of the key point acquisition model is improved, and the decoupling of key point positioning and head posture estimation is achieved, which greatly improves the accuracy and generalization capability of the key point acquisition model.

[0077] In one embodiment of the present application, a bidirectional recurrent neural network is used to map the predicted key points, and 1176 key points can be obtained, further improving the accuracy of the key point acquisition model.

[0078] In one embodiment of the present application, the calculation process of the true value data is as follows:

[0079] The calculated value is further compared with the predicted value, and the data with large errors are screened out, and the final true value is GT. n ={(L n |P n ),(M n |H n )}; where P nRepresents the 2D key point coordinates, H n represents the head posture, u represents the 2D image x coordinate, v represents the 2D image y coordinate, and K represents the camera intrinsic parameter matrix.

[0080] S3, using multiple image acquisition devices to acquire a multi-view image data sequence of the face to be acquired;

[0081] S4. Deploy a key point acquisition model on the local control device, input the multi-view image data sequence into the key point acquisition model, determine the first frame image data, and use the key point mapping model to determine the predicted key point positions of the face to be collected based on the first frame image data, and calculate the head posture of the face to be collected through the predicted key point positions of the face to be collected.

[0082] Specifically, step S4 specifically includes:

[0083] S41, deploying a key point acquisition model on a local control device, using multiple image acquisition devices to obtain multi-view image data of the face and inputting it into a deep neural network model to obtain predicted 2D key points and head postures of each camera;

[0084] S42, calculating the 3D key points of the face according to the depth value provided by the RGBD camera and the 2D key points predicted by the RGBD camera in S41 according to the orthogonal projection;

[0085] S43, using the 3D key points and head posture based on the RGBD camera calculated in S42, according to the orthogonal projection formula, combined with the head posture and internal and external parameters of the remaining cameras, the 3D key points are back-projected onto the images of each camera, and compared and screened with the predicted values ​​based on the remaining cameras in S41 to verify the error;

[0086] S44, repeat S41-43 until the error is less than the set threshold;

[0087] S45, using a bidirectional recurrent neural network to map the 3D key points calculated in S42 to obtain 392 key points to form a dense face network;

[0088] S46, using the 3D key points based on the RGBD camera and the inverse matrix of the head posture, calculating the 3D face template in the standard posture;

[0089] S47, repeating steps S41-S46 until all true value data are acquired after the acquisition, and bidirectionally verifying the validity of the true value according to the orthogonal projection, and deleting the data with large errors;

[0090] S48. Use the filtered true value data to iteratively train the deep neural network model and the bidirectional recurrent neural network until the accuracy of each network model reaches the preset requirements. Then, each network model after training will be the acquisition model used for the next acquisition.

[0091] Furthermore, the calculation process of predicting key points and head posture is as follows:

[0092] The training data is sequentially input into multiple stage layers in the backbone network for data processing to obtain the first feature;

[0093] The first feature is processed using the key point prediction branch and the head posture prediction branch to obtain the initial key point coordinates, the confidence of the initial key point and the rotation matrix of the initial head posture;

[0094] The initial key point coordinates, the initial key point confidences, and the rotation matrix of the initial head posture are normalized to obtain predicted 2D key points and head postures, wherein the predicted key points include 68 key point coordinates and key point confidences.

[0095] In one embodiment of the present application, a bidirectional recurrent neural network is used to map the predicted 2D key points, specifically including:

[0096] Normalize the predicted 2D key points to obtain 68 normalized face coordinates;

[0097] Flatten the normalized 2D coordinates of the 68 face coordinates to form a 204-dimensional 1D vector;

[0098] The 204-dimensional one-dimensional vector is sequentially input into two hidden layers and one fully connected layer of the bidirectional recurrent neural network to obtain a 1176-dimensional vector;

[0099] The 1176-dimensional vector is converted into 3D key points, resulting in 392 key points.

[0100] In one embodiment of the present application, the loss function of the bidirectional recurrent neural network is the mean square error.

[0101] In one embodiment of the present application, a multi-view image data series is input into a key point acquisition model to obtain a predicted output, and a quality assessment is performed on the predicted output to determine the first frame image data;

[0102] Calculate the coordinates of 3D key points based on the predicted output of the first frame image data and the camera parameters of the image acquisition device;

[0103] Use geometric algorithms to verify whether the head posture is consistent:

[0104] If they are not consistent, repeat the above steps until the head postures of the multi-view image data are consistent;

[0105] The predicted output is mapped through a bidirectional recurrent neural network to generate a dense face network and calculate the 3D coordinates of the face in standard poses;

[0106] The collected face moves according to the collection action. The face at each moment will be matched with the standard face through the key point collection model output to obtain the true value of 2D, 3D key points and head posture.

[0107] It is understandable that if the captured face changes, the multi-view image data series of the captured face after the change will be re-input into the key point acquisition model.

[0108] Specifically, a specific embodiment is used to illustrate:

[0109] Using two RGB cameras (RGB A and RGB C ) and an RGBD camera (RGBD B ) obtain three images, namely I A ,I B and I C ;

[0110] Will I A ,I B and I C Input into the deep neural network model to obtain the model prediction value {O A ,O B ,O C |O n =(L n ,M n )}, where O A Represents RGB A The predicted output, O B Represents RGBD B The predicted output, O C Represents RGB C The predicted output, L n represents the predicted key point coordinates, M n represents the predicted head pose matrix;

[0111] Statistics B The confidence of the middle point is set to a threshold of 0.8. If there are more than 98 points below the threshold, repeat this step and use this moment as the first frame;

[0112] According to O B Key point branch and head posture branch, combined with RBGD B Internal reference and RBGD B The depth information D B , calculate the 3D coordinates, the calculation formula is as follows:

[0113]

[0114] Among them, x BIndicates the horizontal coordinate of the key point, y B Indicates the vertical coordinate of the key point, z B Represents the depth information, provided by the RGBD camera, M B -1 Represents the head posture matrix M B The inverse matrix of T B represents the translation matrix, R B Represents the head posture matrix M B The rotation matrix part, K B -1 Represents the camera intrinsic parameter matrix K B The inverse matrix of , s represents the scale factor, L B Indicates the confidence of the key point.

[0115] According to {O A ,O C}The 3D point matching obtained by the key point branch uses the PnP algorithm to calculate the head posture {H A ,H C}, and with {O A ,O C} A ,M C} for comparison. If it is greater than the set threshold, the next frame will repeat the operations 1 and 2 until the mth frame meets the condition. A Indicates the RGB calculated using the PnP algorithm A The head pose corresponding to the camera, H C Indicates the RGB calculated using the PnP algorithm C The head pose corresponding to the camera;

[0116] {O A ,O B ,O C} is mapped to 392 key points using a bidirectional recurrent neural network to form a dense face network Mesh = {(x i ,y i ,z i )|i=(0,1,2,…,391)}; where x i Indicates the x-coordinate and y-coordinate of the key point i Indicates the y coordinate of the key point, z i Indicates the z coordinate of the key point;

[0117] At the mth frame, according to the moment O B The head posture value in the standard posture is used to calculate the 3D face points. The calculation formula is:

[0118]

[0119] Among them, T B represents the translation matrix, x0 represents the x-coordinate of the 3D face point in the standard pose, y0 represents the y-coordinate of the 3D face point in the standard pose, and z0 represents the z-coordinate of the 3D face point in the standard pose;

[0120] At the {t|t>m}th frame, using the formula Substitute the corresponding predicted values ​​of each camera (L n ,M n ), and calculate the corresponding 2D key points and head posture (P n ,H n ).

[0121] By combining the PnP algorithm, multi-frame data comparison and error screening, high-precision true value data is automatically generated, which improves the quality and consistency of the data and significantly reduces the cost and time of building large-scale data sets.

[0122] This application achieves high-precision and high-efficiency facial 3D key point and head posture data collection by using multiple image acquisition devices and deep neural network models, improves the accuracy and robustness of data collection, and reduces the difficulty and cost of collection.

[0123] The present invention provides a multi-sensor based 3D face key point acquisition system, which adopts the above-mentioned 3D face key point acquisition method, including:

[0124] A model training module is configured to construct a deep neural network model and a recurrent neural network model, and train the two network models respectively to obtain a key point acquisition model and a key point mapping model;

[0125] A camera installation calibration module is configured to arrange multiple image acquisition devices, including at least two RGB cameras and one RGBD camera, wherein the image acquisition devices are arranged around the face to be acquired, and the camera external parameters of the multiple image acquisition devices are calibrated respectively through a calibration board;

[0126] A data acquisition module, using multiple image acquisition devices to acquire a multi-view image data sequence of a face to be acquired;

[0127] A key point calculation module is configured to input a multi-view image data sequence into a key point acquisition model, determine the first frame image data, and use a key point mapping model to determine the key point position of the face to be acquired based on the first frame image data;

[0128] The head posture calculation module is configured to calculate the head posture of the face to be collected based on the key point positions of the face to be collected.

[0129] The present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions enable the computer to implement the method for collecting 3D key points of a face as described above.

[0130] The present invention provides an electronic device, comprising at least one processor, a memory, a communication interface and a bus; wherein the processor, the memory and the communication interface communicate with each other via the bus; the memory stores program instructions executable by the processor, and the processor calls the program instructions to implement the method for collecting 3D key points of a face as described above.

[0131] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for collecting 3D key points of a face based on multiple sensors, characterized in that: The following steps are involved: S1. Construct a deep neural network and a recurrent neural network model, and train the deep neural network and the recurrent neural network model respectively to obtain a key point acquisition model and a key point mapping model; S2. Arrange multiple image acquisition devices, including at least two RGB cameras and one RGBD camera, wherein the image acquisition devices are configured around the face to be acquired, and calibrate the camera external parameters of the multiple image acquisition devices respectively through a calibration plate; S3, using multiple image acquisition devices to acquire a multi-view image data sequence of the face to be acquired; S4. Deploy a key point acquisition model on the local control device, input the multi-view image data sequence into the key point acquisition model, determine the first frame image data, and use the key point mapping model to determine the predicted key point positions of the face to be collected based on the first frame image data, and calculate the head posture of the face to be collected through the predicted key point positions of the face to be collected.

2. The method for collecting 3D key points of a face based on multiple sensors as claimed in claim 1, characterized in that: S1 specifically includes: S11, constructing a deep neural network model; wherein the deep neural network model includes a backbone network and a decoupled prediction head, wherein cross-layer links are provided between each stage layer in the backbone network; S12, constructing a bidirectional recurrent neural network; wherein the bidirectional recurrent neural network includes two hidden layers and one fully connected layer; S13. Use the training data to train the deep neural network model and the bidirectional recurrent neural network to obtain a key point acquisition model and a key point mapping model.

3. The method for collecting 3D key points of a face based on multiple sensors as claimed in claim 2, characterized in that: The decoupled prediction head includes a key point prediction branch and a head posture prediction branch.

4. The method for collecting 3D key points of a face based on multiple sensors as claimed in claim 1, characterized in that: The orthogonal projection calculation formula for determining the key point positions of the face to be collected based on the first frame image data is as follows: Among them, x represents the horizontal coordinate of the 3D key point, y represents the vertical coordinate of the 3D key point, z represents the depth information provided by the RGBD camera, M represents the head posture matrix, K represents the camera intrinsic parameter matrix, s represents the scale factor, u represents the horizontal coordinate of the 2D key point, and v represents the vertical coordinate of the 2D key point.

5. The method for collecting 3D key points of a face based on multiple sensors as claimed in claim 3, characterized in that: Step S4 specifically includes: S41, deploying a key point acquisition model on a local control device, using multiple image acquisition devices to obtain multi-view image data of the face and inputting it into a deep neural network model to obtain predicted 2D key points and head postures of each camera; S42, calculating the 3D key points of the face according to the depth value provided by the RGBD camera and the 2D key points predicted by the RGBD camera in S41 according to the orthogonal projection; S43, using the 3D key points and head posture based on the RGBD camera calculated in S42, according to the orthogonal projection formula, combined with the head posture and internal and external parameters of the remaining cameras, the 3D key points are back-projected onto the images of each camera, and compared and screened with the predicted values ​​based on the remaining cameras in S41 to verify the error; S44, repeat S41-43 until the error is less than the set threshold; S45, using a bidirectional recurrent neural network to map the 3D key points calculated in S42 to obtain 392 key points to form a dense face network; S46, using the 3D key points based on the RGBD camera and the inverse matrix of the head posture, calculating the 3D face template in the standard posture; S47, repeating steps S41-S46 until all true value data are acquired after the acquisition, and bidirectionally verifying the validity of the true value according to the orthogonal projection, and deleting the data with large errors; S48. Use the filtered true value data to iteratively train the deep neural network model and the bidirectional recurrent neural network until the accuracy of each network model reaches the preset requirements. Then, each network model after training will be the collection model used for the next collection.

6. The method for collecting 3D key points of a face based on multiple sensors as claimed in claim 5, characterized in that: The calculation process of predicting 2D key points and head posture is as follows: The training data is sequentially input into multiple stage layers in the backbone network for data processing to obtain the first feature; The first feature is processed using the key point prediction branch and the head posture prediction branch to obtain the initial key point coordinates, the confidence of the initial key point and the rotation matrix of the initial head posture; The initial key point coordinates, the initial key point confidences, and the rotation matrix of the initial head posture are normalized to obtain predicted 2D key points and head postures, wherein the predicted key points include 68 key point coordinates and key point confidences.

7. The multi-sensor based face 3D key point acquisition method as claimed in claim 6, characterized in that: The use of a bidirectional recurrent neural network to map the predicted 2D key points specifically includes: Predict the 2D key points and normalize them to get 68 normalized face coordinates; Flatten the normalized 3D coordinates of the 68 face coordinates to form a 204-dimensional 1D vector; The 204-dimensional one-dimensional vector is sequentially input into two hidden layers and one fully connected layer of the bidirectional recurrent neural network to obtain a 1176-dimensional vector; The 1176-dimensional vector is converted into 3D key points, and 392 three-dimensional key points are obtained.

8. A multi-sensor based face 3D key point acquisition system, characterized in that: The method for collecting 3D key points of a face as claimed in any one of claims 1 to 7 comprises: A model training module is configured to construct a deep neural network model and a recurrent neural network model, and train the two network models respectively to obtain a key point acquisition model and a key point mapping model; A camera installation calibration module is configured to arrange multiple image acquisition devices, including at least two RGB cameras and one RGBD camera, wherein the image acquisition devices are arranged around the face to be acquired, and the camera external parameters of the multiple image acquisition devices are calibrated respectively through a calibration board; A data acquisition module, using multiple image acquisition devices to acquire a multi-view image data sequence of a face to be acquired; A key point calculation module is configured to input a multi-view image data sequence into a key point acquisition model, determine the first frame image data, and use a key point mapping model to determine the key point position of the face to be acquired based on the first frame image data; The head posture calculation module is configured to calculate the head posture of the face to be collected based on the key point positions of the face to be collected.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions enable the computer to implement the method for collecting 3D key points of a face as described in any one of claims 1 to 7.

10. An electronic device, characterized in that: Includes at least one processor, a memory, a communication interface and a bus; wherein, The processor, memory, and communication interface communicate with each other via the bus; The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to implement the method for collecting 3D key points of a face as described in any one of claims 1-7.

Images