Three-dimensional face model training method and device, electronic equipment and readable storage medium

Abstract

The application provides a three-dimensional face model training method and device, electronic equipment and readable storage medium. After obtaining a face image, a face region image in the face image is cropped, and the face region image contains multiple face feature points. A virtual perspective camera is constructed, perspective 3D position information and 2D position information of each face feature point in the face region image in the virtual perspective camera space are obtained, and then the constructed neural network model is trained by using the face region image carrying the perspective 3D position information and 2D position information, and a three-dimensional face model capable of detecting three-dimensional information of the face feature points is obtained. In the scheme, the three-dimensional face model capable of detecting three-dimensional information of the face feature points is used to realize accurate detection of 3D key points, and the model is trained in combination with the perspective 3D position information and 2D position information, so that the model obtained by training has better effect on face feature point alignment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a three-dimensional face model training method and device, electronic equipment and readable storage medium. BACKGROUND

[0002] With the rise of live entertainment and short videos, users' demand for face effects such as beautifying and slimming faces and automatic makeup is also increasing. The three-dimensional face reconstruction technology based on deep learning has made great breakthroughs as the basis for realizing these functions. The three-dimensional face reconstruction method used in the prior art is mostly based on a three-dimensional deformable Basel face model, that is, the three-dimensional key point information of a face is reconstructed by predicting an expression basis vector and a shape basis vector of the face. Although the space constraint of this method is relatively strong, it is very difficult to obtain the expression basis vector and the shape basis vector, and currently, the 3D key points and 2D key points of a face are generally calculated by least squares fitting to obtain the expression basis vector and the shape basis vector. This method itself has certain problems and it is difficult to obtain accurate solutions. SUMMARY

[0003] The present application aims to, for example, provide a three-dimensional face model training method, device, electronic equipment and readable storage medium, which can make the three-dimensional face model trained have better effects on face feature point alignment.

[0004] Embodiments of the present application can be implemented as follows:

[0005] In a first aspect, the present application provides a three-dimensional face model training method, which comprises:

[0006] Obtaining a face image and cropping a face region image in the face image, the face region image containing multiple face feature points;

[0007] Constructing a virtual perspective camera to obtain perspective 3D position information and 2D position information of each face feature point in the face region image in the virtual perspective camera space;

[0008] Training a neural network model constructed using the face region image pair carrying the perspective 3D position information and 2D position information to obtain a three-dimensional face model that can be used to detect three-dimensional information of face feature points.

[0009] In an optional implementation, the step of obtaining perspective 3D position information and 2D position information of each face feature point in the face region image in the virtual perspective camera space comprises:

[0010] obtain 3D position information of each of the facial feature points in the world coordinate system in the face region image, and obtain 2D position information of each of the facial feature points in the image coordinate system;

[0011] calculate internal and external parameters of the virtual perspective camera according to the 3D position information and the 2D position information;

[0012] obtain perspective 3D position information and 2D position information of each of the facial feature points in the virtual perspective camera space according to the internal and external parameters and the 3D position information.

[0013] In an optional implementation, the internal and external parameters include a pose parameter and an intrinsic matrix;

[0014] The step of obtaining perspective 3D position information and 2D position information of each of the facial feature points in the virtual perspective camera space according to the internal and external parameters and the 3D position information includes:

[0015] calculate perspective 3D position information of the facial feature points in the virtual perspective camera space according to the pose parameter and the 3D position information;

[0016] calculate 2D position information of the facial feature points in the virtual perspective camera space according to the perspective 3D position information and the intrinsic matrix.

[0017] In an optional implementation, the step of obtaining 2D position information of each of the facial feature points in the image coordinate system includes:

[0018] perform re-projection processing on the 3D position information of each of the facial feature points in the world coordinate system to obtain 2D position information of the facial feature points in the image coordinate system constructed based on the face image;

[0019] revise the obtained 2D position information based on the cropping information of the face region image to obtain 2D position information of the facial feature points in the image coordinate system constructed based on the face region image.

[0020] In an optional implementation, the method further includes:

[0021] scale the resolution size of the cropped face region image to a preset resolution range;

[0022] The step of revising the obtained 2D position information based on the cropping information of the face region image includes:

[0023] revise the obtained 2D position information based on the cropping information of the face region image and the preset resolution range.

[0024] In an optional implementation, the step of training the neural network model constructed using the face region image pair carrying the perspective 3D position information and 2D position information comprises:

[0025] inputting the face region image into the constructed neural network model, and outputting 3D position prediction information of each of the face feature points in the face region image;

[0026] performing re-projection processing on the 3D position prediction information to obtain corresponding 2D position prediction information;

[0027] constructing a loss function based on the perspective 3D position information, the 3D position prediction information, the 2D position information, and the 2D position prediction information, and continuing to train the neural network model guided by the loss function until a preset requirement is met, to obtain the three-dimensional face model.

[0028] In an optional implementation, the step of constructing a loss function based on the perspective 3D position information, the 3D position prediction information, the 2D position information, and the 2D position prediction information comprises:

[0029] constructing a first loss term based on a difference between the perspective 3D position information and the 3D position prediction information;

[0030] constructing a second loss term based on a difference between the 2D position information and the 2D position prediction information;

[0031] adding the first loss term and the second loss term according to a set weight to obtain the loss function.

[0032] In an optional implementation, before the step of inputting the face region image into the constructed neural network model, the method further comprises:

[0033] obtaining pixel values of each pixel point contained in the face region image;

[0034] normalizing the pixel values of each pixel point to a preset pixel range.

[0035] In an optional implementation, the method further comprises:

[0036] obtaining a to-be-detected image, and cropping a to-be-detected region image in the to-be-detected image, the to-be-detected region image containing a face region, and the face region containing a plurality of to-be-detected key points;

[0037] constructing a virtual perspective space, inputting the to-be-detected region image into the three-dimensional face model, and outputting three-dimensional information of each of the to-be-detected key points in the virtual perspective space.

[0038] Secondly, this application provides a three-dimensional face model training device, the device comprising:

[0039] The acquisition module is used to acquire a face image and crop out the face region image from the face image, wherein the face region image contains multiple face feature points;

[0040] The construction module is used to construct a virtual perspective camera to obtain the perspective 3D position information and 2D position information of each facial feature point in the face region image in the virtual perspective camera space;

[0041] The training module is used to train the constructed neural network model using face region images carrying the perspective 3D position information and 2D position information to obtain a 3D face model that can be used to detect 3D information of face feature points.

[0042] Thirdly, this application provides an electronic device including one or more storage media and one or more processors communicating with the storage media. The one or more storage media store machine-executable instructions that can be executed by the processor. When the electronic device is running, the processor executes the machine-executable instructions to perform the method steps described in any of the foregoing embodiments.

[0043] Fourthly, this application provides a computer-readable storage medium storing machine-executable instructions, which, when executed, implement the method steps described in any of the foregoing embodiments.

[0044] The beneficial effects of the embodiments of this application include, for example:

[0045] This application provides a method, apparatus, electronic device, and readable storage medium for training a 3D face model. After acquiring a face image, a face region image containing multiple facial feature points is cropped from the face image. A virtual perspective camera is constructed, and the 3D and 2D perspective position information of each facial feature point in the face region image within the virtual perspective camera space is obtained. The constructed neural network model is then trained using the face region image carrying the 3D and 2D perspective position information, resulting in a 3D face model that can be used to detect facial feature points. In this scheme, a 3D face model capable of detecting facial feature points is trained, achieving accurate detection of 3D key points. Furthermore, the combination of perspective 3D and 2D position information during model training results in a model with better alignment of facial feature points. Attached Figure Description

[0046] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 A schematic diagram illustrating an application scenario of the 3D face model training method provided in this embodiment of the application;

[0048] Figure 2 A flowchart of a three-dimensional face model training method provided in the embodiments of this application;

[0049] Figure 3 for Figure 2 A flowchart of the sub-steps included in step S103;

[0050] Figure 4 for Figure 3 A flowchart of the sub-steps included in step S1031;

[0051] Figure 5 for Figure 2 A flowchart of the sub-steps included in step S105;

[0052] Figure 6 for Figure 5 A flowchart of the sub-steps included in step S1055;

[0053] Figure 7 A flowchart of the detection method provided in the embodiments of this application;

[0054] Figure 8 This refers to the three-dimensional facial information detected from a frontal view in the embodiments of this application;

[0055] Figure 9 This refers to the three-dimensional facial information detected from a side view in the embodiments of this application;

[0056] Figure 10 This refers to the three-dimensional facial information detected from a low-angle view in the embodiments of this application;

[0057] Figure 11 A structural block diagram of the electronic device provided in the embodiments of this application;

[0058] Figure 12 A functional block diagram of the three-dimensional face model training device provided in the embodiments of this application.

[0059] Icon: 100 - live broadcast providing terminal; 110 - processor; 120 - memory; 130 - communication module; 140 - three-dimensional face model training device; 141 - acquisition module; 142 - construction module; 143 - training module; 200 - live broadcast server; 300 - live broadcast receiving terminal. DETAILED DESCRIPTION

[0060] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations.

[0061] Therefore, the detailed description of the embodiments of the present application provided below in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. All other embodiments obtained by a person of ordinary skill in the art without creative work based on the embodiments in the present application belong to the scope of protection of the present application.

[0062] It should be noted that: similar reference numbers and letters represent similar items in the following drawings, therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0063] In the description of the present application, it should be noted that if the terms "first", "second", etc. are used, they are only used for differentiation and description, and cannot be understood as indicating or implying relative importance.

[0064] It should be noted that the features in the embodiments of the present application can be combined with each other without conflict.

[0065] Please refer to Figure 1 A possible application scenario of the three-dimensional face model training method provided by the embodiments of the present application is shown in the figure, which includes a live broadcast providing terminal 100, a live broadcast server 200, and a live broadcast receiving terminal 300. The live broadcast server 200 is in communication connection with the live broadcast providing terminal 100 and the live broadcast receiving terminal 300 respectively, and is used to provide live broadcast services for the live broadcast providing terminal 100 and the live broadcast receiving terminal 300.

[0066] For example, the live broadcast providing terminal 100 can send a live video stream to the live broadcast server 200, and the audience can access the live broadcast server 200 through the live broadcast receiving terminal 300 to watch the live video. Among them, the live video stream pushed by the live broadcast server 200 can be a video stream currently being live broadcasted in the live broadcast platform or a complete video stream formed after the live broadcast is completed.

[0067] It can be understood that Figure 1 The scene shown is only one possible example, and in other possible embodiments, the scene can also only include Figure 1 Some of the components shown or can also include other components.

[0068] In this embodiment, the live broadcast providing terminal 100 and the live broadcast receiving terminal 300 can be, but are not limited to, a smart phone, a personal digital assistant, a tablet computer, a personal computer, a notebook computer, a virtual reality terminal device, an augmented reality terminal device, etc. Among them, the live broadcast providing terminal 100 and the live broadcast receiving terminal 300 can be installed with an Internet product for providing an Internet live broadcast service, for example, the Internet product can be an application APP, a Web page, a mini program, etc. related to the Internet live broadcast service used in a computer or a smart phone.

[0069] In this embodiment, the scene can also include an image acquisition device for acquiring the face image of the anchor, and the image acquisition device can be, but is not limited to, a camera, a digital camera, a depth camera, etc. The image acquisition device can be directly installed or integrated in the live broadcast providing terminal 100. For example, the image acquisition device can be a camera configured on the live broadcast providing terminal 100, and other modules or components in the live broadcast providing terminal 100 can receive the video and image sent from the image acquisition device via an internal bus. Alternatively, the image acquisition device can be independent of the live broadcast providing terminal 100, and the two can communicate through wired or wireless means. It should be noted that the above is only one possible implementation scenario of the three-dimensional face model training method provided by the present application, and in addition, the three-dimensional face model training method can also be used to process a separate picture that has been collected.

[0070] Figure 2 The flowchart of the three-dimensional face model training method provided by the embodiment of the present application is shown, and the three-dimensional face model training method can be executed by a three-dimensional face model training device. The three-dimensional face model training device can be realized by software and / or hardware, and can be configured in an electronic device. The electronic device can be a live broadcast server 200 providing a live broadcast platform or the live broadcast providing terminal 100 described above. The detailed steps of the three-dimensional face model training method are as follows.

[0071] S101, acquiring a face image and cropping a face region image in the face image, the face region image containing multiple face feature points.

[0072] S103, constructing a virtual perspective camera to obtain perspective 3D position information and 2D position information of each face feature point in the face region image in the virtual perspective camera space.

[0073] S105, a neural network model constructed by using the face region image pair carrying the perspective 3D position information and the 2D position information is trained to obtain a three-dimensional face model that can be used to detect three-dimensional information of a face feature point.

[0074] In this embodiment, the image acquisition device can acquire a face image containing the anchor face, and the live broadcast providing terminal 100 can analyze and process the acquired face image to train a three-dimensional face model. Alternatively, the image acquisition device can also transmit the face image to the live broadcast server 200, and the live broadcast server 200 can analyze and process the face image to train a three-dimensional face model.

[0075] The face image acquired by the image acquisition device can include multiple face images of the anchor in different shooting angles and different expression states of the anchor, such as a front face angle, a side face angle, a laughing expression, a pursed mouth expression, and the like. In addition, the acquired multiple face images can also include face images in different lighting conditions, so that the generalization ability of the model trained subsequently can be improved.

[0076] The acquired face image contains not only the anchor face, but also the environment image of the environment where the anchor is located and the image of multiple interference objects. In order to focus on face processing and avoid the interference of environment images or other object images, the face region image in the face image can be cropped.

[0077] In this embodiment, in order to reduce the complexity of processing, there should be only one face in each face image, so that the face region image containing the face can be directly cropped.

[0078] The cropped face region image contains multiple face feature points, which can be key points constituting a face contour and facial features. In the case where the viewpoint of the shooting device and the device parameters of the shooting device are determined, the 3D position information of each face feature point in the face region image can be determined, that is, the 3D position information of each face feature point in the world coordinate system. Based on the 3D position information of the face feature points, projection transformation can be performed to determine the 2D position information of the face feature points in the image coordinate system.

[0079] The face image is cropped to intercept the face region image contained therein, and the coordinate information of each face feature point in the face region image will change in the original coordinate system of the face image. That is, the image coordinate system of the face region image changes relative to the image coordinate system of the face image.

[0080] And, due to the change of the image coordinate system, the view point of each facial feature point in the face region image also changes, meaning that each facial feature point in the face region image needs to be transformed to the camera space under the changed view point. Therefore, in the present embodiment, a virtual perspective camera is constructed, and the view point of the virtual perspective camera is established based on the changed view point. The coordinate information of the facial feature points in the face region image needs to be perspective transformed into the virtual perspective camera space.

[0081] Therefore, in the present embodiment, after the virtual perspective camera is constructed, the perspective 3D position information and the 2D position information of each facial feature point in the virtual perspective camera space can be obtained. For the face region image, a plurality of pairs of perspective 3D position information and 2D position information corresponding to a plurality of facial feature points can be obtained.

[0082] In the present embodiment, a neural network model can be constructed in advance, which can be, but is not limited to, a deep neural network model, such as a ResNet18 network model or a MobileNetV1 network model.

[0083] The face region image is taken as the input of the neural network model, and the pairs of perspective 3D position information and 2D position information of the facial feature points in the face region image are taken as the label information of the face region image, so as to train the neural network model. The neural network model can learn the perspective 3D position information of each facial feature point in the face region image, and the learned perspective 3D position information can achieve accurate projection with its corresponding 2D position information. Therefore, the neural network model is trained based on the pairs of perspective 3D position information and 2D position information, and the trained face three-dimensional model can accurately realize the detection of face three-dimensional information.

[0084] In the present embodiment, when training the neural network model, the loss function can be set as a guide for training, which can be constructed based on the error between the input and the output of the neural network model. In the model training process, the loss function is minimized, and the model parameters of the neural network model can be adjusted after each iteration training and continue to train until the preset requirements are met, and the face three-dimensional model can be obtained.

[0085] The preset requirements can be that the loss function reaches convergence and no longer decreases, or that the number of iterations reaches a preset maximum number of iterations, or that the iteration time length reaches a preset maximum iteration time length, etc.

[0086] The three-dimensional face model training method provided in the embodiment adopts a three-dimensional face model that can detect three-dimensional information of face feature points, and can be used to realize accurate detection of 3D key points. Moreover, the model is trained in combination with perspective 3D position information and 2D position information, so that the model obtained through training has better effect on face feature point alignment.

[0087] As known from the above, after the face region image is cropped from the face image, the view point of each face feature point in the image will change, and a virtual perspective camera needs to be constructed to convert the position information of each feature point into position information in the virtual perspective camera space. Please refer to Figure 3 In the embodiment, the position information of each face feature point in the virtual perspective camera space can be determined in the following manner:

[0088] S1031, 3D position information of each face feature point in the face region image in the world coordinate system is obtained, and 2D position information of each face feature point in the image coordinate system is obtained.

[0089] S1033, internal and external parameters of the virtual perspective camera are calculated according to the 3D position information and the 2D position information.

[0090] S1035, perspective 3D position information and 2D position information of each face feature point in the virtual perspective camera space are obtained according to the internal and external parameters and the 3D position information.

[0091] In the embodiment, the cropping operation for obtaining the face region image from the face image has no effect on the 3D position information of each face feature point in the world coordinate system. However, the cropping operation will cause the 2D position information of each face feature point in the face region image to change relative to the 2D position information of the face feature point in the face image, that is, the image coordinate system of the face region image changes relative to the image coordinate system of the face image.

[0092] When the model is trained by using the 3D position information of the face feature point and the corresponding 2D position information, the 2D position information of the face feature point in the image coordinate system of the face region image needs to be used.

[0093] When the re-projection is performed based on the 3D position information of the face feature point, the 2D position information of the face feature point in the image coordinate system of the face image is obtained. Based on this, please refer to Figure 4 In the embodiment, the 2D position information of each face feature point can be determined in the following manner:

[0094] S10311, re-project the 3D position information of each of the facial feature points in the world coordinate system to obtain 2D position information of the facial feature points in an image coordinate system constructed based on the face image.

[0095] S10313, revise the obtained 2D position information based on the cropping information of the face region image to obtain 2D position information of the facial feature points in an image coordinate system constructed based on the face region image.

[0096] In this embodiment, when re-projecting the 3D position information of each facial feature point in the world coordinate system to obtain 2D position information of the facial feature points in an image coordinate system constructed based on the face image, it can be implemented based on the internal and external parameters of the shooting device. For example, the 3D position information of the facial feature points can be multiplied by the internal parameter matrix of the shooting device, wherein the internal parameter matrix is composed of the focal length information of the shooting device. Then divided by the vector composed of the z-axis position of the 3D position information in the world coordinate system, thereby obtaining a 3xN matrix, wherein N represents the number of facial feature points. Taking the elements of the first 2 rows of the matrix, the 2D position information of the facial key points in the image coordinate system constructed based on the face image can be obtained.

[0097] When cropping the face image to obtain the face region image, an external frame that can frame the face region can be first determined, and then the cropping range is determined based on the external frame. For example, the edge position and center point position of the external frame can be determined, and the length of the cropping can be determined based on the edge position and center point position. Then, a certain range is expanded outward based on the external frame to determine a cropping range. The face region image is obtained by cropping the face image based on the cropping range.

[0098] The 2D position information of the same facial feature point on the face region image will change relative to its 2D position information on the face image. In the above case of obtaining the 2D position information of the facial feature points on the face image, the obtained 2D position information can be revised based on the cropping information, thereby obtaining the 2D position information of the facial feature points on the face region image.

[0099] The cropping information can be the center point position of the external frame, the cropping length, etc. The 2D position information of the facial feature points is revised based on the cropping information, so that the 2D position information of the facial feature points is suitable for the position information of the image coordinate system of the cropped face region image.

[0100] In addition, in the embodiment, in order to facilitate analysis and processing of the model, the resolution size of the cropped face region image can be scaled to a preset resolution range when the face region image is used to train the neural network model subsequently. In this way, the face region images input into the neural network model have a uniform resolution, which facilitates model learning and processing.

[0101] The preset resolution range can be, for example, 120x120 or 160x160, and the embodiment is not limited in particular. In the embodiment, the resolution size of the face region image can be scaled to the preset resolution range by using a bilinear interpolation method.

[0102] On this basis, when the 2D position information of the face feature points on the face image is revised to 2D position information on the face region image, the 2D position information can be revised based on the cropping information of the face region image and the preset resolution range.

[0103] For example, the 2D position information can be revised based on the cropping information in the above manner, and then the abscissa and ordinate in the 2D position information can be multiplied by the ratio between the preset resolution range and the cropping length.

[0104] After the paired 3D position information and 2D position information of each face feature point is determined, the 3D position information of the face feature points in the world coordinate needs to be transformed to the virtual perspective camera space subsequently, and therefore the intrinsic and extrinsic parameters of the virtual perspective camera need to be obtained first.

[0105] In order to reduce the reprojection error between the 3D position information and the 2D position information of the face feature points after perspective transformation, in the embodiment, the intrinsic and extrinsic parameters of the virtual perspective camera can be calculated in combination with the 3D position information and the 2D position information.

[0106] In one possible implementation, the intrinsic matrix K of the virtual perspective camera can be obtained first, and the intrinsic matrix is as follows:

[0107]

[0108] wherein f represents the focal length information of the virtual perspective camera, and can be set to 500 generally. Then, the camera extrinsic parameter of the virtual perspective camera is determined by combining the intrinsic matrix of the virtual perspective camera, the 3D position information of the face feature points, the revised 2D position information, and using a PnP (Perspective-n-Point) solving algorithm. The PnP solving algorithm is a method for determining the camera extrinsic parameter of the virtual perspective camera by using a method of minimizing the reprojection error based on multiple pairs of matched 3D position information and 2D position information.

[0109] In this embodiment, the intrinsic and extrinsic parameters of the virtual perspective camera include a rotation matrix and a translation vector, wherein the rotation matrix is a 3x3 matrix and the translation vector is a 3-dimensional vector.

[0110] In this embodiment, the perspective 3D position information of the face key points in the virtual perspective camera space can be calculated according to the pose parameters and the 3D position information.

[0111] Optionally, based on the obtained rotation matrix and translation vector, the 3D position information of the face feature points in the world coordinate system can be transformed into the virtual perspective camera space through a similarity transformation, and the transformation process can be as follows:

[0112] P c =R·P w +t

[0113] wherein P c represents the virtual 3D position information of the face feature points in the virtual perspective camera space, P w represents the 3D position information of the face feature points in the world coordinate system. P c and P w are both 3xN matrices, wherein N represents the number of face feature points. Each column in the matrix represents the coordinate information (x, y, z) of each face feature point.

[0114] According to the obtained perspective 3D position information and the intrinsic parameter matrix, the 2D position information of the face key points in the virtual perspective camera space is calculated.

[0115] The virtual 3D position information transformed into the virtual perspective camera space is projected to obtain the 2D position information of the face feature points in the virtual perspective camera space. The projection process can be implemented according to the following formula:

[0116]

[0117] wherein Z represents a vector composed of the position of the virtual 3D position information of the face feature points on the z-axis in the virtual perspective camera space (i.e., the third column vector of the above P c matrix). P uv is a 3xN matrix, and the elements of the first two rows of the matrix constitute the 2D position information in the virtual perspective camera space obtained by projection.

[0118] After obtaining the virtual 3D position information and the 2D position information of each face feature point in the virtual perspective camera space in the face region image in the above manner, the face region image, the virtual 3D position information, and the 2D position information can be stored together, for example, stored as a binary file, so as to be used for subsequent model training.

[0119] In this embodiment, before the neural network model is trained by using the face region image, the face region image can be pre-processed as follows:

[0120] The pixel value of each pixel point contained in the face region image is obtained, and the pixel value of each pixel point is normalized to a preset pixel range.

[0121] The face region image is an RGB image, and the pixel value of each pixel point is in the range of 0-255. For the normalization processing of the pixel value of each pixel point, the pixel value can be divided by 255, so as to normalize the pixel value to the preset pixel range [0, 1].

[0122] Through the normalization processing of the pixel value of the pixel point of the face region image, it can be avoided that the input layer of the neural network model is too large, and then the gradient transferred to the input layer is too large when back propagation, which causes gradient overflow, and then causes the problem that the training is easy to converge.

[0123] Please refer to Figure 5 In this embodiment, when the neural network model is trained by using the face region image, the virtual 3D position information, and the 2D position information, the following method can be used:

[0124] S1051, the face region image is imported into the constructed neural network model, and the 3D position prediction information of each face feature point in the face region image is output.

[0125] S1053, the 3D position prediction information is re-projected to obtain the corresponding 2D position prediction information.

[0126] S1055, a loss function is constructed based on the perspective 3D position information, the 3D position prediction information, the 2D position information, and the 2D position prediction information, and the neural network model is continuously trained under the guidance of the loss function until the preset requirement is met, and the three-dimensional face model is obtained.

[0127] In this embodiment, the face region image is used as the input of the neural network model, and the output result of the model is the 3D position prediction information of the face feature point obtained by the model based on the analysis and processing of the face region image.

[0128] Each face feature point of the face region image has real virtual 3D position information, and the purpose of training the model is to make the 3D position prediction information output by the model as close as possible to the real virtual 3D position information.

[0129] In addition, in order to ensure that the obtained 3D position prediction information can realize accurate projection of 2D information, re-projection can also be performed based on the obtained 3D position prediction information to obtain corresponding 2D position prediction information. The re-projection manner can refer to the re-projection manner in the virtual perspective camera space described above, and will not be described here.

[0130] Finally, the model is trained in combination with the real virtual 3D position information and 2D position information of each facial feature point and the 3D position prediction information and 2D position prediction information output by the model.

[0131] In order to minimize the difference between the position information of the facial feature points output by the model and the real position information thereof, a loss function can be constructed based on the output of the model and the real label, and the model is trained based on the loss function as a guide.

[0132] Please refer to Figure 6 In the embodiment, the loss function can be constructed in the following manner:

[0133] S10551, a first loss term is constructed based on the difference between the perspective 3D position information and the 3D position prediction information.

[0134] S10553, a second loss term is constructed based on the difference between the 2D position information and the 2D position prediction information.

[0135] S10555, the first loss term and the second loss term are added according to a set weight to obtain the loss function.

[0136] In the embodiment, the first loss term constructed can be as follows:

[0137]

[0138] Wherein, N represents the number of facial feature points in the facial region image, represents the 3D position prediction information of the i-th facial feature point output by the model, represents the real virtual 3D position information of the i-th facial feature point.

[0139] In addition, the second loss term constructed can be as follows:

[0140]

[0141] Wherein, represents the 2D position prediction information obtained by re-projecting the 3D position prediction information of the i-th facial feature point, represents the real 2D position information of the i-th facial feature point.

[0142] The first loss term and the second loss term are added according to a set weight, and a loss function shown in the following formula is obtained:

[0143] L total =λL 3d +(1-λ)L 2d

[0144] Wherein, λ represents the set weight.

[0145] In this embodiment, the random gradient descent method can be used to globally optimize the loss function, and the gradient matrix of the neural network model is calculated, and then the model parameters of the neural network model are updated according to the back propagation principle. After multiple rounds of iterative training, if the preset requirements are met, such as reaching the set maximum number of iterations or the loss function converges and no longer decreases, the trained three-dimensional face model can be obtained.

[0146] The three-dimensional face model obtained by the above method can be used for three-dimensional information detection of feature points in an image containing a face. Please refer to Figure 7 In this embodiment, the application of the three-dimensional face model can be realized in the following way:

[0147] S201, obtaining a to-be-detected image, and cropping a to-be-detected region image in the to-be-detected image, the to-be-detected region image containing a face region, and the face region containing a plurality of to-be-detected key points.

[0148] S203, constructing a virtual perspective space, importing the to-be-detected region image into the three-dimensional face model, and outputting three-dimensional information of each to-be-detected key point in the virtual perspective space.

[0149] In this embodiment, the to-be-detected image is an image containing a face, for example, an image containing the face of an anchor. For the to-be-detected image, the same way as described above can be used to crop the to-be-detected region image therein, that is, the background image, the image of the interfering object, etc. in the to-be-detected image is cropped and filtered out.

[0150] The virtual perspective space for the to-be-detected region image is constructed, and the three-dimensional face model obtained by the training can analyze and process the to-be-detected region image, so as to output the three-dimensional information of each to-be-detected key point in the to-be-detected region image in the virtual perspective space.

[0151] Wherein, the three-dimensional information of each to-be-detected key point is 3D position information. When the face in the to-be-detected region image is at different angles, the 3D position information of the detected to-be-detected key points can obtain the 3D face image shown in Figure 8 、 9 , 10.

[0152] In this embodiment, the 3D position information of each to-be-detected key point can also be used to construct a face mesh of the face. The face mesh includes each to-be-detected key point and a plurality of triangular slices. Each triangular slice is formed by connecting three to-be-detected key points adjacent to each other as vertices.

[0153] Based on the obtained face mesh information, the face recognition tracking, face expression processing, face special effect processing, and other applications can be applied.

[0154] Figure 11 A structural schematic diagram of an electronic device is provided in this embodiment, which can be the live server described above. As shown in the structural schematic diagram of the electronic device, Figure 11 The electronic device includes a memory 120, a processor 110, and a communication module 130. The memory 120, the processor 110, and the communication module 130 are directly or indirectly electrically connected to each other to realize data transmission or interaction. For example, these elements can be electrically connected to each other through one or more communication buses or signal lines. The memory 120 is configured to store programs or data. The memory 120 can be, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), and the like.

[0155] The processor 110 is configured to read / write data or programs stored in the memory 120, and perform the three-dimensional face model training method of any of the above embodiments.

[0156] The communication module 130 is configured to establish a communication connection between the electronic device and other communication terminals through a network, and to transmit and receive data through the network.

[0157] It should be understood that, Figure 11 The structure shown in the structural schematic diagram of the electronic device can include more or fewer components than those shown in Figure 11 or have a different configuration than that shown in Figure 11 . Figure 11 The components shown in the structural schematic diagram of the electronic device can be implemented in hardware, software, or a combination thereof.

[0158] To perform the corresponding steps in the above-mentioned embodiments and various possible manners, an implementation manner of a three-dimensional face model training apparatus is given below, which can optionally adopt the three-dimensional face model training apparatus shown in the above-mentioned Figure 11 device structure of the electronic device.

[0159] Further, please refer to Figure 12 , Figure 12 A functional module diagram of a three-dimensional face model training apparatus 140 provided by an embodiment of the present application is shown. It should be noted that the three-dimensional face model training apparatus 140 provided by the present embodiment has the same basic principles and technical effects as the above-mentioned embodiments, and for brief description, the part not mentioned in the present embodiment can refer to the corresponding content in the above-mentioned embodiments. The three-dimensional face model training apparatus 140 includes:

[0160] An acquisition module 141 is configured to acquire a face image and crop a face region image in the face image, wherein the face region image contains multiple face feature points.

[0161] It can be understood that the acquisition module 141 can be used to perform the above-mentioned step S101, and the detailed implementation manner of the acquisition module 141 can refer to the content related to step S101.

[0162] A construction module 142 is configured to construct a virtual perspective camera and obtain perspective 3D position information and 2D position information of each face feature point in the face region image in the virtual perspective camera space.

[0163] It can be understood that the construction module 142 can be used to perform the above-mentioned step S103, and the detailed implementation manner of the construction module 142 can refer to the content related to step S103.

[0164] A training module 143 is configured to train a constructed neural network model by using a face region image carrying the perspective 3D position information and 2D position information, to obtain a three-dimensional face model that can be used to detect three-dimensional information of face feature points.

[0165] It can be understood that the training module 143 can be used to perform the above-mentioned step S105, and the detailed implementation manner of the training module 143 can refer to the content related to step S105.

[0166] In a possible implementation manner, the above-mentioned construction module 142 can be configured to:

[0167] obtain 3D position information of each face feature point in the face region image under a world coordinate system, and obtain 2D position information of each face feature point under an image coordinate system;

[0168] According to the 3D position information and the 2D position information, internal and external parameters of the virtual perspective camera are calculated;

[0169] According to the internal and external parameters and the 3D position information, perspective 3D position information and 2D position information of each facial feature point in the virtual perspective camera space are obtained.

[0170] In a possible implementation, the internal and external parameters include pose parameters and an intrinsic matrix, and the construction module 142 can be configured to:

[0171] According to the pose parameters and the 3D position information, perspective 3D position information of the facial feature points in the virtual perspective camera space is calculated;

[0172] According to the perspective 3D position information and the intrinsic matrix, 2D position information of the facial feature points in the virtual perspective camera space is calculated.

[0173] In a possible implementation, the construction module 142 can be configured to:

[0174] The 3D position information of each facial feature point in the world coordinate system is re-projected to obtain 2D position information of the facial feature points in the image coordinate system constructed based on the face image;

[0175] The obtained 2D position information is revised based on the cropping information of the face region image to obtain 2D position information of the facial feature points in the image coordinate system constructed based on the face region image.

[0176] In a possible implementation, the three-dimensional face model training apparatus 140 further includes a scaling module, which can be configured to:

[0177] The resolution size of the cropped face region image is scaled to a preset resolution range;

[0178] The construction module 142 can be configured to:

[0179] The obtained 2D position information is revised based on the cropping information of the face region image and the preset resolution range.

[0180] In a possible implementation, the training module 143 can be configured to:

[0181] The face region image is imported into the constructed neural network model, and 3D position prediction information of each facial feature point in the face region image is output;

[0182] The 3D position prediction information is re-projected to obtain corresponding 2D position prediction information.

[0183] A loss function is constructed based on the perspective 3D position information, 3D position prediction information, 2D position information, and 2D position prediction information. The neural network model is then trained using the loss function as a guide until the preset requirements are met, thus obtaining the 3D face model.

[0184] In one possible implementation, the training module 143 described above can be used for:

[0185] A first loss term is constructed based on the difference between the perspective 3D position information and the 3D position prediction information.

[0186] A second loss term is constructed based on the difference between the 2D location information and the 2D location prediction information;

[0187] The first loss term and the second loss term are added together with a set weight to obtain the loss function.

[0188] In one possible implementation, the 3D face model training device 140 further includes a processing module, which can be used for:

[0189] Obtain the pixel values ​​of each pixel point contained in the face region image;

[0190] The pixel values ​​of each pixel are normalized to a preset pixel range.

[0191] In one possible implementation, the 3D face model training device 140 further includes a detection module, which can be used for:

[0192] Obtain the image to be detected, and crop out the region to be detected image from the image to be detected. The region to be detected image includes a face region, and the face region includes multiple key points to be detected.

[0193] A virtual perspective space is constructed, the image of the region to be detected is imported into the three-dimensional face model, and the three-dimensional information of each key point to be detected within the virtual perspective space is output.

[0194] The 3D face model training device 140 provided in this application embodiment can execute the 3D face model training method provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects of executing the method. Optionally, the above modules can be stored in the form of software or firmware. Figure 11 The memory 120 shown is either stored in or embedded in the operating system (OS) of the electronic device, and can be used by... Figure 11 The processor 110 executes the program. Meanwhile, the data and program code required to execute the above modules can be stored in the memory 120.

[0195] The application further provides a computer readable storage medium comprising computer executable instructions for performing the operations of the three-dimensional face model training method provided by any of the embodiments of the application when executed by a computer processor.

[0196] In summary, the three-dimensional face model training method, device, electronic device and readable storage medium provided by the embodiments of the application cut out the face region image of the face image after obtaining the face image, and the face region image contains multiple face feature points. A virtual perspective camera is constructed, and perspective 3D position information and 2D position information of each face feature point in the face region image in the virtual perspective camera space are obtained. Then, the constructed neural network model is trained by using the face region image carrying the perspective 3D position information and 2D position information, and a three-dimensional face model capable of detecting three-dimensional information of face feature points is obtained. In the scheme, the three-dimensional face model capable of detecting three-dimensional information of face feature points is trained to realize accurate detection of 3D key points, and the model is trained in combination with the perspective 3D position information and 2D position information, so that the model obtained by training has better effect on face feature point alignment.

[0197] The above merely provides a specific implementation of the application, but the protection scope of the application is not limited thereto. Any changes or replacements within the technical scope disclosed by the application can be easily thought of by those skilled in the art, and should be covered within the protection scope of the application. Therefore, the protection scope of the application should be subject to the protection scope of the claims.

Claims

1. A method for training a three-dimensional face model, characterized in that, The method includes: Acquire a face image and crop out the face region image from the face image, the face region image containing multiple face feature points; A virtual perspective camera is constructed to obtain the perspective 3D and 2D position information of each facial feature point in the face region image in the virtual perspective camera space. The perspective 3D and 2D position information is obtained based on the 3D position information of the facial feature points in the world coordinate system and the intrinsic and extrinsic parameters of the virtual perspective camera. The intrinsic and extrinsic parameters are calculated based on the 3D position information of each facial feature point and the 2D position information in the image coordinate system. The 2D position information of each facial feature point in the image coordinate system is obtained by reprojecting the 3D position information of each facial feature point in the world coordinate system to obtain the 2D position information of the facial feature point in the image coordinate system constructed based on the facial image; the obtained 2D position information is revised based on the cropping information of the facial region image to obtain the 2D position information of the facial feature point in the image coordinate system constructed based on the facial region image. The neural network model is trained using face region images carrying the aforementioned perspective 3D position information and 2D position information to obtain a 3D face model that can be used to detect 3D information of face feature points.

2. The three-dimensional face model training method according to claim 1, characterized in that, The intrinsic and extrinsic parameters include pose parameters and intrinsic parameter matrices; Based on the intrinsic and extrinsic parameters and the 3D position information, the perspective 3D position information and 2D position information of each facial feature point within the virtual perspective camera space are obtained, including: Based on the pose parameters and the 3D position information, the perspective 3D position information of the facial feature points in the virtual perspective camera space is calculated; Based on the perspective 3D position information and the intrinsic parameter matrix, the 2D position information of the facial feature points in the virtual perspective camera space is calculated.

3. The three-dimensional face model training method according to claim 1, characterized in that, The method further includes: The resolution of the cropped face region image is scaled down to a preset resolution range; The step of revising the obtained 2D positional information based on the cropping information of the face region image includes: The obtained 2D position information is revised based on the cropping information of the face region image and the preset resolution range.

4. The three-dimensional face model training method according to claim 1, characterized in that, The step of training the constructed neural network model using a face region image carrying the perspective 3D position information and 2D position information includes: The face region image is imported into the constructed neural network model, and the 3D position prediction information of each face feature point in the face region image is output. The 3D position prediction information is reprojected to obtain the corresponding 2D position prediction information. A loss function is constructed based on the perspective 3D position information, 3D position prediction information, 2D position information, and 2D position prediction information. The neural network model is then trained using the loss function as a guide until the preset requirements are met, thus obtaining the 3D face model.

5. The three-dimensional face model training method according to claim 4, characterized in that, The step of constructing a loss function based on the perspective 3D position information, 3D position prediction information, 2D position information, and 2D position prediction information includes: A first loss term is constructed based on the difference between the perspective 3D position information and the 3D position prediction information. A second loss term is constructed based on the difference between the 2D location information and the 2D location prediction information; The first loss term and the second loss term are added together with a set weight to obtain the loss function.

6. The three-dimensional face model training method according to claim 4, characterized in that, Before the step of importing the face region image into the constructed neural network model, the method further includes: Obtain the pixel values ​​of each pixel point contained in the face region image; The pixel values ​​of each pixel are normalized to a preset pixel range.

7. The three-dimensional face model training method according to any one of claims 1-6, characterized in that, The method further includes: Obtain the image to be detected, and crop out the region to be detected image from the image to be detected. The region to be detected image includes a face region, and the face region includes multiple key points to be detected. A virtual perspective space is constructed, the image of the region to be detected is imported into the three-dimensional face model, and the three-dimensional information of each key point to be detected within the virtual perspective space is output.

8. A three-dimensional human face model training device, characterized in that, The apparatus for implementing the three-dimensional face model training method according to any one of claims 1-7 includes: The acquisition module is used to acquire a face image and crop out the face region image from the face image, wherein the face region image contains multiple face feature points; The construction module is used to construct a virtual perspective camera to obtain the perspective 3D position information and 2D position information of each facial feature point in the face region image in the virtual perspective camera space; The training module is used to train the constructed neural network model using face region images carrying the perspective 3D position information and 2D position information to obtain a 3D face model that can be used to detect 3D information of face feature points.

9. An electronic device, characterized in that, The device includes one or more storage media and one or more processors communicating with the storage media. The one or more storage media store machine-executable instructions that can be executed by the processor. When the electronic device is running, the processor executes the machine-executable instructions to perform the steps of the method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores machine-executable instructions that, when executed, implement the steps of the method described in any one of claims 1-7.

Images