A method and electronic device for extracting facial landmarks based on a single image

Abstract

This application provides a method and electronic device for extracting facial key points based on a single image, used to improve the accuracy of 3D feature points. The method includes: inputting a face image into a pre-trained facial key point extraction network to obtain the positions of each 2D facial key point; projecting each labeled 3D facial key point in an initial 3D face model onto the face image to obtain the positions of each 2D projected facial key point; obtaining an error value based on the positions of each 2D projected facial key point and each 2D facial key point; if the error value is greater than a first specified threshold, adjusting the 3D face parameters of the initial 3D face model, and returning to the step of projecting each labeled 3D facial key point in the preset initial 3D face model onto the face image, until the error value is no greater than the first specified threshold, and then obtaining the positions of each 3D facial key point based on the adjusted initial 3D face model and each 2D facial key point.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method and electronic device for extracting facial key points based on a single image. Background Technology

[0002] Facial landmark extraction methods refer to the identification of key facial features such as eyes, nose, mouth, and facial contours in face images based on face detection technology. Currently, mainstream facial landmark extraction methods mainly include deformable template methods, point distribution model methods, graph model methods, and cascaded shape regression methods.

[0003] In practical applications, most existing facial landmark extraction methods first obtain 2D (two-dimensional) facial landmarks and then estimate 3D (three-dimensional) facial landmarks based on a virtually defined camera. However, in existing technologies, such virtually defined cameras are fixed and cannot have the same perspective and distortion relationships as each real image. Therefore, the accuracy of the determined 3D facial landmarks is relatively low. Summary of the Invention

[0004] This application provides a method and electronic device for extracting facial key points based on a single image, which is used to extract 3D facial key points and improve the accuracy of the extracted 3D facial key points.

[0005] In a first aspect, embodiments of this application provide a method for extracting facial key points based on a single image, including:

[0006] For any given face image, the face image is input into a pre-trained facial landmark extraction network model to extract 2D facial landmarks, thereby obtaining the position of each 2D facial landmark in the face image.

[0007] Project each labeled 3D facial key point in the preset initial 3D facial model onto the facial image to obtain the position of each 2D projected facial key point in the facial image;

[0008] Based on the positions of the 2D projected facial key points and the positions of the 2D facial key points, the error value is obtained;

[0009] If the error value is greater than the first specified threshold, the three-dimensional face parameters of the initial three-dimensional face model are adjusted, and the process returns to the step of projecting each labeled 3D face key point in the preset initial three-dimensional face model onto the face image to obtain the position of each 2D projected face key point in the face image, until the error value is not greater than the first specified threshold, and then the adjusted initial three-dimensional face model is determined as the target three-dimensional face model.

[0010] Based on the labeled 3D facial key points in the target 3D facial model and the labeled 2D facial key points, the positions of each 3D facial key point in the facial image are obtained.

[0011] A second aspect of this application provides an electronic device, including a processor and a memory, wherein the processor and the memory are connected via a bus;

[0012] The memory stores a computer program, and the processor is configured to perform the following operations based on the computer program:

[0013] For any given face image, the face image is input into a pre-trained facial landmark extraction network model to extract 2D facial landmarks, thereby obtaining the position of each 2D facial landmark in the face image.

[0014] Project each labeled 3D facial key point in the preset initial 3D facial model onto the facial image to obtain the position of each 2D projected facial key point in the facial image;

[0015] Based on the positions of the 2D projected facial key points and the positions of the 2D facial key points, the error value is obtained;

[0016] If the error value is greater than the first specified threshold, the three-dimensional face parameters of the initial three-dimensional face model are adjusted, and the process returns to the step of projecting each labeled 3D face key point in the preset initial three-dimensional face model onto the face image to obtain the position of each 2D projected face key point in the face image, until the error value is not greater than the first specified threshold, and then the adjusted initial three-dimensional face model is determined as the target three-dimensional face model.

[0017] Based on the labeled 3D facial key points in the target 3D facial model and the labeled 2D facial key points, the positions of each 3D facial key point in the facial image are obtained.

[0018] According to a third aspect of the present invention, a computer storage medium is provided, the computer storage medium storing a computer program for performing the method as described in the first aspect.

[0019] In the above embodiments of this application, a face image is input into a pre-trained facial landmark extraction network model to extract 2D facial landmarks, thereby obtaining the position of each 2D facial landmark in the face image. Then, each labeled 3D facial landmark in a preset initial 3D face model is projected onto the face image to obtain the position of each 2D projected facial landmark in the face image. Based on the error value obtained from the positions of each 2D projected facial landmark and the positions of each 2D facial landmark, the 3D face parameters of the initial 3D face model are adjusted, and the adjusted initial 3D face model is determined as the target 3D face model. Finally, based on each labeled 3D facial landmark in the target 3D face model and each 2D facial landmark, the position of each 3D facial landmark in the face image is obtained. Therefore, in this application, the 3D face parameters of the initial 3D face model are continuously adjusted based on the error values ​​obtained from the positions of the 2D projected face key points and the positions of the 2D face key points, in order to overcome the influence of perspective and distortion of the virtual camera, thereby improving the accuracy of the extracted 3D face key points. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 An exemplary illustration shows one of the application scenarios provided in the embodiments of this application;

[0022] Figure 2 The second example illustration shows a schematic diagram of an application scenario provided in an embodiment of this application;

[0023] Figure 3 This document exemplarily illustrates the third application scenario diagram provided in the embodiments of this application;

[0024] Figure 4 An exemplary flowchart illustrates one of the facial key point extraction methods based on a single image provided in an embodiment of this application;

[0025] Figure 5 An exemplary schematic diagram of dense 2D facial key points provided in an embodiment of this application is shown;

[0026] Figure 6 An exemplary schematic diagram of the process for training a facial landmark extraction network model provided in an embodiment of this application is shown;

[0027] Figure 7 An exemplary schematic diagram illustrates the process of determining virtual face image training samples provided in an embodiment of this application;

[0028] Figure 8 An exemplary schematic diagram of the facial landmark extraction network model provided in an embodiment of this application is shown;

[0029] Figure 9 An exemplary schematic diagram of the process for determining error values ​​provided in an embodiment of this application is shown;

[0030] Figure 10 An exemplary diagram of 3D facial key points provided in an embodiment of this application is shown;

[0031] Figure 11 A second flowchart of a method for extracting facial key points based on a single image, provided in an embodiment of this application, is shown as an example.

[0032] Figure 12 An exemplary schematic diagram of the structure of the face key point extraction device based on a single image provided in an embodiment of this application is shown;

[0033] Figure 13 An exemplary hardware structure diagram of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0034] To make the objectives, implementation methods and advantages of this application clearer, the exemplary implementation methods of this application will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments of this application. Obviously, the described exemplary embodiments are only some embodiments of this application, and not all embodiments.

[0035] Based on the exemplary embodiments described in this application, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of the appended claims. Furthermore, although the disclosures in this application are presented by way of one or more exemplary examples, it should be understood that each aspect of these disclosures can also constitute a complete implementation on its own.

[0036] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0037] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to be omnipresent but not exclusive; for example, a product or device comprising a series of components is not necessarily limited to those explicitly listed, but may include other components not explicitly listed or inherent to such product or device.

[0038] As used in this application, the term "module" refers to any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code capable of performing the functions associated with that element.

[0039] The following is an overview of the ideas behind the embodiments of this application.

[0040] Most existing facial landmark extraction methods first obtain 2D facial landmarks and then estimate 3D facial landmarks based on a virtually defined camera. However, in existing technologies, this virtually defined camera is fixed and cannot have the same perspective and distortion relationships as each real image. Therefore, the accuracy of the determined 3D facial landmarks is relatively low.

[0041] To address the issue of low accuracy in existing 3D facial landmark extraction technologies, this application provides a single-image-based facial landmark extraction method. This method involves inputting a face image into a pre-trained facial landmark extraction network model to extract 2D facial landmarks, obtaining the positions of each 2D facial landmark in the face image. Then, each labeled 3D facial landmark in a preset initial 3D face model is projected onto the face image, obtaining the positions of each 2D projected facial landmark in the face image. Based on the error values ​​obtained from the positions of the 2D projected facial landmarks and the positions of the 2D facial landmarks, the 3D face parameters of the initial 3D face model are adjusted. The adjusted initial 3D face model is then determined as the target 3D face model. Finally, based on the labeled 3D facial landmarks in the target 3D face model and the 2D facial landmarks, the positions of each 3D facial landmark in the face image are obtained. Therefore, in this application, the 3D face parameters of the initial 3D face model are continuously adjusted based on the error values ​​obtained from the positions of the 2D projected face key points and the positions of the 2D face key points, in order to overcome the influence of perspective and distortion of the virtual camera, thereby improving the accuracy of the extracted 3D face key points.

[0042] The embodiments of this application are described in detail below with reference to the accompanying drawings.

[0043] Figure 1 An exemplary illustration shows a schematic diagram of an application scenario provided in an embodiment of this application; such as Figure 1 As shown, this application scenario uses an electronic device as a server as an example. This application scenario includes a camera 110 and a server 120. Server 120 can be implemented using a single server or multiple servers. Server 120 can be implemented using a physical server or a virtual server.

[0044] In one possible application scenario, camera 110 sends the acquired face image to server 120. Upon receiving the face image, server 120 inputs it into a pre-trained facial landmark extraction network model to extract 2D facial landmarks, obtaining the positions of each 2D facial landmark in the face image. Then, server 120 projects each labeled 3D facial landmark from a preset initial 3D face model onto the face image, obtaining the positions of each 2D projected facial landmark in the face image. Based on the positions of the 2D projected facial landmarks and the positions of the 2D facial landmarks, it obtains the error... If the error value is greater than a first specified threshold, the three-dimensional face parameters of the initial three-dimensional face model are adjusted, and the process of projecting each labeled 3D face key point in the preset initial three-dimensional face model onto the face image to obtain the position of each 2D projected face key point in the face image is repeated until the error value is not greater than the first specified threshold. Then, the adjusted initial three-dimensional face model is determined as the target three-dimensional face model. Finally, the server 120 obtains the position of each 3D face key point in the face image based on each labeled 3D face key point in the target three-dimensional face model and each 2D face key point.

[0045] like Figure 2The diagram illustrates another application scenario of this application, including a camera 110, a server 120, and a terminal device 130. The camera 110 sends the acquired face image to the server 120. Upon receiving the face image, the server 120 inputs it into a pre-trained facial landmark extraction network model to extract 2D facial landmarks, obtaining the positions of each 2D facial landmark in the face image. Then, the server 120 projects each labeled 3D facial landmark in a preset initial 3D face model onto the face image, obtaining the positions of each 2D projected facial landmark in the face image. Based on the positions of the 2D projected facial landmarks and the positions of the 2D facial landmarks, an error value is obtained. If the error value is greater than a first specified threshold, the initial... After adjusting the 3D face parameters of the 3D face model, the process returns to the step of projecting each labeled 3D face key point in the preset initial 3D face model onto the face image to obtain the position of each 2D projected face key point in the face image. This process continues until the error value is no greater than the first specified threshold. Then, the adjusted initial 3D face model is determined as the target 3D face model. Finally, the server 120 obtains the position of each 3D face key point in the face image based on each labeled 3D face key point in the target 3D face model and each 2D face key point, and sends the position of each 3D face key point in the face image to the terminal device 130 for display.

[0046] like Figure 3The diagram illustrates another application scenario of this application, which includes a camera 110, a server 120, a terminal device 130, and a memory 140. The camera 110 sends the acquired face image to the server 120. Upon receiving the face image, the server 120 inputs it into a pre-trained facial landmark extraction network model to extract 2D facial landmarks, obtaining the positions of each 2D facial landmark in the face image. Then, the server 120 projects each labeled 3D facial landmark in a preset initial 3D face model onto the face image, obtaining the positions of each 2D projected facial landmark in the face image. Based on the positions of each 2D projected facial landmark and the positions of each 2D facial landmark, an error value is obtained. If the error value is greater than a first specified threshold, the 3D face parameters of the initial 3D face model are adjusted. After adjustment, the process returns to the step of projecting each labeled 3D facial key point in the preset initial 3D face model onto the face image to obtain the position of each 2D projected facial key point in the face image, until the error value is not greater than the first specified threshold. Then, the adjusted initial 3D face model is determined as the target 3D face model. Finally, the server 120 obtains the position of each 3D facial key point in the face image based on each labeled 3D facial key point in the target 3D face model and each 2D facial key point, stores the position of each 3D facial key point in the face image in the memory 140, and then sends the position of each 3D facial key point in the face image to the terminal device 130 for display.

[0047] The description in this application focuses on a single camera 110, a single server 120, a single terminal device 130, and a single memory 140. However, those skilled in the art should understand that the illustrated camera 110, server 120, terminal device 130, and memory 140 are intended to illustrate the operation of these components in the technical solution of this application, and not to imply any limitation on the number, type, or location of the camera 110, server 120, terminal device 130, and memory 140. It should be noted that adding additional modules to or removing individual modules from the illustrated environment does not change the underlying concept of the exemplary embodiments of this application.

[0048] For example, terminal device 130 includes, but is not limited to: large visual screen, tablet computer, laptop computer, handheld computer, mobile Internet device (MID), wearable device, virtual reality (VR) device, augmented reality (AR) device, wireless terminal device in industrial control, wireless terminal device in autonomous driving, wireless terminal device in smart grid, wireless terminal device in transportation safety, wireless terminal device in smart city, or wireless terminal device in smart home, etc.; the terminal device may have a related client installed, which can be software (e.g., browser, short video software, etc.), or web page, mini program, etc.

[0049] It should be noted that the facial landmark extraction method based on a single image proposed in this application is not only applicable to... Figure 1 , Figure 2 as well as Figure 3 The application scenarios shown are also applicable to any device that has a facial landmark extraction device based on a single image.

[0050] The following describes an exemplary embodiment of the face key point extraction method based on a single image, in conjunction with the application scenarios described above and with reference to the accompanying drawings. It should be noted that the above application scenarios are only shown to facilitate understanding of the methods and principles of this application, and the implementation of this application is not limited in any way in this respect.

[0051] like Figure 4 The diagram shown is a flowchart of a method for extracting facial key points based on a single image, which may include the following steps:

[0052] Step 401: For any face image, input the face image into a pre-trained facial landmark extraction network model to extract 2D facial landmarks and obtain the position of each 2D facial landmark in the face image;

[0053] In this embodiment, the number of 2D facial key points extracted is greater than that in existing technologies; this embodiment extracts denser 2D facial key points. For example... Figure 5 As shown, this is a list of the extracted dense 2D facial key points.

[0054] Below, we will first provide a detailed introduction to the training method of the facial landmark extraction network model, such as... Figure 6 The diagram shown illustrates the process of training a facial landmark extraction network model, including the following steps:

[0055] Step 601: Obtain virtual face image training samples, wherein the virtual face image training samples include each virtual face image and the labeled positions of each 2D face key point in each virtual face image;

[0056] Figure 7 A schematic diagram illustrating the specific process for determining training samples for virtual face images includes the following steps:

[0057] Step 701: Obtain an initial 3D face model based on preset initial 3D face parameters;

[0058] The initial three-dimensional face parameters include pose parameters, body parameters, and expression parameters.

[0059] In one embodiment, the initial 3D face model is obtained through a deformable 3D face template. Preset initial parameters are input into the deformable 3D face template to obtain the initial 3D face model. This deformable 3D face template can be any topologically consistent template parameterized model. The FLAME model is used as an example below:

[0060] The FLAME model is a statistically based parametric linear face model that includes pose parameters, body parameters, and expression parameters. The pose parameters include global head nodes, neck nodes, jaw nodes, and left and right eye nodes. By providing the pose parameters for linear blending skinning (LBS) and by providing the pose and expression parameters for linear combination, the final 3D face reconstruction based on the FLAME template can be obtained. The 3D face reconstruction based on the FLAME template can be obtained through formula (1):

[0061]

[0062] Where β is the shape parameter and θ is the attitude parameter. The expression parameters, namely the preset initial three-dimensional face parameters in this embodiment, T P is a linear function in the flame template, w is the preset skin weight, J is a preset function in the flame template related to shape parameters, and W is the linear binding skin function.

[0063] Wherein, the linear function T P It can be determined by formula (2):

[0064]

[0065] Where T is the initial flame template, s is the preset shape parameter base, and B... SIt is a linear function of the linear sum of the shape parameter β and the shape parameter basis S in the flame template, where P is the preset pose parameter basis, and B is the linear sum of the shape parameter β and S. P In the flame template, ξ is a linear function that is the linear sum of the pose parameters θ and the pose parameter basis P, where ξ is the preset expression parameter basis, and B is the expression parameter basis. E These are the emoticon parameters in the Flame template. A linear function that is linearly added to the expression parameter basis ξ.

[0066] Step 702: Based on the received annotation instructions for 3D facial key points, determine the position of each annotated 3D facial key point in the initial 3D facial model;

[0067] The annotation instructions include the position of each annotated 3D face key point in the initial 3D face model, the index of the face patch where each annotated 3D face point is located, and the centroid coordinates of each face patch.

[0068] Step 703: Using multiple preset intermediate 3D face parameters, deform and update the initial 3D face model to obtain multiple updated 3D face models;

[0069] In this embodiment, the intermediate 3D face parameters include multiple combinations of shape parameters. These different combinations of shape parameters deform the initial 3D face model, generating different individuals. This embodiment generates 5000 individuals, and the geometric similarity between each individual is less than 0.5. However, this embodiment does not limit the number of individuals generated; the specific number can be set according to actual circumstances.

[0070] Secondly, the intermediate 3D face parameters in this embodiment include multiple combinations of exp expression parameters. Different expressions are generated by reconstructing the initial 3D face model using multiple combinations of shape expression parameters. Additionally, the intermediate 3D face parameters include multiple combinations of pose parameters. Multiple pose parameters are used to reconstruct the pose of the initial 3D face model, resulting in multiple different poses.

[0071] Then, each individual is randomly combined with a first specified number of facial expressions, and then randomly combined with a second specified number of pose parameters to obtain multiple updated 3D face models.

[0072] It should be noted that in this embodiment, the first specified quantity is 200 and the second specified quantity is 20. However, the first and second specified quantities in this embodiment can be set according to the actual situation, and this embodiment does not limit the specific values ​​of the first and second specified quantities.

[0073] Step 704: Reconstruct the multiple updated 3D face models and the preset virtual face texture images respectively to obtain multiple reconstructed 3D face models;

[0074] In this embodiment, multiple high-precision facial texture images of varying resolutions are acquired. During the acquisition process, the model wears a hairnet to cover its hair, removes glasses, and is presented with bare skin and uniform lighting. The generated high-precision facial texture images are then manually processed and enhanced to create a virtual facial texture image.

[0075] It should be noted that the method of generating virtual face images in this embodiment can be adjusted according to the actual situation, and this embodiment does not limit the method of generating virtual face images.

[0076] Step 705: Render the multiple reconstructed 3D face models with multiple preset background images to obtain multiple virtual face images and the labeled positions of each 2D facial key point in the multiple virtual face images. Determine the multiple virtual face images and the labeled positions of each 2D facial key point in the multiple virtual face images as training samples for the virtual face images. The labeled positions of each 2D facial key point in the multiple virtual face images are obtained based on the labeled 3D facial key points in the multiple reconstructed 3D face models.

[0077] In this embodiment, the reconstructed 3D face model is rendered using a rendering tool with different background images added to generate a large-scale virtual face image. Furthermore, during the rendering process, 1000 different hairstyles and colors are randomly selected for both men and women, and hair accessories, glasses, etc., are also randomly combined.

[0078] Furthermore, in this embodiment, during the rendering process, the 3D facial key points annotated in the reconstructed 3D face model are also projected into the image. During the projection process, the 2D facial key points obtained from each projection are annotated according to the perspective relationship of the camera projection. For the 2D facial key points that are visible in the image, the pixel position is annotated and the confidence level is set to 1. For the 2D facial key points that are not visible, the pixel position is annotated and the confidence level is set to 0. Finally, a large-scale virtual face image training sample is generated.

[0079] It should be noted that the projection of 3D facial key points in this embodiment is a method already in the prior art, and this embodiment does not limit the specific projection method. The visible and invisible 2D facial key points are determined based on user input instructions, and different colors can be used to distinguish between them.

[0080] Step 602: Input the training samples of the virtual face images into the face key point extraction network model to extract 2D face key points and obtain the predicted positions of each 2D face key point in each virtual face image.

[0081] Step 603: Based on the labeled position and the predicted position of each 2D facial key point, the training error value is obtained;

[0082] In one embodiment, for any 2D facial landmark, based on the labeled position and the predicted position of the 2D facial landmark, a sub-error value corresponding to the 2D facial landmark is obtained. The sub-error values ​​corresponding to each 2D facial landmark are then summed to obtain the training error value. The sub-error value of any 2D facial landmark can be obtained using formula (3):

[0083]

[0084] Where x1 is the x-coordinate of the labeled position of the 2D facial landmark, x2 is the x-coordinate of the predicted position of the 2D facial landmark, y1 is the y-coordinate of the labeled position of the 2D facial landmark, and y2 is the y-coordinate of the predicted position of the 2D facial landmark.

[0085] Step 604: Determine whether the training error value is greater than the second specified threshold. If yes, proceed to step 605; otherwise, proceed to step 606.

[0086] It should be noted that the second specified threshold in this embodiment can be set according to the actual situation, and this embodiment does not limit the specific value of the second specified threshold.

[0087] Step 605: After adjusting the specified parameters in the facial key point extraction network model, return to step 602;

[0088] In this embodiment, the adjustment method for the specified parameter can be to increase or decrease the specified value of the specified parameter each time. In this embodiment, there can be one or more specified parameters. The adjustment methods for different specified parameters can be the same or different. The specific adjustment method can be set according to the actual situation. This embodiment does not limit the adjustment method.

[0089] Step 606: End the training of the facial landmark extraction network model to obtain the trained facial landmark extraction network model.

[0090] To make the facial landmark extraction network model in this application more accurate and further improve the accuracy, in one embodiment, after performing step 606, the trained facial landmark extraction network model is retrained using real face image training samples to obtain a retrained facial landmark extraction network model, and the retrained facial landmark extraction network model is determined as the trained facial landmark extraction network model. The real face training samples are obtained based on the initial 3D face model and the acquired real 3D face model.

[0091] In this embodiment, the real face training samples include real face images and the labeled positions of 2D facial key points in each real face image.

[0092] Furthermore, the real face image is obtained by fitting the initial 3D face model with each of the acquired real 3D face models to obtain the fitted real 3D face model, and then projecting the fitted real 3D face model onto the acquired frontal face image to obtain each real face image and the labeled position of each 2D face key point in each real face image.

[0093] Since the initial 3D face model contains labeled 3D face key points, the fitted real 3D face model also contains labeled 3D face key points. When the fitted real 3D face model is projected onto the acquired frontal face image, the labeled 3D face key points are also projected onto the frontal face image, thus obtaining the labeled positions of each 2D face key point.

[0094] It should be noted that the realistic 3D face model and the frontal face image are obtained using real images from multiple perspectives through MVS (Multiple View Stereo) reconstruction. The realistic 3D face model and the face images from each perspective can be precisely aligned and projected using calibrated camera relationships. Furthermore, the fitting method for the 3D face model generally involves establishing a loss function based on the vertex-to-face distances between the initial 3D face model and the realistic 3D face model. Through continuous optimization, the loss function is minimized to obtain the pose parameters, shape parameters, and expression parameters of the fitted realistic 3D face model. This embodiment does not limit the specific model fitting method; the specific fitting method can be set according to the actual situation.

[0095] Therefore, in this embodiment, the facial landmark extraction network model is further trained using real face image training samples. The network structure of the facial landmark extraction network model remains unchanged. Based on the previously trained facial landmark extraction network model, the model is further trained using real face image training samples, thereby refining the facial landmark extraction network model and resulting in higher extraction accuracy and better generalization.

[0096] It should be noted that the method of training the facial landmark extraction network model using real face image training samples is the same as the method described above, and will not be repeated here.

[0097] After introducing the training method of the facial landmark extraction network model, the following describes the specific process of 2D facial landmark extraction from face images based on the facial landmark extraction network model in step 602, such as... Figure 8 The diagram shown is a schematic of the structure of a facial landmark extraction network model. The facial landmark extraction network model 800 includes a max pooling layer 801, a feature pyramid (FPN) layer 802, a transform layer 803, and a fully connected layer 804.

[0098] The following describes the steps for extracting 2D facial landmarks from a face image, based on the structure of the facial landmark extraction network model. First, the face image is downsampled using a max pooling layer 801 to obtain a downsampled face image. Then, a Feature Pyramid Network (FPN) layer 802 is used to extract features from the downsampled face image, resulting in a set of feature vectors. Next, a transform layer 803 filters the vectors in the feature vector set to obtain an intermediate feature vector set. Then, the intermediate feature vector set is fused with the visibility features in the face image to obtain a target feature vector set, where the visibility features are pre-set or obtained based on the previous frame of the face image. Finally, a fully connected layer 804 concatenates the feature vector set to obtain the positions of each 2D facial landmark in the face image.

[0099] If 3D facial landmark extraction is required for each frame of face image, the visibility feature of the current frame of face image is the average confidence of each 2D facial landmark in the previous frame of face image, and the confidence of each 2D facial landmark is obtained based on a pre-trained facial landmark extraction network model. If 3D facial landmark extraction is performed on only one frame of face image, the visibility feature of that face image is pre-set.

[0100] Step 402: Project each labeled 3D facial key point in the preset initial 3D facial model onto the facial image to obtain the position of each 2D projected facial key point in the facial image;

[0101] Step 403: Based on the positions of the 2D projected facial key points and the positions of the 2D facial key points, obtain the error value;

[0102] like Figure 9 The diagram shown illustrates the specific process for determining the error value, including the following steps:

[0103] Step 901: Filter the 2D facial key points using the confidence scores of each 2D facial key point to obtain the filtered 2D facial key points. The confidence scores of each 2D facial key point are obtained based on the facial key point extraction network model pre-trained.

[0104] In this embodiment, 2D facial landmarks with a confidence level greater than a specified confidence level are identified as the filtered 2D facial landmarks. The specified confidence level in this embodiment is 0.8. However, the specific value of the specified confidence level is not limited; in this embodiment, the specific value of the specified confidence level can be set according to the actual situation.

[0105] Step 902: For any 2D facial key point, based on the position coordinates of the 2D facial key point and the position coordinates of the 2D projected facial key point with the same index as the 2D facial key point, obtain the distance between the 2D facial key point and the 2D projected facial key point, and determine the distance as the intermediate error value corresponding to the 2D facial key point;

[0106] Each 2D facial landmark and each labeled 3D facial landmark has an index, and they are in one-to-one correspondence. Furthermore, the index of the 2D projected facial landmark obtained by projecting any labeled 3D facial landmark is the same.

[0107] Step 903: Obtain the error value based on the intermediate error values ​​corresponding to each 2D facial key point.

[0108] In one embodiment, the error value can be obtained in the following two ways:

[0109] Method 1: Add up the intermediate error values ​​corresponding to each 2D facial key point to obtain the error value.

[0110] Method 2: The error value is obtained by weighted summation of the intermediate error values ​​corresponding to each 2D facial key point.

[0111] It should be noted that the specific method for determining the error value can be selected according to the actual situation, and this embodiment does not limit it.

[0112] Step 404: If the error value is greater than the first specified threshold, then after adjusting the three-dimensional face parameters of the initial three-dimensional face model, return to the step of projecting each labeled 3D face key point in the preset initial three-dimensional face model onto the face image to obtain the position of each 2D projected face key point in the face image, until the error value is not greater than the first specified threshold, then determine the adjusted initial three-dimensional face model as the target three-dimensional face model;

[0113] The method for adjusting the three-dimensional face parameters of the initial three-dimensional face model can be to increase or decrease each parameter by a specified value each time. The specified values ​​for increasing or decreasing different three-dimensional face parameters can be the same or different. The specific adjustment method can be set according to the actual situation. This embodiment does not limit the adjustment method.

[0114] Step 405: Based on the labeled 3D facial key points in the target 3D facial model and the labeled 2D facial key points, obtain the position of each 3D facial key point in the facial image.

[0115] In this embodiment, the obtained 3D facial key points are also dense 3D facial key points, such as... Figure 10 The image shown is a schematic diagram of the dense 3D facial key points. Figure 10 As can be seen, the number of 3D facial key points obtained in this embodiment is relatively large.

[0116] In one embodiment, step 405 can be implemented as follows: for any 2D facial key point, determine the labeled 3D facial key point with the same index as the 2D facial key point as the target labeled 3D facial key point corresponding to the 2D facial key point; and determine the target labeled 3D facial key points corresponding to each 2D facial key point as each 3D facial key point in the face image, and determine the current position of each target labeled 3D facial key point in the target 3D face model as the position of each 3D facial key point in the face image, wherein the current position of each target labeled 3D facial key point is obtained based on the position of each vertex of the patch where each target labeled 3D facial key point is located.

[0117] The current position of each target-annotated 3D facial key point in the target 3D facial model is determined by the following method:

[0118] For any target labeled 3D facial key points, using the pre-set correspondence between 3D facial key points and patches, determine the patch corresponding to the index of the target labeled 3D facial key points; based on the position coordinates of the vertices of the patch, obtain the current position of the target labeled 3D facial key points.

[0119] The annotation instructions mentioned above include the position of each annotated 3D facial keypoint in the initial 3D face model, the index of the face patch containing each annotated 3D facial keypoint, and the centroid coordinates of each face patch. Therefore, the spatial relative position between each annotated 3D facial keypoint and the centroid coordinates of its corresponding face patch can be determined based on the centroid coordinates and the annotated 3D facial keypoint, and this spatial relative position includes both distance and direction. Since the spatial relative position relationship between the annotated 3D facial keypoint and the face patch does not change, the current position of the target annotated 3D facial keypoint can be obtained based on the centroid coordinates and spatial relative position of the face patch containing the annotated 3D facial keypoint.

[0120] To further connect the technical solutions in this application, the following is combined with... Figure 11 A detailed explanation may include the following steps:

[0121] Step 1101: Obtain virtual face image training samples, wherein the virtual face image training samples include each virtual face image and the labeled positions of each 2D face key point in each virtual face image;

[0122] Step 1102: Input the training samples of the virtual face image into the face key point extraction network model to extract 2D face key points and obtain the predicted position of each 2D face key point in each virtual face image.

[0123] Step 1103: Based on the labeled position and the predicted position of each 2D facial key point, obtain the training error value;

[0124] Step 1104: Determine whether the training error value is greater than the second specified threshold. If yes, proceed to step 1105; otherwise, proceed to step 1106.

[0125] Step 1105: After adjusting the specified parameters in the facial key point extraction network model, return to step 1102;

[0126] Step 1106: End the training of the facial landmark extraction network model to obtain the trained facial landmark extraction network model;

[0127] Step 1107: Retrain the trained facial landmark extraction network model using real face image training samples to obtain a retrained facial landmark extraction network model, and determine the retrained facial landmark extraction network model as the trained facial landmark extraction network model, wherein the real face image training samples are obtained based on the initial 3D face model and the collected real 3D face model;

[0128] Step 1108: For any face image, input the face image into a pre-trained facial landmark extraction network model to extract 2D facial landmarks and obtain the position of each 2D facial landmark in the face image;

[0129] Step 1109: Project each labeled 3D facial key point in the preset initial 3D facial model onto the facial image to obtain the position of each 2D projected facial key point in the facial image;

[0130] Step 1110: Based on the positions of the 2D projected facial key points and the positions of the 2D facial key points, obtain the error value;

[0131] Step 1111: Determine whether the error value is greater than the first specified threshold. If yes, proceed to step 1112; otherwise, proceed to step 1113.

[0132] After adjusting the 3D face parameters of the initial 3D face model in step 1112, return to step 1109.

[0133] Step 1113: Determine the adjusted initial 3D face model as the target 3D face model;

[0134] Step 1114: Based on the labeled 3D facial key points in the target 3D facial model and the labeled 2D facial key points, obtain the position of each 3D facial key point in the facial image.

[0135] Based on the same inventive concept, the facial landmark extraction method based on a single image as described above can also be implemented by a facial landmark extraction device based on a single image. The effect of this facial landmark extraction based on a single image is similar to that of the aforementioned method, and will not be described again here.

[0136] Figure 12 This is a schematic diagram of a face key point extraction device based on a single image according to an embodiment of the present disclosure.

[0137] like Figure 12As shown, the face key point extraction device 1200 based on a single image disclosed herein may include a 2D face key point extraction module 1210, a projection module 1220, an error value determination module 1230, an adjustment module 1240, and a 3D face key point determination module 1250.

[0138] The 2D facial landmark extraction module 1210 is used to extract 2D facial landmarks from any given facial image by inputting the facial image into a pre-trained facial landmark extraction network model, thereby obtaining the position of each 2D facial landmark in the facial image.

[0139] The projection module 1220 is used to project each labeled 3D facial key point in the preset initial 3D facial model onto the facial image to obtain the position of each 2D projected facial key point in the facial image.

[0140] The error value determination module 1230 is used to obtain an error value based on the position of each 2D projected facial key point and the position of each 2D facial key point;

[0141] The adjustment module 1240 is used to adjust the three-dimensional face parameters of the initial three-dimensional face model if the error value is greater than the first specified threshold, and then return to the step of projecting each marked 3D face key point in the preset initial three-dimensional face model onto the face image to obtain the position of each 2D projected face key point in the face image, until the error value is not greater than the first specified threshold, and then the adjusted initial three-dimensional face model is determined as the target three-dimensional face model;

[0142] The 3D facial key point determination module 1250 is used to obtain the position of each 3D facial key point in the face image based on each labeled 3D facial key point in the target 3D facial model and each 2D facial key point.

[0143] In one embodiment, the apparatus further includes:

[0144] Training module 1260 is used to obtain the trained facial landmark extraction network model in the following manner:

[0145] Obtain virtual face image training samples, wherein the virtual face image training samples include each virtual face image and the labeled positions of each 2D face key point in each virtual face image;

[0146] The training samples of the virtual face images are input into the face landmark extraction network model to extract 2D face landmarks, thereby obtaining the predicted positions of each 2D face landmark in each virtual face image.

[0147] The training error value is obtained based on the labeled position and the predicted position of each 2D facial key point.

[0148] If the training error value is greater than the second specified threshold, the specified parameters in the facial landmark extraction network model are adjusted, and the process returns to the step of inputting the virtual face image training samples into the facial landmark extraction network model to extract 2D facial landmarks and obtain the predicted position of each 2D facial landmark in each virtual face image. This process continues until the error value is no greater than the second specified threshold, at which point the training of the facial landmark extraction network model ends, and the trained facial landmark extraction network model is obtained.

[0149] In one embodiment, the apparatus further includes:

[0150] The retraining module 1270 is used to terminate the training of the facial landmark extraction network model. After obtaining the trained facial landmark extraction network model, the retrained facial landmark extraction network model is retrained using real face image training samples to obtain a retrained facial landmark extraction network model. The retrained facial landmark extraction network model is then identified as the trained facial landmark extraction network model. The real face image training samples are obtained based on the initial 3D face model and the acquired real 3D face model.

[0151] In one embodiment, the apparatus further includes:

[0152] The virtual face image training sample determination module 1280 is used to determine the virtual face image training samples in the following manner before acquiring the virtual face image training samples:

[0153] An initial 3D face model is obtained based on preset initial 3D face parameters;

[0154] Based on the received annotation instructions for 3D facial key points, the position of each annotated 3D facial key point in the initial 3D facial model is determined;

[0155] Using multiple preset intermediate 3D face parameters, the initial 3D face model is deformed and updated to obtain multiple updated 3D face models.

[0156] The multiple updated 3D face models are reconstructed by reconstructing each of the preset virtual face texture images to obtain multiple reconstructed 3D face models.

[0157] The reconstructed 3D face models are rendered with multiple preset background images to obtain multiple virtual face images and the labeled positions of each 2D facial key point in the multiple virtual face images. The multiple virtual face images and the labeled positions of each 2D facial key point in the multiple virtual face images are determined as training samples of the virtual face images. The labeled positions of each 2D facial key point in the multiple virtual face images are obtained based on the labeled 3D facial key points in the multiple reconstructed 3D face models.

[0158] In one embodiment, the facial landmark extraction network model includes a max pooling layer, a feature pyramid (FPN) layer, a transform layer, and a fully connected layer; the 2D facial landmark extraction module 1210 is specifically used for:

[0159] The face image is downsampled using a max-pooling layer to obtain a downsampled face image; and,

[0160] The feature pyramid (FPN) layer is used to extract features from the downsampled face image to obtain a set of feature vectors;

[0161] The vectors in the feature vector set are filtered through the transform layer to obtain an intermediate feature vector set;

[0162] The intermediate feature vector set is fused with the face image visibility features to obtain the target feature vector set, wherein the visibility features are pre-set or obtained based on the previous frame face image;

[0163] The feature vector set is concatenated using a fully connected layer to obtain the positions of each 2D facial key point in the face image.

[0164] In one embodiment, the error value determination module 1230 is specifically used for:

[0165] The confidence scores of each 2D facial key point are used to filter the 2D facial key points to obtain the filtered 2D facial key points. The confidence scores of each 2D facial key point are obtained based on the facial key point extraction network model pre-trained.

[0166] For any 2D facial key point, the distance between the 2D facial key point and the 2D projected facial key point is obtained based on the position coordinates of the 2D facial key point and the position coordinates of the 2D projected facial key point with the same index as the 2D facial key point, and the distance is determined as the intermediate error value corresponding to the 2D facial key point.

[0167] The error value is obtained based on the intermediate error values ​​corresponding to each 2D facial key point.

[0168] In one embodiment, the 3D facial key point determination module 1250 is specifically used for:

[0169] For any 2D facial landmark, the labeled 3D facial landmark with the same index as the 2D facial landmark is determined as the target labeled 3D facial landmark corresponding to the 2D facial landmark; and,

[0170] The target labeled 3D facial key points corresponding to each 2D facial key point are determined as each 3D facial key point in the face image, and the current position of each target labeled 3D facial key point in the target 3D face model is determined as the position of each 3D facial key point in the face image, wherein the current position of each target labeled 3D facial key point is obtained based on the position of each vertex of the patch where each target labeled 3D facial key point is located.

[0171] In one embodiment, the apparatus further includes:

[0172] The current position determination module 1290 is used to determine the current position of each target-annotated 3D facial key point in the target 3D facial model in the following manner:

[0173] For any target labeled with 3D facial key points, the face corresponding to the index of the target labeled with 3D facial key points is determined by using the pre-set correspondence between 3D facial key points and facets.

[0174] Based on the position coordinates of the vertices of the facet, the current position of the target labeled 3D face key points is obtained.

[0175] After introducing a method and apparatus for extracting facial key points based on a single image according to an exemplary embodiment of the present invention, the following describes an electronic device according to another exemplary embodiment of the present invention.

[0176] Those skilled in the art will understand that various aspects of the present invention can be implemented as systems, methods, or program products. Therefore, various aspects of the present invention can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as "circuit", "module", or "system".

[0177] In some possible implementations, the electronic device according to the present invention may include at least one processor and at least one computer storage medium. The computer storage medium stores program code that, when executed by the processor, causes the processor to perform the steps in the single-image-based facial landmark extraction method according to various exemplary embodiments of the present invention described above. For example, the processor may perform actions such as... Figure 4 Steps 401-405 are shown in the diagram.

[0178] The following reference Figure 13 To describe an electronic device 1300 according to this embodiment of the present invention. Figure 13 The electronic device 1300 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0179] like Figure 13 As shown, the electronic device 1300 is presented in the form of a general-purpose electronic device. The components of the electronic device 1300 may include, but are not limited to: at least one processor 1301, at least one computer storage medium 1302, and a bus 1303 connecting different system components (including the computer storage medium 1302 and the processor 1301).

[0180] Bus 1303 represents one or more of several bus structures, including a computer storage media bus or computer storage media controller, peripheral bus, processor, or local bus using any of the various bus structures.

[0181] Computer storage medium 1302 may include readable media in the form of volatile computer storage media, such as random access computer storage medium (RAM) 1321 and / or cache storage medium 1322, and may further include read-only computer storage medium (ROM) 1323.

[0182] The computer storage medium 1302 may also include a program / utility 1325 having a set (at least one) of program modules 1324, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0183] Electronic device 1300 can also communicate with one or more external devices 1304 (e.g., keyboard, pointing device, etc.), one or more devices that enable a user to interact with electronic device 1300, and / or any device that enables electronic device 1300 to communicate with one or more other electronic devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 1305. Furthermore, electronic device 1300 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 1306. As shown, network adapter 1306 communicates with other modules used in electronic device 1300 via bus 1303. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 1300, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0184] In some possible implementations, various aspects of the single-image-based facial landmark extraction method provided by the present invention can also be implemented in the form of a program product, which includes program code that, when the program product is run on a computer device, causes the computer device to perform the steps in the single-image-based facial landmark extraction method according to various exemplary embodiments of the present invention described above.

[0185] The program product may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access computer storage media (RAM), read-only computer storage media (ROM), erasable programmable read-only computer storage media (EPROM or flash memory), optical fibers, portable compact disk read-only computer storage media (CD-ROM), optical computer storage media, magnetic computer storage media, or any suitable combination thereof.

[0186] The facial landmark extraction program product based on a single image according to embodiments of the present invention can be a portable compact disc read-only computer storage medium (CD-ROM) and include program code, and can run on an electronic device. However, the program product of the present invention is not limited thereto. In this document, the readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0187] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. This propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0188] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0189] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's electronic device, partially on the user's device, as a standalone software package, partially on the user's electronic device and partially on a remote electronic device, or entirely on a remote electronic device or server. In cases involving remote electronic devices, the remote electronic device can be connected to the user's electronic device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external electronic device (e.g., via the Internet using an Internet service provider).

[0190] It should be noted that although several modules of the apparatus have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more modules described above can be embodied in one module. Conversely, the features and functions of one module described above can be further divided and embodied by multiple modules.

[0191] Furthermore, although the operations of the method of the present invention are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0192] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk computer storage media, CD-ROMs, optical computer storage media, etc.) containing computer-usable program code.

[0193] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0194] These computer program instructions may also be stored in a computer-readable computer storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable computer storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0195] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0196] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for extracting facial key points based on a single image, characterized in that, include: For any given face image, the face image is input into a pre-trained facial landmark extraction network model to extract 2D facial landmarks, obtaining the positions of each 2D facial landmark in the face image. The facial landmark extraction network model includes a max pooling layer, a Feature Pyramid Network (FPN) layer, a transform layer, and a fully connected layer. The process of inputting the face image into the pre-trained facial landmark extraction network model to extract 2D facial landmarks and obtain the positions of each 2D facial landmark in the face image includes: downsampling the face image using a max pooling layer to obtain... The process involves: obtaining a downsampled face image; extracting features from the downsampled face image using a Feature Pyramid Network (FPN) layer to obtain a set of feature vectors; filtering the vectors in the feature vector set using a transform layer to obtain an intermediate feature vector set; fusing the intermediate feature vector set with the visibility features of the face image to obtain a target feature vector set, wherein the visibility features are pre-set or obtained based on the previous frame of the face image; and concatenating the feature vector set using a fully connected layer to obtain the positions of each 2D facial keypoint in the face image. Project each labeled 3D facial key point in the preset initial 3D facial model onto the facial image to obtain the position of each 2D projected facial key point in the facial image; Based on the positions of the 2D projected facial key points and the positions of the 2D facial key points, the error value is obtained; If the error value is greater than the first specified threshold, the three-dimensional face parameters of the initial three-dimensional face model are adjusted, and the process returns to the step of projecting each labeled 3D face key point in the preset initial three-dimensional face model onto the face image to obtain the position of each 2D projected face key point in the face image, until the error value is not greater than the first specified threshold, and then the adjusted initial three-dimensional face model is determined as the target three-dimensional face model. Based on the labeled 3D facial key points in the target 3D facial model and the labeled 2D facial key points, the positions of each 3D facial key point in the facial image are obtained.

2. The method according to claim 1, characterized in that, The trained facial landmark extraction network model was obtained through the following method: Obtain virtual face image training samples, wherein the virtual face image training samples include each virtual face image and the labeled positions of each 2D face key point in each virtual face image; The training samples of the virtual face images are input into the face landmark extraction network model to extract 2D face landmarks, thereby obtaining the predicted positions of each 2D face landmark in each virtual face image. The training error value is obtained based on the labeled position and the predicted position of each 2D facial key point. If the training error value is greater than the second specified threshold, the specified parameters in the facial landmark extraction network model are adjusted, and the process returns to the step of inputting the virtual face image training samples into the facial landmark extraction network model to extract 2D facial landmarks and obtain the predicted position of each 2D facial landmark in each virtual face image. This process continues until the error value is no greater than the second specified threshold, at which point the training of the facial landmark extraction network model ends, and the trained facial landmark extraction network model is obtained.

3. The method according to claim 2, characterized in that, After completing the training of the facial landmark extraction network model and obtaining the trained facial landmark extraction network model, the method further includes: The trained facial landmark extraction network model is retrained using real face image training samples to obtain a retrained facial landmark extraction network model, and the retrained facial landmark extraction network model is determined as the trained facial landmark extraction network model. The real face image training samples are obtained based on the initial 3D face model and the acquired real 3D face model.

4. The method according to claim 2, characterized in that, Before acquiring virtual face image training samples, the method further includes: The virtual face image training samples are determined in the following manner: An initial 3D face model is obtained based on preset initial 3D face parameters; Based on the received annotation instructions for 3D facial key points, the positions of each annotated 3D facial key point in the initial 3D facial model are determined; Using multiple preset intermediate 3D face parameters, the initial 3D face model is deformed and updated to obtain multiple updated 3D face models. The multiple updated 3D face models are reconstructed by combining them with preset virtual face texture images to obtain multiple reconstructed 3D face models. The reconstructed 3D face models are rendered with multiple preset background images to obtain multiple virtual face images and the labeled positions of each 2D facial key point in the multiple virtual face images. The multiple virtual face images and the labeled positions of each 2D facial key point in the multiple virtual face images are determined as training samples of the virtual face images. The labeled positions of each 2D facial key point in the multiple virtual face images are obtained based on the labeled 3D facial key points in the multiple reconstructed 3D face models.

5. The method according to claim 1, characterized in that, The error value is obtained based on the positions of the 2D projected facial key points and the positions of the 2D facial key points, including: The confidence scores of each 2D facial key point are used to filter the 2D facial key points to obtain the filtered 2D facial key points. The confidence scores of each 2D facial key point are obtained based on the facial key point extraction network model pre-trained. For any 2D facial key point, the distance between the 2D facial key point and the 2D projected facial key point is obtained based on the position coordinates of the 2D facial key point and the position coordinates of the 2D projected facial key point with the same index as the 2D facial key point, and the distance is determined as the intermediate error value corresponding to the 2D facial key point. The error value is obtained based on the intermediate error values ​​corresponding to each 2D facial key point.

6. The method according to claim 1, characterized in that, The step of obtaining the position of each 3D facial key point in the face image based on each labeled 3D facial key point in the target 3D face model and each 2D facial key point includes: For any 2D facial landmark, the labeled 3D facial landmark with the same index as the 2D facial landmark is determined as the target labeled 3D facial landmark corresponding to the 2D facial landmark; and, The target labeled 3D facial key points corresponding to each 2D facial key point are determined as each 3D facial key point in the face image. The current position of each target labeled 3D facial key point in the target 3D face model is determined as the position of each 3D facial key point in the face image. The current position of each target labeled 3D facial key point is obtained based on the position of each vertex of the patch where each target labeled 3D facial key point is located.

7. The method according to any one of claims 1 to 6, characterized in that, The current position of each target-annotated 3D facial key point in the target 3D facial model is determined by the following method: For any target labeled with 3D facial key points, the face corresponding to the index of the target labeled with 3D facial key points is determined by using the pre-set correspondence between 3D facial key points and facets. Based on the position coordinates of the vertices of the facet, the current position of the target labeled 3D face key points is obtained.

8. An electronic device, characterized in that, It includes a processor and a memory, which are connected via a bus; The memory stores a computer program, and the processor is configured to perform the following operations based on the computer program: For any given face image, the face image is input into a pre-trained facial landmark extraction network model to extract 2D facial landmarks, obtaining the positions of each 2D facial landmark in the face image. The facial landmark extraction network model includes a max pooling layer, a feature pyramid (FPN) layer, a transform layer, and a fully connected layer. The processor executes the step of inputting the face image into the pre-trained facial landmark extraction network model to extract 2D facial landmarks, obtaining the positions of each 2D facial landmark in the face image, specifically configured to: use a max pooling layer to extract the facial landmarks from the face image. The face image is downsampled to obtain a downsampled face image; then, features are extracted from the downsampled face image using a Feature Pyramid Network (FPN) layer to obtain a set of feature vectors; the vectors in the set of feature vectors are filtered through a transform layer to obtain an intermediate set of feature vectors; the intermediate set of feature vectors is fused with the visibility features of the face image to obtain a target set of feature vectors, wherein the visibility features are pre-set or obtained based on the previous frame of the face image; the feature vector set is then concatenated using a fully connected layer to obtain the positions of each 2D facial keypoint in the face image; Project each labeled 3D facial key point in the preset initial 3D facial model onto the facial image to obtain the position of each 2D projected facial key point in the facial image; Based on the positions of the 2D projected facial key points and the positions of the 2D facial key points, the error value is obtained; If the error value is greater than the first specified threshold, the three-dimensional face parameters of the initial three-dimensional face model are adjusted, and the process returns to the step of projecting each labeled 3D face key point in the preset initial three-dimensional face model onto the face image to obtain the position of each 2D projected face key point in the face image, until the error value is not greater than the first specified threshold, and then the adjusted initial three-dimensional face model is determined as the target three-dimensional face model. Based on the labeled 3D facial key points in the target 3D facial model and the labeled 2D facial key points, the positions of each 3D facial key point in the facial image are obtained.

9. The electronic device according to claim 8, characterized in that, The processor executes the step of obtaining the positions of each 3D facial key point in the face image based on each annotated 3D facial key point in the target 3D face model and each 2D facial key point, specifically configured as follows: For any 2D facial landmark, the labeled 3D facial landmark with the same index as the 2D facial landmark is determined as the target labeled 3D facial landmark corresponding to the 2D facial landmark; and, The target labeled 3D facial key points corresponding to each 2D facial key point are determined as each 3D facial key point in the face image. The current position of each target labeled 3D facial key point in the target 3D face model is determined as the position of each 3D facial key point in the face image. The current position of each target labeled 3D facial key point is obtained based on the position of each vertex of the patch where each target labeled 3D facial key point is located.

Images