Parametric face reconstruction model training method and key point label data generation method

Abstract

The present disclosure provides a parameterized face reconstruction model training method and a key point label data generation method, which are applied to the field of image processing. The method comprises the following steps: extracting a hidden code set from a face key point annotation sample image hidden code; inputting the hidden code set into an initial parameterized face reconstruction model color recognition network, and processing the face key point annotation sample image by using a first rendering module to obtain a first feature map; inputting the hidden code set into a generator module of the color recognition network to output a style hidden code; processing the first feature map based on the style hidden code by using a first upsampling module in the color recognition network to output a first rendering image; inputting the hidden code set into an initial parameterized face reconstruction model semantic recognition network to output a second rendering image; and adjusting model parameters of the initial parameterized face reconstruction model by using a loss between the first rendering image, the second rendering image and a real face sample image corresponding to the face key point annotation sample image, to obtain a target parameterized face reconstruction model.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and more specifically, to a parametric face reconstruction model training method, a face key point labeled data generation method, apparatus, electronic device, storage medium, and program product. Background Technology

[0002] In recent years, high-resolution face generation models have been proposed in related technologies. The faces generated by these models have good realism and viewpoint consistency. Face reconstruction models represented by HeadNeRF (Head Neural Radiance Fields) can reconstruct a single face from a new viewpoint, while also having a certain degree of semantic editing and multi-viewpoint consistency.

[0003] In realizing the concept disclosed herein, the inventors discovered at least the following problems in the related technologies: the face generated by HeadNeRF has low clarity and realism, and there are problems with poor image quality such as hair adhesion, which leads to low accuracy of the parametric face reconstruction model in annotating key points during the face reconstruction process. Summary of the Invention

[0004] In view of the above problems, this disclosure provides a parametric face reconstruction model training method, a face key point labeled data generation method, apparatus, electronic device, storage medium and program product.

[0005] According to a first aspect of this disclosure, a method for training a parametric face reconstruction model is provided, comprising: extracting implicit codes from a sample image of facial key points to obtain an implicit code set; inputting the implicit code set into a color recognition network of an initial parametric face reconstruction model, and using a first rendering module in the color recognition network to process the sample image of facial key points based on the implicit code set to obtain a first feature map; inputting the implicit code set into a generator module in the color recognition network to output a style implicit code; using a first upsampling module in the color recognition network to process the first feature map based on the style implicit code to output a first rendered image; inputting the implicit code set into a semantic recognition network of the initial parametric face reconstruction model to output a second rendered image; and adjusting the model parameters of the initial parametric face reconstruction model using the loss between the first rendered image and the second rendered image and the real face sample image corresponding to the sample image of facial key points to obtain a target parametric face reconstruction model.

[0006] According to embodiments of this disclosure, the above-mentioned extraction of hidden codes from a facial key point annotation sample image to obtain a hidden code set includes: extracting hidden codes from the facial key point annotation sample image using a preset method to obtain an initial hidden code set, wherein the initial hidden code set includes an initial identity hidden code, an initial expression hidden code, a texture hidden code, and an illumination hidden code; inputting the facial key point annotation sample image into a preset face recognition network to output a target identity hidden code; extracting a preset number of key points from the facial key point annotation sample image; standardizing the key points to obtain a target expression hidden code; and generating a target hidden code set based on the target identity hidden code, the target expression hidden code, the texture hidden code, and the illumination hidden code.

[0007] According to an embodiment of this disclosure, the first rendering module in the color recognition network processes the facial key point annotation sample image based on the hidden code set to obtain a first feature map, including: inputting the target hidden code set into an implicit neural function; predicting the facial key point annotation sample image based on the target identity hidden code and the target expression hidden code; outputting density values ​​and intermediate features; mapping the intermediate features based on the texture hidden code and the illumination hidden code to obtain high-dimensional features; integrating each light sampling point in the facial key point annotation sample image based on the density value to obtain an intermediate density value; and generating the first feature map based on the intermediate density value and the high-dimensional features using the first rendering module.

[0008] According to an embodiment of this disclosure, the above-mentioned use of the first upsampling module in the color recognition network to process the first feature map based on the style code and output a first rendering image includes: converting the first feature map into a Fourier feature distribution based on the mean square error condition to obtain a converted feature map; inputting the converted feature map and the style code into the first upsampling module and outputting the first rendering image.

[0009] According to an embodiment of this disclosure, the above-mentioned inputting the hidden code set into the semantic recognition network of the parameterized face reconstruction model and outputting a second rendering image includes: using the second rendering module in the semantic recognition network to process the face key point annotation sample image based on the hidden code set to obtain a second feature map; and using the second upsampling module in the semantic recognition network to process the second feature map and output a second rendering image.

[0010] According to embodiments of this disclosure, the above-mentioned method of adjusting the model parameters of the initial parameterized face reconstruction model to obtain a target parameterized face reconstruction model by utilizing the loss between the first rendered image and the second rendered image and the real face sample image corresponding to the face key point annotation sample image includes: inputting the first ground truth value corresponding to the real face sample image and the first rendered image, as well as the second ground truth value corresponding to the real face sample image and the second rendered image, into a loss function, and outputting a model loss value; adjusting the model parameters of the initial parameterized face reconstruction model based on the model loss value to obtain the target parameterized face reconstruction model.

[0011] According to embodiments of this disclosure, the step of inputting the first ground truth value corresponding to the real face sample image and the first rendered image, as well as the second ground truth value corresponding to the real face sample image and the second rendered image, into a loss function and outputting a model loss value includes: inputting the first ground truth value and the first rendered image into a pixel loss function and outputting a first pixel loss value; inputting the second ground truth value and the second rendered image into the pixel loss function and outputting a second pixel loss value; and inputting the predicted latent code set corresponding to the first rendered image and the second rendered image, and the latent code set corresponding to the real face sample image, into a decoupling loss function. Output the decoupling loss value; input the first rendered image and the first ground truth value into the image perception loss function, and output the first image perception loss value; input the second rendered image and the second ground truth value into the image perception loss function, and output the second image perception loss value; determine the semantic loss value based on the second pixel loss value and the image perception loss value; input the first feature map and the preset feature distribution into the distribution loss function, and output the distribution loss value; determine the model loss value based on the first pixel loss value, the first image perception loss value, the decoupling loss value, the distribution loss value, and the semantic loss value.

[0012] The second aspect of this disclosure provides a method for generating facial landmark label data, comprising: acquiring a target facial image or a randomly given set of facial hidden codes; inputting the target facial image or the randomly given set of facial hidden codes into a parametric facial reconstruction model to obtain a first output rendering image, a second output rendering image, and a set of facial hidden codes; processing the first output rendering image and the second output rendering image based on the set of facial hidden codes corresponding to the target facial image or the randomly given set of facial hidden codes to obtain a facial landmark annotation map, wherein the parametric facial reconstruction model is obtained using the above training method.

[0013] A third aspect of this disclosure provides a training apparatus for a parametric face reconstruction model, comprising: a latent code extraction module for extracting latent codes from a sample image of facial key points to obtain a latent code set; a first processing module for inputting the latent code set into a color recognition network of an initial parametric face reconstruction model, and using a first rendering module in the color recognition network to process the sample image of facial key points based on the latent code set to obtain a first feature map; a first output module for inputting the latent code set into a generator module in the color recognition network to output a style latent code; a second output module for using a first upsampling module in the color recognition network to process the first feature map based on the style latent code to output a first rendered image; a third output module for inputting the latent code set into a semantic recognition network of the initial parametric face reconstruction model to output a second rendered image; and a training module for adjusting the model parameters of the initial parametric face reconstruction model using the loss between the first rendered image and the second rendered image and the real face sample image corresponding to the sample image of facial key points to obtain a target parametric face reconstruction model.

[0014] The fourth aspect of this disclosure provides a face key point labeling data generation apparatus, comprising: an acquisition module for acquiring a target face image or a randomly given face hidden code set; a fitting module for inputting the target face image or the randomly given face hidden code set into a parametric face reconstruction model to obtain a first output rendering image, a second output rendering image, and a face hidden code set; and a labeling module for processing the first output rendering image and the second output rendering image based on the face hidden code set corresponding to the target face image or the randomly given face hidden code set to obtain a face key point labeling image, wherein the parametric face reconstruction model is obtained using the above training method.

[0015] A fifth aspect of this disclosure provides an electronic device comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the method described above.

[0016] A sixth aspect of this disclosure also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the methods described above.

[0017] The seventh aspect of this disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0018] According to embodiments of this disclosure, a parametric face reconstruction model refines the features of the facial key point annotation sample image in terms of color and semantics, thereby generating a first rendered image and a second rendered image. This provides a large amount of high-quality training data for the training of the parametric face reconstruction model. Furthermore, during the generation of the first rendered image, the parametric face reconstruction model inputs a latent code set into the generator module of the color recognition network, outputting a style latent code; then, using the first upsampling module in the color recognition network, it processes the first feature image based on the style latent code to output the first rendered image. Because the generator module can further capture the style latent code of the facial key point annotation sample image, the first upsampling module effectively reduces aliasing information in the color recognition network when processing the first feature image, maintaining the translation and rotation invariance of information transmission in the color recognition network. This results in images generated by the parametric face reconstruction model having high realism and high clarity, while avoiding hair adhesion problems. Attached Figure Description

[0019] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0020] Figure 1 The illustrations schematically depict application scenarios of the parametric face reconstruction model training method, face key point labeled data generation method, apparatus, device, medium, and program products according to embodiments of the present disclosure.

[0021] Figure 2 A flowchart illustrating a parametric face reconstruction model training method according to an embodiment of the present disclosure is shown schematically.

[0022] Figure 3 A flowchart illustrating a parametric face reconstruction model training method according to another embodiment of the present disclosure is shown schematically.

[0023] Figure 3A The illustration shows a schematic diagram of a reconstruction result obtained using a parametric face reconstruction model based on a first and a second rendered image according to an embodiment of the present disclosure.

[0024] Figure 3B The illustration shows a schematic diagram of obtaining a reconstruction result based on a first and a second rendered image using a parametric face reconstruction model according to another embodiment of the present disclosure;

[0025] Figure 3C The illustration shows a schematic diagram of a reconstruction result obtained using a parametric face reconstruction model based on a first and a second rendered image according to yet another embodiment of the present disclosure.

[0026] Figure 4The illustration shows a comparative diagram of facial landmark annotation maps obtained using a parametric face reconstruction model according to an embodiment of the present disclosure;

[0027] Figure 5 A flowchart illustrating a method for generating facial landmark tag data according to an embodiment of the present disclosure is shown.

[0028] Figure 6 This schematic diagram illustrates the structural block diagram of a parametric face reconstruction model training apparatus according to an embodiment of the present disclosure;

[0029] Figure 7 This schematic diagram illustrates a structural block diagram of a face key point tag data generation apparatus according to an embodiment of the present disclosure;

[0030] Figure 8 A block diagram schematically illustrates an electronic device suitable for implementing a parametric face reconstruction model training method and a face key point labeled data generation method according to embodiments of the present disclosure. Detailed Implementation

[0031] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0032] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0033] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0034] When using expressions such as "at least one of A, B, and C", they should generally be interpreted in accordance with the meaning that is commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B, and C, etc.).

[0035] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure, and application of data (including but not limited to user personal information) comply with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and they do not violate public order and good morals.

[0036] Currently, highly annotated data, such as facial landmark annotation data, facial semantic segmentation data, and facial gaze estimation data, plays a crucial role in downstream face-related tasks, such as face recognition, face parsing, and facial expression analysis. However, existing facial landmark annotation datasets mainly rely on manual annotation, resulting in high annotation costs, low efficiency, and poor annotation stability. This means that the quantity of such annotated data is far from meeting the needs of face-related tasks, thus hindering the training of face-related models.

[0037] Therefore, considering the challenges and limitations caused by limited data, related technologies attempt to leverage computer graphics methods to use synthetic face data instead of traditional manually annotated keypoint data as training data for downstream face recognition tasks. This involves progressively generating synthetic face data with 3D keypoints using rasterization rendering. To enhance the realism of the synthetic faces, rendered synthetic faces can be used instead of real faces for training.

[0038] However, despite employing various methods to enhance the realism of synthetic faces, the generated faces still retain a noticeable anime style, differing significantly from real faces. This stylistic difference results in the face reconstruction model performing significantly worse on synthetic face data compared to training on real face landmark annotation data. Furthermore, rendering synthetic faces is computationally expensive, time-consuming, and costly, making it less than ideal for practical use.

[0039] Face generation and reconstruction models can be used to generate realistic faces. In recent years, related technologies have proposed face generation models with high-resolution generation quality. These models generate faces with good realism and viewpoint consistency; however, their semantic editability is weak, and they can only simply adjust the camera viewpoint, making them unsuitable for generating face data with keypoint annotations. Meanwhile, face reconstruction models, such as HeadNeRF, can reconstruct a single face from a new viewpoint, while also possessing a degree of semantic editability and multi-viewpoint consistency.

[0040] However, due to the weak expressive power of the hidden codes extracted by HeadNeRF from the face in terms of facial features, and the inadequate design of the upsampling layer structure, HeadNeRF's reconstruction effect on difficult expressions is not ideal, and the generated faces suffer from poor image quality issues such as low clarity and realism, and hair adhesion. Furthermore, because HeadNeRF lacks editable semantic attributes and direct connections to facial landmarks, it is impossible to generate new face images with landmark annotations by editing semantic attributes.

[0041] In view of the above, embodiments of this disclosure provide a parametric face reconstruction model training method, a face key point labeled data generation method, a parametric face reconstruction model training device, a face key point labeled data generation device, an electronic device, a storage medium, and a program product. One method for training a parametric face reconstruction model includes: extracting hidden codes from a sample image of facial key points to obtain a hidden code set; inputting the hidden code set into the color recognition network of an initial parametric face reconstruction model, and using the first rendering module in the color recognition network to process the sample image of facial key points based on the hidden code set to obtain a first feature map; inputting the hidden code set into the generator module in the color recognition network to output a style hidden code; using the first upsampling module in the color recognition network to process the first feature map based on the style hidden code to output a first rendered image; inputting the hidden code set into the semantic recognition network of the initial parametric face reconstruction model to output a second rendered image; and using the loss between the first and second rendered images and the real face sample image corresponding to the sample image of facial key points to adjust the model parameters of the initial parametric face reconstruction model to obtain a target parametric face reconstruction model.

[0042] Figure 1 The illustrations schematically depict application scenarios of the parametric face reconstruction model training method, face key point labeled data generation method, apparatus, device, medium, and program products according to embodiments of the present disclosure.

[0043] like Figure 1 As shown, application scenario 100 according to this embodiment may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as a medium for providing communication links between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0044] Users can interact with server 105 via network 104 using at least one of the first terminal device 101, second terminal device 102, and third terminal device 103 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, second terminal device 102, and third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0045] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0046] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0047] It should be noted that the parametric face reconstruction model training method and the face key point labeling data generation method provided in this embodiment can generally be executed by server 105. Correspondingly, the parametric face reconstruction model training device and the face key point labeling data generation device provided in this embodiment can generally be located in server 105. The parametric face reconstruction model training method and the face key point labeling data generation method provided in this embodiment can also be executed by a server or server cluster that is different from server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the parametric face reconstruction model training device and the face key point labeling data generation device provided in this embodiment can also be located in a server or server cluster that is different from server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0048] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0049] Figure 2 A flowchart illustrating a parametric face reconstruction model training method according to an embodiment of the present disclosure is shown.

[0050] like Figure 2 As shown, the method includes operations S201 to S206.

[0051] In operation S201, the hidden code is extracted from the face key point annotation sample image to obtain the hidden code set.

[0052] In operation S202, the hidden code set is input into the color recognition network of the initial parameterized face reconstruction model. The first rendering module in the color recognition network processes the face key point annotation sample map based on the hidden code set to obtain the first feature map.

[0053] In operation S203, the implicit code set is input into the generator module in the color recognition network, and the style implicit code is output.

[0054] In operation S204, the first upsampling module in the color recognition network is used to process the first feature map based on the style latent code, and the first rendered image is output.

[0055] In operation S205, the hidden code set is input into the semantic recognition network of the initial parameterized face reconstruction model, and the second rendering image is output.

[0056] In operation S206, the model parameters of the initial parametric face reconstruction model are adjusted by using the loss between the first and second rendered images and the real face sample images corresponding to the face key point annotation sample images, so as to obtain the target parametric face reconstruction model.

[0057] According to embodiments of this disclosure, the facial landmark annotation sample image is obtained by preprocessing the facial image. The facial image is input into the open-source facial recognition project (InsightFace) for landmark annotation, resulting in the facial landmark annotation sample image. Furthermore, a one-euro filtering algorithm is used during preprocessing to mitigate inter-frame annotation jitter and ensure stability. It should be noted that the acquisition of both the facial image and the facial landmark annotation sample image complies with relevant laws and regulations, and necessary confidentiality measures are taken. Moreover, user authorization or consent is obtained before acquiring or collecting user facial images.

[0058] According to embodiments of this disclosure, a 3D deformation statistical model (3DMM) can be used to extract hidden codes from facial landmark annotation sample images to obtain a hidden code set. This hidden code set includes identity hidden codes, expression hidden codes, texture hidden codes, and illumination hidden codes. In the process of extracting identity hidden codes, AdaFace (a general representation for face recognition and quality assessment) can also be used. In the process of extracting expression hidden codes, expression hidden codes can also be extracted based on key points annotated with InsightFace.

[0059] According to embodiments of this disclosure, the initial parametric face reconstruction model can be denoted as HeadNeRF V2, comprising a color recognition network (RGBNet) and a semantic recognition network (SemanticNet). The color recognition network includes a first rendering module and a first upsampling module. The semantic recognition network includes a second rendering module and a second upsampling module. The network layer distribution in the first and second rendering modules can be the same. The network layer distribution in the first and second upsampling modules is different. Compared to the color recognition network, the semantic recognition network does not contain as much high-frequency information during processing; therefore, the network layer structure in the second upsampling module is not as refined as that in the first rendering module.

[0060] According to embodiments of this disclosure, a latent code set is input into the first rendering module, enabling the model to further learn features from the facial landmark annotation sample image and generate a first feature map. A generator module that avoids information distortion is introduced into the color recognition network to maintain the translation and rotation invariance of the first upsampling module. The first feature map is input into the first upsampling module, and simultaneously, the latent code set is input into the mapping network of the generator module and mapped to a preset dimension, such as 512 dimensions, to obtain the style latent code. The style latent code is input into each network layer of the first upsampling module, thereby guiding the parametric face reconstruction model to effectively learn facial detail information. After the first feature map is processed by the first upsampling module, it is converted into an RGB image with a resolution of 1024, i.e., the first rendered image.

[0061] According to embodiments of this disclosure, a hidden code set is input into a semantic recognition network, which outputs a semantic segmentation map with a resolution of 1024, i.e., the second rendered image. The addition of the semantic recognition network not only enhances the parametric face reconstruction model's learning of facial semantic information but also corrects the error between the first rendered image and the real RGB image, accelerating the learning of the parametric face reconstruction model.

[0062] According to embodiments of this disclosure, the model parameters of the initial parametric face reconstruction model are adjusted using the loss between the first ground truth value and the first rendered image of the real face sample image corresponding to the face key point annotation sample image, and the loss between the second ground truth value and the second rendered image of the real face sample image corresponding to the face key point annotation sample image, to obtain the target parametric face reconstruction model.

[0063] Figure 3 A flowchart illustrating a parametric face reconstruction model training method according to another embodiment of the present disclosure is shown.

[0064] According to embodiments of this disclosure, such as Figure 3As shown, key points are annotated on face image 301 to obtain a sample image of face key point annotations, and hidden codes are extracted from it to obtain a hidden code set 302. The hidden code set 302 includes the identity hidden code z. id , emoji hidden code z lm Texture Hidden Code z alb and illumination hidden code z ill The hidden code set 302 and the camera parameters of the face image are input into the color recognition network 303 (RGB Net), and processed by the first rendering module 304 in the color recognition network 303 to obtain the first feature map 305. Simultaneously, the implicit code set 302 is input into the generator module 306 in the color recognition network 303, and mapped through the mapping network M in the generator module 306 to obtain the style implicit code w. The first feature map 305 and the style implicit code w are input into the first upsampling module 307 to obtain the first rendered map 308.

[0065] According to embodiments of this disclosure, the hidden code set 302 and camera parameters of the face image are input into the semantic recognition network 309 (SemanticNet), and processed by the second rendering module 310 in the semantic recognition network 309 to obtain the second feature map 311. The second feature map 311 is input into the second upsampling module 312 (2D Neural Rendering Module Ψ) to obtain the second rendered map 313 (Rendered Semantic I2).

[0066] According to embodiments of this disclosure, the loss L between the first rendered image 308 and the first real image 314 (GT RGB), and the loss L between the second rendered image 313 and the second real image 315 (GT Semantic) are calculated. sem This facilitates subsequent optimization and training of the parameterized face reconstruction model.

[0067] According to embodiments of this disclosure, a parametric face reconstruction model refines the features of the facial key point annotation sample image in terms of color and semantics, thereby generating a first rendered image and a second rendered image. This provides a large amount of high-quality training data for the training of the parametric face reconstruction model. Furthermore, in the process of generating the first rendered image, the parametric face reconstruction model inputs a latent code set into the generator module of the color recognition network, outputting a style latent code; and utilizes the first upsampling module in the color recognition network to process the first feature map based on the style latent code, outputting the first rendered image. Because the generator module can further capture the style latent code of the facial key point annotation sample image, the first upsampling module effectively reduces aliasing information in the color recognition network when processing the first feature map, maintaining the translation and rotation invariance of information transmission in the color recognition network. This results in images generated by the parametric face reconstruction model having high realism and high clarity, while avoiding hair adhesion problems.

[0068] According to embodiments of this disclosure, extracting hidden codes from a sample image of facial landmark annotations to obtain a hidden code set may include the following operations:

[0069] A set of hidden codes is obtained by extracting hidden codes from the facial landmark annotation sample image using a preset method. The initial hidden code set includes initial identity hidden codes, initial expression hidden codes, texture hidden codes, and illumination hidden codes. The facial landmark annotation sample image is then input into a preset face recognition network, which outputs the target identity hidden code. A preset number of key points are extracted from the facial landmark annotation sample image. The key points are standardized to obtain the target expression hidden code. A target hidden code set is generated based on the target identity hidden code, target expression hidden code, texture hidden code, and illumination hidden code.

[0070] According to embodiments of this disclosure, a 3D deformation statistical model (3DMM) is used to extract implicit codes from facial landmark annotation sample images to obtain an initial implicit code set. To compensate for the insufficient ability of the 3DMM's identity vector to represent facial identities—specifically, the vector's inability to represent details such as teeth and hair—the facial landmark annotation sample images are further input into a preset face recognition network. This preset face recognition network can be a pre-trained face recognition network like AdaFace. This extracts the target identity implicit code, inferring a 128-dimensional identity feature as the target identity implicit code, effectively enhancing the learnable implicit code's ability to represent a large number of identities.

[0071] According to embodiments of this disclosure, a preset number of key points are extracted from a facial landmark annotation sample image, and a facial mesh is generated based on the preset number of key points and an initial latent code set. The preset number can be set to 68. 63*3D key points in a reference space are extracted from the facial mesh. Specifically, the extraction of 3D key points is performed before participating in camera parameter transformation, thereby ensuring that they do not contain camera parameter information, thus naturally decoupling them from camera parameters. The 3D key points are standardized and used as a new expression representation, thereby obtaining the target expression latent code.

[0072] According to embodiments of this disclosure, the first rendering module in a color recognition network processes a facial key point annotation sample map based on a hidden code set to obtain a first feature map, which may include the following operations:

[0073] The target implicit code set is input into the implicit neural function. Based on the target identity implicit code and the target expression implicit code, the facial key point annotation sample map is predicted, and the density value and intermediate features are output. Based on the texture implicit code and the illumination implicit code, the intermediate features are mapped to obtain high-dimensional features. Based on the density value, each light sampling point in the facial key point annotation sample map is integrated to obtain the intermediate density value. Using the first rendering module, the first feature map is generated based on the intermediate density value and the high-dimensional features.

[0074] According to embodiments of this disclosure, the target implicit code set is input into the implicit neural function. Simultaneously, the target identity implicit code from the target implicit code set is again input into the fifth fully connected layer of the implicit neural function network layer, enabling the parameterized face reconstruction model to better learn identity information. Based on the target identity implicit code and the target expression implicit code, prediction is performed on the facial key point annotation sample map to predict the density values ​​and intermediate features of the sampling points. The implicit neural function h... θ as follows:

[0075]

[0076] Where θ represents the network parameters of the color recognition network, z * This represents the hidden code of a sample image containing facial landmark annotations, where z id This indicates the target's identity is hidden, and z lm This indicates the target's coded expression, and z alb This indicates texture implicit code, and z ill This indicates a hidden code for illumination, and remember in, The dimension is indicated by the parentheses, and the concatenation of vectors is indicated by γ(x). γ(x) represents the position encoding of sampling point x in the face landmark annotation sample image. σ represents the density value of sampling point x, and F represents the intermediate feature. According to embodiments of this disclosure, the intermediate feature is mapped based on texture and illumination codes to obtain a high-dimensional feature F(x). The density intermediate value is obtained by integrating the density value across each light sampling point in the face keypoint annotation sample image. The formula for calculating the density intermediate value is as follows:

[0077]

[0078] Where r(t) represents a ray emitted from the center of the camera, t represents the farthest point of the ray, and s represents the integration variable.

[0079] According to embodiments of this disclosure, a first feature map is generated using a first rendering module based on the density median and high-dimensional features. The specific calculation method is as follows:

[0080]

[0081] According to embodiments of this disclosure, the first upsampling module in a color recognition network processes the first feature map based on style latent codes to output a first rendered image, which may include the following operations:

[0082] Based on the mean square error condition, the first feature map is transformed into a Fourier feature distribution to obtain the transformed feature map; the transformed feature map and style code are input into the first upsampling module to output the first rendering map.

[0083] According to embodiments of this disclosure, to avoid information aliasing, the first upsampling module uses a Fourier feature during pre-training. As input, it is made to conform to a specific distribution, thus ensuring the translation and rotation invariance of the network. However, The distribution of Fourier features differs greatly from that of the other feature; therefore, directly using the former instead of the latter as input to the first upsampling module will only generate an image full of noise blocks. Therefore, it is necessary to... Preprocessing is performed.

[0084] According to embodiments of this disclosure, in the first stage, random Fourier features are... The input is placed into the first upsampling module, instead of directly... Input it. Simultaneously, use mean square error to constrain... and The distribution differences between them. In the second stage, using replace and the constrained The data is input into the first upsampling module to ensure training stability. During this stage, due to... Guided by this, the face parameterization model can learn to control camera pose and facial semantic attributes. In the first two stages, the model parameters of the first upsampling module are fixed to avoid interference from relatively large noise during training. In the third stage, the upsampling module... During the processing, the model parameters of the first upsampling module are allowed to change freely, enabling the face parameter model to effectively learn facial details and complex expressions.

[0085] According to embodiments of this disclosure, inputting the hidden code set into the semantic recognition network of a parameterized face reconstruction model and outputting a second rendered image may include the following operations:

[0086] The second rendering module in the semantic recognition network is used to process the face key point annotation sample map based on the hidden code set to obtain the second feature map; the second upsampling module in the semantic recognition network is used to process the second feature map to output the second rendering map.

[0087] According to embodiments of this disclosure, a second rendering module in a semantic recognition network processes a sample image of facial key point annotations based on a hidden code set. The processing flow of the second rendering module is the same as that of the first rendering module, differing only in the network parameters, and will not be elaborated further here. After processing, a second feature map is obtained. Then, it is input into the second upsampling module of the semantic recognition network to obtain a semantic segmentation map with a resolution of 1024, which is also the second rendered image. The second upsampling module is an improvement on HeadNeRF's 2D neural rendering module. The addition of the semantic recognition network not only enhances the face parameterization model's learning of facial semantic information but also corrects the error between the first rendered image and the real RGB image, accelerating the model's learning process.

[0088] According to embodiments of this disclosure, the model parameters of the initial parametric face reconstruction model are adjusted using the loss between the first and second rendered images and the real face sample images corresponding to the face key point annotation sample images to obtain the target parametric face reconstruction model. This can include the following operations:

[0089] The first ground truth value and the first rendered image corresponding to the real face sample image, as well as the second ground truth value and the second rendered image corresponding to the real face sample image, are input into the loss function, and the model loss value is output. Based on the model loss value, the model parameters of the initial parameterized face reconstruction model are adjusted to obtain the target parameterized face reconstruction model.

[0090] According to embodiments of this disclosure, the model is optimized by utilizing the loss between the real value and the rendered image output by the initial parametric face reconstruction model, thereby obtaining a target parametric face reconstruction model, which makes the target parametric face reconstruction model more accurate in image reconstruction.

[0091] According to embodiments of this disclosure, inputting the first ground truth value and the first rendered image corresponding to the real face sample image, as well as the second ground truth value and the second rendered image corresponding to the real face sample image, into a loss function and outputting the model loss value may include the following steps:

[0092] The first ground truth value and the first rendered image are input into the pixel loss function, and the first pixel loss value is output. The second ground truth value and the second rendered image are input into the pixel loss function, and the second pixel loss value is output. The predicted hidden code sets corresponding to the first and second rendered images, and the hidden code sets corresponding to the real face sample images are input into the decoupling loss function, and the decoupling loss value is output. The first rendered image and the first ground truth value are input into the image perception loss function, and the first image perception loss value is output. The second rendered image and the second ground truth value are input into the image perception loss function, and the second image perception loss value is output. The semantic loss value is determined based on the second pixel loss value and the image perception loss value. The first feature map and the preset feature distribution are input into the distribution loss function, and the distribution loss value is output. The model loss value is determined based on the first pixel loss value, the first image perception loss value, the decoupling loss value, the distribution loss value, and the semantic loss value.

[0093] According to embodiments of this disclosure, the pixel loss function is constructed by subtracting the image loss of the predicted image from the true value of the real face sample image. The specific calculation of the first pixel loss value is as follows:

[0094]

[0095] in, I1 represents the image loss of the first pixel, and I2 represents the image loss of the first rendered image. This represents the first true value of a real face sample image for color recognition.

[0096] Similarly, the specific calculation of the second pixel loss value is as follows:

[0097]

[0098] in, I2 represents the image loss value of the second pixel, and I2 represents the image loss of the second rendered image. This represents the second ground truth value of a real face sample image for semantic recognition.

[0099] According to embodiments of this disclosure, the decoupling loss function is constructed by subtracting the learnable predicted latent code sets from the first and second rendered images and the corresponding latent code sets from the real face sample images. The decoupling loss value is specifically calculated as follows:

[0100]

[0101] Among them, z* This represents a set of learnable predictive hidden codes. Represents the implicit code set, w * Indicates the weight.

[0102] According to embodiments of this disclosure, the image perception loss function is constructed by subtracting the image perception loss of the predicted image from the true perception value corresponding to the true value of the real face sample image. Image perception can be obtained using VGG16 (Visual Geometry Group). The first image perception loss value can be denoted as... The perceptual loss value of the second image can be denoted as:

[0103] According to embodiments of this disclosure, the semantic loss value can optimize the learnable implicit code, and changes in the implicit code will adjust the RGB image, thereby helping the color recognition network learn facial semantic information and subtle expressions, thus accelerating model training. The semantic loss value is determined based on the second pixel loss value and the image perception loss value, and the specific calculation method is as follows:

[0104]

[0105] Among them, L sem This represents the semantic loss value.

[0106] According to embodiments of this disclosure, the distribution loss function is based on the square of the difference between the distribution of the first feature map and a preset feature distribution. The distribution loss is crucial for training the first upsampling module. Only when the distribution of the feature map generated by the first rendering module learns the preset feature distribution can the first upsampling module avoid signal interference and exert its powerful generation capability. The preset feature distribution can be set as a Fourier distribution. The specific calculation method for the distribution loss value is as follows:

[0107]

[0108] Among them, L dist Represents the distributed loss value. This represents the distribution of the first feature map. This indicates a preset feature distribution.

[0109] According to embodiments of this disclosure, the model loss value is determined based on the first pixel loss value, the first image perception loss value, the decoupling loss value, the distribution loss value, and the semantic loss value. The specific calculation method for the model loss value is as follows:

[0110]

[0111] Where L represents the model loss.

[0112] According to embodiments of this disclosure, the training of the parameterized face reconstruction model is divided into three stages. In stage one, w dist The value was set relatively large, and the distributed loss value was used as the main body of the loss function. In the next two training phases, w was reduced. dist It also allows the parametric face reconstruction model to focus its attention on learning the facial information of the input image.

[0113] Figure 3A The illustration shows a schematic diagram of a reconstruction result obtained using a parametric face reconstruction model based on a first and a second rendered image according to an embodiment of the present disclosure.

[0114] According to embodiments of this disclosure, the expression code corresponding to the first rendered image is transformed by projection to obtain the key point positions of the first rendered image, thereby generating a face reconstruction image with key point labels. The face reconstruction image with key point labels is then used to edit the attributes of key points, semantic information of the face, and camera pose based on the second rendered image. A parametric face reconstruction model is used to generate a large amount of new semantic information and new perspectives for the face reconstruction image with key point labels. Since the key points corresponding to the newly generated face reconstruction image with key point labels are still 3D key points in the reference space that do not contain camera parameter information, the 3D key points are inversely normalized along the camera ray coordinates, and the resulting image is transformed using camera parameters to obtain a 3D face reconstruction image with key point labels. The coordinates of the 3D key points in the 3D face reconstruction image with key point labels are projected into 68-point 2D key point annotations.

[0115] According to embodiments of this disclosure, such as Figure 3A As shown, Figure 3A The leftmost and rightmost columns of (a) and (b) show the reconstruction results obtained from randomly selected facial landmark annotation sample images from the dataset. The middle column shows the face images generated by interpolating facial attributes from two reconstructed face images with landmark labels, and inputting the interpolation implicit code into the parameterized face reconstruction model. The corresponding landmark information is annotated in the face images. Clearly, the newly generated face images have high-precision landmark annotations and good clarity. The third row of (a) and (b) shows the 3D landmark annotation results, while the rest show the 2D landmark annotation results. Both 2D and 3D landmarks are planar landmarks, and the 3D landmarks include landmarks in occluded areas.

[0116] Figure 3B The illustration shows a schematic diagram of a reconstruction result obtained using a parametric face reconstruction model based on a first and a second rendered image, according to another embodiment of the present disclosure.

[0117] Figure 3CThe illustration shows a schematic diagram of a reconstruction result obtained using a parametric face reconstruction model based on a first and a second rendered image, according to yet another embodiment of the present disclosure.

[0118] According to embodiments of this disclosure, such as Figure 3B and Figure 3C As shown, Figure 3B and Figure 3C The leftmost and rightmost columns show the reconstruction results obtained from randomly selected facial landmark annotation sample images in the dataset. The middle column shows the face images generated by interpolating facial attributes from two faces and inputting the interpolation implicit code into the parameterized face reconstruction model. Figure 3B For the regenerated face with 2D keypoint annotations, Figure 3C These are faces with 3D keypoint annotations regenerated from a synthetic face dataset. These randomly selected and generated results demonstrate that our method can generate not only 2D keypoint-annotated data but also high-precision 3D keypoint-annotated data, exhibiting good generalization ability.

[0119] Figure 4 The illustration shows a comparative diagram of facial landmark annotation maps obtained using a parametric face reconstruction model according to an embodiment of the present disclosure.

[0120] According to embodiments of this disclosure, such as Figure 4 As shown, after generating a large amount of new labeled keypoint image data based on the first and second rendered images, this data can be used to train a parameterized face reconstruction model and improve its prediction accuracy. 400 face images are randomly selected, and a ResNet34 (Residual Network) is used as the parameterized face reconstruction model for training. The trained model is denoted as I. Using the parameterized face reconstruction model, 3000 new labeled face images with new face attributes and perspectives are generated based on the 400 images. The ResNet34 is then trained on these 3400 expanded images. The trained model is denoted as II. Both models are trained for the same number of iterations and tested on randomly selected face images that were not used in the training. Figure 4 As shown, Model II's predictions are significantly more accurate than Model I's predictions at key points of the mouth and eyes. The latter's predictions deviate considerably from the true values, while the former almost perfectly matches the true values. The test results are shown in Table 1, which compares the NME (Normalized Mean Squared) and MSE (Mean Square Error) values ​​of Model I and Model II.

[0121] Table 1

[0122]

[0123] According to embodiments of this disclosure, when comparing with synthetic face data, 3000 synthetic faces are randomly selected from the synthetic faces, and ResNet34 is trained on them, denoted as model III. Similarly, 10000 face images are randomly generated using pre-trained HeadNeRF V2, and 3000 are randomly selected as training data. ResNet34 is trained on them, denoted as model IV. The same experiment is performed ten times, and the average NME and MSE values ​​are calculated on 100 randomly selected face images that were not used in training. In particular, since the keypoints in the synthetic dataset are labeled as 3D keypoints, for fairness, the jawline annotations are converted from 3D to 2D using InsightFace. The test results are shown in Table 2, which shows the comparison results of the NME and MSE values ​​of model III and model IV.

[0124] Table 2

[0125]

[0126] According to embodiments of this disclosure, to quantitatively characterize the improvement effect of data generated by the parameterized face reconstruction model on the training effect of the parameterized face reconstruction model, 400 face images were randomly selected as the original training data, and ResNet34 was used as the parameterized face reconstruction model for training. Then, N, 2N, 4N, 8N, 16N, 32N, and 64N newly generated face images using HeadNeRF V2 were added to the training data, and ResNet34 was trained on them. When N was set to 500, the model was tested on 150 randomly selected face images that were not used in the training. For each sample size, the experiment was repeated 10 times, and the average NME and MSE values ​​were calculated for each trained model. The training results are shown in Table 3. It can be seen that as newly generated data is gradually added, the NME and MSE values ​​of the trained model decrease significantly. This strongly demonstrates that the newly generated labeled data has a significant improvement effect on the training effect of the keypoint prediction model. Table 3 shows the training results data of the parameterized face reconstruction model.

[0127] Table 3

[0128]

[0129] Figure 5 A flowchart illustrating a method for generating facial key point label data according to an embodiment of the present disclosure is shown.

[0130] like Figure 5 As shown, the method includes operations S501 and S502.

[0131] In operation S501, the target face image or a randomly given set of face hidden codes is obtained.

[0132] In operation S502, the target face image or a randomly given face hidden code set is input into the parameterized face reconstruction model to obtain the first output rendering image, the second output rendering image, and the face hidden code set.

[0133] In operation S503, the first output rendering image and the second output rendering image are processed based on the face hidden code set corresponding to the target face image or a randomly given face hidden code set to obtain a face key point annotation map. The parametric face reconstruction model is obtained by using the parametric face reconstruction model training method.

[0134] According to embodiments of this disclosure, the acquisition of a target face image or a randomly given face steganography set complies with relevant laws and regulations, and necessary confidentiality measures are taken. Furthermore, user authorization or consent is obtained before acquiring or collecting the user's target face image or randomly given face steganography set. The target face image or randomly given face steganography set is input into a parametric face reconstruction model. The model fits the steganography values ​​of each attribute in the face steganography set corresponding to the target face image or the randomly given face steganography set to obtain a first output rendering image and a second output rendering image. The steganography values ​​of attributes of interest (such as illumination steganography) in the face steganography set are edited, and then 3D-2D projection is performed on the expression steganography to obtain a facial key point annotation map.

[0135] Based on the above-described parametric face reconstruction model training method, this disclosure also provides a parametric face reconstruction model training device. The following will combine... Figure 6 The device is described in detail.

[0136] Figure 6 A schematic block diagram of a parametric face reconstruction model training apparatus according to an embodiment of the present disclosure is shown.

[0137] like Figure 6 As shown, the parametric face reconstruction model training device 600 of this embodiment includes a hidden code extraction module 610, a first processing module 620, a first output module 630, a second output module 640, a third output module 650, and a training module 650.

[0138] The hidden code extraction module 610 is used to extract hidden codes from the face key point annotation sample image to obtain a hidden code set. In one embodiment, the hidden code extraction module 610 can be used to perform the operation S201 described above, which will not be repeated here.

[0139] The first processing module 620 is used to input the hidden code set into the color recognition network of the initial parameterized face reconstruction model, and use the first rendering module in the color recognition network to process the face key point annotation sample map based on the hidden code set to obtain the first feature map. In one embodiment, the first processing module 620 can be used to perform the operation S202 described above, which will not be repeated here.

[0140] The first output module 630 is used to input the hidden code set into the generator module in the color recognition network and output the style hidden code. In one embodiment, the first output module 630 can be used to perform the operation S203 described above, which will not be repeated here.

[0141] The second output module 640 is used to process the first feature map based on the style latent code using the first upsampling module in the color recognition network, and output the first rendered image. In one embodiment, the second output module 640 can be used to perform the operation S204 described above, which will not be repeated here.

[0142] The third output module 650 is used to input the hidden code set into the semantic recognition network of the initial parameterized face reconstruction model and output the second rendered image. In one embodiment, the third output module 650 can be used to perform the operation S205 described above, which will not be repeated here.

[0143] The training module 660 is used to adjust the model parameters of the initial parameterized face reconstruction model by utilizing the loss between the first and second rendered images and the real face sample images corresponding to the face key point annotation sample images, thereby obtaining the target parameterized face reconstruction model. In one embodiment, the training module 660 can be used to perform the operation S206 described above, which will not be repeated here.

[0144] According to embodiments of this disclosure, a parametric face reconstruction model refines the features of the facial key point annotation sample image in terms of color and semantics, thereby generating a first rendered image and a second rendered image. This provides a large amount of high-quality training data for the training of the parametric face reconstruction model. Furthermore, in the process of generating the first rendered image, the parametric face reconstruction model inputs a latent code set into the generator module of the color recognition network, outputting a style latent code; and utilizes the first upsampling module in the color recognition network to process the first feature map based on the style latent code, outputting the first rendered image. Because the generator module can further capture the style latent code of the facial key point annotation sample image, the first upsampling module effectively reduces aliasing information in the color recognition network when processing the first feature map, maintaining the translation and rotation invariance of information transmission in the color recognition network. This results in images generated by the parametric face reconstruction model having high realism and high clarity, while avoiding hair adhesion problems.

[0145] According to embodiments of this disclosure, the steganography extraction module 610 includes a steganography extraction submodule, an identity determination submodule, an extraction submodule, a standardization submodule, and a steganography set generation submodule.

[0146] The hidden code extraction submodule is used to extract hidden codes from the face key point annotation sample image using a preset method to obtain an initial hidden code set, which includes initial identity hidden code, initial expression hidden code, texture hidden code, and illumination hidden code.

[0147] The identity determination submodule is used to input the sample image of the face key point annotation into the preset face recognition network and output the target identity hidden code.

[0148] The extraction submodule is used to extract a preset number of key points from the face key point annotation sample image.

[0149] The standardization submodule is used to standardize key points to obtain the target facial expression code.

[0150] The steganography set generation submodule is used to generate a target steganography set based on target identity steganography, target expression steganography, texture steganography, and illumination steganography.

[0151] According to embodiments of this disclosure, the first processing module 620 includes a prediction submodule, a mapping submodule, an integration submodule, and a first feature generation submodule.

[0152] The prediction submodule is used to input the target implicit code set into the implicit neural function, predict the facial key point annotation sample map based on the target identity implicit code and the target expression implicit code, and output density value and intermediate features.

[0153] The mapping submodule is used to map intermediate features based on texture latent codes and illumination latent codes to obtain high-dimensional features.

[0154] The integration submodule is used to integrate each light sampling point in the face key point annotation sample image based on the density value to obtain the intermediate density value.

[0155] The first feature generation submodule is used to generate a first feature map based on the density median and high-dimensional features using the first rendering module.

[0156] According to embodiments of this disclosure, the second output module 640 includes a conversion submodule and a first output submodule.

[0157] The transformation submodule is used to transform the first feature map into a Fourier feature distribution based on the mean square error condition, thus obtaining the transformed feature map.

[0158] The first output submodule is used to input the transformed feature map and style code into the first upsampling module and output the first rendered image.

[0159] According to embodiments of this disclosure, the third output module 650 includes a processing submodule and a second output submodule.

[0160] The processing submodule is used to process the face key point annotation sample map based on the hidden code set using the second rendering module in the semantic recognition network to obtain the second feature map;

[0161] The second output submodule is used to process the second feature map using the second upsampling module in the semantic recognition network and output the second rendered map.

[0162] According to embodiments of this disclosure, training module 660 includes a loss submodule and a training submodule.

[0163] The loss submodule is used to input the first ground truth value and the first rendered image corresponding to the real face sample image, as well as the second ground truth value and the second rendered image corresponding to the real face sample image, into the loss function, and output the model loss value.

[0164] The training submodule is used to adjust the model parameters of the initial parameterized face reconstruction model based on the model loss value to obtain the target parameterized face reconstruction model.

[0165] According to embodiments of this disclosure, the loss submodule includes a first pixel loss unit, a second pixel loss unit, a decoupling loss unit, a first perceptual loss unit, a second perceptual loss unit, a semantic loss unit, a distribution loss unit, and a model loss unit.

[0166] The first pixel loss unit is used to input the first real value and the first rendered image into the pixel loss function and output the first pixel loss value.

[0167] The second pixel loss unit is used to input the second true value and the second rendered image into the pixel loss function and output the second pixel loss value.

[0168] The decoupling loss unit is used to input the predicted hidden code sets corresponding to the first and second rendered images, and the hidden code sets corresponding to the real face sample images into the decoupling loss function, and output the decoupling loss value.

[0169] The first perceptual loss unit is used to input the first rendered image and the first real value into the image perceptual loss function and output the first image perceptual loss value.

[0170] The second perceptual loss unit is used to input the second rendered image and the second real value into the image perceptual loss function and output the second image perceptual loss value.

[0171] A semantic loss unit is used to determine the semantic loss value based on the second pixel loss value and the image perception loss value.

[0172] The distribution loss unit is used to input the first feature map and the preset feature distribution into the distribution loss function and output the distribution loss value.

[0173] The model loss unit is used to determine the model loss value based on the first pixel loss value, the first image perception loss value, the decoupling loss value, the distribution loss value, and the semantic loss value.

[0174] Based on the above-described method for generating facial landmark label data, this disclosure also provides a device for generating facial landmark label data. The following will be combined with... Figure 7 The device is described in detail.

[0175] Figure 7 The diagram illustrates a structural block diagram of a face key point labeling data generation apparatus according to an embodiment of the present disclosure.

[0176] like Figure 7 As shown, the parametric face reconstruction model training device 700 of this embodiment includes an acquisition module 710, a fitting module 720, and a labeling module 730.

[0177] The acquisition module 710 is used to acquire a target face image or a randomly given face hidden code set. In one embodiment, the acquisition module 710 can be used to perform the operation S501 described above, which will not be repeated here.

[0178] The fitting module 720 is used to input the target face image or a randomly given face hidden code set into the parameterized face reconstruction model to obtain a first output rendering image, a second output rendering image, and a face hidden code set. In one embodiment, the annotation module 720 can be used to perform the operation S502 described above, which will not be repeated here.

[0179] The annotation module 730 is used to process the first output rendering image and the second output rendering image based on the face hidden code set corresponding to the target face image or a randomly given face hidden code set to obtain a face key point annotation map. The parametric face reconstruction model is obtained using a training method. In one embodiment, the annotation module 730 can be used to perform the operation S503 described above, which will not be repeated here.

[0180] According to embodiments of this disclosure, any multiple modules among the hidden code extraction module 610, first processing module 620, first output module 630, second output module 640, third output module 650, training module 650, acquisition module 710, fitting module 720, and annotation module 730 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the following modules—hidden code extraction module 610, first processing module 620, first output module 630, second output module 640, third output module 650, and training module 650—and the following modules—acquisition module 710, fitting module 720, and annotation module 730—can be at least partially implemented as hardware circuits, such as field-programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems-on-a-chip, systems-on-a-substrate, systems-on-package, application-specific integrated circuits (ASICs), or any other reasonable means of integrating or packaging circuits, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three methods. Alternatively, at least one of the following modules—hidden code extraction module 610, first processing module 620, first output module 630, second output module 640, third output module 650, and training module 650—and the following modules—acquisition module 710, fitting module 720, and annotation module 730—can be at least partially implemented as computer program modules, which, when run, can perform corresponding functions.

[0181] Figure 8 The diagram illustrates an electronic device suitable for implementing a parametric face reconstruction model training method and a face generation method with key point labels, according to embodiments of the present disclosure.

[0182] like Figure 8 As shown, an electronic device 800 according to an embodiment of this disclosure includes a processor 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage portion 808 into a random access memory (RAM) 803. The processor 801 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 801 may also include onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this disclosure.

[0183] RAM 803 stores various programs and data required for the operation of electronic device 800. Processor 801, ROM 802, and RAM 803 are interconnected via bus 804. Processor 801 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 802 and / or RAM 803. It should be noted that the programs may also be stored in one or more memories other than ROM 802 and RAM 803. Processor 801 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0184] According to embodiments of this disclosure, the electronic device 800 may further include an input / output (I / O) interface 805, which is also connected to a bus 804. The electronic device 800 may also include one or more of the following components connected to the input / output (I / O) interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the input / output (I / O) interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 810 as needed so that computer programs read from it can be installed into the storage section 808 as needed.

[0185] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0186] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 802 and / or RAM 803 and / or one or more memories other than ROM 802 and RAM 803 described above.

[0187] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code enables the computer system to implement the parametric face reconstruction model training method and the face key point labeling data generation method provided in the embodiments of this disclosure.

[0188] When the computer program is executed by the processor 801, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0189] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 809, and / or installed from a removable medium 811. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0190] In such an embodiment, the computer program can be downloaded and installed from a network via communication section 809, and / or installed from removable medium 811. When the computer program is executed by processor 801, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0191] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0192] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0193] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0194] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A method for training a parametric face reconstruction model, comprising: Hidden codes are extracted from the face landmark annotation sample image to obtain the hidden code set; The hidden code set is input into the color recognition network of the initial parameterized face reconstruction model. The first rendering module in the color recognition network processes the face key point annotation sample map based on the hidden code set to obtain the first feature map. The hidden code set is input into the generator module in the color recognition network, and the style hidden code is output. Using the first upsampling module in the color recognition network, the first feature map is processed based on the style code to output the first rendered image; The hidden code set is input into the semantic recognition network of the initial parameterized face reconstruction model, and a second rendering image is output. The model parameters of the initial parameterized face reconstruction model are adjusted by using the loss between the first and second rendered images and the real face sample images corresponding to the face key point annotation sample images to obtain the target parameterized face reconstruction model. The process of extracting hidden codes from the face key point annotation sample image yields a hidden code set, including: Generate a target implicit code set based on target identity implicit code, target expression implicit code, texture implicit code, and illumination implicit code; The first rendering module in the color recognition network processes the facial key point annotation sample map based on the hidden code set to obtain a first feature map, including: The target implicit code set is input into the implicit neural function, and the facial key point annotation sample map is predicted based on the target identity implicit code and the target expression implicit code, and the density value and intermediate features are output. Based on the texture code and the illumination code, the intermediate features are mapped to obtain high-dimensional features; Based on the density value, the intermediate density value is obtained by integrating each light sampling point in the face key point annotation sample image; The first feature map is generated using the first rendering module based on the intermediate density value and the high-dimensional features.

2. The method according to claim 1, wherein, The process of extracting hidden codes from the face key point annotation sample image yields a hidden code set, including: The facial key point annotation sample image is extracted using a preset method to obtain an initial hidden code set, which includes an initial identity hidden code, an initial expression hidden code, a texture hidden code, and an illumination hidden code. The facial key point annotation sample image is input into a preset facial recognition network, and the target identity hidden code is output. Extract a preset number of key points from the facial key point annotation sample image; The key points are standardized to obtain the target facial expression code; A target steganography set is generated based on the target identity steganography, the target facial expression steganography, the texture steganography, and the illumination steganography.

3. The method according to claim 1, wherein, The step of using the first upsampling module in the color recognition network to process the first feature map based on the style latent code and output the first rendered image includes: Based on the mean square error condition, the first feature map is transformed into a Fourier feature distribution to obtain the transformed feature map; The transformed feature map and the style code are input into the first upsampling module, and the first rendered image is output.

4. The method according to claim 1, wherein, The step of inputting the hidden code set into the semantic recognition network of the parameterized face reconstruction model and outputting the second rendered image includes: Using the second rendering module in the semantic recognition network, the facial key point annotation sample map is processed based on the hidden code set to obtain the second feature map; The second feature map is processed using the second upsampling module in the semantic recognition network to output the second rendered map.

5. The method according to claim 1, wherein, The step of adjusting the model parameters of the initial parameterized face reconstruction model using the loss between the first rendered image and the second rendered image and the real face sample image corresponding to the face key point annotation sample image to obtain the target parameterized face reconstruction model includes: The first ground truth value and the first rendered image corresponding to the real face sample image, as well as the second ground truth value and the second rendered image corresponding to the real face sample image, are input into the loss function, and the model loss value is output. The model parameters of the initial parameterized face reconstruction model are adjusted based on the model loss value to obtain the target parameterized face reconstruction model.

6. The method according to claim 5, wherein, The step of inputting the first ground truth value and the first rendered image corresponding to the real face sample image, as well as the second ground truth value and the second rendered image corresponding to the real face sample image, into the loss function, and outputting the model loss value, includes: The first true value and the first rendered image are input into the pixel loss function, and the first pixel loss value is output. The second true value and the second rendered image are input into the pixel loss function, and the second pixel loss value is output. Input the predicted hidden code sets corresponding to the first and second rendered images, and the hidden code set corresponding to the real face sample image into the decoupling loss function, and output the decoupling loss value; The first rendered image and the first real value are input into the image perception loss function, and the first image perception loss value is output. The second rendered image and the second real value are input into the image perception loss function, and the second image perception loss value is output. The semantic loss value is determined based on the second pixel loss value and the image perception loss value; The first feature map and the preset feature distribution are input into the distribution loss function, and the distribution loss value is output. The model loss value is determined based on the first pixel loss value, the first image perception loss value, the decoupling loss value, the distribution loss value, and the semantic loss value.

7. A method for generating facial landmark-labeled data, comprising: Obtain the target face image or a randomly given set of hidden face codes; The target face image or the randomly given face hidden code set is input into the parameterized face reconstruction model to obtain the first output rendering image and the second output rendering image. The first output rendering image and the second output rendering image are processed based on the face hidden code set corresponding to the target face image or the randomly given face hidden code set to obtain a face key point annotation image. The parameterized face reconstruction model is obtained using the training method described in any one of claims 1 to 6.

8. A parametric face reconstruction model training device, comprising: The hidden code extraction module is used to extract hidden codes from the face key point annotation sample image to obtain the hidden code set; The first processing module is used to input the hidden code set into the color recognition network of the initial parameterized face reconstruction model, and use the first rendering module in the color recognition network to process the face key point annotation sample map based on the hidden code set to obtain the first feature map. The first output module is used to input the hidden code set into the generator module in the color recognition network and output the style hidden code; The second output module is used to process the first feature map based on the style code using the first upsampling module in the color recognition network and output the first rendering image. The third output module is used to input the hidden code set into the semantic recognition network of the initial parameterized face reconstruction model and output the second rendering image. The training module is used to adjust the model parameters of the initial parameterized face reconstruction model by using the loss between the first rendered image and the second rendered image and the real face sample image corresponding to the face key point annotation sample image, so as to obtain the target parameterized face reconstruction model. The steganography set generation submodule is used to generate a target steganography set based on target identity steganography, target expression steganography, texture steganography, and illumination steganography. The prediction submodule is used to input the target hidden code set into the implicit neural function, predict the facial key point annotation sample map based on the target identity hidden code and the target expression hidden code, and output density value and intermediate features. The mapping submodule is used to map the intermediate features based on the texture code and the illumination code to obtain high-dimensional features; The integration submodule is used to integrate each light sampling point in the face key point annotation sample image based on the density value to obtain an intermediate density value; The first feature generation submodule is used to generate the first feature map based on the density intermediate value and the high-dimensional feature using the first rendering module.

9. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors perform the method according to any one of claims 1 to 7.

Images