Facial model generation method, device, equipment and storage medium

By acquiring multiple face images and iterating the parameters, combining multi-view features, the ambiguity problem caused by a single image is solved, the accuracy and fineness of the face model are improved, and more stable model generation is achieved.

CN116012513BActive Publication Date: 2024-11-29TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111221324.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-10-20
Publication Date
2024-11-29
Estimated Expiration
2041-10-20

AI Technical Summary

Technical Problem

In the prior art, the ambiguity problem caused by a single face image leads to inaccurate face models and lack of multi-view image features, which affects the accuracy and fineness of model generation.

Method used

By acquiring multiple face images of the target object, extracting shared parameters and independent parameters, generating rendered images, calculating matching losses, iterating the parameters to generate the target face model, and improving the accuracy and fineness of the model with multi-view features.

Benefits of technology

The accuracy and precision of face model generation is improved, the ambiguity problem caused by single images is solved, and the richness and stability of the model are enhanced through multi-view features.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116012513B_ABST
    Figure CN116012513B_ABST
Patent Text Reader

Abstract

The present application discloses a facial model generation method, device, equipment and storage medium, and relates to the field of artificial intelligence technology. The method includes: obtaining shared parameters corresponding to multiple facial images of a target object, and independent parameters corresponding to the multiple facial images; based on the shared parameters and the independent parameters, obtaining rendering images corresponding to the multiple facial images; based on the multiple facial images and the multiple rendering images, determining the matching loss; with the convergence matching loss as the goal, obtaining the target shared parameters corresponding to the multiple facial images and the target independent parameters corresponding to the multiple facial images; on the basis of the target shared parameters corresponding to the multiple facial images, combining the target independent parameters corresponding to the multiple facial images, generating the facial model of the target object. The present application improves the accuracy and refinement of facial model generation by associating multiple facial images of the target object to obtain the facial model of the target object.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to a facial model generation method, device, equipment and storage medium. Background Art

[0002] With the development of artificial intelligence technology, developers can now automatically reconstruct human facial models by adopting computer vision algorithms.

[0003] In the related art, the automatic reconstruction of a human face model is achieved by using 3DMM (3D Morphable Model). For example, for a given face picture, a set of parameters is first set based on 3DMM to parameterize the face picture, and then a face model corresponding to the face picture is generated by 3DMM according to the set of parameters, and then the face model is rendered into a rendered picture, and finally the set of parameters is adjusted to make the rendered picture as close as possible to the face picture, thereby obtaining a final face model corresponding to the face picture.

[0004] However, since a single image is often ambiguous, the face model obtained by related technologies is not accurate enough. Summary of the invention

[0005] The embodiments of the present application provide a facial model generation method, apparatus, device and storage medium, which can improve the accuracy and refinement of facial model generation. The technical solution is as follows:

[0006] According to one aspect of an embodiment of the present application, a method for generating a face model is provided, the method comprising:

[0007] Acquire multiple facial images of the target object;

[0008] Acquire shared parameters corresponding to the multiple facial images, and independent parameters corresponding to the multiple facial images respectively; wherein the shared parameters are used to characterize the common features between the multiple facial images, and the independent parameters are used to characterize the private features corresponding to the multiple facial images respectively;

[0009] Obtaining rendering images corresponding to the multiple facial images respectively based on the shared parameters corresponding to the multiple facial images and the independent parameters corresponding to the multiple facial images respectively;

[0010] Determining a matching loss based on the plurality of facial images and the rendered images respectively corresponding to the plurality of facial images, wherein the matching loss is used to characterize a degree of difference between the plurality of facial images and the rendered images respectively corresponding to the plurality of facial images;

[0011] With the goal of converging the matching loss, adjusting the shared parameters corresponding to the multiple facial images and the independent parameters respectively corresponding to the multiple facial images to obtain target shared parameters corresponding to the multiple facial images and target independent parameters respectively corresponding to the multiple facial images;

[0012] Based on the target shared parameters corresponding to the multiple facial images, the target independent parameters corresponding to the multiple facial images are combined to generate a facial model of the target object.

[0013] According to one aspect of an embodiment of the present application, a facial model generation device is provided, the device comprising:

[0014] A facial image acquisition module, used to acquire multiple facial images of a target object;

[0015] An initial parameter acquisition module, used to acquire shared parameters corresponding to the multiple facial images, and independent parameters corresponding to the multiple facial images respectively; wherein the shared parameters are used to characterize the common features between the multiple facial images, and the independent parameters are used to characterize the private features corresponding to the multiple facial images respectively;

[0016] A rendering image acquisition module, configured to obtain rendering images corresponding to the plurality of facial images respectively based on the shared parameters corresponding to the plurality of facial images and the independent parameters corresponding to the plurality of facial images respectively;

[0017] a matching loss acquisition module, configured to determine a matching loss based on the plurality of facial images and the rendered images respectively corresponding to the plurality of facial images, wherein the matching loss is used to characterize the difference between the plurality of facial images and the rendered images respectively corresponding to the plurality of facial images;

[0018] a target parameter acquisition module, configured to adjust the shared parameters corresponding to the plurality of facial images and the independent parameters respectively corresponding to the plurality of facial images with the goal of converging the matching loss, so as to obtain target shared parameters corresponding to the plurality of facial images and target independent parameters respectively corresponding to the plurality of facial images;

[0019] The face model generation module is used to generate a face model of the target object based on the target shared parameters corresponding to the multiple face images and in combination with the target independent parameters respectively corresponding to the multiple face images.

[0020] According to one aspect of an embodiment of the present application, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the above-mentioned facial model generation method.

[0021] The computer device is a terminal or a server.

[0022] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the above-mentioned facial model generation method.

[0023] According to one aspect of the embodiments of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above-mentioned face model generation method.

[0024] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0025] By parameterizing and representing multiple facial images of a target object, and associating the multiple facial images of the target object through shared parameters (such as shared shape parameters and shared texture parameters) to obtain rendered images corresponding to the multiple facial images, and then iteratively optimizing the facial model of the target object based on the difference between the multiple facial images and the rendered images corresponding to the multiple facial images, the ambiguity problem caused by a single image in related technologies can be solved, thereby improving the accuracy of facial model generation, and by collecting multiple facial images, rich image features corresponding to the target object (such as multi-view image features) can be obtained, thereby improving the refinement of facial model generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0027] Figure 1 It is a flow chart of a solution implementation environment provided by an embodiment of the present application;

[0028] Figure 2 is a flowchart of a facial model generation method provided by an embodiment of the present application;

[0029] Figure 3 is a schematic diagram of a face model provided by an embodiment of the present application;

[0030] Figure 4 is a schematic diagram of a facial image provided by an embodiment of the present application;

[0031] Figure 5 is a schematic diagram of a facial topology structure of a converted facial model provided by an embodiment of the present application;

[0032] Figure 6 is a schematic diagram of a method for obtaining total loss provided by an embodiment of the present application;

[0033] Figure 7 is a schematic diagram of a comparison result based on public test data provided by an embodiment of the present application;

[0034] Figure 8 is a block diagram of a facial model generation device provided by an embodiment of the present application;

[0035] Fig. 9 is a block diagram of a facial model generating device provided by another embodiment of the present application;

[0036] Fig.10 It is a block diagram of a computer device provided by one embodiment of the present application. DETAILED DESCRIPTION

[0037] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0038] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0039] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0040] Computer vision technology (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision such as using cameras and computers to replace human eyes to identify and measure targets, and further perform graphic processing to make computer processing into images that are more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR (Optical Character Recognition), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and mapping, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0041] The technical solution provided in the embodiment of the present application involves artificial intelligence computer vision technology, which uses computer vision technology to parameterize multiple facial images of a target object and associate the multiple facial images, thereby obtaining a facial model of the target object based on the multiple facial images of the target object.

[0042] In the method provided in the embodiment of the present application, the execution subject of each step may be a computer device, which refers to an electronic device with data calculation, processing and storage capabilities. The computer device may be a terminal such as a PC (Personal Computer), a tablet computer, a smart phone, a wearable device, an intelligent robot, etc.; or it may be a server. Among them, the server may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0043] The technical solution provided by the embodiment of the present application is applicable to any scenario where facial model generation is required, such as facial model reconstruction scenario, face pinching system (such as generating a corresponding facial model with a set style (such as a game style) based on a facial image uploaded by a user), facial recognition scenario, etc. The technical solution provided by the embodiment of the present application can effectively improve the accuracy and refinement of the facial model.

[0044] In one example, if Figure 1 As shown, taking the face pinching system as an example, the system may include a terminal 10 and a server 20.

[0045] The terminal 10 may be an electronic device such as a mobile phone, a digital camera, a tablet computer, a PC, a wearable device, etc. The user may access the server 20 through the terminal 10 and perform a facial model generation operation. For example, a client of a target application may be installed in the terminal 10, and the user may access the server 20 through the client and perform a facial model generation operation. The target application may be a facial model generation application, a face pinching application, a game application, an entertainment application, etc., which is not limited in the embodiments of the present application.

[0046] The server 20 may be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The server 20 is used to provide background services for the client of the target application in the terminal 10. For example, the server 20 may be a background server of the above-mentioned target application (such as a face model generation application).

[0047] The terminal 10 and the server 20 can communicate with each other via the network 30 .

[0048] Exemplarily, the user inputs multiple facial images of the target object in the client of the target application, the client sends the multiple facial images to the server 20, the server 20 generates a facial model of the target object based on the multiple facial images, and the server 20 then sends the facial model to the client.

[0049] Of course, in some other examples, the facial model generation method can also be executed in the terminal 10, such as in the client of the above-mentioned target application. For example, the client can directly generate the facial image of the target object based on multiple facial images of the target object input by the user, and the embodiment of the present application is not limited to this.

[0050] Please refer to Figure 2 , which shows a flowchart of a facial model generation method provided by an embodiment of the present application. The execution subject of each step of the method can be Figure 1 In the terminal 10 or server 20 in the implementation environment of the solution shown, the method may include the following steps (201-206).

[0051] Step 201: Acquire multiple facial images of a target object.

[0052] In the embodiment of the present application, the target object refers to the object for which a face model needs to be generated, which can be any object, such as a person, an animal, a virtual character, a virtual animal, etc. The embodiment of the present application does not limit the target object. The face model refers to the model corresponding to the face area of ​​the target object. For example, refer to Figure 3, face model 301 is a left side 3D face model of the target object, face model 302 is a front 3D face model of the target object, and face model 303 is a right side 3D face model of the target object.

[0053] A facial image refers to an image including the facial area of ​​the target object. For example, the facial image may be an image corresponding to the target object in different facial postures, or an image corresponding to the target object in different facial expressions, or an image corresponding to the target object in different facial illuminations.

[0054] In one example, the selected high-quality facial images can be used as facial images required for generating a facial model of the target object to improve the quality of facial model generation. The selection requirements of the multiple facial images can be as follows: 1. The multiple facial images can include images corresponding to the target object in various facial postures to avoid multiple facial images being concentrated in frontal or side facial postures; 2. For facial expressions, extremely exaggerated expressions need to be avoided; 3. For facial lighting, extreme lighting conditions need to be avoided. Among them, for facial expressions and facial lighting, individual extreme expressions and extreme lighting conditions will not have a serious impact on the generation of the facial model. Exemplarily, the multiple facial images can simultaneously include facial images with a frontal facial posture and a side facial posture that differ by 90 degrees. Optionally, in the case where there are only facial images in frontal facial posture and side facial posture, the facial image in the side facial posture can be horizontally mirrored to obtain other facial images of the target object. For example, refer to Figure 4 , facial image 401 is a right side facial image, facial image 402 is a left side facial image, and facial image 403 is a frontal facial image. Facial image 401 can be obtained by horizontally mirroring facial image 402. Optionally, the frontal facial image can also be obtained by horizontally mirroring the side facial image.

[0055] Optionally, when only one original image of the target object is obtained, a second original image can be constructed based on the original image by adopting a symmetry method (for example, flipping the original image left to right), thereby obtaining multiple facial images of the target object, and then converting the single-view optimization problem into a pseudo-multi-view optimization problem.

[0056] Optionally, facial images may be selected by manual selection or by using a trained network model, which is not limited in the embodiments of the present application.

[0057] In one example, after selecting multiple facial images of the target object, key point detection can be performed on the multiple facial images to obtain key point position sets corresponding to the multiple facial images, and then the multiple facial images can be aligned based on the key point position sets corresponding to the multiple facial images to further improve the quality of the facial images, thereby improving the quality of the facial model generation. Figure 4 , facial image 401 , facial image 402 , and facial image 403 are aligned at key points (such as eyes, nose, mouth, etc.).

[0058] Optionally, a segmentation network can be used to detect the facial region of the image, thereby segmenting a facial image focused on the facial region to further improve the quality of the facial image. At the same time, the image features (such as facial features) corresponding to each facial image can be extracted through a recognition network.

[0059] Step 202, obtaining shared parameters corresponding to the multiple facial images and independent parameters corresponding to the multiple facial images respectively; wherein the shared parameters are used to characterize the common features between the multiple facial images, and the independent parameters are used to characterize the private features corresponding to the multiple facial images respectively.

[0060] In the embodiment of the present application, multiple facial images can be parameterized by using computer vision technology. For example, taking 3DMM as an example, a facial scanner can be used to obtain a plurality of facial models with natural expressions (such as 3D facial models), and then a registration algorithm can be used to convert the topological structures of the plurality of facial models into a consistent one, which can be used to obtain the facial models. 1 , S 2 , S 3 , …, S N ). Assuming that any facial shape S with natural expressions can be approximated by linear interpolation of known facial models, we have:

[0061]

[0062] Among them, S i is the i-th face model, N is the number of face models, α i is the shape parameter (ie, facial shape coefficient) corresponding to the i-th face model.

[0063] Then the facial model generation problem is transformed into solving the linear difference coefficient α i Optionally, the matrix (S 1 , S 2 , S 3, …, S N ) is compressed to obtain:

[0064]

[0065] Among them, S is the shape mean representation corresponding to N face models obtained based on PCA technology, s i It is the former M 1 The principal components (M 1 Generally, we choose less than N), we will It is called the facial shape base. is called the shape coefficient (i.e. a shared set of shape parameters).

[0066] For any facial texture T, we also assume that it can be obtained by linear interpolation of known facial textures. Using operations similar to those for facial shapes, we can obtain:

[0067]

[0068] in, is the texture mean representation corresponding to the N face models obtained based on PCA technology, and ti is the first M 2 The principal components (M 2 Generally, we choose less than N), we will It is called facial texture base. It is called texture coefficients (i.e. a shared texture parameter set).

[0069] For any facial expression E, we also assume that it can be obtained by linear interpolation of known facial expressions. Using operations similar to those for facial shapes, we can obtain:

[0070]

[0071] in, is the mean expression representation corresponding to the N face models obtained based on PCA technology, ei is the first M 3 The principal components (M 3 Generally, we choose less than N), we will It is called facial texture base. It is called the expression coefficient (i.e., sub-expression parameter set).

[0072] In this way, through 3DMM, we can parameterize the facial model. In the embodiment of the present application, considering that multiple facial images correspond to the same target object, the facial shapes and facial textures corresponding to the multiple facial images should be consistent, while facial expressions, facial postures and facial illuminations should be independent. Therefore, facial shapes and facial textures can be used as shared parameters, and facial expressions, facial postures and facial illuminations can be used as independent parameters, that is, the shared parameters can include shared shape parameters and shared texture parameters, and the independent parameters include expression parameters, posture parameters and illumination parameters. Among them, the shared shape parameters are used to characterize the facial shape of the target object, the shared texture parameters are used to characterize the facial texture of the target object, the expression parameters are used to characterize the facial expressions of the target object in each facial image, the posture parameters are used to characterize the facial postures of the target object in each facial image, and the illumination parameters are used to characterize the facial illumination of the target object in each facial image.

[0073] Therefore, the generation task of the face model is transformed into Tasks. Among them, is a shared shape parameter set, For shared texture parameter sets, is the sub-expression parameter set corresponding to the i-th facial image, is the posture parameter set corresponding to the i-th facial image, is the illumination parameter set corresponding to the i-th facial image.

[0074] Exemplarily, in the initial stage of the solution, the initial shared shape parameter set and the initial shared texture parameter set corresponding to the target object can be obtained by randomly setting the shared shape parameters and the shared texture parameters, or by randomly setting the shared shape parameters and the shared texture parameters based on the target distribution, to obtain the initial shared shape parameter set and the initial shared texture parameter set corresponding to the target object. The sum of the shared shape parameters in the initial shared shape parameter set can be 1, and the sum of the shared texture parameters in the initial shared texture parameter set can be 1. Optionally, for the first facial image among the multiple facial images, the initial sub-expression parameter set corresponding to the first facial image can be obtained by randomly setting the expression parameters, and the sum of the expression parameters in the initial sub-expression parameter set can be 1. The initial posture parameters and the initial illumination parameters corresponding to the first facial image can be obtained by randomly setting the posture parameters and the illumination parameters.

[0075] Step 203 : obtaining rendered images corresponding to the multiple facial images respectively based on the shared parameters corresponding to the multiple facial images and the independent parameters corresponding to the multiple facial images respectively.

[0076] The rendered image refers to an image generated by rendering the face model into the image plane. Optionally, a differentiable rendering technique may be used to render the face model into the image plane to generate the rendered image.

[0077] In one example, the process of acquiring a rendered image can be as follows: for a first facial image among multiple facial images, a first sub-expression parameter set, a first posture parameter, and a first lighting parameter corresponding to the first facial image are set; according to a shared shape parameter set and a shared texture parameter set corresponding to the multiple facial images, and a first sub-expression parameter set, a first posture parameter, and a first lighting parameter corresponding to the first facial image, a rendered image corresponding to the first facial image is rendered.

[0078] The first facial image may be any facial image of the plurality of facial images. Optionally, the sub-expression parameters, posture parameters and illumination parameters of the first facial image may be randomly set to obtain a first sub-expression parameter set, a first posture parameter and a first illumination parameter corresponding to the first facial image.

[0079] Optionally, the facial shape of the first facial model corresponding to the first facial image can be constructed based on the shared shape parameter set, and then the facial texture of the first facial model can be constructed based on the shared texture parameter set on the basis of the above steps. Subsequently, based on the first sub-expression parameter set, the facial expression construction of the first facial model is completed. Finally, based on the first posture parameter and the first illumination parameter, the facial posture and facial illumination construction of the first facial model are completed, so as to obtain the final first facial model corresponding to the first facial image.

[0080] After the final first facial model corresponding to the first facial image is acquired, the first facial model is rendered into the image plane by using a differentiable rendering technique to obtain a rendered image corresponding to the first facial image.

[0081] Step 204 : determining a matching loss based on the plurality of facial images and the rendered images respectively corresponding to the plurality of facial images, where the matching loss is used to characterize the difference between the plurality of facial images and the rendered images respectively corresponding to the plurality of facial images.

[0082] Optionally, the matching loss includes a total re-rendering loss and a total key point loss. The total re-rendering loss refers to the sum of the re-rendering losses corresponding to the multiple facial images, which is used to characterize the total difference in image features between the multiple facial images and the rendered images corresponding to the multiple facial images, and the total key point loss refers to the sum of the key point losses corresponding to the multiple facial images, which is used to characterize the total difference in key point positions between the multiple facial images and the rendered images corresponding to the multiple facial images.

[0083] In one example, the process of obtaining the total re-rendering loss and the total key point loss can be as follows: obtain the re-rendering losses corresponding to multiple facial images respectively, and the re-rendering losses are used to characterize the differences in image features between the facial images and the rendered images corresponding to the facial images; sum the re-rendering losses corresponding to the multiple facial images to obtain the total re-rendering loss; obtain the key point losses corresponding to the multiple facial images respectively, and the key point losses are used to characterize the differences in key point positions between the facial images and the rendered images corresponding to the facial images; sum the key point losses corresponding to the multiple facial images to obtain the total key point loss.

[0084] Optionally, a method for obtaining a re-rendering loss may be as follows: for a first facial image among multiple facial images, performing feature extraction on the first facial image to obtain a first feature map, and performing feature extraction on a rendered image corresponding to the first facial image to obtain a second feature map; obtaining a first intermediate parameter based on the first feature map and the second feature map, the first intermediate parameter being used to characterize the difference between the first facial image and the rendered image corresponding to the first facial image; performing facial recognition on the first facial image to obtain a first facial region, and performing facial recognition on the rendered image corresponding to the first facial image to obtain a second facial region; obtaining a second intermediate parameter based on the first facial region and the second facial region, the second intermediate parameter being used to characterize an effective calculation region between the first facial region and the second facial region; obtaining a re-rendering loss corresponding to the first facial image based on the first intermediate parameter and the second intermediate parameter.

[0085] The first feature map refers to image feature information corresponding to the first facial image, and the second feature map refers to image feature information corresponding to the rendered image corresponding to the first facial image. The first intermediate parameter may refer to the difference between the first feature map and the second feature map. The first facial region refers to the facial region corresponding to the first facial image, and the second facial region refers to the facial region corresponding to the rendered image corresponding to the first facial image. The second intermediate parameter may refer to the product between the first feature map and the second feature map.

[0086] In an example, the re-rendering loss can be expressed as follows:

[0087]

[0088] Where N is the number of face images, I Render,i is the image feature of the rendered image corresponding to the i-th facial image (i.e., the second feature map), I i is the image feature of the i-th facial image (i.e., the first feature map), M i is the facial region corresponding to the ith facial image (i.e., the first facial region), M Render,i is the facial region of the rendered image corresponding to the i-th facial image (ie, the second facial region).

[0089] Optionally, the process of obtaining key point loss can be as follows: for a first facial image among multiple facial images, key point detection is performed on the first facial image to obtain a first key point position set, and key point detection is performed on a rendered image corresponding to the first facial image to obtain a second key point position set; based on the difference between the first key point position set and the second key point position set, the key point loss corresponding to the first facial image is obtained.

[0090] The first key point position set includes key point positions corresponding to the first facial image, and the second key point position set includes key point positions corresponding to the rendered image corresponding to the first facial image (such as the projection positions of key points on the facial model on the rendered image). The key point loss can be obtained based on the difference between the first key point position set and the second key point position set.

[0091] In an example, the keypoint loss can be expressed as follows:

[0092]

[0093] Where N is the number of face images, q Render,i is the key point position set of the rendered image corresponding to the i-th facial image, q i is the key point position set of the i-th facial image.

[0094] Optionally, after obtaining the total re-rendering loss and the total key point loss, the regularization loss can also be obtained to generate the total loss, and the process can be as follows: obtain the regularization loss corresponding to multiple facial images, and the regularization loss is used to regularize the shared shape parameters, shared texture parameters and expression parameters to maintain the facial structure of the facial model; obtain the total loss according to the matching loss and the regularization loss.

[0095] Among them, the process of obtaining the regularization loss can be as follows: based on a shared shape parameter set corresponding to multiple facial images, obtain the shape regularization loss corresponding to multiple facial images; based on a shared texture parameter set corresponding to multiple facial images, obtain the texture regularization loss corresponding to multiple facial images; based on the sub-expression parameter sets corresponding to the multiple facial images, obtain the expression regularization loss corresponding to the multiple facial images; perform weighted summation of the shape regularization loss, texture regularization loss and expression regularization loss to obtain the regularization loss corresponding to the multiple facial images.

[0096] The shape regularization loss is used to regularize the shared shape parameters, the texture regularization loss is used to regularize the shared texture parameters, and the expression regularization loss is used to regularize the expression parameters.

[0097] In an example, the regularization loss can be expressed as follows:

[0098]

[0099] Among them, w shp 、w tex and w ex They are the weights corresponding to shape regularization loss, texture regularization loss and expression regularization loss, N shp refers to the number of shared shape parameters in the shared shape parameter set, α j is the jth shared shape parameter in the shared shape parameter set, σ j is α j The corresponding standard deviation, N tex refers to the number of shared texture parameters in the shared texture parameter set, β j is the jth shared texture parameter in the shared texture parameter set, ρ j β j The corresponding standard deviation, N is the number of corresponding face images, N tex refers to the number of expression parameters in the sub-expression parameter set, γ i,j is the jth expression parameter in the sub-expression parameter set corresponding to the i-th facial image, τ i,j For γ i,j The corresponding standard deviation.

[0100] Optionally, α j , β j and γ i,j It can refer to the standard deviation corresponding to each principal component of the facial shape basis, facial texture basis and facial expression basis respectively.

[0101] In an example, the total loss can be expressed as follows:

[0102] L=w Render ·L Render +w Lm ·L Lm +w Reg ·L Reg ;

[0103] Among them, w Render 、w Lm 、w Reg are the weights corresponding to the total re-rendering loss, the total key point loss and the regularization loss, L Render is the total weight rendering loss, which can make the rendered image corresponding to the face image and the face image as close as possible, L Lm is the total key point loss, which can maintain the correct facial posture corresponding to the face model, L Reg It is a regularization loss, which can maintain the facial structure of the face model.

[0104] Step 205 , with the convergence matching loss as the goal, adjusting the shared parameters corresponding to the multiple facial images and the independent parameters corresponding to the multiple facial images respectively, to obtain the target shared parameters corresponding to the multiple facial images and the target independent parameters corresponding to the multiple facial images respectively.

[0105] The target shared parameters refer to the shared parameters after the initial shared parameters are iteratively optimized, which may include target shared shape parameters and target shared texture parameters. The target independent parameters refer to the independent parameters after the initial independent parameters are iteratively optimized, which may include target expression parameters, target posture parameters and target illumination parameters.

[0106] Exemplarily, with the convergence matching loss as the goal, the above-mentioned initial shared shape parameter set and initial shared texture parameter set, as well as the initial sub-expression parameter sets, initial posture parameters and initial lighting parameters corresponding to multiple facial images are adjusted to obtain a target shared shape parameter set and a target shared texture parameter set, as well as a target sub-expression parameter set, target posture parameters and target lighting parameters corresponding to multiple facial images.

[0107] Optionally, with the goal of converging the total re-rendering loss and the total key point loss, the shared parameters corresponding to the multiple facial images and the independent parameters corresponding to the multiple facial images are adjusted to obtain target shared parameters corresponding to the multiple facial images and target independent parameters corresponding to the multiple facial images. Alternatively, with the goal of converging the total loss, the shared parameters corresponding to the multiple facial images and the independent parameters corresponding to the multiple facial images are adjusted to obtain target shared parameters corresponding to the multiple facial images and target independent parameters corresponding to the multiple facial images.

[0108] In one example, the process of acquiring the target shared parameters and the target independent parameters can be as follows: in the process of converging the total loss, adjust the shared shape parameter set, shared texture parameter set, expression parameter set, posture parameter set and lighting parameter set; if it is detected that the total re-rendering loss stops changing, the convergence of the total loss is completed, and the target shared shape parameter set, target shared texture parameter set, target expression parameter set, target posture parameter set and target lighting parameter set are obtained; wherein the expression parameter set includes sub-expression parameter sets corresponding to the multiple facial images respectively, the posture parameter set includes posture parameters corresponding to the multiple facial images respectively, and the lighting parameter set includes lighting parameters corresponding to the multiple facial images respectively.

[0109] Optionally, at the beginning of the iteration, the weight of the total re-rendering loss can be set to be greater than the weight of the total key point loss and the weight of the regularization loss, and the total loss is minimized with the first iteration step; during the iteration process, the weight of the total key point loss, the weight of the regularization loss, and the first iteration step are gradually reduced.

[0110] Exemplarily, in the early stage of iteration, a larger weight and iteration step size can be used to make the shared parameters and independent parameters converge quickly to the vicinity of the optimal solution, and then the weight of the total key point loss and the weight of the regularization loss can be gradually reduced, while the iteration step size is reduced, so that the shared parameters and independent parameters can slowly and accurately converge to the optimal solution. Optionally, the total re-rendering loss (or matching loss) can be used as the termination condition for the end of the iteration. In response to several optimizations, it is recognized that the total re-rendering loss (or matching loss) no longer decreases, and the iteration can be stopped.

[0111] Step 206 , based on the target shared parameters corresponding to the multiple facial images, combined with the target independent parameters corresponding to the multiple facial images respectively, a facial model of the target object is generated.

[0112] Optionally, the facial model acquisition process can be as follows: for a first facial image among multiple facial images, based on a target shared shape parameter set and a target shared texture parameter set corresponding to the multiple facial images, combined with a target sub-expression parameter set, target posture parameters and target lighting parameters corresponding to the first facial image, a facial model corresponding to the first facial image is generated.

[0113] Exemplarily, for a first facial image among multiple facial images, the facial shape of the first facial model is obtained based on a target shared shape parameter set and a facial shape basis, the facial texture of the first facial model is obtained based on a target shared texture parameter set and a facial texture basis, the facial expression of the first facial model is obtained based on a target sub-expression parameter set and a facial expression basis corresponding to the first facial image, the facial posture of the first facial model is obtained based on a target posture parameter corresponding to the first facial image, the facial lighting corresponding to the first facial image is obtained based on a target illumination parameter corresponding to the first facial image, and finally, a first facial model corresponding to the first facial image is generated based on the facial shape, facial texture, facial expression, facial posture and facial illumination of the first facial model.

[0114] Optionally, the embodiment of the present application does not limit the number of generated face models. For example, face models corresponding to multiple face images can be generated, or face models of only some face images among multiple face images (such as a frontal face model and a side face model) can be generated.

[0115] Optionally, after generating the facial model of the target object, the facial model may be adjusted as required, and the specific process may be as follows: obtaining the target facial topology structure; adjusting the facial model according to the target facial topology structure to obtain an adjusted facial model, wherein the facial topology structure of the adjusted facial model is the same as the target facial topology structure.

[0116] The target facial topology refers to the facial topology required by the user, such as a game-type facial topology, a custom-type facial topology, an average facial model of BFM (Basel Face Model, an open source face database), etc. Exemplarily, when the facial topology of the obtained facial model is inconsistent with the facial topology required by the designer, a registration algorithm is used to deform the target facial topology onto the facial model based on the facial topology required by the designer, so that the facial topology of the adjusted facial model is the same as the target facial topology. For example, referring to Figure 5 , the target facial topology structure 502 is deformed onto the facial model 501 using a registration algorithm to obtain an adjusted facial model 503 , and the facial topology structure of the adjusted facial model 503 is the same as the target facial topology structure 502 .

[0117] In this way, the designer only needs to design how to convert the facial topology of the facial model into the required facial topology, without having to redesign the 3DMM in different scenes (such as game scenes), thereby improving the flexibility of facial model generation and reducing the workload of facial model generation. Alternatively, when the scene is fixed, the 3DMM can also be directly designed into the required facial topology, thereby saving the process of facial topology conversion and further reducing the workload of facial model generation.

[0118] In summary, the technical solution provided by the embodiment of the present application parametrically represents multiple facial images of the target object, and associates the multiple facial images of the target object through shared parameters (such as shared shape parameters and shared texture parameters) to obtain rendered images corresponding to the multiple facial images, and then iteratively optimizes the facial model of the target object based on the difference between the multiple facial images and the rendered images corresponding to the multiple facial images. This can not only solve the ambiguity problem caused by a single image in the related technology, thereby improving the accuracy of facial model generation, but also can obtain rich image features (such as multi-view image features) corresponding to the target object by collecting multiple facial images, thereby improving the fineness of facial model generation.

[0119] In addition, by converting the facial topology structure according to needs after obtaining the facial model, designers only need to design how to convert the facial topology structure of the facial model into the required facial topology structure, without having to redesign the 3DMM in different scenarios, thereby improving the flexibility of facial model generation and reducing the workload of facial model generation.

[0120] In addition, since the technical solution provided by the embodiment of the present application integrates the prior information of the facial shape basis, the facial texture basis and the facial expression basis, the stability of the facial model generation can be improved. At the same time, a facial model with a consistent facial topological structure can be obtained, thereby reducing the difficulty of converting the facial model.

[0121] In an exemplary embodiment, reference Figure 6 , which shows a schematic diagram of a method for obtaining total loss provided by an embodiment of the present application.

[0122] Obtain N facial images of the target object 601, record I 1 to I N The N facial images may be facial images of the target object 601 with different facial expressions, different facial postures or different facial illuminations. Optionally, the N facial images are processed by key point position alignment, and N is an integer greater than 1.

[0123] Perform key point detection on N facial images respectively, and obtain the key point position set corresponding to N facial images, denoted by q 1 To q N .

[0124] Randomly set the initial shared shape parameter set α and the initial shared texture parameter set p corresponding to the N facial images, and randomly set the initial sub-expression parameter set (γ N ), initial posture parameters (p N ) and initial lighting parameters (1 N ).

[0125] Based on the initial shared shape parameter set α and the initial shared texture parameter set β, as well as the initial sub-expression parameter sets (γ), initial posture parameters (p) and initial illumination parameters (1) corresponding to the N facial images, the initial facial models corresponding to the N facial images are obtained, and then the initial facial models corresponding to the N facial images are rendered using the differentiable rendering technique to obtain the initial facial rendering images corresponding to the N facial images, which are denoted as I Render,1 to I Render,N For example, based on the initial shared shape parameter set α and the initial shared texture parameter set β, and the initial sub-expression parameter set (γ N ), initial posture parameters (p N ) and initial lighting parameters (1 N ), obtain the initial face model corresponding to the Nth face image, and then obtain the initial rendered image I corresponding to the Nth face image Render,N .

[0126] Based on the total difference between the N face images and the initial face rendering images corresponding to the N face images, the total re-rendering loss L is obtained. Render For example, based on I 1 and I Render,1 The difference between 2 and I Render,2 The difference between, ..., I N and I Render,N Sum the differences between them and get L Render .

[0127] Perform key point detection on the rendered images corresponding to the N facial images respectively, and obtain the key point position sets corresponding to the rendered images corresponding to the N facial images, denoted as q Render,1 To q Render,N .

[0128] Based on the total difference between the key point position sets corresponding to the N facial images and the key point position sets corresponding to the initial facial rendering images corresponding to the N facial images, the total key point loss L is obtained. Lm For example, based on q 1 and q Render,1 The difference between 2 and q Render,2 The difference between, ..., q N and q Render,N Sum the differences between them and get L Lm .

[0129] Based on the initial shared shape parameter set α, the initial shared texture parameter set β and the initial sub-expression parameter set (γ) corresponding to the N facial images, the regularization loss L is obtained. Reg .

[0130] Rendering loss L for the total weight Render , total key point loss L Lm and the regularization loss L Reg Perform weighted summation to get the total loss.

[0131] With the goal of converging the total loss, the initial shared shape parameter set and the initial shared texture parameter set corresponding to the N facial images, as well as the initial sub-expression parameter set, initial posture parameter and initial lighting parameter corresponding to the N facial images are adjusted; in response to the total re-rendering loss stopping changing, the convergence of the total loss is completed, and the target shared shape parameter set and the target shared texture parameter set, as well as the target sub-expression parameter set, target posture parameter and target lighting parameter corresponding to the N facial images are obtained.

[0132] On the basis of the facial shape base, the facial texture base and the facial expression base, the facial images of the N facial images are obtained by combining the target shared shape parameter set and the target shared texture parameter set, as well as the target sub-expression parameter set, the target posture parameter and the target illumination parameter corresponding to the N facial images respectively.

[0133] In an exemplary embodiment, the technical solution provided in the embodiment of the present application can be used to assist art designers. Art designers only need to collect several facial images of the target object, and then based on the technical solution provided in the embodiment of the present application, they can automatically reconstruct a high-quality facial model, which reduces the workload of facial model generation compared to designing facial images from scratch. At the same time, art designers can further process the facial topology of the facial model to obtain a more refined model.

[0134] In another exemplary embodiment, the technical solution provided in the embodiment of the present application can be applied to a face pinching system. The face pinching system in the related art often requires art designers to pinch a lot of faces in advance to build a face model library, and then find the closest face model from the face model library based on the photos uploaded by the user. However, by adopting the technical solution provided in the embodiment of the present application, a high-quality face model can be directly generated based on the photos uploaded by the user. All the art designer has to do is to design how to convert the face model into a face model of the required style, which can avoid the trouble of redesigning the face model library for different scenes.

[0135] In an exemplary embodiment, in order to further compare the technical solutions provided by the embodiments of the present application with the technical solutions provided by the related art, we selected the public dataset FG2018 as the evaluation dataset (a very reliable evaluation dataset). The public dataset FG2018 provides test images, test codes, and GT (Ground Truth) data. We mainly compare the open source algorithms D3DFR (Deep 3D Face Reconstruction) and DECA (Detailed Expression Capture and Animation). Reference Figure 7, which shows a schematic diagram of the comparison results based on the public data set FG2018 provided in an embodiment of the present application. The comparison results include a test image 701, a face model 702 corresponding to GT, a face model 703 corresponding to D3DFR, a face model 704 corresponding to DECA, and a face model 705 corresponding to the present application. From the comparison results, it can be seen that the face model 705 obtained by the technical solution provided in the embodiment of the present application can better capture the geometric information of the character, and the face model 705 has a high degree of identity recognition. In addition, the entire process of the face model in the embodiment of the present application is fully automatic, which can reduce 1-2 days compared to the manual generation of the face model.

[0136] In addition, since the technical solution provided in the embodiment of the present application uses the idea of ​​iterative optimization, it can be seen from the comparison results that compared with D3DFR and DECA, the technical solution provided in the embodiment of the present application can obtain a more refined facial model, which is obviously more like the facial model corresponding to GT in terms of visual effect.

[0137] At the same time, this application also tested the average errors between the face models corresponding to D3DFR, the face models corresponding to DECA, the face models corresponding to this application, and the face models corresponding to GT by using test codes. The average error corresponding to D3DFR is 2.31, the average error corresponding to DECA is 1.89 (public SOTA (State of the Art, current technical level)), and the average error corresponding to this application is 1.56, which is much lower than D3DFR and DECA. This proves that the technical solution provided by this application (i.e., multi-view optimization) is superior to the current best open source algorithm in terms of both visual effect and reconstruction accuracy.

[0138] In summary, the technical solution provided by the embodiment of the present application parametrically represents multiple facial images of the target object, and associates the multiple facial images of the target object through shared parameters (such as shared shape parameters and shared texture parameters) to obtain rendered images corresponding to the multiple facial images, and then iteratively optimizes the facial model of the target object based on the difference between the multiple facial images and the rendered images corresponding to the multiple facial images. This can not only solve the ambiguity problem caused by a single image in the related technology, thereby improving the accuracy of facial model generation, but also can obtain rich image features (such as multi-view image features) corresponding to the target object by collecting multiple facial images, thereby improving the fineness of facial model generation.

[0139] The following are device embodiments of the present application, which can be used to execute the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0140] refer to Figure 8 , which shows a block diagram of a facial model generation device provided by an embodiment of the present application. The device has the function of implementing the above method example, and the function can be implemented by hardware, or by hardware executing corresponding software. The device can be the computer device introduced above, or it can be set in a computer device. Figure 8 As shown, the device 800 includes: a facial image acquisition module 801, an initial parameter acquisition module 802, a rendering image acquisition module 803, a matching loss acquisition module 804, a target parameter acquisition module 805 and a facial model generation module 806.

[0141] The facial image acquisition module 801 is used to acquire multiple facial images of the target object.

[0142] The initial parameter acquisition module 802 is used to obtain shared parameters corresponding to the multiple facial images and independent parameters corresponding to the multiple facial images respectively; wherein the shared parameters are used to characterize the common features between the multiple facial images, and the independent parameters are used to characterize the private features corresponding to the multiple facial images respectively.

[0143] The rendering image acquisition module 803 is used to obtain the rendering images corresponding to the multiple facial images respectively based on the shared parameters corresponding to the multiple facial images and the independent parameters corresponding to the multiple facial images respectively.

[0144] The matching loss acquisition module 804 is used to determine the matching loss based on the multiple facial images and the rendered images corresponding to the multiple facial images respectively, and the matching loss is used to characterize the difference between the multiple facial images and the rendered images corresponding to the multiple facial images respectively.

[0145] The target parameter acquisition module 805 is used to adjust the shared parameters corresponding to the multiple facial images and the independent parameters corresponding to the multiple facial images respectively with the goal of converging the matching loss, so as to obtain the target shared parameters corresponding to the multiple facial images and the target independent parameters corresponding to the multiple facial images respectively.

[0146] The face model generation module 806 is used to generate a face model of the target object based on the target shared parameters corresponding to the multiple face images and in combination with the target independent parameters respectively corresponding to the multiple face images.

[0147] In an exemplary embodiment, the shared parameters include shared shape parameters and shared texture parameters, and the independent parameters include expression parameters, posture parameters and lighting parameters; wherein the shared shape parameters are used to characterize the facial shape of the target object, the shared texture parameters are used to characterize the facial texture of the target object, the expression parameters are used to characterize the facial expression of the target object in each facial image, the posture parameters are used to characterize the facial posture of the target object in each facial image, and the lighting parameters are used to characterize the facial lighting of the target object in each facial image.

[0148] In an exemplary embodiment, the matching loss includes a total re-rendering loss and a total key point loss; the matching loss acquisition module 804 is further used to:

[0149] Obtaining re-rendering losses respectively corresponding to the plurality of facial images, wherein the re-rendering losses are used to characterize the difference in image features between the facial image and the rendered image corresponding to the facial image;

[0150] Summing the re-rendering losses corresponding to the multiple facial images respectively to obtain the total re-rendering loss;

[0151] Obtaining key point losses respectively corresponding to the plurality of facial images, wherein the key point losses are used to characterize the difference between the facial image and the rendered image corresponding to the facial image in terms of key point positions;

[0152] The key point losses corresponding to the multiple facial images are summed to obtain the total key point loss.

[0153] In an exemplary embodiment, the matching loss acquisition module 804 is further used to:

[0154] For a first facial image among the multiple facial images, extract features from the first facial image to obtain a first feature map, and extract features from a rendered image corresponding to the first facial image to obtain a second feature map;

[0155] Based on the first feature map and the second feature map, obtaining a first intermediate parameter, where the first intermediate parameter is used to characterize a difference between the first facial image and a rendered image corresponding to the first facial image;

[0156] Performing facial recognition on the first facial image to obtain a first facial region, and performing facial recognition on the rendered image corresponding to the first facial image to obtain a second facial region;

[0157] Based on the first facial region and the second facial region, obtaining a second intermediate parameter, where the second intermediate parameter is used to characterize a valid calculation region between the first facial region and the second facial region;

[0158] A re-rendering loss corresponding to the first facial image is obtained based on the first intermediate parameter and the second intermediate parameter.

[0159] In an exemplary embodiment, the matching loss acquisition module 804 is further used to:

[0160] For a first facial image among the multiple facial images, performing key point detection on the first facial image to obtain a first key point position set, and performing key point detection on a rendered image corresponding to the first facial image to obtain a second key point position set;

[0161] Based on the difference between the first key point position set and the second key point position set, a key point loss corresponding to the first facial image is obtained.

[0162] In an exemplary embodiment, Fig. 9 As shown, the device 800 also includes: a regularization loss acquisition module 807 and a total loss acquisition module 808.

[0163] The regularization loss acquisition module 807 is used to acquire the regularization loss corresponding to the multiple facial images, and the regularization loss is used to regularize the shared shape parameters, shared texture parameters and expression parameters to maintain the facial structure of the facial model.

[0164] The total loss acquisition module 808 is used to obtain the total loss according to the matching loss and the regularization loss.

[0165] The target parameter acquisition module 805 is further used to adjust the shared parameters corresponding to the multiple facial images and the independent parameters corresponding to the multiple facial images respectively with the goal of converging the total loss, so as to obtain the target shared parameters corresponding to the multiple facial images and the target independent parameters corresponding to the multiple facial images respectively.

[0166] In an exemplary embodiment, the regularization loss acquisition module 807 is further used to:

[0167] Based on the shared shape parameter set corresponding to the multiple facial images, obtaining shape regularization losses corresponding to the multiple facial images;

[0168] Based on the shared texture parameter set corresponding to the multiple facial images, obtaining texture regularization losses corresponding to the multiple facial images;

[0169] Based on the sub-expression parameter sets respectively corresponding to the multiple facial images, obtaining expression regularization losses corresponding to the multiple facial images;

[0170] A weighted sum is performed on the shape regularization loss, the texture regularization loss, and the expression regularization loss to obtain regularization losses corresponding to the multiple facial images.

[0171] In an exemplary embodiment, the target parameter acquisition module 805 is further used to:

[0172] In the process of converging the total loss, adjusting the shared shape parameter set, the shared texture parameter set, the expression parameter set, the posture parameter set and the lighting parameter set;

[0173] If it is detected that the total re-rendering loss stops changing, the convergence of the total loss is completed, and a target shared shape parameter set, a target shared texture parameter set, a target expression parameter set, a target posture parameter set, and a target lighting parameter set are obtained;

[0174] The expression parameter set includes sub-expression parameter sets corresponding to the multiple facial images respectively, the posture parameter set includes posture parameters corresponding to the multiple facial images respectively, and the illumination parameter set includes illumination parameters corresponding to the multiple facial images respectively.

[0175] In an exemplary embodiment, the target parameter acquisition module 805 is further used to:

[0176] The weight of the total re-rendering loss is set to be greater than the weight of the total key point loss and the weight of the regularization loss, and the total loss is minimized with a first iteration step size;

[0177] During the iteration process, the weight of the total key point loss, the weight of the regularization loss, and the first iteration step are gradually reduced.

[0178] In an exemplary embodiment, the rendered image acquisition module 803 is further used to:

[0179] For a first facial image among the plurality of facial images, setting a first sub-expression parameter set, a first posture parameter, and a first illumination parameter corresponding to the first facial image;

[0180] A rendered image corresponding to the first facial image is obtained by rendering according to the shared shape parameter set and the shared texture parameter set corresponding to the multiple facial images, and the first sub-expression parameter set, the first posture parameter and the first illumination parameter corresponding to the first facial image.

[0181] In an exemplary embodiment, the facial model generation module 806 is further used to generate, for a first facial image among the multiple facial images, a facial model corresponding to the first facial image based on a target shared shape parameter set and a target shared texture parameter set corresponding to the multiple facial images, in combination with a target sub-expression parameter set, a target posture parameter and a target lighting parameter corresponding to the first facial image.

[0182] In an exemplary embodiment, Fig. 9 As shown, the device 800 further includes: a target topology acquisition module 809 and a topology structure adjustment module 810 .

[0183] The target topology acquisition module 809 is used to acquire the target facial topology structure.

[0184] The topology structure adjustment module 810 is used to adjust the face model according to the target face topology structure to obtain an adjusted face model, wherein the face topology structure of the adjusted face model is the same as the target face topology structure.

[0185] In summary, the technical solution provided by the embodiment of the present application parametrically represents multiple facial images of the target object, and associates the multiple facial images of the target object through shared parameters (such as shared shape parameters and shared texture parameters) to obtain rendered images corresponding to the multiple facial images, and then iteratively optimizes the facial model of the target object based on the difference between the multiple facial images and the rendered images corresponding to the multiple facial images. This can not only solve the ambiguity problem caused by a single image in the related technology, thereby improving the accuracy of facial model generation, but also can obtain rich image features (such as multi-view image features) corresponding to the target object by collecting multiple facial images, thereby improving the fineness of facial model generation.

[0186] It should be noted that the device provided in the above embodiment, when implementing its functions, is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0187] Please refer to Fig.10 , which shows a structural block diagram of a computer device provided in one embodiment of the present application. The computer device can be used to implement the face model generation method provided in the above embodiment. Specifically, it can include the following contents.

[0188] The computer device 1000 includes a central processing unit (such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit) and an FPGA (Field Programmable Gate Array)) 1001, a system memory 1004 including a RAM (Random-Access Memory) 1002 and a ROM (Read-Only Memory) 1003, and a system bus 1005 connecting the system memory 1004 and the central processing unit 1001. The computer device 1000 also includes a basic input / output system (Input Output System, I / O system) 1006 for helping various devices in the server to transmit information, and a large-capacity storage device 1007 for storing an operating system 1013, an application program 1014 and other program modules 1015.

[0189] The basic input / output system 1006 includes a display 1008 for displaying information and an input device 1009 such as a mouse and a keyboard for user inputting information. The display 1008 and the input device 1009 are connected to the central processing unit 1001 through an input / output controller 1010 connected to the system bus 1005. The basic input / output system 1006 may also include an input / output controller 1010 for receiving and processing inputs from a plurality of other devices such as a keyboard, a mouse, or an electronic stylus. Similarly, the input / output controller 1010 also provides output to a display screen, a printer, or other types of output devices.

[0190] The mass storage device 1007 is connected to the central processing unit 1001 via a mass storage controller (not shown) connected to the system bus 1005. The mass storage device 1007 and its associated computer readable medium provide non-volatile storage for the computer device 1000. That is, the mass storage device 1007 may include a computer readable medium (not shown) such as a hard disk or a CD-ROM (Compact Disc Read-Only Memory) drive.

[0191] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory or other solid-state storage technologies, CD-ROM, DVD (Digital Video Disc) or other optical storage, cassettes, tapes, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that the computer storage medium is not limited to the above. The above-mentioned system memory 1004 and mass storage device 1007 can be collectively referred to as memory.

[0192] According to the embodiment of the present application, the computer device 1000 can also be connected to a remote computer on the network through a network such as the Internet. That is, the computer device 1000 can be connected to the network 1012 through the network interface unit 1011 connected to the system bus 1005, or the network interface unit 1011 can be used to connect to other types of networks or remote computer systems (not shown).

[0193] The memory also includes at least one instruction, at least one program, code set or instruction set, which is stored in the memory and configured to be executed by one or more processors to implement the above-mentioned face model generation method.

[0194] In an exemplary embodiment, a computer-readable storage medium is also provided, wherein the storage medium stores at least one instruction, at least one program, a code set or an instruction set, and when the at least one instruction, the at least one program, the code set or the instruction set is executed by a processor, the above-mentioned face model generation method is implemented.

[0195] Optionally, the computer readable storage medium may include: ROM (Read-Only Memory), RAM (Random-Access Memory), SSD (Solid State Drives) or optical disks, etc. Among them, the random access memory may include ReRAM (Resistance Random Access Memory) and DRAM (Dynamic Random Access Memory).

[0196] In an exemplary embodiment, a computer program product or a computer program is also provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above-mentioned face model generation method.

[0197] It should be understood that the "multiple" mentioned in this article refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. In addition, the step numbers described in this article only illustrate a possible execution sequence between the steps. In some other embodiments, the above steps may not be executed in the order of the numbers, such as two steps with different numbers are executed at the same time, or two steps with different numbers are executed in the opposite order to the diagram. The embodiments of the present application are not limited to this.

[0198] The above description is only an exemplary embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A facial model generation method, characterized in that: The method comprises: Acquire multiple facial images of the target object; Performing parameterized representation on the initial facial model obtained by scanning the facial images to obtain a facial shape basis, a facial texture basis, and a facial expression basis, wherein the facial shape basis, the facial texture basis, and the facial expression basis are respectively shape mean representations, texture mean representations, and expression mean representations of the initial facial model corresponding to the multiple facial images; Acquire shared parameters corresponding to the multiple facial images, and independent parameters corresponding to the multiple facial images respectively; wherein the shared parameters are used to characterize the common features between the multiple facial images, the shared parameters include a shared shape parameter set and a shared texture parameter set corresponding to the target object, the shared shape parameters in the shared shape parameter set are obtained by random setting, the shared texture parameters in the shared texture parameter set are obtained by random setting, the shared shape parameters are used to characterize the facial shape of the target object, the shared texture parameters are used to characterize the facial texture of the target object, and the independent parameters are used to characterize the private features corresponding to the multiple facial images respectively; For a first facial image among the plurality of facial images, obtaining a rendered image corresponding to the first facial image based on the facial shape basis, the facial texture basis, the facial expression basis, the shared shape parameter set, the shared texture parameter set, and independent parameters corresponding to the first facial image; Performing facial recognition on the first facial image to obtain a first facial region, and performing facial recognition on the rendered image corresponding to the first facial image to obtain a second facial region; determining a product of the first facial region and the second facial region as a second intermediate parameter, wherein the second intermediate parameter is used to characterize a valid calculation region between the first facial region and the second facial region; obtaining a re-rendering loss corresponding to the first facial image based on the second intermediate parameter, the first facial image and the rendered image corresponding to the first facial image, wherein the re-rendering loss is used to characterize a difference between the first facial image and the rendered image corresponding to the first facial image in terms of image features; Sum the re-rendering losses corresponding to the multiple facial images respectively to obtain a total re-rendering loss; Summing the key point losses corresponding to the multiple facial images respectively to obtain a total key point loss, wherein the key point loss is used to characterize the difference between the facial image and the rendered image corresponding to the facial image in terms of key point positions; Determine a matching loss based on the total re-rendering loss and the total key point loss, where the matching loss is used to characterize the difference between the multiple facial images and the rendered images respectively corresponding to the multiple facial images; With the goal of converging the matching loss, adjusting the shared parameters corresponding to the multiple facial images and the independent parameters corresponding to the multiple facial images respectively, to obtain target shared parameters corresponding to the multiple facial images and target independent parameters corresponding to the multiple facial images respectively, wherein the target shared parameters are obtained by iteratively optimizing the shared parameters, the target shared parameters include a target shared shape parameter set and a target shared texture parameter set, and the target independent parameters are obtained by iteratively optimizing the independent parameters; On the basis of the target shared shape parameter set and the target shared texture parameter set, the target independent parameters corresponding to the multiple facial images are combined to generate a facial model of the target object.

2. The method according to claim 1, characterized in that: The obtaining, based on the second intermediate parameter, the first facial image and the rendered image corresponding to the first facial image, a re-rendering loss corresponding to the first facial image includes: Performing feature extraction on the first facial image to obtain a first feature map, and performing feature extraction on a rendered image corresponding to the first facial image to obtain a second feature map; Based on the first feature map and the second feature map, obtaining a first intermediate parameter, where the first intermediate parameter is used to characterize a difference between the first facial image and a rendered image corresponding to the first facial image; A re-rendering loss corresponding to the first facial image is obtained based on the first intermediate parameter and the second intermediate parameter.

3. The method according to claim 1, characterized in that Before summing the key point losses respectively corresponding to the plurality of facial images to obtain the total key point loss, the method further includes: For a first facial image among the multiple facial images, performing key point detection on the first facial image to obtain a first key point position set, and performing key point detection on a rendered image corresponding to the first facial image to obtain a second key point position set; Based on the difference between the first key point position set and the second key point position set, a key point loss corresponding to the first facial image is obtained.

4. The method according to claim 1, characterized in that: The method further comprises: Obtaining regularization losses corresponding to the multiple facial images, where the regularization losses are used to regularize the shared shape parameters, the shared texture parameters, and the expression parameters to maintain the facial structure of the facial model; According to the matching loss and the regularization loss, a total loss is obtained; The step of adjusting the shared parameters corresponding to the multiple facial images and the independent parameters corresponding to the multiple facial images respectively with the goal of converging the matching loss to obtain target shared parameters corresponding to the multiple facial images and target independent parameters corresponding to the multiple facial images respectively includes: With the goal of converging the total loss, the shared parameters corresponding to the multiple facial images and the independent parameters respectively corresponding to the multiple facial images are adjusted to obtain target shared parameters corresponding to the multiple facial images and target independent parameters respectively corresponding to the multiple facial images.

5. The method according to claim 4, characterized in that The obtaining of regularization losses corresponding to the plurality of facial images includes: Based on the shared shape parameter set corresponding to the multiple facial images, obtaining shape regularization losses corresponding to the multiple facial images; Based on the shared texture parameter set corresponding to the multiple facial images, obtaining texture regularization losses corresponding to the multiple facial images; Based on the sub-expression parameter sets respectively corresponding to the multiple facial images, obtaining expression regularization losses corresponding to the multiple facial images; A weighted sum is performed on the shape regularization loss, the texture regularization loss, and the expression regularization loss to obtain regularization losses corresponding to the multiple facial images.

6. The method according to claim 4, characterized in that The step of adjusting the shared parameters corresponding to the plurality of facial images and the independent parameters respectively corresponding to the plurality of facial images with the goal of converging the total loss to obtain target shared parameters corresponding to the plurality of facial images and target independent parameters respectively corresponding to the plurality of facial images includes: In the process of converging the total loss, adjusting the shared shape parameter set, shared texture parameter set, expression parameter set, posture parameter set and illumination parameter set corresponding to the multiple facial images; If it is detected that the total re-rendering loss in the matching loss stops changing, the convergence of the total loss is completed, and the target shared shape parameter set, the target shared texture parameter set, the target expression parameter set, the target posture parameter set and the target lighting parameter set are obtained; The expression parameter set includes sub-expression parameter sets corresponding to the multiple facial images respectively, the posture parameter set includes posture parameters corresponding to the multiple facial images respectively, and the illumination parameter set includes illumination parameters corresponding to the multiple facial images respectively.

7. The method according to claim 6, characterized in that The method further comprises: The weight of the total re-rendering loss is set to be greater than the weight of the total key point loss and the weight of the regularization loss, and the total loss is minimized with a first iteration step size; During the iteration process, the weight of the total key point loss, the weight of the regularization loss, and the first iteration step are gradually reduced.

8. The method according to claim 1, characterized in that The obtaining, based on the facial shape basis, the facial texture basis, the facial expression basis, the shared shape parameter set, the shared texture parameter set, and the independent parameters corresponding to the first facial image, a rendered image corresponding to the first facial image comprises: Setting a first sub-expression parameter set, a first posture parameter, and a first illumination parameter corresponding to the first facial image, wherein the independent parameters corresponding to the first facial image include the first sub-expression parameter set, the first posture parameter, and the first illumination parameter; A rendered image corresponding to the first facial image is rendered based on the facial shape base, the facial texture base, the facial expression base, the shared shape parameter set, the shared texture parameter set, the first sub-expression parameter set, the first posture parameter, and the first lighting parameter.

9. The method according to claim 1, characterized in that: The method of generating a face model of the target object based on the target shared shape parameter set and the target shared texture parameter set and combining target independent parameters corresponding to the plurality of face images respectively comprises: For a first facial image among the multiple facial images, based on the target shared shape parameter set and the target shared texture parameter set, a facial model corresponding to the first facial image is generated in combination with a target sub-expression parameter set, a target posture parameter and a target lighting parameter corresponding to the first facial image.

10. The method according to claim 1, characterized in that After generating the face model of the target object based on the target shared shape parameter set and the target shared texture parameter set and combining the target independent parameters corresponding to the plurality of facial images, the method further includes: Obtain the target face topology; The facial model is adjusted according to the target facial topology to obtain an adjusted facial model, wherein the facial topology of the adjusted facial model is the same as the target facial topology.

11. A facial model generation device, characterized in that: The device comprises: A facial image acquisition module, used to acquire multiple facial images of a target object; An initial parameter acquisition module is used to parameterize the initial face model obtained by scanning the face image, and obtain a face shape basis, a face texture basis and a face expression basis, wherein the face shape basis, the face texture basis and the face expression basis are respectively the shape mean representation, the texture mean representation and the expression mean representation of the initial face model corresponding to the multiple face images; and obtain shared parameters corresponding to the multiple face images, and independent parameters corresponding to the multiple face images respectively; wherein the shared parameters are used to represent the common features between the multiple face images, and the shared parameters include a shared shape parameter set and a shared texture parameter set corresponding to the target object, the shared shape parameters in the shared shape parameter set are obtained by random setting, the shared texture parameters in the shared texture parameter set are obtained by random setting, the shared shape parameters are used to represent the face shape of the target object, the shared texture parameters are used to represent the face texture of the target object, and the independent parameters are used to represent the private features corresponding to the multiple face images respectively; a rendering image acquisition module, configured to obtain, for a first facial image among the plurality of facial images, a rendering image corresponding to the first facial image based on the facial shape basis, the facial texture basis, the facial expression basis, the shared shape parameter set, the shared texture parameter set, and the independent parameters corresponding to the first facial image; a matching loss acquisition module, configured to perform facial recognition on the first facial image to obtain a first facial region, and perform facial recognition on the rendered image corresponding to the first facial image to obtain a second facial region; determine the product of the first facial region and the second facial region as a second intermediate parameter, the second intermediate parameter being used to characterize the effective calculation region between the first facial region and the second facial region; obtain a re-rendering loss corresponding to the first facial image based on the second intermediate parameter, the first facial image and the rendered image corresponding to the first facial image, the re-rendering loss being used to characterize the difference in image features between the first facial image and the rendered image corresponding to the first facial image; sum the re-rendering losses corresponding to the plurality of facial images to obtain a total re-rendering loss; sum the key point losses corresponding to the plurality of facial images to obtain a total key point loss, the key point loss being used to characterize the difference in key point positions between the facial image and the rendered image corresponding to the facial image; determine a matching loss based on the total re-rendering loss and the total key point loss, the matching loss being used to characterize the difference between the plurality of facial images and the rendered images corresponding to the plurality of facial images; a target parameter acquisition module, configured to adjust the shared parameters corresponding to the multiple facial images and the independent parameters corresponding to the multiple facial images respectively with the goal of converging the matching loss, to obtain target shared parameters corresponding to the multiple facial images and target independent parameters corresponding to the multiple facial images respectively, wherein the target shared parameters are obtained by iteratively optimizing the shared parameters, the target shared parameters include a target shared shape parameter set and a target shared texture parameter set, and the target independent parameters are obtained by iteratively optimizing the independent parameters; The face model generation module is used to generate a face model of the target object based on the target shared shape parameter set and the target shared texture parameter set and in combination with the target independent parameters respectively corresponding to the multiple face images.

12. A computer device, characterized in that: The computer device includes a processor and a memory, wherein at least one program is stored in the memory, and the at least one program is loaded and executed by the processor to implement the facial model generation method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one program, and the at least one program is loaded and executed by the processor to implement the face model generation method according to any one of claims 1 to 10.

14. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium. A processor reads and executes the computer instructions from the computer-readable storage medium to implement the face model generation method according to any one of claims 1 to 10.