Method, device, electronic device, and storage medium for generating three-dimensional results of human face

Through multiple iterations, the modeling coefficient group and rendering coefficient group are adjusted, combined with the comparison results and reference evaluation results, the problem of poor face three-dimensional reconstruction effect in the existing technology is solved, and more accurate face three-dimensional result generation is achieved.

CN116563451BActive Publication Date: 2024-11-01TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210112620.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-01-29
Publication Date
2024-11-01
Estimated Expiration
2042-01-29

AI Technical Summary

Technical Problem

When the prior art generates a three-dimensional face based on two-dimensional face image reconstruction, due to the differences in shooting conditions, environment and angles of different face images, the target face reconstruction effect is poor, and the three-dimensional face results cannot be effectively generated.

Method used

A three-dimensional result generation method for faces is proposed. By obtaining multiple face image groups of target objects, including the main face image and the auxiliary face image, iterates through multiple iterations to adjust the modeling coefficient group and rendering coefficient group, and uses the comparison results and Refer to the evaluation results to adjust the modeling coefficient group, reduce the impact of conflicts, and generate effective three-dimensional face results.

Benefits of technology

By differentiating the learning of different face image features, the impact of conflict is reduced, and the accuracy and effectiveness of face three-dimensional reconstruction are improved, so as to ensure the accuracy of the generated three-dimensional results for the target face.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116563451B_ABST
    Figure CN116563451B_ABST
Patent Text Reader

Abstract

The present application relates to the field of artificial intelligence technology, and in particular to a method, device, electronic device, and storage medium for generating a three-dimensional result of a human face. The method comprises: obtaining a group of human face images collected for a target object, wherein the group of human face images includes a main human face image and at least one auxiliary human face image, iteratively adjusting a modeling coefficient group and a rendering coefficient, and in the process of iterative adjustment, configuring a reference evaluation result of the auxiliary human face image according to a first comparison result between corresponding main human face images and an offset between a second comparison result between the auxiliary human face image and the auxiliary rendering image, and then calculating a loss value based on the first comparison result, the second comparison result, and the reference evaluation result, and adjusting the modeling coefficient group and the rendering coefficient group based on the loss value. In this way, by configuring the reference evaluation result, the influence of the auxiliary human face image that has a conflicting influence with the main human face image can be reduced, providing a guarantee for finally generating an effective three-dimensional result of the human face.
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0002] In the related art, when generating a three-dimensional (3D) face reconstruction result based on two-dimensional (2D) face image reconstruction, a face modeling result is generated with the help of a basic face set obtained and a modeling coefficient group configured for the basic face set. For example, a three-dimensional transformation model (3D Morphable Models, 3DMM) face set and optimized 3DMM coefficients are used to generate a three-dimensional face result constructed by a face image group corresponding to the target object.

[0003] However, in the process of iterative optimization, since different face images in the face image group usually correspond to different shooting conditions, shooting environments, shooting times, and shooting angles, the organ shapes of the target objects are different in different face images. As a result, when optimizing according to the feature differences between each face image and the mapping image, a conflict relationship is introduced due to the same organ corresponding to different organ shapes.

[0004] In this way, when face reconstruction is performed based on the optimized modeling coefficient group and the basic face set, effective face reconstruction cannot be performed on the target face. Summary of the invention

[0005] The embodiments of the present application provide a method, device, electronic device, and storage medium for generating a three-dimensional result of a face, so as to solve the problem that the existing technology cannot effectively reconstruct the target face.

[0006] In the first aspect, a method for generating a three-dimensional result of a face is proposed, comprising:

[0007] Acquire a face image group associated with the target object, wherein the face image group includes a main face image and at least one auxiliary face image;

[0008] Based on the face image group and the basic face set, the initial modeling coefficient group is adjusted for multiple rounds of iterations, and based on the adjusted target modeling coefficient group and the basic face set, the target face three-dimensional result is determined; wherein the following operations are performed in each round of iterative adjustment process:

[0009] Based on the three-dimensional face result determined according to the current modeling coefficient group and the basic face set, obtaining a first comparison result between the main face image and the associated main rendered image, and respectively obtaining a second comparison result between the at least one auxiliary face image and the associated auxiliary rendered image;

[0010] Based on the obtained at least one second comparison result and the degree of deviation between each of the second comparison results and the first comparison result, configuring a reference evaluation result of a corresponding auxiliary facial image;

[0011] The modeling coefficient group is adjusted based on a first loss value determined according to the first comparison result and a second loss value determined according to the at least one second comparison result and a corresponding reference evaluation result, respectively.

[0012] In a second aspect, a device for generating a three-dimensional result of a face is proposed, comprising:

[0013] An acquisition unit, configured to acquire a face image group associated with a target object, wherein the face image group includes a main face image and at least one auxiliary face image;

[0014] The adjustment unit is used to perform multiple rounds of iterative adjustments on the initial modeling coefficient group based on the face image group and the basic face set, and determine the target face three-dimensional result based on the adjusted target modeling coefficient group and the basic face set; wherein the following operations are performed in each round of iterative adjustment process:

[0015] Based on the three-dimensional face result determined according to the current modeling coefficient group and the basic face set, obtaining a first comparison result between the main face image and the associated main rendered image, and respectively obtaining a second comparison result between the at least one auxiliary face image and the associated auxiliary rendered image;

[0016] Based on the obtained at least one second comparison result and the degree of deviation between each of the second comparison results and the first comparison result, configuring a reference evaluation result of a corresponding auxiliary facial image;

[0017] The modeling coefficient group is adjusted based on a first loss value determined according to the first comparison result and a second loss value determined according to the at least one second comparison result and a corresponding reference evaluation result. Optionally, when the basic face set is a 3DMM face set, the modeling coefficient group is a 3DMM coefficient.

[0018] Optionally, after acquiring the face image group collected for the target object and before performing multiple rounds of iterative adjustments on the initial modeling coefficient group based on the face image group and the basic face set, the acquisition unit is further used to:

[0019] Using facial key point detection technology and facial segmentation technology, respectively for each facial image included in the facial image group, determine a facial key point set and a facial segmentation result;

[0020] For a main face image and at least one corresponding auxiliary face image determined in each of the face images, a key point importance weight set corresponding to the face key point set and a corresponding pixel point importance weight map are configured according to the corresponding face segmentation results.

[0021] Optionally, when configuring a key point importance weight set corresponding to a face key point set and configuring a corresponding pixel point importance weight map according to corresponding face segmentation results, the acquisition unit is used to:

[0022] According to the corresponding face segmentation results, the following operations are performed respectively:

[0023] Determine the facial key points in each corresponding segmented area respectively, and generate a key point importance weight set based on the key point importance weights respectively configured for the facial key points in each segmented area;

[0024] The importance weights of the pixels in each segmented area are configured, and a pixel importance weight map is generated based on the importance weights of the pixels in each segmented area.

[0025] Optionally, when obtaining a first comparison result between the main face image and the associated main rendered image based on the three-dimensional face result determined according to the current modeling coefficient group and the basic face set, and respectively obtaining a second comparison result between the at least one auxiliary face image and the associated auxiliary rendered image, the adjustment unit is used to:

[0026] Determine a three-dimensional face result according to the current modeling coefficient group and the basic face set, and generate a main rendered image associated with the main face image and a secondary rendered image associated with the at least one secondary face image based on the three-dimensional face result and the currently acquired rendering coefficient group, wherein the rendering coefficient group includes each group of illumination coefficients and each group of posture coefficients;

[0027] Determine respectively the main face image, the main rendered image, the at least one auxiliary face image, and the at least one auxiliary rendered image, and the corresponding facial key point sets and face segmentation results;

[0028] A corresponding first comparison result is determined based on the pixel value differences and key point position differences between the main rendered image and the main facial image, and a corresponding second comparison result is determined based on the pixel value differences and key point position differences between the at least one auxiliary rendered image and the corresponding auxiliary facial image.

[0029] Optionally, when determining the corresponding first comparison result according to the pixel value difference and key point position difference between the main rendered image and the main face image, the adjustment unit is used to:

[0030] Determine the pixel value difference between each pixel point with the same relative position in the main face image and the main rendering image, and determine the key point position difference between each face key point with the same label;

[0031] According to the differences in each pixel value and each key point position, a corresponding first comparison result is determined.

[0032] Optionally, when configuring the reference evaluation result of the corresponding auxiliary facial image based on the degree of deviation between the at least one second comparison result obtained and the first comparison result, the adjustment unit is used to:

[0033] For at least one second comparison result, perform the following operations respectively:

[0034] Determine, according to a first face segmentation result corresponding to the first comparison result and a second face segmentation result corresponding to a second comparison result, groups of segmented regions that match each other, and configure, for each group of segmented regions, reference evaluation results of pixel values ​​of each pixel point in the corresponding segmented regions according to first offset degrees between corresponding pixel value difference means;

[0035] According to the facial key point sets corresponding to the second comparison result and the first comparison result, each group of facial key point subsets that matches within the each group of segmentation areas is determined respectively; and for each group of facial key point subsets, according to the second offset degree between the corresponding key point position difference means, a key point reference evaluation result of the facial key points within the corresponding segmentation area is configured.

[0036] Optionally, when determining the first loss value according to the first comparison result, the adjustment unit is used to:

[0037] Determine pixel importance weights respectively configured for pixels in each segmented area in the master face image, wherein the segmented area is determined by a corresponding face segmentation result;

[0038] Calculating a first pixel value loss based on each pixel value difference included in the first comparison result and a pixel point importance weight configured corresponding to each pixel point;

[0039] Calculating a first key point loss based on the position differences of each key point included in the first comparison result and the key point importance weights corresponding to each key point configuration;

[0040] A corresponding first loss value is determined according to the first pixel value loss and the first key point loss.

[0041] Optionally, when determining the second loss value according to the at least one second comparison result and the corresponding reference evaluation result respectively, the adjustment unit is used to:

[0042] Obtaining pixel importance weights respectively configured for pixels in each segmented area in the at least one auxiliary face image, wherein the segmented area is determined by a corresponding face segmentation result;

[0043] Calculating a second pixel value loss based on each pixel value difference included in the at least one second comparison result, and a pixel point importance weight configured corresponding to each pixel point and a pixel value reference evaluation result;

[0044] Calculating a second key point loss based on the position differences of each key point included in the at least one second comparison result, and the key point importance weights configured corresponding to each face key point and the key point reference evaluation results;

[0045] A corresponding second loss value is determined based on each second pixel value loss and each second key point loss.

[0046] In a third aspect, an electronic device is proposed, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for generating a three-dimensional face result described in any one of the first aspects is implemented.

[0047] In a fourth aspect, a computer-readable storage medium is proposed, on which a computer program is stored. When the computer program is executed by a processor, the method for generating a three-dimensional face result described in any one of the first aspects is implemented.

[0048] In a fifth aspect, a computer program product is proposed, comprising a computer program, which, when executed by a processor, implements the method for generating a three-dimensional face result as described in any one of the first aspects above.

[0049] The beneficial effects of this application are as follows:

[0050] In the embodiments of the present application, a method, device, electronic device and storage medium for generating a three-dimensional face result are proposed. Since the constructed face image group includes a main face image and at least one auxiliary face image, it is equivalent to determining the face images with the main influence and the auxiliary influence in each face image, respectively. This provides a processing basis for differentially learning the features in different face images in the process of iteratively adjusting the modeling coefficient group and the rendering coefficient group. In addition, when calculating the loss value, the main face image can be used as a reference basis, and a reference evaluation of the auxiliary face image can be achieved according to the offset between the corresponding comparison results. Moreover, by configuring the reference evaluation result, the influence of the auxiliary face image that has a conflicting influence with the main face image can be reduced, thereby providing a guarantee for the final generation of a valid three-dimensional face result. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A schematic diagram of a possible application scenario in an embodiment of the present application;

[0052] Figure 2 This is a schematic diagram of an interface for a relevant object to select a face image on a terminal device in a possible application scenario in an embodiment of the present application;

[0053] Figure 3 A schematic diagram of a process for performing face reconstruction in an embodiment of the present application;

[0054] Figure 4 A schematic diagram of the process of configuring a pixel importance weight map and a key point importance weight set in an embodiment of the present application;

[0055] Figure 5 This is a schematic diagram of the result of face segmentation in an embodiment of the present application;

[0056] Figure 6 This is a schematic diagram of the results of face key point detection in an embodiment of the present application;

[0057] Figure 7 A schematic diagram of a process for performing a round of iterative adjustment in an embodiment of the present application;

[0058] Figure 8 A schematic diagram of a process for determining a comparison result in an embodiment of the present application;

[0059] Fig. 9 Schematic diagram of two facial key points corresponding to the same number in the embodiment of the present application;

[0060] Fig.10 A schematic diagram of a process for determining a reference evaluation result of an auxiliary face image in an embodiment of the present application;

[0061] Fig.11 Schematic diagram of each group of segmented areas in the embodiment of the present application;

[0062] Fig.12 This is a schematic diagram of an application of face reconstruction based on 3DMM in an embodiment of the present application;

[0063] Fig.13 This is a schematic diagram of the logical structure of the face reconstruction device in the embodiment of the present application;

[0064] Fig.14 A schematic diagram of a hardware structure of an electronic device according to an embodiment of the present application;

[0065] Fig.15 Schematic diagram of the structure of a computing device in an embodiment of the present application. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.

[0067] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the invention described herein can be implemented in sequences other than those illustrated or described herein.

[0068] Some of the terms used in the embodiments of the present application are explained below to facilitate understanding by those skilled in the art.

[0069] 3D Morphable Models (3DMM): also known as a 3D deformable face model, is a general 3D face parametric model that uses a fixed number of points to represent a face. Its core idea is that faces can be matched one-to-one in three-dimensional space, and can be obtained by weighted linear addition of many other faces by orthogonal basis. In the embodiment of the present application, the 3D face result corresponding to the target object can be obtained based on the 3DMM face set and the 3DMM coefficients determined by the final adjustment, wherein the obtained 3D face result is specifically a modeled 3D face, and the 3DMM coefficients specifically include face shape coefficients and face texture coefficients.

[0070] Optimization algorithm: refers to the processing algorithm used in the process of iteratively adjusting the modeling coefficient group and the rendering coefficient group, such as the one used in the gradient descent process. The optimization algorithm can be understood as, given an objective function f, the range of the independent variable is A, and the range of the function value is a real number R. The optimization algorithm aims to determine a target independent variable in the range of the independent variable A, so that the function value under the target independent variable is greater than the function value of each other independent variable in the range of the independent variable A, or the function value under the target independent variable is less than the function value of each other independent variable in the range of the independent variable A. At this time, the target independent variable can be called the optimal solution.

[0071] Face key point detection, also known as two-dimensional face key point detection algorithm, is used to locate a set of predefined face reference points in a two-dimensional image. Specifically, a set of face reference points can be defined in the face image in advance according to actual processing needs, and then when performing face key point detection on the image, each face reference point included in the image can be located, and then each face key point included in the image can be determined. Among them, the number of reference points included in the predefined set of face reference points is set according to actual processing needs, and this application does not make specific restrictions.

[0072] Basic face set: refers to a set composed of various basic faces. Basic faces are the basis for constructing a three-dimensional face result. In the embodiment of the present application, the basic face set can specifically be a 3DMM face set. In the present application, the open source 3DMM face set can be directly obtained, wherein the 3DMM face set includes various pre-scanned three-dimensional face models, and a three-dimensional face model corresponds to a face orthogonal basis.

[0073] Modeling coefficient group: refers to the coefficients closely related to the generation of the face reconstruction model, which are used to weight each basic face in the basic face set so that the three-dimensional face result generated by the face image of the corresponding target object can be obtained. In the embodiment of the present application, the modeling coefficient group includes face shape coefficients and face texture coefficients. Specifically, based on the processing concept of 3DMM, it can be known that the three-dimensional face model can be divided into two aspects: shape and texture. Assuming that the three-dimensional face model is composed of n vertices, the shape vector (Shape Vector) can be represented as: S = (X1, Y1, Z1, X2, Y2, Z2, ..., Yn, Zn); the texture vector (Texture Vector) can be represented as: T = (R1, G1, B1, R2, G2, B2, ..., Rn, Bn). Then, the generated three-dimensional face result can be specifically the linear superposition of the shape vector and the texture vector. The specific formula is: in, is the shape mean, is the texture mean, Si represents the shape vector of the i-th face 3D model in the 3DMM face set, Ti represents the texture vector of the i-th face 3D model in the 3DMM face set, ai represents the face shape coefficient corresponding to the i-th face 3D model, and bi represents the face texture coefficient corresponding to the i-th face 3D model.

[0074] The rendering coefficient group refers to the coefficients used when reconstructing a two-dimensional image based on the three-dimensional face result after obtaining the three-dimensional face result based on the face image. In the embodiment of the present application, when the basic face set is a 3DMM face set, the rendering coefficient group can specifically be each group of illumination coefficients and each group of posture coefficients, wherein the number of groups of illumination coefficients and posture coefficients is the same, and the value of the number of groups is consistent with the total number of face images included in the face image group.

[0075] The following is a brief introduction to the design concept of the embodiment of the present application:

[0076] Under the related technology, when performing three-dimensional reconstruction based on the two-dimensional image of the target object to generate a face reconstruction result, the 3DMM face set and the face image set of the target object can be used to obtain the three-dimensional face result of the target object based on the face image set and the adjusted 3DMM coefficients.

[0077] In a round of coefficient adjustment of the relevant 3DMM model, a set of 3DMM parameters, multiple sets of pose coefficients and illumination coefficients, and a set of 3DMM faces are combined to determine the corresponding three-dimensional face result, wherein one input image corresponds to one set of pose coefficients and one set of illumination coefficients. Then, based on the obtained three-dimensional face result, and the determined pose coefficients and illumination coefficients, the rendered images corresponding to the respective face images of the target object are rendered, and then, according to the difference between each rendered image and the corresponding face image, the loss value is independently calculated, and the 3DMM coefficients, the pose coefficients of each set, and the pose coefficients of each set are adjusted, which is equivalent to utilizing the information difference between multiple face images and the corresponding rendered images, and constraining the 3DMM coefficients, the pose coefficients of each set, and the illumination coefficients of each set at the same time.

[0078] However, in the related art, when adjusting the 3DMM coefficients, each group of posture coefficients, and each group of illumination coefficients, for the multiple face images of the target object used in the adjustment and optimization process, different face images usually have different shooting conditions, where the different shooting conditions are specifically reflected in the shooting time, shooting environment, and shooting angle. This makes the facial features information provided by multiple face images similar but not the same. In other words, there are differences in the organ shapes of the target object shown in different face images, and there are conflicting influences between the organ shape information provided by each face image.

[0079] In this way, when the correlation coefficient is adjusted based on multiple face images with conflicting effects, contradictory constraints are introduced, which will greatly affect the modeling effect and make it impossible to effectively reconstruct the target face.

[0080] In view of this, the present application proposes a face reconstruction method, device, electronic device and storage medium, firstly obtaining a face image group collected for a target object, wherein the face image group includes a main face image and at least one auxiliary face image, so that each face image of the target object is divided differently, providing a processing basis for considering the influence of each face image when adjusting the coefficient by difference value. Then, based on the face image group and the basic face set, the initial modeling coefficient group and rendering coefficient group are adjusted for multiple rounds of iterations, and the three-dimensional result of the target face is determined based on the adjusted target modeling coefficient group and the basic face set, so that the face reconstruction of the target object can be completed based on the modeling coefficient group obtained after the final adjustment from the perspective of adjusting and optimizing the modeling coefficient group and the rendering coefficient group. In addition, during each round of iterative adjustment, based on the three-dimensional face result determined according to the current modeling coefficient group and the basic face set, a first comparison result between the main face image and the associated main rendering image is obtained, and a second comparison result between at least one auxiliary face image and the associated auxiliary rendering image is obtained respectively. Based on the degree of deviation between the at least one second comparison result obtained and the first comparison result, a reference evaluation result of the corresponding auxiliary face image is configured. Then, based on the first loss value determined according to the first comparison result and the second loss value determined according to the at least one second comparison result and the corresponding reference evaluation result, the modeling coefficient group is adjusted.

[0081] In this way, since the constructed face image group includes a main face image and at least one auxiliary face image, it is equivalent to determining the face images with the main influence and the auxiliary influence in each face image, respectively. This provides a processing basis for differentially learning the features in different face images in the process of iteratively adjusting the modeling coefficient group and the rendering coefficient group. In addition, when calculating the loss value, the main face image can be used as a reference, and a reference evaluation of the auxiliary face image can be achieved according to the offset between the corresponding comparison results. Moreover, by configuring the reference evaluation results, the influence of the auxiliary face image that has a conflicting influence with the main face image can be reduced, thereby providing a guarantee for the final generation of an effective three-dimensional face result.

[0082] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. In addition, the embodiments of the present application and the features in the embodiments may be combined with each other if there is no conflict.

[0083] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0084] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0085] In an embodiment of the present application, after completing the iterative adjustment of the modeling coefficient group with the help of artificial intelligence technology, the three-dimensional result of the target face reconstructed and generated according to the target object is determined based on the final modeling coefficient group and the basic face set.

[0086] See also Figure 1 As shown, it is a schematic diagram of a possible application scenario in the implementation of this application. Figure 1 The application scenario shown includes a terminal device 110 (including a terminal device 1101 of object 1, a terminal device 1102 of object 2 . . . a terminal device 110n of object n) and a processing device 120 .

[0087] In the embodiment of the present application, the terminal device 110 includes but is not limited to electronic devices such as desktop computers, mobile phones, mobile computers, tablet computers, media players, smart wearable devices, smart televisions, vehicle-mounted devices, personal digital assistants (PDAs), etc., or may be various image acquisition devices, such as cameras.

[0088] The processing device 120 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, as well as big data and artificial intelligence platforms.

[0089] In the embodiment of the present application, a communication connection is established between the terminal device 110 and the processing device 120 through a communication network using a wired connection or a wireless connection.

[0090] In a possible application scenario of the present application, when the processing device 120 is used as a device for performing face reconstruction, after the object 1 selects or uploads the main face image and at least one auxiliary face image for generating a three-dimensional face result on the terminal device 110, the main face image and the corresponding at least one auxiliary face image are triggered to be uploaded to the processing device 120. The processing device 120 generates a three-dimensional face result based on the obtained main face image and the corresponding at least one auxiliary face image, the basic face set, and the initially configured modeling coefficient group and rendering coefficient group, and then based on the difference between the rendered image reconstructed from the three-dimensional face result and the corresponding face image, the modeling coefficient group and the rendering coefficient group are iteratively adjusted for multiple rounds until it is determined that the preset convergence condition is met, and the corresponding target face three-dimensional result is generated based on the determined modeling coefficient group and the basic face set.

[0091] For details, see Figure 2 As shown, it is a schematic diagram of an interface for a related object to select a face image on a terminal device in a possible application scenario in an embodiment of the present application, and the terminal device presents Figure 2 The illustrated operation interface enables the relevant object to select multiple facial images of the target object in the operation interface, and send the multiple facial images to the processing device by triggering the completion button.

[0092] In addition, in a possible embodiment of the present application, each facial image obtained by the processing device 120 can be obtained directly from the image acquisition device, that is, connecting to the image acquisition device and exporting each captured facial image, and then the processing device 120 determines the main facial image and the auxiliary facial image in each facial image by itself.

[0093] The technical solution proposed in this application can realize face reconstruction by the terminal device itself when the processing capability of the terminal device meets the processing requirements. This application does not make specific restrictions. The following description only takes the processing device as the executor of face reconstruction as an example to illustrate the face reconstruction process.

[0094] In an embodiment of the present application, the processing device can, based on the processing needs in various application scenarios, after obtaining a group of facial images collected for the target object, generate a corresponding three-dimensional result of the target face by means of a target modeling coefficient group determined by iteratively adjusting the modeling coefficient group and the rendering coefficient group, and a basic face set. The possible application scenarios are as follows.

[0095] Scenario 1: Applied to generate solid models for 3D printing.

[0096] Specifically, after obtaining the face image group of the target object and determining the main face image and the auxiliary face image in the face image group, the processing device uses the target modeling coefficient group and the basic face set determined by the final optimization to obtain the corresponding three-dimensional face result, and then the three-dimensional face result can be applied to 3D printing technology to obtain a solid model based on 3D printing technology.

[0097] Scenario 2: Applied for face recognition.

[0098] Specifically, it can be applied to scenarios where facial recognition is used for identity recognition, such as facial recognition unlocking, face payment, etc.

[0099] After the processing device constructs a three-dimensional face model for the target object, it can implement identity authentication based on the mapping result of the three-dimensional face model in the two-dimensional image during subsequent face verification.

[0100] Scene 3: Applied to generate special effects images.

[0101] Specifically, after generating the corresponding three-dimensional facial result based on the main facial image and at least one auxiliary facial image of the target object, the three-dimensional facial result can be edited for special effects according to actual processing needs to generate a three-dimensional result with special effects information added, wherein the special effects information includes stickers, added expressions, etc.

[0102] Scenario 4: Applied to generating game characters in virtual space.

[0103] Specifically, in order to enrich the game experience of the target object, a game character carrying the three-dimensional face result of the target object can be generated in a virtual game space based on the three-dimensional face result of the target object.

[0104] See also Figure 3 As shown in the figure, it is a schematic diagram of the process of performing face reconstruction in the embodiment of the present application. Figure 3 , the training process of the face reconstruction model is explained:

[0105] Step 301: The processing device obtains a face image group associated with a target object.

[0106] In some possible embodiments of the present application, when constructing a facial image group, a corresponding main facial image and at least one auxiliary facial image can be collected for the target object, and then a corresponding facial image group is established based on the collected main facial image and at least one auxiliary facial image, wherein the facial images in the facial image group can specifically be red, green, and blue (RGB) images, and the number of main facial images and auxiliary facial images is set according to actual processing needs, such as setting the number of main facial images to 1.

[0107] It should be noted that when the constructed face image group only includes the face image of the target object, in the subsequent iterative adjustment process, when more information corresponding to the face image is needed, corresponding content can be generated according to actual processing needs.

[0108] In some other possible embodiments of the present application, when constructing a facial image group, after determining a main facial image and at least one auxiliary facial image included in the facial image group, a pixel importance weight map corresponding to each pixel and a key point importance weight set corresponding to each key point are configured respectively, wherein the facial image is an RGB image, and the number of main facial images and auxiliary facial images is set according to actual needs.

[0109] See also Figure 4 As shown, it is a schematic diagram of the process of configuring the pixel importance weight map and the key point importance weight set in the embodiment of the present application. Figure 4 , the process of determining a main face image and at least one auxiliary face image in each face image, and configuring pixel importance weight maps and key point importance weight sets corresponding to each face image respectively, in an embodiment of the present application is described:

[0110] Step 3011: The processing device uses facial key point detection technology and facial segmentation technology to determine a facial key point set and a facial segmentation result for each facial image included in the facial image group.

[0111] In an embodiment of the present application, the processing device can preset a set of facial reference points that need to be detected according to actual processing needs, wherein the total number of reference points included in the preset set of facial reference points is set according to actual processing needs, and this application does not impose any specific restrictions.

[0112] After the processing device obtains the face image group collected by the corresponding target object, it can use the face key point detection algorithm to generate a corresponding face key point set for each face key point detected in each face image, and use the face segmentation algorithm to determine the face segmentation result corresponding to each face image. Among them, the face key point detection algorithm and the face segmentation algorithm are mature technologies in this field, and this application does not specifically describe the face key point detection algorithm and the face segmentation algorithm.

[0113] For example, see Figure 5 As shown, it is a schematic diagram of the result of face segmentation in the embodiment of the present application. Figure 5As shown in the figure, after the processing device processes a face image using face segmentation technology, the corresponding face segmentation regions can be obtained. As shown in Figure 5, after the face image is processed by face segmentation, a face segmentation result aligned with the face image can be obtained. Based on the obtained face segmentation result, the organ region, cheek region, hair region, and background region in the face image can be determined. Figure 5 For easy distinction, the area indicated by the right slash is the hair area, the area indicated by the left slash is the organ area and the cheek area, and the area indicated by the white part is the background area.

[0114] For example, see Figure 6 As shown, it is a schematic diagram of the result of face key point detection in the embodiment of the present application. Figure 6 As shown, the processing device uses a predefined set of facial reference points, including 86 facial reference points, as the identification basis for key point detection. When performing key point detection on a face image, a face key point set is generated based on each face key point detected in the face image. When generating the face key point set, since the labels corresponding to different face key points remain unchanged, the corresponding face key point set can be established based on the identified key point labels.

[0115] Step 3012: The processing device configures a key point importance weight set corresponding to the face key point set and a corresponding pixel point importance weight map for a main face image and at least one corresponding auxiliary face image determined in each face image, respectively, according to the corresponding face segmentation results.

[0116] In an embodiment of the present application, a processing device may select a main face image and at least one corresponding auxiliary face image from each face image according to actual processing needs, wherein the processing device may receive a selection indication for the main face image, determine a main face image selected from each face image, and then use each other face image in each face image except the main face image as a corresponding auxiliary face image.

[0117] It should be noted that, considering that in the subsequent rounds of iterative adjustment of the modeling coefficient group and the rendering coefficient group, the total number of facial images referenced at the same time is set according to actual processing needs, so the number of images corresponding to the determined auxiliary facial images is at least one. In the embodiment of the present application, the determined main facial image is used to play a main supervisory role when iteratively adjusting the modeling coefficient group, and the determined auxiliary facial image plays an auxiliary supervisory role.

[0118] When the processing device determines the key point importance weight set and pixel importance weight map corresponding to the main face image and at least one auxiliary face image respectively, it performs the following operations according to the corresponding face segmentation results: first, determine the facial key points in each corresponding segmented area respectively, and generate a key point importance weight set based on the key point importance weights configured for the facial key points in each segmented area, then configure the pixel importance weights in each segmented area, and generate a pixel importance weight map based on the pixel importance weights in each segmented area.

[0119] It should be noted that in the embodiments of the present application, in order to be able to refer to the facial feature information provided by the main face image and the auxiliary face image to varying degrees, the present application creatively proposes to differentially configure corresponding importance weights for the pixels and facial key points in the main face image and the auxiliary face image, so as to differentially reference the pixel value information and key point information provided by the main face image and the auxiliary face image.

[0120] In addition, in the embodiment of the present application, the key point importance weights can also be recorded in the form of an array. According to the labels of each detected facial key point, a corresponding label array is generated, and the key point importance weights are configured corresponding to the facial key points at each position in the label array of the facial key points, thereby obtaining the corresponding key point importance weight array.

[0121] In an embodiment of the present application, when configuring the key point importance weight set and the pixel importance weight map corresponding to the facial image, compared with the configuration of the auxiliary facial image, the pixels in different segmented areas of the main facial image can be differentially configured with larger pixel importance weights, and similarly, larger key point importance weights can be configured for the facial key points in different segmented areas.

[0122] For example, for the main face image, for the background area of ​​the face segmentation result, the pixel importance weight of each pixel in the background area can be set to 0, and for the face area including the cheek area and the organ area, the pixel importance weight of each pixel in the face area can be set to 1. Then, based on the pixel importance weights set for the pixels in each area, a key point importance weight map can be synthesized, wherein the synthesized key point importance weight map is specifically a single-channel image in which the pixel values ​​corresponding to the pixels are 0 or 1.

[0123] For another example, taking the processing of an auxiliary face image as an example, the processing device can set the pixel importance weight of each pixel in the background area in the face segmentation result to 0, and the pixel importance weight of each pixel in the cheek area can be set to 1. For each organ area: left and right eyebrow area, left and right eye area, nose area, and upper and lower lip area, the pixel importance weight of each pixel in each organ area can be set to a value less than 1. For example, assuming that the total number of auxiliary face images is X, and X is greater than 1, then the pixel importance weight set for the pixels in each organ area is 1 / X. Then, according to the pixel importance weights corresponding to the pixels included in each segmented area, a key point importance weight map is synthesized.

[0124] For another example, when configuring the key point importance weights for the main face image, the facial key point detection technology is used. It is assumed that the main face image includes K 2D facial key points, and then the corresponding key point importance weights are set for the determined K facial key points. After that, a key point importance weight array composed of the corresponding K key point importance weights can be obtained, wherein the set key point importance weights are set according to actual processing needs, such as being set to 1.

[0125] For another example, taking the processing of an auxiliary face image as an example, when configuring the key point importance weights for the auxiliary face image, the face key point detection technology is used. It is assumed that the auxiliary face image includes M 2D face key points, and then the corresponding key point importance weights are configured for the determined M face key points. In the specific settings, the face key points in different areas can be set differently. For example, the face key points in the contour part can be set to a key point importance weight of 1. The face key points of each organ part of the face are set to a number with a weight less than 1. For example, assuming that the total number of auxiliary face images is Y, and Y is greater than 1, the pixel importance weight set for the pixels in each organ area is 1 / Y, and then the corresponding face key point weight array is established.

[0126] It should be noted that, in the embodiment of the present application, the values ​​of the pixel importance weights and key point importance weights configured for the main face image and the auxiliary face image respectively can be configured according to actual processing needs, but the principle followed in configuring the corresponding values ​​is: compared with the main face image, the pixel importance weights and key point importance weights configured for the auxiliary face image are not higher than the pixel importance weights and key point importance weights configured in the same segmentation area in the main face image.

[0127] For example, the nose areas of the auxiliary face image and the main face image can be used as a corresponding similar segmentation area. Then, the configuration of the pixel importance weight of each pixel in the nose area of ​​the auxiliary face image is smaller than that of each pixel in the nose area of ​​the main face image, and the pixel importance weight of each facial pixel in the nose area of ​​the auxiliary face image is lower than that of each facial pixel in the nose area of ​​the main face.

[0128] In addition, in order to reduce to a certain extent the interference caused by the facial feature information (organ information) provided by the auxiliary facial image that conflicts with the main facial image, the pixel importance weight values ​​and key point importance weight values ​​corresponding to the segmented areas of each organ in the auxiliary facial image can be configured with smaller values ​​than other segmented areas in the auxiliary facial image.

[0129] In this way, based on the pixel importance weight configuration and key point importance weight configuration for the main face image and the auxiliary face image, the different importance levels of the main face image and the auxiliary face image can be concretely distinguished in the form of weight values, and the method of generating a key point importance weight set and a pixel importance weight map before performing multiple rounds of iterative adjustments can avoid repeated generation of the same information content in different rounds of iterations, thereby reducing the amount of calculation to a certain extent.

[0130] At the same time, when configuring the pixel importance weights and key point importance weights, the various segmented areas obtained by means of face segmentation technology can better distinguish different facial areas in the main face image and the auxiliary face image, providing a processing basis for targeted configuration of different segmented areas.

[0131] Step 302: The processing device performs multiple rounds of iterative adjustments on the initial modeling coefficient group based on the face image group and the basic face set, and determines the three-dimensional result of the target face based on the adjusted target modeling coefficient group and the basic face set.

[0132] In executing step 302, the processing device performs multiple rounds of iterative adjustments on the initial modeling coefficient group based on the face image group and the basic face set, wherein when the basic face set is a 3DMM face set, the modeling coefficient group is a face shape coefficient and a face texture coefficient.

[0133] It should be noted that in the embodiment of the present application, other basic face sets that can work together to generate three-dimensional facial results can be selected according to actual processing needs, and the modeling coefficient group used for weighted fusion of each basic face in the basic face set can be iteratively adjusted.

[0134] In addition, in the embodiment of the present application, while iteratively adjusting the modeling coefficient group, the rendering coefficient group is synchronously adjusted, wherein the rendering coefficient group is used to render and reconstruct the three-dimensional result of the face to obtain a two-dimensional image, and the present application does not impose specific restrictions on this. For the rendering coefficient group, the rendering coefficient group is reversely optimized and adjusted using the loss value. Therefore, the main face image and at least one auxiliary face image used when calculating the loss value in the rendering coefficient group are respectively determined to have a corresponding set of rendering coefficients, so that when calculating the loss value, the main rendering image obtained by rendering the main face image corresponding to the three-dimensional result of the face can be obtained, and the auxiliary rendering images obtained by rendering the corresponding at least one auxiliary face image can be obtained.

[0135] In the embodiment of the present application, considering that the image information that a single image can provide is too single, in the process of constructing the three-dimensional result of the face, it is necessary to use the information of multiple face images to comprehensively reduce the loss between the face image and the rendered image obtained by rendering the corresponding face image, so as to achieve comprehensive adjustment of the modeling coefficient group. Therefore, in the process of iterative optimization and adjustment of the modeling coefficient group, it is necessary to use the face image group including multiple face images collected according to the corresponding target object. Therefore, the technical solution proposed in this application is applied to the scenario of using the information of multiple face images to calculate the loss value in the iterative adjustment process.

[0136] See attached Figure 7 As shown, it is a schematic diagram of a process flow for performing a round of iterative adjustment in an embodiment of the present application. Figure 7 , the process of performing a round of iterative adjustments is described in detail:

[0137] Step 3021: The processing device obtains a first comparison result between the main face image and the associated main rendering image based on the three-dimensional face result determined according to the current modeling coefficient group and the basic face set, and obtains a second comparison result between at least one auxiliary face image and each of the associated auxiliary rendering images.

[0138] Specifically, when executing step 3021, refer to Figure 8 As shown, it is a schematic diagram of the process of determining the comparison result in the embodiment of the present application. Figure 8 , the process of determining a first comparison result between a main face image and an associated main rendered image, and respectively determining a second comparison result between at least one auxiliary face image and each of the associated auxiliary rendered images is described:

[0139] Step 3021-a: The processing device determines a face reconstruction result according to the current modeling coefficient group and the basic face set, and based on the face reconstruction result and the currently acquired rendering coefficient group, generates a main rendering image associated with the main face image, and generates a secondary rendering image associated with the at least one secondary face image.

[0140] It should be noted that in the initial iterative adjustment process, the current modeling coefficient group and rendering coefficient group specifically refer to the initialized modeling coefficient group and the initialized rendering coefficient group. In the subsequent iterative process, the modeling coefficient group and rendering coefficient group obtained in the previous round of iterative adjustment are used as the current modeling coefficient group and rendering coefficient group.

[0141] Specifically, the processing device can obtain a three-dimensional face result after weighted superposition of each basic face in the basic face set according to the current modeling coefficient group, and then obtain the current rendering coefficient group. Based on the three-dimensional face result and the current rendering coefficient group, the three-dimensional face result can be reconstructed to generate a main rendered image associated with the main face image, and a secondary rendered image associated with the secondary face image, wherein the rendering coefficient group includes each group of illumination coefficients and each group of posture coefficients.

[0142] For example, with the help of the 3DMM face set, as well as the current face shape coefficient and face texture coefficient, the corresponding three-dimensional face result can be determined, and then with the help of each group of illumination coefficients and each group of posture coefficients, each rendered image rendered by the three-dimensional face result can be determined respectively, wherein each rendered image includes a main rendered image and an auxiliary rendered image.

[0143] Step 3021-b: The processing device determines the main face image, the main rendering image, at least one auxiliary face image, and at least one auxiliary rendering image, and their corresponding facial key point sets and face segmentation results.

[0144] In some possible embodiments of the present application, when face recognition processing and face segmentation processing are not performed on the main face image and the at least one auxiliary face image respectively before multiple rounds of iterations, when executing step 3021-b, the processing device can use face key point detection technology and face segmentation technology to process the main face image and the at least one auxiliary face image respectively, and determine the face key point sets and face segmentation results included in the main face image and the at least one auxiliary face image respectively.

[0145] In this way, there is no need to additionally store the main face image and at least one auxiliary face image, their corresponding facial key point sets and face segmentation results, which reduces the storage space occupied and avoids the abnormal situation of being unable to obtain the facial key point set and face segmentation results due to storage abnormalities.

[0146] In some other possible embodiments of the present application, when face recognition processing and face segmentation processing are performed on the main face image and at least one auxiliary face image respectively before multiple rounds of iterations, when executing step 3021-b, the processing device can directly obtain the face key point set and face segmentation result corresponding to the main face image, and can directly obtain the face key point set and face segmentation result corresponding to at least one auxiliary face image.

[0147] In this way, considering that the facial key point sets and facial segmentation results in the primary face image and the auxiliary face image need to be used in each round of iterative adjustment, the relevant content can be stored in advance to avoid repeated calculations.

[0148] After the processing device determines the facial key point sets and facial segmentation results corresponding to the main facial image and at least one auxiliary facial image, it can determine the corresponding facial three-dimensional results and facial segmentation results in the established facial three-dimensional result, so that in the process of obtaining the rendered image, the corresponding facial key points and facial segmentation results in the rendered image can be directly mapped and determined, wherein the rendered image in this application is a general term for the main rendered image and at least one auxiliary rendered image, and is used to describe the common content in the main rendered image and at least one auxiliary rendered image.

[0149] Specifically, in some possible implementation methods of the present application, taking the processing process of the main face image as an example: after the processing device determines the two-dimensional coordinates of each facial key point in the main face image under the coordinates of the main face image, it can adopt a two-dimensional to three-dimensional shape dense correspondence algorithm to respectively determine the corresponding three-dimensional coordinates of each facial key point in the main face image in the three-dimensional face image, and when the main rendered image is obtained based on the three-dimensional face result rendering, determine the two-dimensional coordinates of the three-dimensional coordinates of each facial key point when it is projected onto the main rendered image, and then determine each facial key point included in the main rendered image and the corresponding two-dimensional coordinates.

[0150] Similar to the process of determining each facial key point in the main rendered image, the processing device can separately determine the edge pixel points of each segmented area in the main face image after face segmentation, and determine the corresponding three-dimensional coordinates on the three-dimensional face result based on the two-dimensional coordinates of the edge pixel points. Then, when the main rendered image is rendered based on the three-dimensional face result, the various segmented areas in the main rendered image are synchronously mapped and determined.

[0151] It should be noted that, since the processing method for determining each facial key point and the face segmentation result in the auxiliary rendering image is the same as the above process, it will not be described in detail here.

[0152] In some other possible embodiments of the present application, when determining the facial key points and face segmentation results included in the rendered image, it is also possible to selectively directly use the facial key point detection technology to determine each facial key point in the rendered image respectively, and use the face segmentation technology to determine the face segmentation results of each rendered image respectively.

[0153] In this way, each facial key point and facial segmentation result included in the facial image and the associated rendering image can be determined respectively, providing a processing basis for the subsequent calculation of the loss value.

[0154] Step 3021-c: The processing device determines a corresponding first comparison result based on the pixel value differences and key point position differences between the main rendered image and the main facial image, and determines a corresponding second comparison result based on the pixel value differences and key point position differences between at least one auxiliary rendered image and the corresponding auxiliary facial image.

[0155] When determining the first comparison result, the processing device determines the difference between the main facial image and the main rendered image at the level of pixel values ​​of the pixels and at the level of key point positions, and then determines the comparison result between the rendered image and the corresponding facial image based on the pixel value differences between each pixel and the key point position differences between each facial key point with the same label.

[0156] It should be noted that, when the face image is an RGB image, each face image includes pixel values ​​of three channels, namely, the pixel value of each pixel point in the R channel, the pixel value of each pixel point in the G channel, and the pixel value of each pixel point in the B channel. Therefore, in determining the pixel value difference, it is necessary to determine the pixel value difference between pixels with the same relative position in the corresponding channels for each of the three channels.

[0157] In an embodiment of the present application, the processing device determines the comparison result between the rendered image and the corresponding facial image based on the pixel value differences between each pixel point and the key point position differences between each facial key point with the same label.

[0158] Specifically, when the processing device determines the corresponding first comparison result based on the pixel value differences and key point position differences between the main rendered image and the main facial image, the processing device determines the pixel value differences between each pixel point with the same relative position in the main facial image and the main rendered image, and determines the key point position differences between each facial key point with the same label, and then determines the corresponding first comparison result based on each pixel value difference and each key point position difference.

[0159] When determining the pixel value difference, the processing device can use the L1 error between each pixel point with the same relative position in the same channel of the main rendered image and the main face image as the pixel value difference corresponding to the pixel points at the corresponding position. Similarly, the pixel value difference corresponding to each pixel point in the R channel, G channel, and B channel between the main rendered image and the main face image can be determined.

[0160] When determining the difference in key point positions, the processing device can determine the coordinate difference between the facial key points with the same label in the main rendered image and the main face image as the corresponding key point position difference. Since there is a benchmark facial key point as a reference when performing facial key point detection, the same label of the facial key point mentioned in this application specifically refers to the two facial key points in the main rendered image and the main face image corresponding to the same benchmark facial key point.

[0161] It should be noted that when calculating the key point position difference between facial key points, it can be determined by calculating the L1 error of the pixel point in the same constructed coordinate system, or by calculating the distance between two facial key points in the same coordinate system.

[0162] For example, see Fig. 9 As shown in FIG. 1 , it is a schematic diagram of two facial key points corresponding to the same number in an embodiment of the present application. For the main facial image and the main rendered image, since the sizes of the two images are exactly the same, when establishing the coordinate system as shown in the figure, the coordinate position of the facial key point labeled 81 in the main rendered image can be determined to be (x1, y1), and the coordinate position of the facial key point labeled 81 in the main facial image can be determined to be (x2, y2). Then, when determining the difference in the key point position between the two facial key points labeled 81, the L1 error between the two coordinates can be directly calculated, or, it can be calculated by To determine the distance between two facial key points, to characterize the difference in key point positions.

[0163] Furthermore, after determining the pixel value difference of each pixel point between the main rendered image and the main face image, and determining the key point position difference between each facial key point between the main rendered image and the main face image, the determined pixel value difference and key point position difference are used as the corresponding first comparison result.

[0164] Similarly, the processing device can adopt the same processing method as that for determining the first comparison result to respectively determine the second comparison result between at least one auxiliary rendering image and the corresponding auxiliary facial image, which is not described in detail in this application.

[0165] In addition, in the embodiment of the present application, the number of pixels included in the face image, the rendered image, and the pixel importance weight image configured for the face image and the rendered image is exactly the same.

[0166] In this way, by calculating the differences in pixel values ​​and the coordinate differences between facial key points, the difference between the facial image and the rendered image can be expressed.

[0167] Step 3022: The processing device configures a reference evaluation result of a corresponding auxiliary facial image based on the degree of deviation between the at least one second comparison result obtained and the first comparison result.

[0168] See also Fig.10 As shown in FIG. 1 , it is a schematic diagram of the process of determining the reference evaluation result of the auxiliary face image in the embodiment of the present application. Fig.10 , the process of determining the reference evaluation result based on at least one second comparison result obtained when executing step 3022 is described:

[0169] Step 3022-a: The processing device determines each group of matching segmentation areas based on a first face segmentation result corresponding to a first comparison result and a second face segmentation result corresponding to a second comparison result, and configures a pixel value reference evaluation result for each pixel point in the corresponding segmentation area for each group of segmentation areas according to a first offset degree between corresponding pixel value difference means.

[0170] Specifically, the processing device takes the face segmentation result corresponding to the main face image as the first face segmentation result, and since the first comparison result is determined based on the pixel value difference between the corresponding pixel points in the main face image and the main rendering image, as well as the key point coordinate difference of the corresponding face key points, the face analysis result corresponding to the first comparison result is the face segmentation result of the main face image.

[0171] Similarly, the processing device names the face segmentation result corresponding to at least one auxiliary facial image as a second face segmentation result. For at least one second comparison result, since a second comparison result is determined based on the difference in pixel values ​​between corresponding pixel points in an auxiliary rendering image and a corresponding auxiliary facial image, and the difference in key point coordinates of corresponding facial key points, a second comparison result corresponds to the face segmentation result of an auxiliary facial image.

[0172] Based on the above description, the distribution of the first comparison results in each segmented area of ​​the primary face image can be determined, and the distribution of the second comparison results in each segmented area of ​​the auxiliary face image can be determined.

[0173] Furthermore, the processing device determines the groups of segmented regions that match in the first face segmentation result and the second face segmentation result, wherein when determining the groups of segmented regions that match, the segmented regions corresponding to the same organ region can be used as a group of segmented regions that match. Then, based on the first degree of deviation between the mean values ​​of the pixel value differences in each group of segmented regions that match, the pixel value reference evaluation result of each pixel point in the corresponding segmented region in the second comparison result is configured.

[0174] For example, see Fig.11 As shown, it is a schematic diagram of each group of segmented areas in the embodiment of the present application. For the first face segmentation result and the second face segmentation result, the two segmented areas corresponding to the mouth area can be used as a group of segmented areas. Similarly, the two segmented areas corresponding to the nose area can be used as a group of segmented areas, the two segmented areas corresponding to the eyebrow area can be used as a group of segmented areas, and the two segmented areas corresponding to the eye area can be used as a group of segmented areas.

[0175] In the embodiment of the present application, when determining the pixel value reference evaluation result of each pixel point in each segmented area in the second comparison result, the processing device can perform targeted configuration for the first offset degree in each group of segmented areas. When the first offset degree is used to characterize the offset of the second comparison result relative to the first comparison result, if the first offset degree is greater than 0, when determining the pixel value reference evaluation result of each pixel point in a group of segmented areas, the following formula is used:

[0176]

[0177] Wherein, err is the mean difference of pixel values ​​in a corresponding set of segmented areas in the second comparison result after normalization (i.e., the L1 error mean), ref_err is the mean difference of pixel values ​​in a corresponding set of segmented areas in the first comparison result after normalization (i.e., the L1 error mean), and the normalization is achieved by dividing the difference of pixel values ​​by 255.

[0178] On the contrary, if it is determined that within a set of segmented areas, the first offset degree of the second comparison result compared to the first comparison result is less than 0, then the pixel value difference of the corresponding area between the corresponding auxiliary face image and the auxiliary rendering image is smaller than the pixel value difference between the main face image and the main rendering image, indicating that the pixel value difference between the auxiliary face image and the auxiliary rendering image is smaller, so the pixel value reference evaluation result of each pixel point in the corresponding segmented area can be set to 1.

[0179] In addition, for areas that have little impact on the generation of the three-dimensional face model, such as the cheek area and the background area, the reference evaluation result corresponding to the cheek area can be set to 1, and the reference evaluation result of the background area can be set to 0.

[0180] It should be noted that since the main face image plays the main reference role in the iterative adjustment process, this application does not describe the reference evaluation results of each pixel in the main face image. According to actual processing needs, the reference evaluation results of each pixel in the face area of ​​the main face image can be set to 1 by default, and the reference evaluation results of the pixels in the invalid background area can be set to 0.

[0181] It should be noted that since in the embodiment of the present application, both the facial image and the rendered image are three-channel images, the first comparison result and the second comparison result are essentially also three-channel. When determining the offset situation based on the pixel value difference in the comparison result, the pixel value difference of the three channels can be considered at the same time, and the average value of the pixel point difference in the same segmented area in the three channels can be taken for consideration, and finally a single-channel pixel value reference evaluation map is constructed for the auxiliary facial image.

[0182] In this way, since the corresponding reference evaluation result is configured for each pixel, a single-channel reference evaluation result map can be generated based on the reference evaluation result corresponding to each pixel point as the pixel value of the corresponding pixel point.

[0183] Step 3022-b: The processing device determines each group of facial key point subsets that match within each group of segmented areas based on a second comparison result and a set of facial key points corresponding to each of the first comparison results, and configures key point reference evaluation results for the facial key points within the corresponding segmented area for each group of facial key point subsets based on a second offset degree between the corresponding key point position difference means.

[0184] Specifically, after determining each group of segmentation intervals, the processing device can determine each group of facial key point subsets that match within each group of segmentation areas based on a second comparison result and a set of facial key points corresponding to the first comparison result, and determine the second degree of offset between the corresponding key point position difference means for each group of facial key point subsets, wherein a group of key point subsets includes each facial key point within the same type of segmentation area.

[0185] For example, continue to see Fig.11As shown, after dividing each group of target areas, taking a group of facial key point subsets corresponding to the eyebrow area as an example, a facial key point subset can be generated based on each facial key point located in the eyebrow area in the primary face image, and a facial key point subset can be generated based on each facial key point located in the eyebrow area in the auxiliary face image, and then a group of facial key point subsets can be generated based on the determined two facial key point subsets.

[0186] Furthermore, the processing device determines the key point position differences corresponding to each facial key point in the auxiliary facial image according to the second comparison result, and determines the key point position differences corresponding to each facial key point in the main facial image according to the first comparison result, and then determines the mean value of the key point position difference corresponding to each of the two facial key point subsets included in each group of facial key point subsets, and then configures the key point reference evaluation result of each facial key point in the corresponding segmented area according to the second offset degree between the determined two key point position difference means.

[0187] For example, taking the key point reference evaluation result of each facial key point in the mouth area as an example, the processing device calculates the mean value of the key point position difference for each facial key point in the mouth area in the main face image and the auxiliary face image respectively. Assuming that the mean value of the key point position difference calculated for the main face image is recorded as ref_err1, and the mean value of the key point position difference calculated for the auxiliary face image is recorded as err1, then assuming that the difference between err1 and ref_err1 is greater than 0, the key point reference evaluation result for each facial key point in the mouth area in the auxiliary face image is set to e (-(err1-ref_err1)) Otherwise, if it is determined that the difference between err1 and ref_err1 is less than 0, the key point reference evaluation result of each facial key point in the mouth area is set to 1.

[0188] For another example, considering that the key points of the face contour part have little difference when generating a three-dimensional face model, the key point reference evaluation result of the face key points located in the face contour part in the auxiliary face image can be set to 1.

[0189] It should be noted that since the master face image plays the main reference role in the iterative adjustment process, this application does not provide a specific description of the key point reference evaluation results of each facial key point in the master face image, and they can all be set to 1.

[0190] In this way, by calculating the reference evaluation results corresponding to each facial key point, the reference situation for the corresponding facial key point in the auxiliary facial image can be determined.

[0191] Step 3023: The processing device adjusts the modeling coefficient group based on a first loss value determined according to the first comparison result and a second loss value determined according to at least one second comparison result and a corresponding reference evaluation result.

[0192] In an embodiment of the present application, when the processing device calculates the first loss value determined according to the first comparison result, the processing device determines the pixel importance weights respectively configured for the pixel points in each segmented area in the main face image, wherein the segmented area is determined by the corresponding face segmentation result, and then calculates the first pixel value loss based on the differences in each pixel value included in the first comparison result and the pixel importance weights configured for each pixel point.

[0193] It should be noted that, before performing multiple rounds of iterations, when a pixel importance weight map corresponding to the master face image is configured, the processing device can directly obtain the pixel importance weight map corresponding to the master face image to determine the pixel importance weights corresponding to each pixel in the master face image; conversely, if before performing multiple rounds of iterations, when a pixel importance weight map corresponding to the master face image is not configured, the processing device configures pixel importance weights for the pixels in each segmented area in the master face image, wherein the relevant configuration process is the same as the configuration process of step 3012, and will not be repeated here.

[0194] Specifically, when calculating the first pixel value loss, considering that the main face image is an RGB image, the following formulas can be used to calculate the corresponding first pixel value sub-loss for the three channels of the main face image:

[0195]

[0196] Among them, L1_photo represents the calculated first pixel sub-loss, |Mask| represents the sum of the pixel importance weights configured for each pixel in the main face image, ∑() represents the processing and summation of the pixels in the face area in one channel of the main face image, Ci represents the pixel value reference evaluation result of any pixel i in the face area, and the value is 1, Mask i Characterizes the importance weight of the pixel corresponding to pixel i, I' i Represents the pixel value of pixel i in the main rendered image, I i Represents the pixel value of pixel i in the main face image.

[0197] Similarly, the first pixel sub-loss in different channels can be obtained according to the above formula, and then the sum of the first pixel sub-loss of the three channels R, G, and B is determined as the first pixel loss.

[0198] At the same time, the processing device calculates the first key point loss based on the position differences of each key point included in the first comparison result and the key point importance weights corresponding to each key point configuration.

[0199] It should be noted that, before performing multiple rounds of iterations, when a set of key point importance weights corresponding to the master face image is configured, the processing device can directly obtain the corresponding set of key point importance weights; conversely, if a set of key point importance weights corresponding to the master face image is not configured before performing multiple rounds of iterations, the processing device configures the key point importance weights for each facial key point in the master face image, wherein the relevant configuration process is the same as the configuration process of step 3012 and will not be repeated here.

[0200]

[0201] Among them, |M| represents the sum of the key point importance weights configured for each facial key point in the main face image, ∑() represents the sum of the processing of each facial key point in the main face image, CLi represents the key point reference evaluation result of any facial key point i, and its value is 1, W i Characterizes the key point importance weight corresponding to key point i, K' i -K i Characterizes the pixel coordinate L1 error between the face key point i in the main rendered image and the corresponding face key point i in the main face image.

[0202] Further, the processing device determines a corresponding first loss value according to the first pixel value loss and the first key point loss. Specifically, the sum of the first pixel value loss and the first key point loss is used as the corresponding first loss value.

[0203] While determining the first loss based on the first comparison result, the processing device may also use the above calculation formula to determine a corresponding second loss value for at least one second comparison result.

[0204] Specifically, the processing device obtains pixel importance weights respectively configured for pixels in each segmented area in at least one face image, wherein the segmented area is determined by the corresponding face segmentation result.

[0205] Among them, the method for determining the importance weights of the pixels corresponding to each pixel in the auxiliary facial image is the same as the method for determining the importance weights of the pixels corresponding to each pixel in the main facial image as mentioned above, and this application does not make any specific restrictions.

[0206] Furthermore, the processing device calculates the second pixel value loss based on each pixel value difference included in at least one second comparison result, and the pixel importance weight configured corresponding to each pixel point and the pixel value reference evaluation result.

[0207] Specifically, considering that both the auxiliary face image and the corresponding auxiliary rendering image are RGB images, the corresponding second comparison result is also three-channel. Therefore, similar to the calculation result of the first pixel value loss, the processing device can calculate the second pixel sub-loss of each channel, and add the second pixel sub-loss of each channel to obtain the second pixel loss. The formula used when calculating a second pixel sub-loss is as follows:

[0208]

[0209] Wherein, L2_photo represents the calculated second pixel sub-loss, |Mask| represents the sum of the pixel importance weights configured for each pixel in the auxiliary face image, ∑() represents the sum of the pixel processing in the face area in one channel of the auxiliary face image, Cj represents the reference evaluation result of the pixel value of any pixel j in the face area, Mask j Characterizes the importance weight of the pixel corresponding to pixel j, I' j Represents the pixel value of pixel j in the auxiliary rendering image, I j Represents the pixel value of pixel j in the auxiliary face image.

[0210] Similarly, according to the above formula, the second pixel sub-loss in different channels can be obtained, and then the sum of the first pixel sub-loss of the three channels R, G, and B is determined as the second pixel loss.

[0211] At the same time, the processing device can calculate the second key point loss based on the position differences of each key point included in at least one second comparison result, as well as the key point importance weights corresponding to each facial key point configuration and the key point reference evaluation results.

[0212] Similar to the above process of calculating the first key point loss, the processing device may calculate the second key point loss using the following formula for at least one second comparison result:

[0213]

[0214] Among them, |W| represents the sum of the key point importance weights configured for each facial key point in the auxiliary face image, ∑() represents the sum of the processing of each facial key point in the auxiliary face image, CLj represents the key point reference evaluation result of any facial key point j, and W j Characterizes the key point importance weight corresponding to key point j, K'j -K j Characterizes the pixel coordinate L1 error between the face key point j in the auxiliary rendered image and the corresponding face key point j in the main face image.

[0215] Further, the processing device determines a corresponding second loss value based on each second pixel value loss and each second key point loss. Specifically, the sum of each second pixel value loss and each second key point loss is used as the corresponding second loss value.

[0216] After obtaining the first loss value and the second loss value, the processing device reversely optimizes and adjusts the modeling coefficient group and the rendering coefficient group based on the sum of the first loss value and each second loss value, wherein the processing device can select an optimizer to achieve optimization adjustment, and the selected optimizer can specifically be an Adam optimizer.

[0217] In this way, in each iterative step, the pixel value differences and key point position differences in the facial feature areas of the main face image and the secondary face image are considered, and an adaptive loss function scheme is implemented, so that in the process of iterative optimization calculation of the loss value, the image information of a main face image is mainly used, and among the image information of the secondary face image, the secondary face image with less difference than the main face image, that is, the part similar to the main face image is used more as the supervision information for optimization adjustment, and the dissimilar part is used less.

[0218] Furthermore, the processing device executes the iterative optimization process in a loop until a preset convergence condition is met, obtains a final adjusted target modeling coefficient group, and generates a target face three-dimensional result corresponding to the target object based on the target modeling coefficient group and the basic face set.

[0219] The preset convergence condition can be: the number of iterative adjustments reaches a first set threshold, or the sum of the first loss value and the second loss value is continuously lower than the first threshold value for a number of times reaching a second set threshold, wherein the values ​​of the first set threshold, the first threshold value, and the second set threshold are set according to actual processing needs, and this application does not impose any specific restrictions.

[0220] See also Fig.12 As shown in the figure, it is an application diagram of realizing face reconstruction based on 3DMM in the embodiment of the present application. Fig.12 Provide detailed explanation.

[0221] according to Fig.12 In the illustrated content, when reconstructing the face of the target object, it is assumed that when reconstructing the face of the target object Z, the target object: X+1 face images of the object Z are obtained, and the main face image and X auxiliary face images are specified in the obtained X+1 face images, wherein, for the convenience of illustration, Fig.12Only the auxiliary facial image 1 and the auxiliary facial image X are shown.

[0222] During the initial face reconstruction process, the processing device weightedly fuses each 3DMM face in the 3DMM face set based on the initial 3DMM coefficient, Pose coefficient, and illumination coefficient to obtain a three-dimensional face result, and then uses each group of Pose coefficients and each group of illumination coefficients to perform mapping and rendering based on the three-dimensional face result to obtain a rendered image group, and then determines the reference evaluation result corresponding to the auxiliary face image according to the comparison result between the rendered image and the corresponding face image, and the offset between the main rendered image and the corresponding main face image, and then calculates the corresponding second loss value based on the reference evaluation result and the second comparison result.

[0223] Then, based on the calculated loss value, the optimizer is used to adjust the 3DMM coefficients, each group of Pose coefficients, and each group of illumination coefficients. The above operations are performed repeatedly until convergence is determined, the target 3DMM coefficients are obtained, and the three-dimensional face result is determined based on the target 3DMM coefficients and the 3DMM face set.

[0224] Based on the same inventive concept, refer to Fig.13 As shown, it is a schematic diagram of the logical structure of the device for generating a three-dimensional face result in an embodiment of the present application. The device for generating a three-dimensional face result 1300 includes an acquisition unit 1301 and an adjustment unit 1302, wherein:

[0225] An acquisition unit 1301 is used to acquire a face image group collected for a target object, wherein the face image group includes a main face image and at least one auxiliary face image;

[0226] The adjustment unit 1302 is used to perform multiple rounds of iterative adjustments on the initial modeling coefficient group based on the face image group and the basic face set, and determine the target face 3D result based on the adjusted target modeling coefficient group and the basic face set; wherein the following operations are performed in each round of iterative adjustment process:

[0227] Based on the three-dimensional face result determined according to the current modeling coefficient group and the basic face set, a first comparison result between the main face image and the associated main rendering image is obtained, and a second comparison result between at least one auxiliary face image and the associated auxiliary rendering image is obtained respectively;

[0228] Based on the obtained at least one second comparison result and the degree of deviation between each of the second comparison results and the first comparison result, configuring a reference evaluation result of a corresponding auxiliary facial image;

[0229] The modeling coefficient group is adjusted based on a first loss value determined according to the first comparison result and a second loss value determined according to at least one second comparison result and a corresponding reference evaluation result, respectively.

[0230] Optionally, when the basic face set is a 3DMM face set, the modeling coefficient group is a 3DMM coefficient.

[0231] Optionally, after acquiring the face image group collected for the target object, before performing multiple rounds of iterative adjustments on the initial modeling coefficient group based on the face image group and the basic face set, the acquiring unit 1301 is further used to:

[0232] Using facial key point detection technology and facial segmentation technology, respectively for each facial image included in the facial image group, determine a facial key point set and a facial segmentation result;

[0233] For a main face image and at least one corresponding auxiliary face image determined in each face image, a key point importance weight set corresponding to the face key point set and a corresponding pixel point importance weight map are configured according to the corresponding face segmentation results.

[0234] Optionally, when configuring a key point importance weight set corresponding to a face key point set and configuring a corresponding pixel point importance weight map according to corresponding face segmentation results, the acquisition unit 1301 is used to:

[0235] According to the corresponding face segmentation results, the following operations are performed respectively:

[0236] Determine the facial key points in each corresponding segmented area respectively, and generate a key point importance weight set based on the key point importance weights respectively configured for the facial key points in each segmented area;

[0237] The importance weights of the pixels in each segmented area are configured, and based on the importance weights of the pixels in each segmented area, a pixel importance weight map is generated.

[0238] Optionally, based on the three-dimensional face result determined according to the current modeling coefficient group and the basic face set, when obtaining a first comparison result between the main face image and the associated main rendering image, and respectively obtaining a second comparison result between at least one auxiliary face image and the associated auxiliary rendering image, the adjustment unit 1302 is used to:

[0239] Determine a three-dimensional face result according to the current modeling coefficient group and the basic face set, and generate a main rendered image associated with the main face image and a secondary rendered image associated with the at least one secondary face image based on the three-dimensional face result and the currently acquired rendering coefficient group, wherein the rendering coefficient group includes each group of illumination coefficients and each group of posture coefficients;

[0240] Determine respectively the main face image, the main rendered image, the at least one auxiliary face image, and the at least one auxiliary rendered image, and the corresponding facial key point sets and face segmentation results;

[0241] A corresponding first comparison result is determined based on the pixel value differences and key point position differences between the main rendered image and the main facial image, and a corresponding second comparison result is determined based on the pixel value differences and key point position differences between the at least one auxiliary rendered image and the corresponding auxiliary facial image.

[0242] Optionally, when determining the corresponding first comparison result according to the pixel value difference and key point position difference between the main rendering image and the main face image, the adjustment unit 1302 is configured to:

[0243] Determine the pixel value difference between each pixel point with the same relative position in the main face image and the main rendering image, and determine the key point position difference between each face key point with the same label;

[0244] According to the differences in each pixel value and each key point position, a corresponding first comparison result is determined.

[0245] Optionally, when configuring the reference evaluation result of the corresponding auxiliary facial image based on the degree of deviation between the at least one second comparison result obtained and the first comparison result, the adjustment unit 1302 is used to:

[0246] For at least one second comparison result, perform the following operations respectively:

[0247] Determine, according to a first face segmentation result corresponding to the first comparison result and a second face segmentation result corresponding to the second comparison result, groups of matched segmentation regions, and configure, for each group of segmentation regions, pixel value reference evaluation results for each pixel point in the corresponding segmentation region according to a first offset degree between corresponding pixel value difference means;

[0248] According to the facial key point sets corresponding to the second comparison result and the first comparison result, each group of facial key point subsets matched within each group of segmentation areas is determined respectively, and for each group of facial key point subsets, according to the second offset degree between the corresponding key point position difference means, the key point reference evaluation results of the facial key points within the corresponding segmentation area are configured.

[0249] Optionally, when determining the first loss value according to the first comparison result, the adjusting unit 1302 is configured to:

[0250] Determine pixel importance weights respectively configured for pixels in each segmented region in the master face image, wherein the segmented region is determined by the corresponding face segmentation result;

[0251] Calculating a first pixel value loss based on each pixel value difference included in the first comparison result and a pixel point importance weight configured corresponding to each pixel point;

[0252] Calculate the first key point loss based on the position differences of each key point included in the first comparison result and the key point importance weights corresponding to each key point configuration;

[0253] A corresponding first loss value is determined according to the first pixel value loss and the first key point loss.

[0254] Optionally, when determining the second loss value according to at least one second comparison result and a corresponding reference evaluation result, the adjusting unit 1302 is configured to:

[0255] Obtaining pixel importance weights respectively configured for pixels in each segmented area in the at least one auxiliary face image, wherein the segmented area is determined by a corresponding face segmentation result;

[0256] Calculating a second pixel value loss based on each pixel value difference included in the at least one second comparison result, and a pixel point importance weight configured corresponding to each pixel point and a pixel value reference evaluation result;

[0257] Calculating a second key point loss based on the position differences of each key point included in the at least one second comparison result, and the key point importance weights configured corresponding to each face key point and the key point reference evaluation results;

[0258] A corresponding second loss value is determined based on each second pixel value loss and each second key point loss.

[0259] After introducing the method and device for generating a three-dimensional result of a face according to an exemplary embodiment of the present application, next, an electronic device according to another exemplary embodiment of the present application is introduced.

[0260] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.

[0261] Based on the same inventive concept as the above method embodiment, an electronic device is also provided in the embodiment of the present application, referring to Fig.14As shown, it is a schematic diagram of a hardware structure of an electronic device using an embodiment of the present application, and the electronic device 1400 may include at least a processor 1401 and a memory 1402. Among them, the memory 1402 stores a program code, and when the program code is executed by the processor 1401, the processor 1401 performs any of the above steps for generating a three-dimensional face result.

[0262] In some possible implementations, the computing device according to the present application may include at least one processor and at least one memory. The memory stores program code, and when the program code is executed by the processor, the processor executes the steps of generating the three-dimensional face result according to various exemplary implementations of the present application described above in this specification. For example, the processor may execute the following steps: Figure 3 , Figure 4 Follow the steps shown in .

[0263] Refer to the following Fig.15 1500 according to this embodiment of the present application is described. Fig.15 As shown, the computing device 1500 is in the form of a general computing device. The components of the computing device 1500 may include, but are not limited to: at least one processing unit 1501, at least one storage unit 1502, and a bus 1503 connecting different system components (including the storage unit 1502 and the processing unit 1501).

[0264] Bus 1503 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a processor, or a local bus using any of a variety of bus architectures.

[0265] The storage unit 1502 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 15021 and / or a cache memory unit 15022 , and may further include a read-only memory (ROM) 15023 .

[0266] The storage unit 1502 may also include a program / utility 15025 having a set (at least one) of program modules 15024, such program modules 15024 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0267] The computing device 1500 may also communicate with one or more external devices 1504 (e.g., keyboards, pointing devices, etc.), one or more devices that enable objects to interact with the computing device 1500, and / or any device that enables the computing device 1500 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 1505. In addition, the computing device 1500 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 1506. As shown, the network adapter 1506 communicates with other modules for the computing device 1500 via a bus 1503. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the computing device 1500, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0268] Based on the same inventive concept as the above-mentioned method embodiment, various aspects of the method for generating a three-dimensional face result provided by the present application can also be implemented in the form of a program product, which includes a program code. When the program product is run on an electronic device, the program code is used to enable the electronic device to execute the steps of the method for generating a three-dimensional face result according to various exemplary embodiments of the present application described above in this specification. For example, the electronic device can execute the following steps: Figure 3 , Figure 4 Follow the steps shown in .

[0269] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0270] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0271] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for generating a three-dimensional result of a face, characterized in that: include: Acquire a face image group associated with the target object, wherein the face image group includes a main face image and at least one auxiliary face image; Based on the face image group and the basic face set, the initial modeling coefficient group is adjusted for multiple rounds of iterations, and based on the adjusted target modeling coefficient group and the basic face set, the target face three-dimensional result is determined; wherein the following operations are performed in each round of iterative adjustment process: Based on the three-dimensional face result determined according to the current modeling coefficient group and the basic face set, obtaining a first comparison result between the main face image and the associated main rendered image, and respectively obtaining a second comparison result between the at least one auxiliary face image and the associated auxiliary rendered image; Based on the obtained at least one second comparison result and the degree of deviation between each of the second comparison results and the first comparison result, configuring a reference evaluation result of a corresponding auxiliary facial image; Based on a first loss value determined according to the first comparison result and a second loss value determined according to the at least one second comparison result and the corresponding reference evaluation result, the modeling coefficient group is adjusted; wherein, when the basic face set is a 3DMM face set, the modeling coefficient group includes a face shape coefficient and a face texture coefficient.

2. The method according to claim 1, characterized in that After acquiring the face image group collected for the target object, and before performing multiple rounds of iterative adjustments on the initial modeling coefficient group based on the face image group and the basic face set, the method further includes: Using facial key point detection technology and facial segmentation technology, respectively for each facial image included in the facial image group, determine a facial key point set and a facial segmentation result; For a main face image and at least one corresponding auxiliary face image determined in each of the face images, a key point importance weight set corresponding to the face key point set and a corresponding pixel point importance weight map are configured according to the corresponding face segmentation results.

3. The method according to claim 2, characterized in that The configuring of a key point importance weight set corresponding to a face key point set and a corresponding pixel point importance weight map according to the corresponding face segmentation results respectively includes: According to the corresponding face segmentation results, the following operations are performed respectively: Determine the facial key points in each corresponding segmented area respectively, and generate a key point importance weight set based on the key point importance weights respectively configured for the facial key points in each segmented area; The importance weights of the pixels in each segmented area are configured, and a pixel importance weight map is generated based on the importance weights of the pixels in each segmented area.

4. The method according to any one of claims 1 to 3, characterized in that: The method of obtaining a first comparison result between the main face image and the associated main rendered image based on the human face 3D result determined according to the current modeling coefficient group and the basic face set, and obtaining a second comparison result between each of the at least one auxiliary human face image and the associated auxiliary rendered image, comprises: Determine a three-dimensional face result according to the current modeling coefficient group and the basic face set, and generate a main rendered image associated with the main face image and a secondary rendered image associated with the at least one secondary face image based on the three-dimensional face result and the currently acquired rendering coefficient group, wherein the rendering coefficient group includes each group of illumination coefficients and each group of posture coefficients; Determine respectively the main face image, the main rendered image, the at least one auxiliary face image, and the at least one auxiliary rendered image, and the corresponding facial key point sets and face segmentation results; A corresponding first comparison result is determined based on the pixel value differences and key point position differences between the main rendered image and the main facial image, and a corresponding second comparison result is determined based on the pixel value differences and key point position differences between the at least one auxiliary rendered image and the corresponding auxiliary facial image.

5. The method according to claim 4, characterized in that The determining a corresponding first comparison result according to a difference in pixel values ​​and a difference in key point positions between the main rendered image and the main face image comprises: Determine the pixel value difference between each pixel point with the same relative position in the main face image and the main rendering image, and determine the key point position difference between each face key point with the same label; According to the differences in each pixel value and each key point position, a corresponding first comparison result is determined.

6. The method according to claim 4, characterized in that The configuring a reference evaluation result of a corresponding auxiliary face image based on the obtained at least one second comparison result and the degree of deviation between each of the second comparison results and the first comparison result comprises: For at least one second comparison result, perform the following operations respectively: Determine, according to a first face segmentation result corresponding to the first comparison result and a second face segmentation result corresponding to a second comparison result, groups of segmented regions that match each other, and configure, for each group of segmented regions, reference evaluation results of pixel values ​​of each pixel point in the corresponding segmented regions according to first offset degrees between corresponding pixel value difference means; According to the facial key point sets corresponding to the second comparison result and the first comparison result, each group of facial key point subsets that matches within the each group of segmentation areas is determined respectively; and for each group of facial key point subsets, according to the second offset degree between the corresponding key point position difference means, a key point reference evaluation result of the facial key points within the corresponding segmentation area is configured.

7. The method according to claim 6, characterized in that Determining a first loss value according to the first comparison result includes: Determine pixel importance weights respectively configured for pixels in each segmented area in the master face image, wherein the segmented area is determined by a corresponding face segmentation result; Calculating a first pixel value loss based on each pixel value difference included in the first comparison result and a pixel point importance weight configured corresponding to each pixel point; Calculating a first key point loss based on the position differences of each key point included in the first comparison result and the key point importance weights corresponding to each key point configuration; A corresponding first loss value is determined according to the first pixel value loss and the first key point loss.

8. The method according to claim 6, characterized in that When determining the second loss value according to the at least one second comparison result and the corresponding reference evaluation result respectively, it includes: Obtaining pixel importance weights respectively configured for pixels in each segmented area in the at least one auxiliary face image, wherein the segmented area is determined by a corresponding face segmentation result; Calculating a second pixel value loss based on each pixel value difference included in the at least one second comparison result, and a pixel point importance weight configured corresponding to each pixel point and a pixel value reference evaluation result; Calculating a second key point loss based on the position differences of each key point included in the at least one second comparison result, and the key point importance weights configured corresponding to each face key point and the key point reference evaluation results; A corresponding second loss value is determined based on each second pixel value loss and each second key point loss.

9. A device for generating a three-dimensional result of a human face, characterized in that: include: An acquisition unit, configured to acquire a face image group associated with a target object, wherein the face image group includes a main face image and at least one auxiliary face image; The adjustment unit is used to perform multiple rounds of iterative adjustments on the initial modeling coefficient group based on the face image group and the basic face set, and determine the target face three-dimensional result based on the adjusted target modeling coefficient group and the basic face set; wherein the following operations are performed in each round of iterative adjustment process: Based on the three-dimensional face result determined according to the current modeling coefficient group and the basic face set, obtaining a first comparison result between the main face image and the associated main rendered image, and respectively obtaining a second comparison result between the at least one auxiliary face image and the associated auxiliary rendered image; Based on the obtained at least one second comparison result and the degree of deviation between each of the second comparison results and the first comparison result, configuring a reference evaluation result of a corresponding auxiliary facial image; Based on a first loss value determined according to the first comparison result and a second loss value determined according to the at least one second comparison result and the corresponding reference evaluation result, the modeling coefficient group is adjusted; wherein, when the basic face set is a 3DMM face set, the modeling coefficient group includes a face shape coefficient and a face texture coefficient.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method for generating a three-dimensional face result as described in any one of claims 1 to 8 is implemented.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for generating a three-dimensional face result as described in any one of claims 1 to 8 is implemented.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for generating a three-dimensional face result as described in any one of claims 1 to 8 is implemented.