Method, device and equipment for reconstructing three-dimensional face model and storage medium

Abstract

The application provides a three-dimensional face model reconstruction method, device and equipment and a storage medium. The method comprises the following steps: obtaining N face images of the same user, and obtaining a first face image from the N face images; performing first joint optimization on preset face shape parameters, first parameters corresponding to the first face image, and second pose parameters corresponding to N-1 second face images; and reconstructing a three-dimensional face model of the user according to the face shape parameters after the first joint optimization. That is, when reconstructing the three-dimensional face model, the first face image of the user is used as a main image, and other face images are used as auxiliary images, so that the information conflict between multiple images is reduced, the face shape parameters, the first parameters corresponding to the first face image, and the second pose parameters corresponding to the N-1 second face images are subjected to first joint optimization, the coupling degree of the face shape parameters and the expression parameters is reduced, and the reconstruction accuracy of the three-dimensional face model is improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of image processing, and particularly relate to a three-dimensional face model reconstruction method, device, equipment and storage medium. BACKGROUND

[0002] Face three-dimensional model reconstruction can be understood as establishing a three-dimensional model of a face, which has an additional dimension relative to a two-dimensional face image and is widely used in the fields of movies and games. For example, a three-dimensional face model of a user is reconstructed through multiple face images of the user.

[0003] However, due to the differences between different face images, such as different expressions, different poses, and even different face shapes at different times, there is conflicting information between these face images. In addition, pose coefficients, 3DMM shape coefficients and 3DMM expression coefficients are severely coupled, which makes the optimization of the above parameters insufficient, and further leads to inaccurate three-dimensional face model reconstruction. SUMMARY

[0004] The present application provides a three-dimensional face model reconstruction method, device, equipment and storage medium, which improves the accuracy of three-dimensional face model reconstruction.

[0005] In a first aspect, the present application provides a three-dimensional face model reconstruction method, comprising:

[0006] Obtaining N face images of the same user, and obtaining a first face image from the N face images, wherein N is a positive integer greater than 1;

[0007] Performing first joint optimization on a preset face shape parameter, a first parameter corresponding to the first face image, and second pose parameters corresponding to N-1 second face images, wherein the N-1 second face images are other face images in the N face images except the first face image;

[0008] Reconstructing a three-dimensional face model of the user according to the face shape parameter after the first joint optimization.

[0009] In a second aspect, a model training device is provided, comprising:

[0010] An obtaining unit, configured to obtain N face images of the same user, and obtain a first face image from the N face images, wherein N is a positive integer greater than 1;

[0011] an optimization unit, configured to perform first joint optimization on preset face shape parameters, first parameters corresponding to the first face image, and second pose parameters corresponding to N-1 second face images, the N-1 second face images being other face images in the N face images except the first face image;

[0012] a reconstruction unit, configured to reconstruct a three-dimensional face model of the user according to the face shape parameters after the first joint optimization.

[0013] In a third aspect, a computing device is provided, including a processor and a memory. The memory is configured to store a computer program, and the processor is configured to invoke and run the computer program stored in the memory to execute the method in the first aspect or any implementation manner thereof.

[0014] In a fourth aspect, a chip is provided for implementing the method in any one of the first aspect to the second aspect or any implementation manner thereof. Specifically, the chip includes a processor configured to invoke and run a computer program from a memory, so that a device installed with the chip executes the method in any one of the first aspect or any implementation manner thereof.

[0015] In a fifth aspect, a computer readable storage medium is provided for storing a computer program, the computer program causing a computer to execute the method in any one of the first aspect or any implementation manner thereof.

[0016] In a sixth aspect, a computer program product is provided, including computer program instructions, the computer program instructions causing a computer to execute the method in any one of the first aspect or any implementation manner thereof.

[0017] In a seventh aspect, a computer program is provided, when running on a computer, causing the computer to execute the method in any one of the first aspect or any implementation manner thereof.

[0018] In summary, the present application obtains N face images of the same user, and obtains a first face image from the N face images; performs first joint optimization on preset face shape parameters, first parameters corresponding to the first face image, and second pose parameters corresponding to N-1 second face images; and reconstructs a three-dimensional face model of the user according to the face shape parameters after the first joint optimization. That is, the present application takes the first face image of the user as a main image and other face images as auxiliary images when reconstructing the three-dimensional face model, reduces information conflicts between the multiple images, performs first joint optimization on the face shape parameters, the first parameters corresponding to the first face image, and the second pose parameters corresponding to the N-1 second face images, reduces the coupling degree of the face shape parameters and the expression parameters, accurately optimizes the face shape parameters, and improves the reconstruction accuracy of the three-dimensional face model when reconstructing the three-dimensional face model based on the accurately optimized face shape parameters. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0020] Figure 1 A face reference point diagram related to the embodiments of the present application;

[0021] Figure 2 An application scenario diagram related to the embodiments of the present application;

[0022] Figure 3 A flowchart of the three-dimensional face model reconstruction method provided by an embodiment of the present application;

[0023] Figure 4 A flowchart of the three-dimensional face model reconstruction method provided by an embodiment of the present application;

[0024] Figure 5 A schematic block diagram of a three-dimensional face model reconstruction device provided by an embodiment of the present application;

[0025] Figure 6 A schematic block diagram of a computing device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0026] The technical solutions in the embodiments of the present application will be described below with reference to the drawings.

[0027] It should be understood that, in the embodiments of the present application, “B corresponding to A” means that B is associated with A. In an implementation, B can be determined according to A. However, it should also be understood that determining B according to A does not mean that B is determined only according to A, but B can also be determined according to A and / or other information.

[0028] In the description of the present application, “a plurality of” means two or more, unless otherwise specified.

[0029] In addition, in order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the same items or similar items with basically the same functions and effects are distinguished by using “first”, “second”, etc. The skilled in the art can understand that “first”, “second”, etc. do not limit the quantity and execution order, and “first”, “second”, etc. also do not necessarily mean different.

[0030] In order to facilitate the understanding of the embodiments of the present application, first, the related concepts involved in the embodiments of the present application are briefly introduced as follows:

[0031] Artificial intelligence (AI) is to use digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which tries to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0032] Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, both hardware and software technologies. Artificial intelligence basic technologies generally include technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc.

[0033] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithmic complexity theory, etc. It is a specialized study of how computers simulate or implement human learning behavior to acquire new knowledge or skills, reorganize existing knowledge structure, and continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental approach to making computers intelligent. Its applications are widespread in various fields of artificial intelligence. Machine learning and deep learning generally include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and rule-based learning.

[0034] Face three-dimensional reconstruction can be understood as establishing a three-dimensional model of the face. It has an additional dimension compared to two-dimensional face images and is widely used in movies, games, etc. Currently, there are three main methods to obtain a three-dimensional face model: software modeling, instrument acquisition, and image-based modeling.

[0035] Three-dimensional deformable face model (3D Morphable models, 3DMM) is a general three-dimensional face model that represents a face with a fixed number of points. Its core idea is that a face can be one-to-one matched in three-dimensional space and can be linearly added by many face orthogonal bases.

[0036] Each three-dimensional face can be represented in a basis vector space composed of all faces in a database. Solving the model of any three-dimensional face is actually equivalent to solving the coefficient problem of each basis vector.

[0037] The basic attributes of a face include shape and texture, and each face can be represented as a linear superposition of shape and texture vectors.

[0038] Shape Vector: S = (X1, Y1, Z1, X2, Y2, Z2,..., Yn, Zn),

[0039] Texture Vector: T = (R1, G1, B1, R2, G2, B2,..., Rn, Bn),

[0040] where n is the number of face samples in the dataset, Xi, Yi, Zi are the coordinates of the shape vector of the ith face sample in the dataset, and Ri, Gi, Bi are the coordinates of the texture vector of the ith face sample in the dataset.

[0041] Any face model can be combined by weighting m face models in the dataset as follows:

[0042]

[0043]

[0044] where S model is a three-dimensional face shape model, a i is the target value of the face shape parameter, i = 1…m, m is the number of face samples in the dataset, S i is the shape vector of the i-th face sample in the dataset, S is the mean of the shape vectors of all face samples in the dataset, Tmodel is a three-dimensional face texture model, b i is the target value of the face texture parameter, T i is the texture vector of the i-th face sample in the dataset, T is the mean of the texture vectors of all face samples in the dataset.

[0045] Optimization: Given a function f: A → R, find an element x0∈ A such that for all x in A, f(x 0 ) ≤ f(x) (minimization); or f(x0) ≥ f(x) (maximization). Such problems are sometimes also called "mathematical programming" (e.g., linear programming). Many real-world and theoretical problems can be modeled as such a general framework. Elements of A are called feasible solutions. The function f is called the objective function, or criterion. A feasible solution that minimizes (or maximizes) the objective function is called an optimal solution.

[0046] 2D face key point detection: The purpose is to automatically locate a set of pre-defined face fiducial points (such as eye corner points, mouth corner points), as shown in Figure 1 .

[0047] Figure 2 Fig. 1 is a schematic diagram of an application scenario related to an embodiment of the present application, comprising a collection device 101, a computing device 102, and a display device 103.

[0048] The collection device 101 is configured to collect N face images of a user. The N face images can be face images of the user at different times, in different scenes, under different light, etc.

[0049] The computing device 102 is configured to reconstruct a three-dimensional face model according to the N face images collected by the collection device 101, by using a three-dimensional face model reconstruction method provided in an embodiment of the present application.

[0050] The display device 103 is configured to display the three-dimensional face model reconstructed by the computing device 102.

[0051] Optionally, the computing device 102 can be a user device, such as a mobile phone, a tablet computer, a notebook computer, a palm computer, a mobile internet device (MID), or other terminal device with a browser installed.

[0052] Optionally, the technical device described above can be a server. The server can be one or more. When the server is more than one, there are at least two servers for providing different services, and / or there are at least two servers for providing the same service, such as providing the same service in a load balancing manner, and the embodiments of the present application do not limit this. The server can be provided with a reconstruction model, and the server supports the training and application process of the reconstruction model. Among them, the server described above can be a stand-alone physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms. The server can also become a node of the blockchain.

[0053] In some embodiments, when the computing device 102 has a display function, the display device 103 described above can be a display in the computing device 102.

[0054] In some embodiments, the display device 103 and the computing device 102 are different devices, and the display device 103 is connected to the computing device 102 through a network. The network can be an intranet, the Internet, a global system for mobile communication (GSM), wideband code division multiple access (WCDMA), a 4G network, a 5G network, Bluetooth, Wi-Fi, a call network, and other wireless or wired networks.

[0055] The application scenarios of the present application include but are not limited to the face reconstruction of characters in 3D games / 3D film works.

[0056] In some embodiments, the scheme of reconstructing the 3D face model in the face image by using the optimization method first projects the original image onto the image by using the initial 3DMM coefficient, the illumination coefficient, and the pose coefficient, then calculates the gradient information according to the difference (loss function) between the projected image and the original image, adjusts the 3DMM coefficient, the pose coefficient, and the illumination coefficient by using the gradient information, so as to be more accurate next time, and stops until the result converges. The difference between the projected image and the original image generally reflects the difference in color information between the two images, and the difference between the key points detected in the original image and the projection results of the key points in the projected image. Generally, the multi-RGB face image optimization scheme considers multiple images and optimizes a 3DMM shape coefficient (shared by multiple face images), multiple 3DMM expression coefficients, 3DMM texture coefficients, illumination coefficients, and pose coefficients. Finally, the 3D face model is obtained by combining the 3DMM shape coefficient with the 3DMM model.

[0057] However, due to the differences between different face images, such as different expressions, different poses, and even different face shapes at different times, there is conflicting information between these face images. In addition, the pose coefficient, the 3DMM shape coefficient, and the 3DMM expression coefficient are severely coupled, which makes the optimization of the above parameters insufficient, and further leads to inaccurate reconstruction of the three-dimensional face model.

[0058] To solve the above technical problems, in the reconstruction of the three-dimensional face model, the first face image of the user is taken as the main image, and the other face images are taken as auxiliary images, so as to reduce the information conflict between multiple images. The face shape parameter, the first parameter corresponding to the first face image, and the second pose parameter corresponding to the N-1 second face images are subjected to first joint optimization, the coupling degree of the face shape parameter and the expression parameter is reduced, the face shape parameter is accurately optimized, and the reconstruction accuracy of the three-dimensional face model is improved when the three-dimensional face model is reconstructed based on the accurately optimized face shape parameter.

[0059] The technical solutions of the embodiments of the present application will be described in detail in the following embodiments. The following embodiments can be combined with each other, and the same or similar concepts or processes can not be described in detail in some embodiments.

[0060] Figure 3 The flowchart of the three-dimensional face model reconstruction method provided by an embodiment of the present application is shown.

[0061] The execution subject of the embodiments of the present application is a device with a face model reconstruction function, for example, a face model reconstruction device. The face model reconstruction device can be a computing device or a part of a computing device. The following will be described by taking the computing device as an example.

[0062] As Figure 3 shown, the method of the embodiments of the present application comprises:

[0063] S301, acquiring N face images of the same user, and acquiring a first face image from the N face images.

[0064] The N face images can be face images of different poses, different expressions, etc. taken by the user at different times, different places, and different lighting conditions. That is, the embodiments of the present application do not limit the N face images, which can be any N face images of the user. That is, the method of the present application can reconstruct a three-dimensional face model of the user from any N face images of the user.

[0065] In reconstructing the three-dimensional face model, at least one of the parameters involved includes a face shape parameter, an expression parameter, a texture parameter, a lighting parameter, and a pose parameter.

[0066] Since the N face images are face images of the same user, the face shape is a parameter that remains basically unchanged, and therefore the N face images correspond to one face shape parameter S, and each of the N face images corresponds to one expression parameter E, one texture parameter T, one lighting parameter L, and one pose parameter P.

[0067] The above parameters are coupled, for example, there is a coupling between the face shape parameter S and the pose parameter P, that is, there is an influence between the face shape coefficient S and the pose parameter P, for example, if the face is raised, the face raising information can be reflected in the face shape parameter S or the face pose parameter S. If other information remains unchanged, and the face raising information is more reflected in the pose information, the pose parameter P becomes larger, and the face shape coefficient S becomes smaller, and if the face raising information is more reflected in the face shape information, the face shape coefficient S becomes larger, and the pose parameter P becomes smaller.

[0068] When the above parameters are coupled, it will cause the problem of inaccurate face model reconstruction. To solve the technical problem, the embodiments of the present application first select a first face image from the N face images as a main image, and other images in the N face images as auxiliary images, and reconstruct a three-dimensional face model.

[0069] In one example, the first face image is any front face image in the N face images.

[0070] In one example, the first face image is a front face image with the smallest expression or no expression in the N face images.

[0071] S302, first joint optimization is performed on the preset face shape parameter, the first parameter corresponding to the first face image, and the pose parameters corresponding to the N-1 second face images.

[0072] The N-1 second face images are other face images in the N face images except the first face image.

[0073] The application first initializes the following parameters:

[0074] A face shape parameter S is initialized.

[0075] For the first face image, a first parameter corresponding to the first face image is initialized. Optionally, the first parameter includes at least one of a first texture parameter T1, a first pose parameter P1, and a first lighting coefficient L1 corresponding to the first face image.

[0076] For each of the N-1 second face images, a second parameter for each second face image is initialized, and the second parameter includes a second pose parameter P2.

[0077] Optionally, the second parameter of the second face image further includes at least one of a second texture parameter T2, a second lighting coefficient L2, and an expression parameter E2 corresponding to the second face image.

[0078] Optionally, the initial values of the parameters are empirical values.

[0079] The initialized face shape parameter S, the first parameter corresponding to the first face image, and the second pose parameters P2 corresponding to the N-1 second face images are jointly optimized, that is, in the parameter updating process, only the face shape parameter S, the first parameter, and the second pose parameters P2 corresponding to the second face images are updated to realize face model reconstruction with the first face image as the main and other face images as the auxiliary, effectively reduce the information conflict between multiple face images, and realize the decoupling of the face shape parameter S and the pose parameter P, so as to realize accurate optimization of the face shape parameter S. When the face three-dimensional model is reconstructed based on the accurately optimized face shape parameter S, the reconstruction accuracy of the face model can be improved.

[0080] The first joint optimization of the face shape parameter, the first parameter corresponding to the first face image, and the pose parameters corresponding to the N-1 second face images in S302 includes but is not limited to the following methods:

[0081] In a first mode, the face shape parameter, the first parameter, and the pose parameters corresponding to the N-1 second face images are input into the neural network model, and the reconstruction values of the N face images output by the neural network are obtained. The face shape parameter, the first parameter, and the pose parameters corresponding to the N-1 second face images are updated according to the deviation between the reconstruction values of the N face images and the N face images, so as to realize the first joint optimization of the face shape parameter, the first parameter, and the pose parameters corresponding to the N-1 second face images.

[0082] In a second mode, the S302 includes the following steps:

[0083] S302-A1, determining the first projection image corresponding to each of the N face images according to the face shape parameter, the first parameter, and the second pose parameters corresponding to the N-1 second face images;

[0084] S302-A2, determining the first loss between the N face images and the N first projection images, and updating the face shape parameter, the first parameter, and the N-1 second pose parameters according to the first loss;

[0085] S302-A3, taking the updated face shape parameter, first parameter, and second pose parameter as new face shape parameter, first parameter, and second pose parameter respectively, and repeating the above steps to update the face shape parameter, first parameter, and second pose parameter multiple times.

[0086] In the second implementation mode, the first parameter (such as at least one of the first texture parameter T1, the first pose parameter P1, and the first light coefficient L1) corresponding to the first face image is used to project the first face image, and the first projection image corresponding to the first face image is obtained. For each of the N second face images, the second parameter (such as at least one of the first pose parameter P2, the second texture parameter T2, the second light coefficient L2, and the expression parameter E2) corresponding to the second face image is used to project the second face image, and the first projection image corresponding to the second face image is obtained, and then the N first projection images are obtained.

[0087] Then, the first loss between the N face images and the N first projection images is determined, and the face shape parameter, the first parameter, and the N-1 second pose parameters are updated according to the first loss.

[0088] The updated face shape parameters, the first parameters, and the second pose parameters are taken as new face shape parameters, first parameters, and second pose parameters respectively, and the steps of S302-A1 to S302-A3 are returned to be executed, for example, n1 times of loop execution, to realize n1 times of updating of the face shape parameters, the first parameters, and N-1 second pose parameters respectively, to realize the first joint optimization of the face shape parameters, the first parameters, and the N-1 second pose parameters corresponding to the N-1 second face images respectively.

[0089] In some embodiments, the first parameters include first pose parameters corresponding to the first face image, and in the above-mentioned mode two, when the face shape parameters, the first parameters, and the second pose parameters are updated, the update step of the first pose parameters and the second pose parameters is greater than the update step of other parameters in the face shape parameters and the first parameters except the first pose parameters, to realize the rapid update of the pose parameters and effectively decouple the face shape parameters and the pose parameters.

[0090] The application does not limit the specific way of determining the first loss in S302-A2.

[0091] In an example, S302-A2 includes: comparing each pixel point in the first face image with each pixel point in the first projection image corresponding to the first face image one by one, and determining the deviation between the pixel value of each pixel point in the first face image and the pixel value of each pixel point in the first projection image as the first loss.

[0092] In another example, S302-A2 includes the steps of S302-A21 to S302-A23 as follows:

[0093] S302-A21, according to the deviation between the pixel value of each pixel point in the first face image and the pixel value of each pixel point in the projection image corresponding to the first face image, determining the first pixel loss between the first face image and the corresponding projection image.

[0094] For example, each pixel point in the first face image I is compared with each pixel point in the first projection image I' corresponding to the first face image I point by point, and the deviation between the pixel value of each pixel point in the first face image I and the pixel value of each pixel point in the first projection image I' is determined as the first pixel loss between the first face image and the corresponding projection image.

[0095] Optionally, the standard deviation between the pixel value of each pixel point in the first face image I and the pixel value of each pixel point in the first projection image I' is determined as the first pixel loss between the first face image and the corresponding projection image.

[0096] S302-A22, for each of the N face images, determine the projection coordinates of each key point in the corresponding projection image, and determine the first key point loss corresponding to the face image according to the deviation between the coordinates of each key point in the face image and the projection coordinates in the corresponding projection image.

[0097] For example, the face image I1 includes k key points, which can be mouth, eyes, nose, eyebrows, chin, etc. The projection points of the above k key points in the first projection image I'1 of the face image I1 are denoted as k', and the deviation between the coordinates of the above k key points in the face image I1 and the coordinates of the above k' key points in the projection image I'1 is determined, which is determined as the first key point loss corresponding to the face image I1.

[0098] Optionally, the standard deviation between the coordinates of the above k key points in the face image I1 and the coordinates of the above k' key points in the projection image I'1 is determined as the first key point loss corresponding to the face image I1.

[0099] S302-A23, determine the first loss according to the first pixel loss and the first key point loss corresponding to the N face images respectively.

[0100] For example, the sum (or weighted sum) of the first pixel loss and the first key point loss corresponding to the N face images respectively is determined as the first loss.

[0101] For another example, the sum of the first pixel loss and the first key point loss corresponding to the N face images respectively is divided by 2 to determine the first loss.

[0102] Optionally, the first loss can also be obtained by performing a preset arithmetic operation on the above first pixel loss and the first key point loss corresponding to the N face images respectively, and the embodiments of the present application do not limit this.

[0103] In the second mode, the first loss determined by the above steps is used to update the face shape parameters, the first parameters and the N-1 second pose parameters at each update. Since the first loss focuses on the pixel information of the first face image when calculated, the update of the above parameters is mainly affected by the pixel information of the first face image, and the first face image is a main image selected from the N face images, for example, a neutral face image, which has the most complete face shape information and is least affected by expression information and pose information. Therefore, when the first face image information is used to jointly optimize the face shape parameters, the first parameters corresponding to the first face image and the second pose parameters corresponding to the N-1 second face images, not only the face shape parameters can be accurately optimized, but also the face shape parameters and the pose parameters can be decoupled and optimized, so that the reconstruction accuracy of the face model can be improved when the three-dimensional face model is reconstructed based on the optimized face shape parameters.

[0104] S303, reconstructing a three-dimensional face model of the user according to the first joint-optimized face shape parameters.

[0105] The implementation modes of S303 include but are not limited to the following modes:

[0106] In the first mode, the first joint-optimized face shape parameters are input into a three-dimensional face neural network to obtain the three-dimensional face model of the user.

[0107] In the second mode, the three-dimensional face model of the user is reconstructed using 3DMM. At this time, S303 includes the following steps S303-A and S303-B:

[0108] S303-A, determining a target value of the face shape parameters according to the first joint-optimized face shape parameters.

[0109] The target value of the face shape parameters can be understood as the value of the face shape parameters after optimization.

[0110] S303-B, reconstructing a three-dimensional face model of the user according to the target value of the face shape parameters and a preset face three-dimensional morphable model.

[0111] In the second mode, the target value of the face shape parameters is determined according to the first joint-optimized face shape parameters, and the target value of the face shape parameters is ai, i = 1…m. Then, the three-dimensional face model of the user is reconstructed according to the target value of the face shape parameters and the preset face three-dimensional morphable model.

[0112] In an example, the three-dimensional face model of the user is obtained by the following formula (1):

[0113]

[0114] wherein Smodelis the three-dimensional face shape model of the user, aiis the target value of the face shape parameter, i = 1…m, m is the number of face samples in the data set, Siis the shape vector of the i-th face sample in the data set, and S is the mean of the shape vectors of all face samples in the data set.

[0115] In an example, the three-dimensional face model of the user can also be reconstructed by the 3DMM model according to the target value of the face shape parameter, and at least one of the target value of the texture parameter, the target value of the pose parameter, the target value of the expression parameter, and the target value in the illumination parameter.

[0116] The process of determining the target value of the face shape parameter according to the first joint-optimized face shape parameter in S303-A is described below.

[0117] In example 1, the first joint-optimized face shape parameter is determined as the target value of the face shape parameter.

[0118] In example 2, S303-A includes S303-A1:

[0119] S303-A1, determining the target value of the face shape parameter according to the first joint-optimized face shape parameter and the first parameter.

[0120] The implementation of S303-A1 includes but is not limited to the following:

[0121] In mode one, the first joint-optimized face shape parameter and the first parameter are jointly optimized, and the joint-optimized face shape parameter is determined as the target value of the face shape parameter.

[0122] For example, the first face image is projected using the first joint-optimized face shape parameter and the first parameter to obtain a projected image of the first face image. The loss between the first face image and the corresponding projected image is determined, and the face shape parameter and the first parameter are updated according to the loss. Then, the first face image is projected using the updated face shape parameter and the first parameter, the loss between the first face image and the projected image is determined, and the face shape parameter and the first parameter are updated according to the loss. The above steps are repeated to update the face shape parameter and the first parameter for multiple steps, and the final face shape parameter is obtained. The value corresponding to the final face shape parameter is determined as the target value of the face shape parameter.

[0123] Optionally, in this step, the loss between the first face image and the corresponding projected image can be determined by determining a first deviation 1 between each pixel point in the first face image and each pixel point in the projected image, and determining a second deviation 2 between the positions of each key point in the first face image and the projected positions of the key points in the projected image, and determining the loss between the first face image and the corresponding projected image according to the first deviation 1 and the second deviation 2, for example, determining the sum of the first deviation 1 and the second deviation 2 as the loss between the first face image and the corresponding projected image. Optionally, the first deviation 1 between each pixel point in the first face image and each pixel point in the projected image can be the standard deviation between each pixel point in the first face image and each pixel point in the projected image. Optionally, the second deviation 2 between the positions of each key point in the first face image and the projected positions of the key points in the projected image can be the standard deviation between the positions of each key point in the first face image and the projected positions of the key points in the projected image.

[0124] In the second mode, the S303-A1 includes the following steps of S303-A11 and S303-A12:

[0125] The S303-A11 performs second joint optimization on the second parameters corresponding to the N-1 second face images and the first joint-optimized face shape parameters.

[0126] The second parameters include the expression parameters corresponding to the second face images and the second joint-optimized pose parameters.

[0127] In actual applications, there is coupling between the expression parameters and the face shape parameters, for example, when the facial expression is smiling, the face shape information of the user will change, which makes the expression information affect the accuracy of the face shape information, and further causes the inaccuracy of the three-dimensional face model reconstruction based on the face shape parameters. In order to avoid this problem, it is necessary to decouple the expression parameters and the face shape parameters.

[0128] The step S303-A11 is to perform joint optimization on the second parameters corresponding to the second face images, for example, the expression parameters, and the face shape parameters based on the first joint-optimized parameters, so as to decouple the expression parameters and the face shape parameters, and further realize accurate optimization of the face shape parameters. Therefore, when the three-dimensional face model is reconstructed based on the accurately optimized face shape parameters, the accuracy of the reconstruction can be improved.

[0129] In some embodiments, the S303-A11 includes the following steps of S303-A111 to S303-A113:

[0130] S303-A111, determine N-1 second projection images corresponding to the N-1 second facial images according to the second parameters corresponding to the N-1 second facial images respectively and the first joint-optimized face shape parameters.

[0131] For example, project each of the N-1 second facial images using the second parameters corresponding to the N-1 second facial images respectively and the first joint-optimized face shape parameters to obtain a projection image of each of the N-1 second facial images, and record the projection image corresponding to the second facial image as a second projection image, thereby obtaining N-1 second projection images.

[0132] S303-A112, determine a second loss between the N-1 second facial images and the N-1 second projection images, and update the second parameters corresponding to the N-1 second facial images respectively and the first joint-optimized face shape parameters according to the second loss.

[0133] In this step, the manner of determining the second loss between the N-1 second facial images and the N-1 second projection images includes but is not limited to the following examples:

[0134] Example one, for each of the N-1 second facial images, determine a second pixel loss corresponding to the second facial image according to the deviation between the pixel value of each pixel point in the second facial image and the pixel value of each pixel point in the corresponding projection image. Determine the sum of the second pixel losses corresponding to the N-1 second facial images as the second loss between the N-1 second facial images and the N-1 second projection images.

[0135] Example two, for each of the N-1 second facial images, determine the projection coordinates of each key point in the corresponding projection image in the second facial image, and determine a second key point loss corresponding to the second facial image according to the deviation between the coordinates of each key point in the second facial image and the projection coordinates in the corresponding projection image. Determine the sum of the second key point losses corresponding to the N-1 second facial images as the second loss between the N-1 second facial images and the N-1 second projection images.

[0136] In Example Three, for each of the N-1 second face images, a second pixel loss corresponding to the second face image is determined according to a deviation between pixel values of each pixel point in the second face image and pixel values of each pixel point in the corresponding projection image. In addition, projection coordinates of each key point in the second face image in the corresponding projection image are determined, and a second key point loss corresponding to the second face image is determined according to a deviation between the coordinates of each key point in the second face image and the projection coordinates of each key point in the corresponding projection image. A sum of the second pixel loss and the second key point loss corresponding to the N-1 second face images is determined as the second loss.

[0137] Optionally, the deviation between the pixel values of each pixel point in the second face image and the pixel values of each pixel point in the corresponding projection image can be a standard deviation between the pixel values of each pixel point in the second face image and the pixel values of each pixel point in the corresponding projection image.

[0138] Optionally, the deviation between the coordinates of each key point in the second face image and the projection coordinates of each key point in the corresponding projection image can be a standard deviation between the coordinates of each key point in the second face image and the projection coordinates of each key point in the corresponding projection image.

[0139] In S303-A113, the updated second parameters and the face shape parameters are returned as new second parameters and the face shape parameters after the first joint optimization, and the step of S303-A111 is executed to cyclically update the face shape parameters and the second parameters for multiple times.

[0140] In some embodiments, the update step of the expression parameters and the face shape parameters is smaller than the update step of other parameters in the second parameters except the expression parameters. In this way, not only the expression parameters and the face shape parameters are sufficiently optimized and decoupled, but also the expression parameters and the face shape parameters are decoupled from the pose parameters.

[0141] In this implementation, the expression parameters corresponding to the N-1 second face images and the face shape parameters after the first joint optimization are updated by the second loss corresponding to the N-1 second face images, which can realize the decoupling of the expression parameters and the face shape parameters, and sufficiently optimize the face shape parameters. In this way, when a three-dimensional face model is reconstructed based on the face shape parameters, the accuracy of the reconstruction can be improved.

[0142] It should be noted that the manner of performing the second joint optimization on the N-1 second parameters respectively corresponding to the N-1 second face images and the face shape parameters after the first joint optimization in the above S303-A11 includes but is not limited to the manners of the above S303-A111 to S303-A113. For example, the N-1 second parameters respectively corresponding to the N-1 second face images and the first parameters after the first joint optimization can also be used to project the corresponding face images in the N face images to obtain N projection images of the N face images, and the N-1 second parameters respectively corresponding to the N-1 second face images and the face shape parameters after the first joint optimization are secondly joint optimized according to the loss between the N projection images and the N face images. Optionally, the manner of calculating the loss is basically the same as the manner of calculating the second loss.

[0143] S303-A12, determining a target value of the face shape parameters according to the face shape parameters after the second joint optimization.

[0144] For example, in some embodiments, the value of the face shape parameters after the second joint optimization can be determined as the target value of the face shape parameters.

[0145] For another example, in some embodiments, the face shape parameters after the second joint optimization are further joint optimized with other parameters. Optionally, the target value of the face shape parameters is determined by the steps of the following S303-A121 and S303-A122 in the above S303-A12:

[0146] S303-A121, performing a third joint optimization on the face shape parameters after the second joint optimization, the N-1 second parameters, and the first parameters after the first joint optimization.

[0147] In an example, the third joint optimization in the above S303-A121 is implemented by the steps of the following S303-A1211 to S303-A1213:

[0148] S303-A1211, determining N third projection images of the N face images according to the first parameters after the first joint optimization and the face shape parameters after the second joint optimization and the N-1 second parameters;

[0149] S303-A1212, determining a third loss between the N face images and the N third projection images, and updating the first parameters after the first joint optimization and the face shape parameters after the second joint optimization and the N-1 second parameters according to the third loss;

[0150] S303-A1213, repeat the execution of S303-A1211 to update the face shape parameters, the first parameters and the second parameters for multiple times, by taking the updated face shape parameters and the second parameters as new second joint-optimized face shape parameters and second parameters, and taking the updated first parameters as new first joint-optimized first parameters.

[0151] Specifically, the first face image is projected according to the second joint-optimized face shape parameters and the first joint-optimized first parameters to obtain a third projected image of the first face image, and for each of the N-1 second face images, the second joint-optimized face shape parameters and the second joint-optimized second parameters corresponding to the second face image are used to project the second face image to obtain a third projected image of the second face image. In this way, N third face images can be obtained for the N face images.

[0152] A third loss between the N face images and the N third face images is determined. For example, for each of the N face images, a third pixel loss corresponding to the face image is determined according to the deviation between the pixel value of each pixel point in the face image and the pixel value of each pixel point in the corresponding projected image, and the sum of the third pixel losses corresponding to the N face images is determined as the third loss. For example, for each of the N face images, the projection coordinates of each key point in the face image in the corresponding projected image are determined, and a third key point loss corresponding to the face image is determined according to the deviation between the coordinates of each key point in the face image and the projection coordinates of each key point in the corresponding projected image, and the sum of the third key point losses corresponding to the N face images is determined as the third loss. For example, for each of the N face images, a third pixel loss corresponding to the face image is determined according to the deviation between the pixel value of each pixel point in the face image and the pixel value of each pixel point in the corresponding projected image; the projection coordinates of each key point in the face image in the corresponding projected image are determined, and a third key point loss corresponding to the face image is determined according to the deviation between the coordinates of each key point in the face image and the projection coordinates of each key point in the corresponding projected image; and the sum of the third pixel losses and the third key point losses corresponding to the N face images is determined as the third loss.

[0153] According to the third loss determined above, the first parameter after the first joint optimization, and the face shape parameter and the N-1 second parameters after the second joint optimization are updated. Optionally, the first parameter after the first joint optimization, and the face shape parameter and the N-1 second parameters after the second joint optimization can be updated by using a gradient update method. In some embodiments, the first parameter after the first joint optimization, and the face shape parameter and the N-1 second parameters after the second joint optimization can also be updated by using an optimizer F according to the third loss, and the optimizer F can be an adaptive moment estimation (Adam) optimizer.

[0154] It is determined whether the number of iterations reaches a preset value. If the number of iterations does not reach the preset value, N third projection images of the N face images are determined by using the first parameter, the face shape parameter and the N-1 second parameters updated above, and the above steps are repeated to update the face shape parameter, the first parameter and the second parameter multiple times.

[0155] In this example, the optimization accuracy of the face shape parameter is further improved by integrating and optimizing all the face images.

[0156] It should be noted that the manner of third joint optimization of the face shape parameter and the N-1 second parameters after the second joint optimization, and the first parameter after the first joint optimization in S303-A121 above includes but is not limited to the manners of S303-A1211 to S303-A1213 above. For example, the face shape parameter and the N-1 second parameters after the second joint optimization, and the first parameter after the first joint optimization can also be used to project part of the second face images in the first face image and the N-1 second face images, and the face shape parameter and the N-1 second parameters after the second joint optimization, and the first parameter after the first joint optimization are third joint optimized according to the loss between the projection image of the first face image and the first face image, and the loss between the part of the second face images and the corresponding projection images.

[0157] According to the above method, after the face shape parameter and the N-1 second parameters after the second joint optimization, and the first parameter after the first joint optimization are third joint optimized, S303-A122 is executed to determine a target value of the face shape parameter.

[0158] S303-A122, a target value of the face shape parameter is determined according to the face shape parameter after the third joint optimization.

[0159] For example, the face shape parameter after the third joint optimization is determined as the target value of the face shape parameter.

[0160] For example, the face shape parameters after the third joint optimization are further optimized, and the first face image is optionally used to further optimize the face shape parameters after the third joint optimization. For example, S303-A122 can determine the target value of the face shape parameters by the following steps of S303-A1221 and S303-A1222:

[0161] S303-A1221, the third joint optimization is performed on the face shape parameters after the third joint optimization and the first parameters;

[0162] S303-A1222, the face shape parameters after the fourth joint optimization are determined as the target value of the face shape parameters.

[0163] For example, S303-A1221 includes the following steps of S303-A12211 to S303-A12213:

[0164] S303-A12211, the first face image is projected according to the face shape parameters after the third joint optimization and the first parameters to determine a fourth projection image corresponding to the first face image;

[0165] S303-A12212, a fourth loss between the first face image and the fourth projection image is determined, and the face shape parameters after the third joint optimization and the first parameters are updated according to the fourth loss.

[0166] For example, the face shape parameters and the first parameters are updated by the optimizer F according to the fourth loss.

[0167] The way of determining the fourth loss in this step includes but is not limited to the following examples:

[0168] Example 1, a fourth pixel loss between the first face image and the fourth projection image is determined according to the deviation between the pixel value of each pixel point in the first face image and the pixel value of the corresponding pixel point in the fourth projection image, and the fourth pixel loss is determined as the fourth loss.

[0169] Example 2, the projection coordinates of each key point in the first face image in the fourth projection image are determined, and a fourth key point loss corresponding to the first face image is determined according to the deviation between the coordinates of each key point in the first face image and the projection coordinates in the fourth projection image, and the fourth key point loss is determined as the fourth loss.

[0170] In Example 3, a fourth pixel loss between the first face image and the fourth projection image is determined according to a deviation between a pixel value of each pixel point in the first face image and a pixel value of a corresponding pixel point in the fourth projection image; projection coordinates of each key point in the first face image in the fourth projection image are determined, and a fourth key point loss corresponding to the first face image is determined according to a deviation between the coordinates of each key point in the first face image and the projection coordinates of each key point in the fourth projection image; and a sum of the fourth pixel loss and the fourth key point loss corresponding to the first face image is determined as the fourth loss.

[0171] In S303-A12213, the updated face shape parameter and the first parameter are taken as new third joint-optimized face shape parameter and first parameter, and the above S303-A12211 is repeatedly executed to perform multiple loop updates on the face shape parameter and the first parameter.

[0172] After the face shape parameter and the first parameter are updated multiple times according to the above steps, the fourth joint optimization is performed on the third joint-optimized face shape parameter and the first parameter, the fourth joint-optimized face shape parameter is determined as the target value of the face shape parameter, and finally the three-dimensional face model of the user is reconstructed according to the target value of the face shape parameter and the preset face three-dimensional deformation statistical model, which is specifically described in the above S303-B.

[0173] The three-dimensional face model reconstruction method provided by the embodiment of the present application comprises the following steps: obtaining N face images of the same user, N being a positive integer greater than 1; obtaining a first face image from the N face images; performing first joint optimization on a preset face shape parameter, a first parameter corresponding to the first face image, and second pose parameters respectively corresponding to N-1 second face images, the N-1 second face images being other face images in the N face images except the first face image; and reconstructing a three-dimensional face model of the user according to the first joint-optimized face shape parameter. That is, the first face image of the user is taken as a main image and other face images are taken as auxiliary images when the three-dimensional face model is reconstructed, so as to reduce information conflicts between multiple images, and the first joint optimization is performed on the face shape parameter, the first parameter corresponding to the first face image, and the second pose parameters respectively corresponding to the N-1 second face images, so as to reduce the coupling degree of the face shape parameter and the expression parameter, to realize accurate optimization of the face shape parameter, and to improve the reconstruction accuracy of the three-dimensional face model when the three-dimensional face model is reconstructed based on the accurately optimized face shape parameter.

[0174] Figure 4 The three-dimensional face model reconstruction method provided by the embodiment of the present application comprises the following steps: obtaining N face images of the same user, N being a positive integer greater than 1; obtaining a first face image from the N face images; performing first joint optimization on a preset face shape parameter, a first parameter corresponding to the first face image, and second pose parameters respectively corresponding to N-1 second face images, the N-1 second face images being other face images in the N face images except the first face image; and reconstructing a three-dimensional face model of the user according to the first joint-optimized face shape parameter. That is, the first face image of the user is taken as a main image and other face images are taken as auxiliary images when the three-dimensional face model is reconstructed, so as to reduce information conflicts between multiple images, and the first joint optimization is performed on the face shape parameter, the first parameter corresponding to the first face image, and the second pose parameters respectively corresponding to the N-1 second face images, so as to reduce the coupling degree of the face shape parameter and the expression parameter, to realize accurate optimization of the face shape parameter, and to improve the reconstruction accuracy of the three-dimensional face model when the three-dimensional face model is reconstructed based on the accurately optimized face shape parameter. Figure 4 As shown in FIG. 1, the three-dimensional face model reconstruction method comprises the following steps:

[0175] S401, acquire N face images of the same user, and acquire a first face image from the N face images.

[0176] Optionally, the first face image is a face image with the least expression among the N face images.

[0177] For details, refer to the description of S301 above, which will not be repeated here.

[0178] S402, perform first joint optimization on the preset face shape parameters, the first parameters corresponding to the first face image, and the second pose parameters corresponding to the N-1 second face images.

[0179] For example, according to the face shape parameters, the first parameters, and the second pose parameters corresponding to the N-1 second face images, determine the first projection image corresponding to each of the N face images; determine the first loss between the N face images and the N first projection images, and update the face shape parameters, the first parameters, and the N-1 second pose parameters according to the first loss; take the updated face shape parameters, the first parameters, and the second pose parameters as new face shape parameters, first parameters, and second pose parameters, respectively, and repeat the above steps to perform multiple loop updates on the face shape parameters, the first parameters, and the second pose parameters.

[0180] For details, refer to the related description of S302 above, which will not be repeated here.

[0181] S403, perform second joint optimization on the second parameters corresponding to the N-1 second face images and the face shape parameters after the first joint optimization, the second parameters including the expression parameters corresponding to the second face images and the second pose parameters after the first joint optimization.

[0182] For example, according to the second parameters corresponding to the N-1 second face images and the face shape parameters after the first joint optimization, determine N-1 second projection images corresponding to the N-1 second face images; determine the second loss between the N-1 second face images and the N-1 second projection images, and update the second parameters corresponding to the N-1 second face images and the face shape parameters after the first joint optimization according to the second loss; take the updated second parameters and face shape parameters as new second parameters and face shape parameters after the first joint optimization, and repeat the above steps to perform multiple loop updates on the face shape parameters and the second parameters.

[0183] For details, refer to the related description of S303-A11 above, which will not be repeated here.

[0184] S404, performing third joint optimization on the second joint-optimized face shape parameters and the N-1 second parameters, and the first joint-optimized first parameters.

[0185] For example, according to the first joint-optimized first parameters and the second joint-optimized face shape parameters and the N-1 second parameters, N third projection images of the N face images are determined; a third loss between the N face images and the N third projection images is determined, and the first joint-optimized first parameters and the second joint-optimized face shape parameters and the N-1 second parameters are updated according to the third loss; the updated face shape parameters and the second parameters are taken as new second joint-optimized face shape parameters and second parameters, and the updated first parameters are taken as new first joint-optimized first parameters, and the above steps are repeatedly executed to perform multiple loop updates on the face shape parameters, the first parameters and the second parameters.

[0186] For details, refer to the related description of S303-A121 above, which will not be repeated here.

[0187] S405, performing fourth joint optimization on the third joint-optimized face shape parameters and the first parameters.

[0188] For example, according to the third joint-optimized face shape parameters and the first parameters, a fourth projection image corresponding to the first face image is determined; a fourth loss between the first face image and the fourth projection image is determined, and the third joint-optimized face shape parameters and the first parameters are updated according to the fourth loss; the updated face shape parameters and the first parameters are taken as new third joint-optimized face shape parameters and first parameters, and the above steps are repeatedly executed to perform multiple loop updates on the face shape parameters and the first parameters.

[0189] For details, refer to the related description of S303-A1221 above, which will not be repeated here.

[0190] S406, reconstructing a three-dimensional face model of the user according to the fourth joint-optimized face shape parameters and a preset face three-dimensional deformation statistical model.

[0191] For example, the fourth joint-optimized face shape parameters are substituted into the face three-dimensional deformation statistical model shown in the above formula (1) to obtain the three-dimensional face model of the user.

[0192] The reconstruction method of the three-dimensional face model according to the embodiment of the application decouples information mainly from the first face image, then integrates and optimizes all the pictures, and finally optimizes the first face image alone, so as to achieve the purpose of taking the first face image as the main one and other face images as the auxiliary ones, thereby effectively reducing the information conflict among the multiple face images, and realizing the decoupling of the posture parameter, the face shape parameter and the expression parameter, further improving the optimization accuracy of the face shape parameter, and improving the accuracy of the three-dimensional face model reconstructed based on the face shape parameter.

[0193] The preferred embodiments of the application are described in detail above with reference to the drawings, but the application is not limited to the specific details in the above-described embodiments. Within the technical concept of the application, various simple modifications can be made to the technical solutions of the application, and these simple modifications all belong to the protection scope of the application. For example, in the above-described specific embodiments, various specific technical features can be combined in any appropriate manner without contradiction. In order to avoid unnecessary repetition, various possible combination manners are not described again in the application. For example, various different embodiments of the application can also be combined in any manner, as long as it does not deviate from the idea of the application, and it should also be considered as the disclosed content of the application.

[0194] It should also be understood that in various method embodiments of the application, the size of the sequence number of the above-described processes does not mean the order of execution. The execution order of the processes should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the application.

[0195] The method embodiments of the application are described in detail above with reference to Figures 3 to 4 , and the device embodiments of the application are described in detail below with reference to Figures 5 to 6 .

[0196] Figure 5 is a schematic block diagram of a three-dimensional face model reconstruction device according to an embodiment of the application. The device 10 can be a computing device or a part of a computing device.

[0197] As shown in Figure 5 , the three-dimensional face model reconstruction device 10 can include:

[0198] The acquisition unit 11 is configured to acquire N face images of the same user, and acquire a first face image from the N face images, where N is a positive integer greater than 1.

[0199] The optimization unit 12 is configured to perform first joint optimization on the preset face shape parameters, the first parameters corresponding to the first face image, and second pose parameters corresponding to N-1 second face images, the N-1 second face images being other face images except the first face image in the N face images.

[0200] The reconstruction unit 13 is configured to reconstruct a three-dimensional face model of the user according to the face shape parameters after the first joint optimization.

[0201] In some embodiments, the optimization unit 12 is specifically configured to determine a first projection image corresponding to each of the N face images according to the face shape parameters, the first parameters, and the second pose parameters corresponding to the N-1 second face images; update the face shape parameters, the first parameters, and the N-1 second pose parameters according to a first loss between the N face images and the N first projection images; and repeatedly perform the above steps to perform multiple loop updates on the face shape parameters, the first parameters, and the second pose parameters, by taking the updated face shape parameters, the first parameters, and the second pose parameters as new face shape parameters, first parameters, and second pose parameters, respectively.

[0202] In some embodiments, the optimization unit 12 is specifically configured to determine a first pixel loss between the first face image and the projection image corresponding to the first face image according to a deviation between a pixel value of each pixel point in the first face image and a pixel value of each pixel point in the projection image corresponding to the first face image; determine, for each of the N face images, a projection coordinate of each key point in the face image in the corresponding projection image, and determine a first key point loss corresponding to the face image according to a deviation between a coordinate of each key point in the face image and a projection coordinate of the key point in the corresponding projection image; and determine the first loss according to the first pixel loss and the first key point loss corresponding to the N face images.

[0203] In some embodiments, the first parameters include first pose parameters corresponding to the first face image, and an update step of the first pose parameters and the second pose parameters is greater than an update step of the face shape parameters and other parameters in the first parameters except the first pose parameters.

[0204] In some embodiments, the reconstruction unit 13 is specifically configured to determine a target value of the face shape parameters according to the face shape parameters after the first joint optimization; and reconstruct the three-dimensional face model of the user according to the target value of the face shape parameters and a preset face three-dimensional deformation statistical model.

[0205] In some embodiments, the reconstruction unit 13 is specifically configured to determine a target value of the face shape parameter according to the first joint-optimized face shape parameter and the first parameter.

[0206] In some embodiments, the reconstruction unit 13 is specifically configured to perform second joint optimization on the first joint-optimized face shape parameter and N-1 second parameters respectively corresponding to the N-1 second face images, the second parameters including expression parameters corresponding to the second face images and the second joint-optimized pose parameters; and determine the target value of the face shape parameter according to the second joint-optimized face shape parameter.

[0207] In some embodiments, the reconstruction unit 13 is specifically configured to perform third joint optimization on the second joint-optimized face shape parameter and the N-1 second parameters and the first joint-optimized first parameter; and determine the target value of the face shape parameter according to the third joint-optimized face shape parameter.

[0208] In some embodiments, the reconstruction unit 13 is specifically configured to perform fourth joint optimization on the third joint-optimized face shape parameter and the first parameter; and determine the fourth joint-optimized face shape parameter as the target value of the face shape parameter.

[0209] In some embodiments, the reconstruction unit 13 is specifically configured to determine N-1 second projection images corresponding to the N-1 second face images according to the second parameters respectively corresponding to the N-1 second face images and the first joint-optimized face shape parameter; determine a second loss between the N-1 second face images and the N-1 second projection images, and update the second parameters respectively corresponding to the N-1 second face images and the first joint-optimized face shape parameter according to the second loss; and repeat the above steps by taking the updated second parameters and face shape parameter as new second parameters and first joint-optimized face shape parameter, to perform multiple loop updates on the face shape parameter and the second parameters.

[0210] In some embodiments, the reconstruction unit 13 is specifically configured to, for each of the N-1 second face images, determine a second pixel loss corresponding to the second face image according to a deviation between pixel values of each pixel point in the second face image and pixel values of each pixel point in a corresponding projection image; determine projection coordinates of each key point in the second face image in the corresponding projection image, and determine a second key point loss corresponding to the second face image according to a deviation between coordinates of the key points in the second face image and the projection coordinates in the corresponding projection image; and determine the second loss as a sum of the second pixel losses and the second key point losses corresponding to the N-1 second face images.

[0211] In some embodiments, the update step of the expression parameters and the face shape parameters is smaller than the update step of the other parameters in the second parameters except the expression parameters.

[0212] In some embodiments, the reconstruction unit 13 is specifically configured to determine N third projection images of the N face images according to the first parameters after the first joint optimization and the face shape parameters and the N-1 second parameters after the second joint optimization, determine a third loss between the N face images and the N third projection images, and update the first parameters after the first joint optimization and the face shape parameters and the N-1 second parameters after the second joint optimization according to the third loss; repeat the above steps by taking the updated face shape parameters and the second parameters as new face shape parameters and second parameters after the second joint optimization and taking the updated first parameters as new first parameters after the first joint optimization, to perform multiple loop updates on the face shape parameters, the first parameters and the second parameters.

[0213] In some embodiments, the reconstruction unit 13 is specifically configured to, for each of the N face images, determine a third pixel loss of the face image according to the deviation between the pixel value of each pixel point in the face image and the pixel value of each pixel point in the corresponding projection image, determine the projection coordinates of each key point in the face image in the corresponding projection image, and determine a third key point loss of the face image according to the deviation between the coordinates of each key point in the face image and the projection coordinates of each key point in the corresponding projection image; and determine the third loss as the sum of the third pixel loss and the third key point loss of the N face images.

[0214] In some embodiments, the reconstruction unit 13 is specifically configured to determine a fourth projection image corresponding to the first face image according to the face shape parameters and the first parameters after the third joint optimization, determine a fourth loss between the first face image and the fourth projection image, and update the face shape parameters and the first parameters after the third joint optimization according to the fourth loss; repeat the above steps by taking the updated face shape parameters and the first parameters as new face shape parameters and first parameters after the third joint optimization, to perform multiple loop updates on the face shape parameters and the first parameters.

[0215] In some embodiments, the reconstruction unit 13 is specifically configured to determine a fourth pixel loss between the first face image and the fourth projection image according to a deviation between a pixel value of each pixel point in the first face image and a pixel value of a corresponding pixel point in the fourth projection image; determine projection coordinates of each key point in the first face image in the fourth projection image, and determine a fourth key point loss corresponding to the first face image according to a deviation between coordinates of the each key point in the first face image and the projection coordinates in the fourth projection image; and determine the fourth loss by summing the fourth pixel loss and the fourth key point loss corresponding to the first face image.

[0216] In some embodiments, the first face image is a face image with the least expression among the N face images.

[0217] It should be understood that the device embodiments and the method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, no longer described here. Specifically, Figure 5 The device shown can perform the embodiments of the above method, and the foregoing and other operations and / or functions of each module in the device are respectively for realizing the corresponding method embodiments of the computing device, and for the sake of brevity, no longer described here.

[0218] The device of the embodiments of the present application is described above in conjunction with the drawings from the perspective of functional modules. It should be understood that the functional modules can be realized by hardware, or by instructions in the form of software, or by a combination of hardware and software modules. Specifically, each step of the method embodiments in the embodiments of the present application can be completed by integrated logic circuits and / or software instructions in the hardware of the processor, and the steps of the method disclosed in the embodiments of the present application can be directly embodied as hardware code processor execution completion, or executed by a combination of hardware and software modules in the code processor. Alternatively, the software module can be located in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, register, etc. The storage medium is located in the memory, and the processor reads the information in the memory, and combines the hardware to complete the steps in the above method embodiments.

[0219] Figure 6 is a schematic block diagram of a computing device provided by the embodiments of the present application, which is used to execute the above method embodiments.

[0220] As Figure 6 shown, the computing device 30 can include:

[0221] a memory 31 for storing a computer program 33 and transferring the program code 33 to the processor 32. In other words, the processor 32 can call and run the computer program 33 from the memory 31 to implement the method in the embodiments of the present application.

[0222] For example, the processor 32 can be configured to perform the above-mentioned method steps according to the instructions in the computer program 33.

[0223] In some embodiments of the present application, the processor 32 can include but is not limited to:

[0224] a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, and the like.

[0225] In some embodiments of the present application, the memory 31 can include but is not limited to:

[0226] volatile memory and / or non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically EPROM (EEPROM) or flash memory. The volatile memory can be random access memory (RAM) used as an external cache. By way of example, and not limitation, many forms of RAM can be used, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synch link DRAM (SLDRAM) and direct Rambus RAM (DR RAM).

[0227] In some embodiments of the present application, the computer program 33 can be divided into one or more modules stored in the memory 31 and executed by the processor 32 to complete the method of recording a page provided by the present application. The one or more modules can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program 33 in the computing device.

[0228] As shown in Figure 6 The computing device 30 can further include:

[0229] A transceiver 34, which can be connected to the processor 32 or the memory 31.

[0230] The processor 32 can control the transceiver 34 to communicate with other devices, specifically, can send information or data to other devices, or receive information or data sent by other devices. The transceiver 34 can include a transmitter and a receiver. The transceiver 34 can further include an antenna, and the number of antennas can be one or more.

[0231] It should be understood that various components in the computing device 30 are connected through a bus system, wherein the bus system includes a data bus, a power supply bus, a control bus and a state signal bus in addition to the data bus.

[0232] According to an aspect of the present application, a computer storage medium is provided, and the computer storage medium stores a computer program. When the computer program is executed by a computer, the computer can execute the method of the above-mentioned method embodiments. Alternatively, the embodiments of the present application also provide a computer program product containing instructions, and the instructions are executed by a computer to execute the method of the above-mentioned method embodiments.

[0233] According to another aspect of the present application, a computer program product or computer program is provided, and the computer program product or computer program includes computer instructions stored in a computer readable storage medium. The processor of the computing device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to make the computing device execute the method of the above-mentioned method embodiments.

[0234] In other words, when implemented using software, the functions can be implemented in whole or in part using a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the whole or part of the flow or function according to the embodiments of the present application is generated. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a digital video disc (DVD)), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0235] Those skilled in the art can realize that the modules and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0236] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the above described device embodiments are only schematic, for example, the division of the modules is only a logical function division, and actual implementation can have another division manner, for example, a plurality of modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed each other can be indirect coupling or communication connection through some interface, device or module, and can be electrical, mechanical or other forms.

[0237] The modules illustrated as separate components may or may not be physically separate, and the components illustrated as modules may or may not be physical modules, i.e., may be located in one place, or may be distributed to multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application. For example, the functional modules in various embodiments of the present application can be integrated in one processing module, or each module can exist physically separately, or two or more modules can be integrated in one module.

[0238] The above is merely specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for reconstructing a three-dimensional human face model, characterized in that, include: Obtain N face images of the same user, and extract the first face image from the N face images, where N is a positive integer greater than 1; A first joint optimization is performed on the preset face shape parameters, the first parameters corresponding to the first face image, and the second pose parameters corresponding to N-1 second face images respectively. The N-1 second face images are other face images among the N face images besides the first face image. The first parameters include at least one of the first texture parameters, the first pose parameters, and the first illumination parameters. Based on the first jointly optimized facial shape parameters, the user's three-dimensional facial model is reconstructed.

2. The method according to claim 1, characterized in that, The first joint optimization of the preset face shape parameters, the first parameters corresponding to the first face image, and the second pose parameters corresponding to N-1 second face images includes: Based on the face shape parameters, the first parameter, and the second pose parameters corresponding to the N-1 second face images, determine the first projection image corresponding to each face image in the N face images; Determine the first loss between the N face images and the N first projection images, and update the face shape parameters, the first parameters, and N-1 second pose parameters based on the first loss; The updated face shape parameters, the first parameter, and the second pose parameters are used as the new face shape parameters, the first parameter, and the second pose parameters, respectively. The above steps are repeated to update the face shape parameters, the first parameter, and the second pose parameters multiple times.

3. The method according to claim 2, characterized in that, The determination of the first loss between the N face images and the N first projection images includes: The first pixel loss between the first face image and the corresponding projection image is determined based on the deviation between the pixel value of each pixel in the first face image and the pixel value of each pixel in the corresponding projection image of the first face image. For each of the N face images, determine the projection coordinates of each key point in the face image in the corresponding projection image, and determine the first key point loss corresponding to the face image based on the deviation between the coordinates of each key point in the face image and the projection coordinates in the corresponding projection image. The first loss is determined based on the first pixel loss and the first key point loss corresponding to each of the N face images.

4. The method according to claim 2, characterized in that, The first parameter includes a first pose parameter corresponding to the first face image. The update step size of the first pose parameter and the second pose parameter is greater than the update step size of the face shape parameter and the other parameters in the first parameter besides the first pose parameter.

5. The method according to any one of claims 1-4, characterized in that, The step of reconstructing the user's 3D face model based on the first jointly optimized face shape parameters includes: Based on the first jointly optimized face shape parameters, the target value of the face shape parameters is determined; Based on the target values ​​of the facial shape parameters and the preset three-dimensional facial deformation statistical model, the user's three-dimensional facial model is reconstructed.

6. The method according to claim 5, characterized in that, The step of determining the target value of the face shape parameters based on the first jointly optimized face shape parameters includes: The target value of the face shape parameter is determined based on the first jointly optimized face shape parameter and the first parameter.

7. The method according to claim 6, characterized in that, The step of determining the target value of the face shape parameter based on the first jointly optimized face shape parameter and the first parameter includes: The second parameters corresponding to the N-1 second face images and the first jointly optimized face shape parameters are subjected to a second joint optimization. The second parameters include the expression parameters corresponding to the second face images and the second pose parameters after the first joint optimization. Based on the second jointly optimized face shape parameters, the target value of the face shape parameters is determined.

8. The method according to claim 7, characterized in that, The step of determining the target value of the face shape parameters based on the second jointly optimized face shape parameters includes: A third joint optimization is performed on the face shape parameters after the second joint optimization, the N-1 second parameters, and the first parameters after the first joint optimization. Based on the third jointly optimized face shape parameters, the target value of the face shape parameters is determined.

9. The method according to claim 8, characterized in that, The step of determining the target value of the face shape parameters based on the third jointly optimized face shape parameters includes: A fourth joint optimization is performed on the face shape parameters after the third joint optimization and the first parameter; The fourth jointly optimized face shape parameter is determined as the target value of the face shape parameter.

10. The method according to claim 7, characterized in that, The second joint optimization of the second parameters corresponding to the N-1 second face images and the first jointly optimized face shape parameters includes: Based on the second parameters corresponding to the N-1 second face images respectively, and the first jointly optimized face shape parameters, determine the N-1 second projection images corresponding to the N-1 second face images; Determine the second loss between the N-1 second face images and the N-1 second projection images, and update the second parameters corresponding to the N-1 second face images and the first jointly optimized face shape parameters according to the second loss; The updated second parameter and the face shape parameter are used as the new second parameter and the first jointly optimized face shape parameter. The above steps are repeated to update the face shape parameter and the second parameter multiple times.

11. The method according to claim 10, characterized in that, The determination of the second loss between the N-1 second face images and the N-1 second projected images includes: For each of the N-1 second face images, the second pixel loss corresponding to the second face image is determined based on the deviation between the pixel value of each pixel in the second face image and the pixel value of each pixel in the corresponding projection image. Determine the projection coordinates of each key point in the second face image in the corresponding projection image, and determine the second key point loss corresponding to the second face image based on the deviation between the coordinates of each key point in the second face image and the projection coordinates in the corresponding projection image. The sum of the second pixel loss and the second key point loss corresponding to the N-1 second face images is determined as the second loss.

12. The method according to claim 10, characterized in that, The update step size of the expression parameter and the face shape parameter is smaller than the update step size of the other parameters in the second parameter group besides the expression parameter.

13. The method according to claim 8, characterized in that, The third joint optimization of the face shape parameters after the second joint optimization, the N-1 second parameters, and the first parameters after the first joint optimization includes: Based on the first parameter after the first joint optimization, the face shape parameter after the second joint optimization, and N-1 second parameters, determine the N third projection images of the N face images; Determine the third loss between the N face images and the N third projection images, and update the first parameters after the first joint optimization, the face shape parameters after the second joint optimization, and N-1 second parameters based on the third loss; The updated face shape parameters and second parameters are used as the new second jointly optimized face shape parameters and second parameters, and the updated first parameter is used as the new first jointly optimized first parameter. The above steps are repeated to update the face shape parameters, first parameter and second parameter multiple times.

14. The method according to claim 13, characterized in that, The determination of the third loss between the N face images and the N third projection images includes: For each of the N face images, the third pixel loss corresponding to the face image is determined based on the deviation between the pixel value of each pixel in the face image and the pixel value of each pixel in the corresponding projection image. Determine the projection coordinates of each key point in the face image in the corresponding projection image, and determine the third key point loss corresponding to the face image based on the deviation between the coordinates of each key point in the face image and the projection coordinates in the corresponding projection image. The sum of the third pixel loss and the third key point loss corresponding to the N face images is determined as the third loss.

15. The method according to claim 9, characterized in that, The fourth joint optimization of the face shape parameters after the third joint optimization and the first parameter includes: Based on the third jointly optimized face shape parameters and the first parameter, determine the fourth projection image corresponding to the first face image; A fourth loss is determined between the first face image and the fourth projected image, and the third jointly optimized face shape parameters and the first parameter are updated based on the fourth loss. The updated face shape parameters and the first parameter are used as the new third jointly optimized face shape parameters and the first parameter. The above steps are repeated to update the face shape parameters and the first parameter multiple times.

16. The method according to claim 15, characterized in that, The determination of the fourth loss between the first face image and the fourth projected image includes: The fourth pixel loss between the first face image and the fourth projection image is determined based on the deviation between the pixel value of each pixel in the first face image and the pixel value of the corresponding pixel in the fourth projection image. The projection coordinates of each key point in the first face image in the fourth projection image are determined, and the fourth key point loss corresponding to the first face image is determined based on the deviation between the coordinates of each key point in the first face image and the projection coordinates in the fourth projection image. The fourth loss is determined by summing the fourth pixel loss and the fourth key point loss corresponding to the first face image.

17. The method according to any one of claims 1-4, characterized in that, The first face image is the frontal face image with the smallest expression among the N face images.

18. A device for reconstructing a three-dimensional human face model, characterized in that, include: The acquisition unit is used to acquire N face images of the same user and acquire a first face image from the N face images, where N is a positive integer greater than 1; An optimization unit is used to perform a first joint optimization on preset face shape parameters, a first parameter corresponding to the first face image, and second pose parameters corresponding to N-1 second face images respectively. The N-1 second face images are other face images among the N face images besides the first face image. The first parameter includes at least one of a first texture parameter, a first pose parameter, and a first illumination parameter. The reconstruction unit is used to reconstruct the user's three-dimensional face model based on the first jointly optimized face shape parameters.

19. A computing device, characterized in that, Including processor and memory; The memory is used to store computer programs; The processor is configured to execute the computer program to implement the method as described in any one of claims 1 to 17.

20. A computer-readable storage medium, characterized in that, Used to store computer programs that cause a computer to perform the method as described in any one of claims 1 to 17.

Images