A three-dimensional face model training method and device

Abstract

The embodiment of the present invention provides a 3D face model training method and device, the method comprising: calculating the loss penalty item information of the contour points to be adjusted and the corresponding standard contour points; inputting the loss penalty item information into the 3D face reconstruction network to obtain an updated 3D face model to be adjusted; returning to execute the step of calculating the loss penalty item information of the contour points to be adjusted and the corresponding standard contour points until the latest loss penalty item information meets the contour fitting condition. Through the feature points in the 3D face model to be adjusted and the feature points in the standard 3D face model, the loss penalty item information is repeatedly calculated iteratively to enhance the learning of the contour points by the 3D face reconstruction network, so that the edge contour points of the constructed 3D face model fit the standard contour points as closely as possible.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional face reconstruction, and in particular to a three-dimensional face model training method and device. Background Art

[0002] With the development of artificial intelligence and three-dimensional technology, three-dimensional face reconstruction technology has emerged. Three-dimensional face reconstruction is an important research topic in computer vision and computer graphics. It is widely used in face recognition, human-computer interaction, facial expression drive, AI (Artificial Intelligence) and three-dimensional animation effects.

[0003] Currently, in many applications, users use 3D animation effects. When the face is deflected at a large angle, the 3D animation effect does not fit the face well. Summary of the invention

[0004] The purpose of the present invention includes, for example, providing a three-dimensional face model training method and device, which can solve the problem that the three-dimensional animation effect does not fit the face when the face is deflected at a large angle.

[0005] In order to achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:

[0006] In a first aspect, an embodiment of the present application provides a 3D face model training method, the method comprising:

[0007] Calculating loss penalty item information of contour points to be adjusted and corresponding standard contour points, wherein the contour points to be adjusted are feature points in a three-dimensional face model to be adjusted generated by a three-dimensional face reconstruction network based on an unlabeled training image; the standard contour points are feature points in a standard three-dimensional face model generated based on a labeled image; and the visual content of the unlabeled training image is consistent with that of the labeled image;

[0008] Inputting the loss penalty item information into the 3D face reconstruction network to obtain an updated 3D face model to be adjusted;

[0009] Return to the step of calculating the loss penalty item information of the contour point to be adjusted and the corresponding standard contour point until the latest loss penalty item information meets the contour fitting condition.

[0010] In an optional implementation, the loss penalty item information is the loss penalty item information of facial feature points. When the unlabeled training image is a front view, the step of calculating the loss penalty item information of the contour point to be adjusted and the corresponding standard contour point includes:

[0011] Calculate the loss penalty item information of the facial feature points of the contour points to be adjusted and the corresponding standard contour points, wherein the contour points to be adjusted are the facial feature points in the three-dimensional face model to be adjusted generated by the three-dimensional face reconstruction network, the standard contour points are the facial feature points in the standard three-dimensional face model generated based on the marked image, and the loss penalty item information of the facial feature points is the difference between the facial feature points in the three-dimensional face model to be adjusted and the facial feature points in the standard three-dimensional face model;

[0012] The step of inputting the loss penalty item information into the 3D face reconstruction network to obtain an updated 3D face model to be adjusted comprises:

[0013] The loss penalty item information of the facial feature points is input into the three-dimensional face reconstruction network to obtain an updated three-dimensional face model to be adjusted.

[0014] In an optional implementation, the loss penalty item information includes loss penalty item information of facial feature points and loss penalty item information of contour points. When the unlabeled training image is a side view, the step of calculating the loss penalty item information of the contour point to be adjusted and the corresponding standard contour point includes:

[0015] Calculate the loss penalty item information of the contour point to be adjusted and the corresponding standard contour point;

[0016] Calculate the loss penalty item information of the facial feature points of the to-be-adjusted contour points and the corresponding standard contour points, wherein the to-be-adjusted contour points are the facial feature points and contour points in the to-be-adjusted three-dimensional face model generated by the three-dimensional face reconstruction network, the standard contour points are the facial feature points and contour points in the standard three-dimensional face model generated based on the labeled image, the loss penalty item information of the contour points is the difference between the contour points in the to-be-adjusted three-dimensional face model and the contour points in the standard face model, and the loss penalty item information of the facial feature points is the difference between the facial feature points in the to-be-adjusted three-dimensional face model and the facial feature points in the standard face model;

[0017] The step of inputting the loss penalty item information into the 3D face reconstruction network to obtain an updated 3D face model to be adjusted comprises:

[0018] The loss penalty item information of the facial feature points and the loss penalty item information of the contour points are input into the three-dimensional face reconstruction network to obtain an updated three-dimensional face model to be adjusted.

[0019] In an optional implementation manner, the loss penalty item information of the facial feature points satisfies the following formula:

[0020] Among them, predictcontour label is the coordinates of the facial feature points in the 3D face model to be adjusted. contour are the coordinates of the facial feature points in the standard 3D face model.

[0021] In an optional implementation manner, the loss penalty item information of the contour point satisfies the following formula:

[0022]

[0023] Among them, predict kpt is the coordinate of the contour point in the 3D face model to be adjusted, label kpt The coordinates of the contour points in the standard 3D face model.

[0024] In an optional embodiment, the method further comprises:

[0025] Determining the coordinates of each grid point in the three-dimensional face model to be adjusted or the standard three-dimensional face model;

[0026] Determine the ordinate of each of the grid points, and sort the grid points according to the value of the ordinate;

[0027] Acquire a first target grid point group having the same vertical coordinate;

[0028] For each group of the first target grid points, determining a horizontal coordinate of each grid point of the first target grid point group;

[0029] Sort the grid points in the first target grid point group according to the size of the horizontal coordinates;

[0030] After sorting, the grid points with the largest and smallest horizontal coordinates are obtained to obtain the feature points in the three-dimensional face model to be adjusted or the feature points in the standard three-dimensional face model.

[0031] In a second aspect, an embodiment of the present application provides a three-dimensional face reconstruction network, and the three-dimensional face network is obtained by the three-dimensional face model training method.

[0032] In a third aspect, an embodiment of the present application provides a 3D face model training device, the device comprising:

[0033] A calculation module, used to calculate the loss penalty item information of the contour points to be adjusted and the corresponding standard contour points, wherein the contour points to be adjusted are feature points in the three-dimensional face model to be adjusted generated by the unlabeled training image through the three-dimensional face reconstruction network; the standard contour points are feature points in the standard three-dimensional face model generated based on the labeled image; the visual content of the unlabeled training image is consistent with that of the labeled image;

[0034] An input module, used for inputting the loss penalty item information into the 3D face reconstruction network to obtain an updated 3D face model to be adjusted;

[0035] The execution module is used to return to the step of calculating the loss penalty item information of the contour point to be adjusted and the corresponding standard contour point until the latest loss penalty item information meets the contour fitting condition.

[0036] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the three-dimensional face model training method when executing the computer program.

[0037] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the three-dimensional face model training method.

[0038] This application has the following beneficial effects:

[0039] This application calculates the loss penalty information of the contour points to be adjusted and the corresponding standard contour points, inputs the loss penalty information into the 3D face reconstruction network to obtain an updated 3D face model to be adjusted, and returns to execute the step of calculating the loss penalty information of the contour points to be adjusted and the corresponding standard contour points until the latest loss penalty information meets the contour fitting condition. The loss penalty information is iteratively calculated by the feature points in the 3D face model to be adjusted and the feature points in the standard 3D face model, and the learning of the contour points by the 3D face reconstruction network is enhanced, so that the edge contour points of the constructed 3D face model fit the standard contour points as closely as possible. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0041] Figure 1 A block diagram of an electronic device provided by an embodiment of the present invention;

[0042] Figure 2 One of the flow charts of a 3D face model training method provided by an embodiment of the present invention;

[0043] Figure 3 A second flowchart of a three-dimensional face model training method provided by an embodiment of the present invention;

[0044] Figure 4 A flowchart of a three-dimensional face model training method provided by an embodiment of the present invention;

[0045] Figure 5 A schematic diagram of contour points of a three-dimensional face model provided by an embodiment of the present invention;

[0046] Figure 6 A structural block diagram of a three-dimensional face model training device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0048] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0050] In the description of the present invention, it should be noted that if the terms "upper", "lower", "inside", "outside", etc. appear to indicate an orientation or position relationship, they are based on the orientation or position relationship shown in the accompanying drawings, or are the orientation or position relationship in which the product of the invention is usually placed when used. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0051] In addition, the terms “first”, “second”, etc., if used, are merely used to distinguish between the descriptions and should not be understood as indicating or implying relative importance.

[0052] In the description of this application, it should also be noted that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0053] After extensive research, the inventors found that in many applications, when users use three-dimensional animation effects, when the angle of the face is greatly deflected, the three-dimensional animation effect does not fit the face well.

[0054] In view of the discovery of the above problems, the present embodiment provides a three-dimensional face model training method and device, which can iteratively calculate the loss penalty item information through the feature points in the three-dimensional face model to be adjusted and the feature points in the standard three-dimensional face model, so as to enhance the learning of the contour points by the three-dimensional face reconstruction network, so that the edge contour points of the constructed three-dimensional face model are as close as possible to the standard contour points. The solution provided by this embodiment is elaborated in detail below.

[0055] This embodiment provides an electronic device that can train a three-dimensional face model. In a possible implementation, the electronic device can be a user terminal, for example, the electronic device can be, but is not limited to, a server, a smart phone, a personal computer (PC), a tablet computer, a personal digital assistant (PDA), a mobile Internet device (MID), and an image acquisition device.

[0056] Please refer to Figure 1 , Figure 1 1 is a schematic diagram of the structure of the electronic device 100 provided in the embodiment of the present application. The electronic device 100 may also include Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown. Figure 1 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0057] The electronic device 100 includes a 3D face model training device 110 , a memory 120 and a processor 130 .

[0058] The components of the memory 120 and the processor 130 are electrically connected to each other directly or indirectly to realize data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines. The three-dimensional face model training device 110 includes at least one software function module that can be stored in the memory 120 in the form of software or firmware or solidified in the operating system (OS) of the electronic device 100. The processor 130 is used to execute the executable modules stored in the memory 120, such as the software function modules and computer programs included in the three-dimensional face model training device 110.

[0059] The memory 120 may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), etc. The memory 120 is used to store a program, and the processor 130 executes the program after receiving an execution instruction.

[0060] Please refer to Figure 2 , Figure 2 For application Figure 1 A flowchart of a three-dimensional face model training method for an electronic device 100 is provided, and the method including each step is described in detail below.

[0061] Step 201: Calculate the loss penalty item information of the contour point to be adjusted and the corresponding standard contour point.

[0062] The contour points to be adjusted are feature points in the 3D face model to be adjusted generated by the 3D face reconstruction network based on the unlabeled training image; the standard contour points are feature points in the standard 3D face model generated based on the labeled image; the visual content of the unlabeled training image is consistent with that of the labeled image.

[0063] Step 202: Input the loss penalty item information into the 3D face reconstruction network to obtain an updated 3D face model to be adjusted.

[0064] Step 203: Return to the step of calculating the loss penalty item information of the contour point to be adjusted and the corresponding standard contour point, until the latest loss penalty item information meets the contour fitting condition.

[0065] The unlabeled training image is a two-dimensional image of a face. The unlabeled training image is input into the three-dimensional face reconstruction network, and the three-dimensional face reconstruction network has corresponding prediction parameters of the unlabeled training image, and the three-dimensional face model to be adjusted is generated based on the prediction parameters. Since the initial value of the three-dimensional face reconstruction network is inaccurate, the generated three-dimensional face model to be adjusted is greatly different from the corresponding standard three-dimensional face model.

[0066] The standard 3D face model is generated from a labeled image. The labeled image is an image whose parameters are pre-calibrated. Based on the labeled image, label parameters corresponding to the labeled image are obtained, and the standard 3D face model is directly generated based on the label parameters.

[0067] In order to reduce the difference between the to-be-adjusted 3D face model output by the 3D face reconstruction network and the standard 3D face model, it is necessary to calculate the loss penalty information of the to-be-adjusted contour points and the corresponding standard contour points.

[0068] The 3D face reconstruction network may be a mobilenetV3 network model.

[0069] The visual content of the unlabeled training image is consistent with that of the labeled image, that is, the angle and image content of the unlabeled training image are consistent with those of the labeled image. When the unlabeled training image is a front view of the face, the labeled image is also a front view of the face. When the unlabeled training image is a side view of the face, the labeled image is also a side view of the face.

[0070] The calculated loss penalty item information is re-input into the three-dimensional face reconstruction network, and the parameters of the three-dimensional face reconstruction network are changed based on the loss penalty item information, so as to obtain an updated three-dimensional face model to be adjusted.

[0071] Return to the step of calculating the loss penalty item information of the contour point to be adjusted and the corresponding standard contour point until the latest loss penalty item information meets the contour fitting condition.

[0072] The latest loss penalty item information meets the contour fitting condition, that is, the latest loss penalty item information is input into the 3D face reconstruction network to obtain the updated 3D face model to be adjusted. The contour points of the final 3D face model fit the contour points of the standard 3D face model, and the 3D face reconstruction network training is completed.

[0073] This application calculates the loss penalty information of the contour points to be adjusted and the corresponding standard contour points, inputs the loss penalty information into the 3D face reconstruction network to obtain an updated 3D face model to be adjusted, and returns to execute the step of calculating the loss penalty information of the contour points to be adjusted and the corresponding standard contour points until the latest loss penalty information meets the contour fitting condition. The loss penalty information is iteratively calculated by the feature points in the 3D face model to be adjusted and the feature points in the standard 3D face model, and the learning of the contour points by the 3D face reconstruction network is enhanced, so that the edge contour points of the constructed 3D face model fit the standard contour points as closely as possible.

[0074] When the unlabeled training image is a front view, with respect to the above steps 201 and 202, in another embodiment of the present application, Figure 3 As shown, a 3D face model training method is provided, which specifically includes the following steps:

[0075] Step 201 - 1 : Calculate the loss penalty item information of the facial feature points of the contour point to be adjusted and the corresponding standard contour point.

[0076] Among them, the wheel contour points to be adjusted are the facial feature points in the three-dimensional face model to be adjusted generated by the three-dimensional face reconstruction network, the standard contour points are the facial feature points in the standard three-dimensional face model generated based on the labeled image, and the loss penalty item information of the facial feature points is the difference between the facial feature points in the three-dimensional face model to be adjusted and the facial feature points in the standard three-dimensional face model.

[0077] The loss penalty information of facial feature points satisfies the following formula:

[0078] Among them, predict contour label is the coordinates of the facial feature points in the 3D face model to be adjusted. contour are the coordinates of the facial feature points in the standard 3D face model.

[0079] Step 202 - 1 : Input the loss penalty item information of the facial feature points into the 3D face reconstruction network to obtain an updated 3D face model to be adjusted.

[0080] Since the front view has no contour points, when the unlabeled training image is a front view, the unlabeled training image is input into the 3D face reconstruction network to obtain the 3D face model to be adjusted, the facial feature points in the 3D face model to be adjusted are determined, and the facial feature points in the 3D face model to be adjusted are randomly selected. The facial feature points in the standard 3D face model corresponding to the facial feature points in the 3D face model to be adjusted are determined, and the facial feature point loss penalty information of the facial feature points in the 3D face model to be adjusted and the facial feature points of the standard 3D face model are calculated.

[0081] Among them, the facial feature points may be eye feature points, mouth feature points, nose feature points, ear feature points, and the like.

[0082] The loss penalty information of the facial feature points is input into the 3D face reconstruction network, the parameters in the 3D face reconstruction network are modified, and the 3D face reconstruction network outputs the updated prediction parameters. Based on the following formula, the updated 3D face model to be adjusted is obtained:

[0083]

[0084] f is the scale factor, Pr is the orthogonal projection matrix, which is a 3*3 unit matrix, R is a 3*3 unit orthogonal matrix, and its semantics is a rotation matrix. is the average face, A id is the face shape PCA orthogonal matrix, a id is the regressed face shape coefficient, A exp is the PCA orthogonal matrix of facial expression, a exp is the facial expression coefficient obtained by regression, t 2d It is the regressed mesh offset matrix.

[0085] Return to the step of calculating the loss penalty item information of the contour point to be adjusted and the corresponding standard contour point until the latest loss penalty item information meets the contour fitting condition.

[0086] The present application calculates the loss penalty item information of the facial feature points of the to-be-adjusted contour points and the corresponding standard contour points, where the to-be-adjusted contour points are the facial feature points of the to-be-adjusted three-dimensional face model, and the standard contour points are the facial feature points of the standard three-dimensional face model. The loss penalty item information of the facial feature points is input into the three-dimensional face reconstruction network to obtain an updated three-dimensional face model to be adjusted, and the step of calculating the loss penalty item information of the to-be-adjusted contour points and the corresponding standard contour points is returned until the latest loss penalty item information meets the contour fitting condition. The loss penalty item information of the facial feature points is iteratively calculated through the facial feature points in the to-be-adjusted three-dimensional face model and the facial feature points in the standard three-dimensional face model, so as to enhance the learning of the contour points by the three-dimensional face reconstruction network, so that the facial feature points of the constructed three-dimensional face model fit the standard facial feature points as closely as possible.

[0087] When the unlabeled training image is a side view, with respect to the above steps 201 and 202, in another embodiment of the present application, Figure 4 As shown, a 3D face model training method is provided, which specifically includes the following steps:

[0088] Step 201 - 2 : Calculate the loss penalty item information of the contour point to be adjusted and the corresponding standard contour point.

[0089] Step 201 - 3 : Calculate the loss penalty item information of the facial feature points of the contour point to be adjusted and the corresponding standard contour point.

[0090] Among them, the contour points to be adjusted are the facial feature points and contour points in the three-dimensional face model to be adjusted generated by the three-dimensional face reconstruction network, the standard contour points are the facial feature points and contour points in the standard three-dimensional face model generated based on the labeled image, the loss penalty item information of the contour points is the difference between the contour points in the three-dimensional face model to be adjusted and the contour points in the standard face model, and the loss penalty item information of the facial feature points is the difference between the facial feature points in the three-dimensional face model to be adjusted and the facial feature points in the standard face model.

[0091] Step 202-2: Input the loss penalty item information of the contour points and the loss penalty item information of the facial feature points into the three-dimensional face reconstruction network to obtain an updated three-dimensional face model to be adjusted.

[0092] When the unlabeled training image is a side view, the side view is an image with a side face angle greater than 20 degrees.

[0093] The unlabeled training image is input into the 3D face reconstruction network to obtain the 3D face model to be adjusted, and the contour points and facial feature points in the 3D face model to be adjusted are determined.

[0094] The contour points in the three-dimensional face model to be adjusted are determined based on the following method: Figure 5 As shown, it is a schematic diagram of the contour points of the three-dimensional face model; determine the coordinates of each grid point in the three-dimensional face model to be adjusted or the standard three-dimensional face model; determine the ordinate of each grid point, and sort the grid points according to the value of the ordinate; obtain a first target grid point group with the same ordinate; for each group of first target grid point groups, determine the abscissa of each grid point in the first target grid point group; sort the grid points in the first target grid point group according to the size of the abscissa; obtain the grid points with the largest and smallest abscissas after sorting, and obtain the feature points in the three-dimensional face model to be adjusted or the feature points in the standard three-dimensional face model.

[0095] It should be noted that the contour points in the embodiment of the present application are edge points of the constructed three-dimensional face model.

[0096] Randomly select facial feature points and contour points in the three-dimensional face model to be adjusted, determine the contour points and facial feature points of the standard three-dimensional face model corresponding to the contour points and facial feature points in the three-dimensional face model to be adjusted, and calculate the loss penalty item information of the contour points in the three-dimensional face model to be adjusted and the contour points of the standard three-dimensional face model, and the loss penalty item information of the facial feature points in the three-dimensional face model to be adjusted and the facial feature points in the standard three-dimensional face model.

[0097]

[0098] Among them, predict kpt is the coordinate of the contour point in the 3D face model to be adjusted, label kpt The coordinates of the contour points in the standard 3D face model.

[0099] The loss penalty information of facial feature points satisfies the following formula:

[0100] Among them, predict contour label is the coordinates of the facial feature points in the 3D face model to be adjusted. contour are the coordinates of the facial feature points in the standard 3D face model.

[0101] The loss penalty item information of the contour points and the loss penalty item information of the facial feature points are input into the three-dimensional face reconstruction network, and the parameters in the three-dimensional face reconstruction network are modified to obtain an updated three-dimensional face model to be adjusted.

[0102] Return to the step of calculating the loss penalty item information of the contour point to be adjusted and the corresponding standard contour point until the latest loss penalty item information meets the contour fitting condition.

[0103] The present application calculates the loss penalty item information of the contour points of the to-be-adjusted contour points and the corresponding standard contour points, and calculates the loss penalty item information of the facial feature points of the to-be-adjusted facial features and the corresponding standard facial feature points. The loss penalty item information of the contour points and the loss penalty item information of the facial feature points are input into the three-dimensional face reconstruction network to obtain an updated three-dimensional face model to be adjusted, and the step of calculating the loss penalty item information of the contour points to be adjusted and the corresponding standard contour points is returned until the latest loss penalty item information meets the contour fitting condition. The loss penalty item information of the facial feature points and the loss penalty item information of the contour points are repeatedly iterated by calculating the contour points and facial feature points in the to-be-adjusted three-dimensional face model and the standard three-dimensional face model constructed from the side view, while enhancing the learning of the contour points by the three-dimensional face reconstruction network, the facial feature points are also adjusted, so that the constructed three-dimensional face model to be adjusted fits the standard three-dimensional face model as closely as possible.

[0104] The present application also provides a three-dimensional face model reconstruction method, and the three-dimensional face model is reconstructed through the three-dimensional face model training method.

[0105] Please refer to Figure 6 The present application embodiment also provides a method for applying Figure 1 The 3D face model training device 110 of the electronic device 100 includes:

[0106] The calculation module 111 is used to calculate the loss penalty item information of the contour points to be adjusted and the corresponding standard contour points, wherein the contour points to be adjusted are feature points in the three-dimensional face model to be adjusted generated by the unlabeled training image through the three-dimensional face reconstruction network; the standard contour points are feature points in the standard three-dimensional face model generated based on the labeled image; the visual content of the unlabeled training image is consistent with that of the labeled image.

[0107] In this embodiment, the calculation module 111 can be used to perform Figure 2 As shown in step 201 , the collective description of the calculation module 111 can refer to the description of step 201 .

[0108] The input module 112 is used to input the loss penalty item information into the 3D face reconstruction network to obtain an updated 3D face model to be adjusted.

[0109] In this embodiment, the input module 112 can be used to execute Figure 2 As shown in step 202 , for a detailed description of the input module 112 , reference may be made to the description of step 202 .

[0110] The execution module 113 is used to return to the step of calculating the loss penalty item information of the contour point to be adjusted and the corresponding standard contour point until the latest loss penalty item information meets the contour fitting condition.

[0111] In this embodiment, the execution module 113 can be used to execute Figure 2 As shown in step 203 , for a detailed description of the execution module 113 , reference may be made to the description of step 203 .

[0112] Optionally, in some possible implementations, the calculation module 111 is specifically used for:

[0113] When the unlabeled training image is a front view, calculating the loss penalty item information of the facial feature points of the to-be-adjusted contour points and the corresponding standard contour points, wherein the to-be-adjusted contour points are the facial feature points in the to-be-adjusted three-dimensional face model generated by the three-dimensional face reconstruction network, the standard contour points are the facial feature points in the standard three-dimensional face model generated based on the labeled image, and the loss penalty item information of the facial feature points is the difference between the facial feature points in the to-be-adjusted three-dimensional face model and the facial feature points in the standard three-dimensional face model;

[0114] The execution module 113 is specifically used for:

[0115] The loss penalty item information of the facial feature points is input into the three-dimensional face reconstruction network to obtain an updated three-dimensional face model to be adjusted.

[0116] Optionally, in some possible implementations, the calculation module 111 is specifically used for:

[0117] When the unlabeled training image is a side view, calculating the loss penalty item information of the contour point to be adjusted and the corresponding standard contour point;

[0118] Calculate the loss penalty item information of the facial feature points of the to-be-adjusted contour points and the corresponding standard contour points, wherein the to-be-adjusted contour points are the facial feature points and contour points in the to-be-adjusted three-dimensional face model generated by the three-dimensional face reconstruction network, the standard contour points are the facial feature points and contour points in the standard three-dimensional face model generated based on the labeled image, the loss penalty item information of the contour points is the difference between the contour points in the to-be-adjusted three-dimensional face model and the contour points in the standard face model, and the loss penalty item information of the facial feature points is the difference between the facial feature points in the to-be-adjusted three-dimensional face model and the facial feature points in the standard face model;

[0119] The execution module 113 is specifically used for:

[0120] The loss penalty item information of the facial feature points and the loss penalty item information of the contour points are input into the three-dimensional face reconstruction network to obtain an updated three-dimensional face model to be adjusted.

[0121] Optionally, in some possible implementations, the loss penalty item information of the facial feature points satisfies the following formula:

[0122] Among them, predict contour label is the coordinates of the facial feature points in the 3D face model to be adjusted. contour are the coordinates of the facial feature points in the standard 3D face model.

[0123] Optionally, in some possible implementations, the loss penalty item information of the contour point satisfies the following formula:

[0124]

[0125] Among them, predict kpt is the coordinate of the contour point in the 3D face model to be adjusted, label kpt The coordinates of the contour points in the standard 3D face model.

[0126] Optionally, in some possible implementations, the calculation module 111 is further configured to:

[0127] Determining the coordinates of each grid point in the three-dimensional face model to be adjusted or the standard three-dimensional face model;

[0128] Determine the ordinate of each of the grid points, and sort the grid points according to the value of the ordinate;

[0129] Acquire a first target grid point group having the same vertical coordinate;

[0130] For each group of the first target grid points, determining a horizontal coordinate of each grid point of the first target grid point group;

[0131] Sort the grid points in the first target grid point group according to the size of the horizontal coordinates;

[0132] After sorting, the grid points with the largest and smallest horizontal coordinates are obtained to obtain the feature points in the three-dimensional face model to be adjusted or the feature points in the standard three-dimensional face model.

[0133] In summary, the present application calculates the loss penalty information of the contour points to be adjusted and the corresponding standard contour points, inputs the loss penalty information into the 3D face reconstruction network to obtain an updated 3D face model to be adjusted, and returns to execute the step of calculating the loss penalty information of the contour points to be adjusted and the corresponding standard contour points until the latest loss penalty information meets the contour fitting condition. The loss penalty information is iteratively calculated by repeatedly calculating the feature points in the 3D face model to be adjusted and the feature points in the standard 3D face model to enhance the learning of the contour points by the 3D face reconstruction network, so that the edge contour points of the constructed 3D face model fit the standard contour points as closely as possible.

[0134] The present application also provides an electronic device 100, which includes a processor 130 and a memory 120. The memory 120 stores computer executable instructions, and when the computer executable instructions are executed by the processor 130, the three-dimensional face model training method is implemented.

[0135] The embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by the processor 130, the three-dimensional face model training method is implemented.

[0136] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0137] In addition, each functional module in each embodiment of the present application can be integrated together to form an independent part, or each module can exist separately, or two or more modules can be integrated to form an independent part. If the function is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk.

[0138] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0139] The above are only various implementations of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A three-dimensional face model training method, characterized in that: The method comprises: When the unlabeled training image is a side view, calculate the loss penalty item information of the contour points to be adjusted and the corresponding standard contour points; calculate the loss penalty item information of the facial feature points of the contour points to be adjusted and the corresponding standard contour points, wherein the contour points to be adjusted are the facial feature points and contour points in the three-dimensional face model to be adjusted generated by the unlabeled training image through the three-dimensional face reconstruction network, and the standard contour points are the facial feature points and contour points in the standard three-dimensional face model generated based on the labeled image, the loss penalty item information of the contour points is the difference between the contour points in the three-dimensional face model to be adjusted and the contour points in the standard three-dimensional face model, and the loss penalty item information of the facial feature points is the difference between the facial feature points in the three-dimensional face model to be adjusted and the facial feature points in the standard three-dimensional face model; the visual content of the unlabeled training image is consistent with that of the labeled image; Inputting the loss penalty item information into the 3D face reconstruction network to obtain an updated 3D face model to be adjusted; Return to the step of calculating the loss penalty item information of the contour point to be adjusted and the corresponding standard contour point until the latest loss penalty item information meets the contour fitting condition.

2. The method according to claim 1, characterized in that When the unlabeled training image is a front view, calculate the loss penalty item information of the facial feature points of the to-be-adjusted contour points and the corresponding standard contour points, wherein the to-be-adjusted contour points are the facial feature points in the to-be-adjusted three-dimensional face model generated by the unlabeled training image through the three-dimensional face reconstruction network, the standard contour points are the facial feature points in the standard three-dimensional face model generated based on the labeled image, and the loss penalty item information of the facial feature points is the difference between the facial feature points in the to-be-adjusted three-dimensional face model and the facial feature points in the standard three-dimensional face model; The step of inputting the loss penalty item information into the 3D face reconstruction network to obtain an updated 3D face model to be adjusted comprises: The loss penalty item information of the facial feature points is input into the three-dimensional face reconstruction network to obtain an updated three-dimensional face model to be adjusted.

3. The method according to claim 1, characterized in that The step of inputting the loss penalty item information into the 3D face reconstruction network to obtain an updated 3D face model to be adjusted comprises: The loss penalty item information of the facial feature points and the loss penalty item information of the contour points are input into the three-dimensional face reconstruction network to obtain an updated three-dimensional face model to be adjusted.

4. The method according to claim 2 or 3, characterized in that: The loss penalty item information of the facial feature points satisfies the following formula: Loss penalty item information of the facial feature points = ; in, are the coordinates of the facial feature points in the 3D face model to be adjusted, are the coordinates of the facial feature points in the standard 3D face model.

5. The method according to claim 3, characterized in that: The loss penalty item information of the contour point satisfies the following formula: The loss penalty information of the contour point = ; in, are the coordinates of the contour points in the 3D face model to be adjusted, The coordinates of the contour points in the standard 3D face model.

6. The method according to claim 1, characterized in that The method further comprises: Determining the coordinates of each grid point in the three-dimensional face model to be adjusted or the standard three-dimensional face model; Determine the ordinate of each of the grid points, and sort the grid points according to the value of the ordinate; Acquire a first target grid point group having the same vertical coordinate; For each group of the first target grid points, determining a horizontal coordinate of each grid point of the first target grid point group; Sort the grid points in the first target grid point group according to the size of the horizontal coordinates; After sorting, the grid points with the largest and smallest horizontal coordinates are obtained to obtain the feature points in the three-dimensional face model to be adjusted or the feature points in the standard three-dimensional face model.

7. A three-dimensional face reconstruction method, characterized in that: Three-dimensional face reconstruction is performed by the method described in any one of claims 1 to 6.

8. A 3D face model training device, characterized in that: The device comprises: A calculation module, used for calculating the loss penalty item information of the contour points of the to-be-adjusted contour points and the corresponding standard contour points when the unlabeled training image is a side view; calculating the loss penalty item information of the facial feature points of the to-be-adjusted contour points and the corresponding standard contour points, wherein the to-be-adjusted contour points are the facial feature points and contour points in the to-be-adjusted three-dimensional face model generated by the unlabeled training image through the three-dimensional face reconstruction network, the standard contour points are the facial feature points and contour points in the standard three-dimensional face model generated based on the labeled image, the loss penalty item information of the contour points is the difference between the contour points in the to-be-adjusted three-dimensional face model and the contour points in the standard three-dimensional face model, the loss penalty item information of the facial feature points is the difference between the facial feature points in the to-be-adjusted three-dimensional face model and the facial feature points in the standard three-dimensional face model, and the visual content of the unlabeled training image is consistent with that of the labeled image; An input module, used for inputting the loss penalty item information into the 3D face reconstruction network to obtain an updated 3D face model to be adjusted; The execution module is used to return to the step of calculating the loss penalty item information of the contour point to be adjusted and the corresponding standard contour point until the latest loss penalty item information meets the contour fitting condition.

9. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and wherein the processor implements the steps of the method according to any one of claims 1 to 6 when executing the computer program.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images