Digital face makeup retargeting method, digital human generation method, and digital human video generation method

Abstract

This invention provides a method for digital facial makeup retargeting, a method for digital human generation, and a method for digital human video generation, relating to the field of digital human technology. This method decouples the retargeting process of the template makeup from the facial mesh. Instead of migrating the template makeup based on a pre-established correspondence between the facial mesh and the template makeup, it utilizes the deformation results of the target face's facial mesh for template makeup retargeting. During the retargeting process, the deformation results of the target face's facial mesh are introduced, considering the impact of the target face's own deformation on the makeup. This ensures that the display effect of the template makeup on the template face is the same as the display effect of the target makeup on the target face, without being affected by differences in facial shape. The target makeup obtained after the template makeup retargeting process can adapt to target faces with different face shapes, meeting user needs and improving user experience.

Description

Technical Field

[0001] This invention relates to the field of digital human technology, and in particular to a method for digital human facial makeup retargeting, a method for digital human generation, and a method for digital human video generation. Background Technology

[0002] With the continuous development of 3D digital human technology, 3D video generation platforms allow users to create personalized 3D digital human avatars as characters in 3D videos. Users can choose different hairstyles, face shapes, facial features, eyebrow shapes, makeup, clothing, etc., to create personalized 3D digital humans.

[0003] In terms of technical implementation, the platform pre-stores different types of hairstyles, face shapes, facial features, eyebrow shapes, makeup, and clothing, and displays them on the template face of the 3D digital avatar for users to choose from. When creating a personalized 3D digital avatar, the hairstyle, face shape, facial features, eyebrow shapes, makeup, and clothing that are adapted to the template face of the 3D digital avatar need to be transferred to the face of the personalized 3D digital avatar created by the user.

[0004] Existing makeup transfer methods involve applying texture mapping to the target face's facial network using a 2D facial texture image of a template face with a makeup pattern. However, when the facial shapes of the template and target faces change, the makeup pattern also deforms accordingly. For example, if the target face appears fuller than the template face, the makeup pattern on the target face will be stretched, resulting in a visually different makeup effect between the target and template faces, severely impacting the visual quality of the 3D digital facial makeup. Summary of the Invention

[0005] This invention provides a method for digital human face makeup retargeting, a method for digital human generation, and a method for digital human video generation, in order to overcome the deficiencies existing in related technologies.

[0006] This invention provides a digital facial makeup repositioning method, comprising: Obtain the template face corresponding to the target face; Based on the face mesh deformation result of the target face, the template makeup of the template face is retargeted to obtain the target makeup of the target face.

[0007] The present invention also provides a method for generating a digital human, comprising: Based on the above-described digital facial makeup retargeting method, the target makeup of the target face is determined; Based on the target face and target body, the target digital person is determined.

[0008] The present invention also provides a method for generating digital human videos, characterized in that it includes: Based on the above-described digital human generation method, the target digital human is determined; Based on the target digital human, a digital human video is generated.

[0009] The present invention also provides a digital facial makeup repositioning device, comprising: The template face acquisition module is used to acquire the template face corresponding to the target face; The makeup retargeting module is used to retarget the template makeup of the template face based on the face mesh deformation result of the target face to obtain the target makeup of the target face.

[0010] The present invention also provides a digital human generation device, characterized in that it comprises: The makeup determination module is used to determine the target makeup of the target face based on the above-mentioned digital face makeup retargeting method; The first digital human generation module is used to determine the target digital human based on the target face and the target body.

[0011] The present invention also provides a digital human video generation apparatus, comprising: The second digital human generation module is used to determine the target digital human based on the above-described digital human generation method; The digital human video generation module is used to generate digital human videos based on the target digital human.

[0012] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the digital human face makeup retargeting method, or the digital human generation method, or the digital human video generation method as described above.

[0013] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the digital human face makeup retargeting method, the digital human generation method, or the digital human video generation method as described above.

[0014] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the digital human face makeup retargeting method, the digital human generation method, or the digital human video generation method as described above.

[0015] The present invention provides a digital face makeup retargeting method, a digital human generation method, and a digital human video generation method. First, a template face corresponding to a target face is obtained. Then, using the face mesh deformation results of the target face, the template makeup of the template face is retargeted to obtain the target makeup of the target face. This method decouples the retargeting process of the template makeup from the face mesh, no longer relying on a pre-established correspondence between the face mesh and the template makeup for migration. Instead, it uses the face mesh deformation results of the target face for retargeting. During the retargeting process, the face mesh deformation results of the target face are introduced, considering the influence of the target face's own face mesh deformation on the makeup. This ensures that the display effect of the template makeup on the template face is the same as the display effect of the target makeup on the target face. The shapes of the template makeup and the target makeup remain unchanged and are not affected by differences in face shape. The target makeup obtained after the template makeup retargeting process can adapt to target faces with different face shapes, meeting user needs and improving user experience. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this invention or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of the face mesh of the 3D digital human provided by the present invention.

[0018] Figure 2 This is an example image of a two-dimensional human face texture image provided by the present invention.

[0019] Figure 3 This is a schematic diagram showing the texture mapping process applied to the 3D face mesh of a 3D digital human using a 2D face texture image.

[0020] Figure 4 This is a flowchart illustrating a digital facial makeup repositioning method provided by the present invention.

[0021] Figure 5 This is a schematic diagram of a human face grid covered with preset makeup patterns such as freckles and moles, provided by the present invention.

[0022] Figure 6 This is a template makeup provided by the present invention, and a schematic diagram of the makeup effect before and after template makeup retargeting.

[0023] Figure 7 This is a flowchart illustrating a digital human generation method provided by the present invention.

[0024] Figure 8 This is a flowchart illustrating a digital human video generation method provided by the present invention.

[0025] Figure 9 This is a schematic diagram of the structure of a digital facial makeup repositioning device provided by the present invention.

[0026] Figure 10 This is a schematic diagram of the structure of a digital human generation device provided by the present invention.

[0027] Figure 11 This is a schematic diagram of the structure of a digital human video generation device provided by the present invention.

[0028] Figure 12 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0030] like Figure 1 As shown, in a 3D digital human creation platform or a 3D video generation platform, the face model of a 3D digital human can be viewed as an irregular curved surface in three-dimensional space. To facilitate the processing of this irregular curved surface, it is divided into multiple unit mesh units, which can form a 3D face mesh. Each unit mesh unit can be a triangle or a quadrilateral, with three vertices for triangles and four vertices for quadrilaterals. Each vertex has three-dimensional coordinates in 3D space. Therefore, the 3D face mesh can include face mesh lines and face mesh vertices.

[0031] Here, the 3D face mesh is equivalent to adding latitude and longitude lines to the face. Based on the segmentation of the face using the 3D face mesh, different positions on the face can be associated with different unit grid cells. In other words, a coordinate system similar to "latitude and longitude" is built on the face using the 3D face mesh, so that different positions on the face can be determined according to the 3D face mesh.

[0032] After generating the 3D face mesh, it is necessary to perform texture mapping on the 3D face mesh, that is, to cover the surface of the 3D face mesh with a two-dimensional face texture image, so that the resulting 3D digital human face model has a photorealistic feel.

[0033] like Figure 2 The image shown is an example of a two-dimensional face texture image. Figure 3 As shown, for use Figure 2 This is a schematic diagram illustrating the effect of applying texture mapping to a 3D face mesh using a 2D face texture image. Figure 3 Partial mosaic processing has been applied. From Figure 3 As can be seen, the face model of a 3D digital human includes a 3D face mesh and a corresponding 2D face texture image. The 3D face mesh mainly determines the face shape and facial features of the 3D digital human, while the 2D face texture image mainly determines the facial skin of the 3D digital human.

[0034] It's important to note that a 2D face texture image is a two-dimensional planar image, while a 3D face mesh is a mesh model in three-dimensional space. To texture map a 3D face mesh using a 2D face texture image, a 2D texture mapping coordinate system, different from traditional 2D and 3D coordinate systems, can be introduced to accurately map the 2D face texture image onto the 3D face mesh. The coordinate axes of the 2D texture mapping coordinate system are represented by U and V, where U represents the horizontal coordinate and V represents the vertical coordinate. By assigning UV coordinates to each vertex of the 3D face mesh, the unfolding method of the 2D face texture image on the 3D face mesh is defined. Correspondingly, pixels at specific locations in the 2D face texture image are selected, and UV coordinates are assigned to these selected pixels. This establishes a correspondence between the vertices of the 3D face mesh and the pixels in the 2D face texture image based on the UV coordinates. Then, through interpolation calculations, the correspondence between the entire 3D face mesh and the 2D face texture image is established, thus achieving texture mapping processing of the 3D face mesh.

[0035] Therefore, to add facial makeup to a 3D digital human's face model, such as blush, T-zone highlight, cheek highlight, nose contouring, jaw contouring, freckles on the bridge of the nose, moles on the bridge of the nose, butterfly freckles, and mid-face moles, the corresponding makeup effects need to be drawn on the 3D digital human's face model. This means drawing the corresponding makeup patterns on the facial areas of the 2D facial texture image of the 3D digital human. For example, to add different blush effects to the 3D digital human, simply draw different types of blush patterns on the cheek area of ​​the 2D facial texture image; to add different freckle effects, simply draw the corresponding type of freckle patterns in the corresponding areas of the 2D facial texture image. If the user needs to adjust the 3D digital human's facial makeup, they only need to modify the facial makeup drawn in the corresponding areas of the 2D facial texture image.

[0036] In the field of 3D digital human technology, users can not only adjust the facial makeup of 3D digital humans, but also adjust the facial shape of 3D digital humans, that is, "face sculpting", so that users can create personalized 3D digital humans.

[0037] However, when the shape of the 3D digital human's face changes, the makeup pattern will also be deformed accordingly. For example, if the face shape changes from thin to fat, the makeup pattern will be stretched, making the visual effect of the 3D digital human's face makeup different from that of the 3D digital human before the change. In particular, makeup patterns such as freckles and moles are composed of individual spots. These spots are stretched and lengthened, causing deformation and seriously affecting the visual effect of the face makeup.

[0038] Specifically, when a face changes from thin to plump, the vertices in the 3D face mesh move away from the center of the face. The 3D coordinates of these vertices change, but their UV coordinates remain unchanged. The pixels in the corresponding 2D face texture image also remain unchanged. However, because the vertices in the 3D face mesh move away from the face, the distance between adjacent vertices increases, causing the 2D face texture image covering the 3D face mesh to be stretched. The makeup pattern in the 2D face texture image is also stretched accordingly. Since the stretching ratio varies in different areas and directions, the makeup pattern will be severely distorted. In other words, when the face shape changes, the correspondence between the vertices in the 3D face mesh based on UV coordinates and the pixels in the 2D face texture image remains unchanged. This means that even after the face shape changes, texture mapping is still performed according to this unchanged correspondence, which will distort the makeup pattern.

[0039] Based on this, this embodiment of the invention provides a digital facial makeup repositioning method. For example... Figure 4 As shown, the method includes: S11, Obtain the template face corresponding to the target face; S12, based on the face mesh deformation result of the target face, the template makeup of the template face is redirected to obtain the target makeup of the target face.

[0040] Specifically, the digital face makeup retargeting method provided in this embodiment of the invention is executed by a digital face makeup retargeting device. This device can be configured in a 3D digital human creation platform or a 3D video generation platform. The 3D digital human creation platform or 3D video generation platform can be installed on a local computer or in the cloud. The local computer can be a computer, tablet, etc., and is not specifically limited here.

[0041] First, execute step S11 to obtain the template face corresponding to the target face. Here, the target face refers to the face of the personalized 3D digital avatar created by the user, and the user needs to generate the target makeup on the target face. The template face refers to the face of the 3D digital avatar template adapted to the template makeup selected by the user for the target face. The template face is configured with the template makeup selected by the user so that the user can intuitively feel the display effect of different makeup and choose their favorite template makeup.

[0042] A template face is a pre-designed 3D face with a specific shape and features. A template face may include a template face mesh and a template 2D face texture image; template makeup can be overlaid on the template 2D face texture image. The target face may include an initial face mesh. It is understood that different face models within the same software system typically employ the same modeling techniques; therefore, the face meshes included in different face models have the same number of vertices and network topology. Accordingly, in this embodiment of the invention, the initial face mesh and the template face mesh have the same number of vertices and network topology. Both the template face mesh and the initial face mesh include multiple unit mesh cells, and each unit mesh cell and each vertex in the template face mesh and the initial face mesh corresponds one-to-one.

[0043] Each unit grid cell can be either a triangle or a quadrilateral.

[0044] Then, step S12 is executed, where the template makeup of the template face is redirected using the deformation result of the target face's face mesh to obtain the target makeup of the target face. Here, before redirecting the template makeup, the initial face mesh of the target face can be deformed to ensure that the mesh units corresponding to the makeup area in the deformed face mesh are uniform, and that the relative sizes between the mesh units corresponding to the makeup area in the deformed face mesh remain unchanged.

[0045] In this embodiment of the invention, the initial face mesh is deformed, that is, some or all of the unit meshes in the initial face mesh are scaled or rotated. The deformed face mesh, the template face mesh, and each unit mesh and vertex in the initial face mesh are all in one-to-one correspondence. For example, if the template face is thin and the target face is fat, then only by making each unit mesh cell in the initial face mesh larger than the corresponding unit mesh cell in the template face mesh can the space enclosed by the target face mesh be larger. In other words, it is necessary to enlarge each unit mesh cell in the initial face mesh.

[0046] The uniformity of each unit grid cell corresponding to the makeup area in the face mesh deformation result means that the absolute size of each unit grid cell corresponding to the makeup area in the face mesh deformation result remains relatively consistent. That is, the absolute size of each unit grid cell corresponding to the makeup area is as close as possible, avoiding the existence of excessively large or small unit grid cells.

[0047] Furthermore, when retargeting the template makeup, the face mesh deformation result of the target face can be used to retarget the template makeup of the template face. This retargeting process refers to using the face mesh deformation result to map the template makeup of the template face onto the target face, thereby obtaining the target makeup of the target face.

[0048] During the retargeting process, the position information (UV coordinates) of each vertex in the initial face mesh in the 2D texture mapping coordinate system can be retargeted to determine the UV coordinates of each vertex in the face mesh deformation result. By using the UV coordinates of each vertex in the face mesh deformation result as a medium, a correspondence is established between the surface of the target face's face mesh deformation result and the template 2D face texture image. Based on this correspondence, texture mapping processing of the target face's face mesh deformation result can be performed. Combined with the corresponding area of ​​the template makeup on the template face, the retargeting processing of the template makeup on the target face can be achieved, resulting in the target makeup on the target face.

[0049] The digital facial makeup retargeting method provided in this embodiment of the invention first obtains a template face corresponding to the target face; then, using the facial mesh deformation result of the target face, the template makeup of the template face is retargeted to obtain the target makeup of the target face. This method decouples the retargeting process of the template makeup from the facial mesh, no longer relying on a pre-established correspondence between the facial mesh and the template makeup for migration. Instead, it uses the facial mesh deformation result of the target face for retargeting. During the retargeting process, the facial mesh deformation result of the target face is introduced, considering the influence of the deformation of the target face's own facial mesh on the makeup. This ensures that the display effect of the template makeup on the template face is the same as the display effect of the target makeup on the target face. The shapes of the template makeup and the target makeup remain unchanged and are not affected by differences in facial shape. The target makeup obtained after the template makeup retargeting process can adapt to target faces with different face shapes, meeting user needs and improving user experience.

[0050] Based on the above embodiments, the template makeup of the template face is retargeted based on the face mesh deformation result of the target face to obtain the target makeup of the target face. This process includes: The target face is registered with the template face to obtain the registration result; Based on the registration results, the face mesh deformation energy constraint is applied to perform a geometric transformation on the initial face mesh of the target face to obtain the face mesh deformation result.

[0051] Specifically, when determining the face mesh deformation result of the target face, the target face and the template face can be registered first to obtain the registration result. The purpose of registration is to minimize the spatial distance between the target face and the template face, and the registration result can be a transformation matrix between the target face and the template face. This transformation matrix is ​​used to represent the rigid transformation between the target face and the template face.

[0052] Then, using the registration results and combining them with the face mesh deformation energy constraint, a geometric transformation is performed on the initial face mesh of the target face to obtain the face mesh deformation result. The geometric deformation can include position transformation and deformation operations. Deformation operations can include rotation and scaling operations. Rotation operations can be represented based on a rotation matrix, and scaling operations can be represented by a scaling factor.

[0053] Face mesh deformation energy constraint refers to the restriction conditions corresponding to the deformation operation in the geometric transformation process. It is used to ensure the uniformity of each unit mesh cell corresponding to the makeup region in the face mesh deformation result, as well as the relative size invariance among the unit mesh cells corresponding to the makeup region in the face mesh deformation result. Specifically, the face mesh deformation energy constraint can be represented by minimizing the error between the face mesh deformation result and the corresponding connected edges in the initial face mesh corresponding to the makeup region.

[0054] In this embodiment of the invention, when determining the face mesh deformation result, the target face and the template face are first registered, which can reduce the difficulty of geometric transformation of the initial face mesh. Moreover, when performing geometric transformation, the introduction of face mesh deformation energy constraints can suppress excessive deformation of the initial face mesh, improve the effect of the face mesh deformation result, and maintain the overall visual effect of the makeup pattern.

[0055] Based on the above embodiments, the registration result includes a transformation matrix; Based on the registration results, a face mesh deformation energy constraint is applied to perform a geometric transformation on the initial face mesh of the target face, resulting in the face mesh deformation result, including: Based on the transformation matrix, the initial face mesh is transformed to obtain the face mesh to be deformed. Based on the energy constraint of face mesh deformation, a deformation operation is performed on the face mesh to be deformed to obtain the face mesh deformation result.

[0056] Specifically, in this embodiment of the invention, the initial face mesh can be transformed using the transformation matrix obtained by registering the target face with the template face, so as to obtain the face mesh to be deformed and make the face mesh to be deformed overlap with the template face mesh as much as possible.

[0057] Subsequently, using the deformation energy constraint of the face mesh, rotation and scaling operations are performed on the face mesh to be deformed to obtain the face mesh deformation result. The deformation operation on the face mesh can be understood as the deformation operation on the connecting edges within the face mesh.

[0058] In this embodiment of the invention, geometric transformation is split into position transformation and deformation operation, and the deformation operation is restricted by the deformation energy constraint of the face mesh, which can improve the efficiency of geometric transformation.

[0059] Based on the above embodiments, the face mesh deformation energy constraint includes a constraint constructed based on the first deformation energy of the unit to be deformed in the face mesh to be deformed relative to the corresponding template unit in the template face mesh; Each deformable element and each template element includes a vertex, adjacent vertices of the vertex, connecting edges between the vertex and adjacent vertices, and connecting edges between adjacent vertices, or includes a unit mesh element and adjacent unit mesh elements of the unit mesh element.

[0060] Specifically, to simplify the construction process of the deformation energy constraint for the face mesh, several unit mesh elements corresponding to the makeup area can be selected as the deformable elements in the face mesh to be deformed, and several unit mesh elements corresponding to the makeup area can be selected as template elements in the template face mesh, with a one-to-one correspondence between the deformable elements and the template elements. The deformable elements and the corresponding template elements are determined in the same way, either by centering on a vertex or by a single unit mesh element.

[0061] For example, each deformable unit and each template unit can be defined with a vertex as the center. In this case, each deformable unit and each template unit can include a vertex, the adjacent vertices of the vertex, the connecting edges between the vertex and the adjacent vertices, and the connecting edges between adjacent vertices.

[0062] Each deformable element and each template element can also be defined with a unit grid element as the center. In this case, each deformable element and each template element can include a unit grid element and its adjacent unit grid elements.

[0063] Furthermore, the face mesh deformation energy constraint can use the unit to be deformed and the template unit as the basic units for data processing. It can be a constraint constructed using the first deformation energy of the unit to be deformed in the face mesh relative to the corresponding template unit in the template face mesh. The first deformation energy can be the error between the unit to be deformed and the corresponding connecting edge in the template unit after the deformation operation.

[0064] In this embodiment of the invention, by selecting several unit mesh units corresponding to the makeup area in the face mesh to be deformed as the units to be deformed, and selecting several unit mesh units corresponding to the makeup area in the template face mesh as the template units, the mesh area corresponding to the makeup area can be restricted by the face mesh deformation energy constraint, thereby further avoiding excessive deformation of the mesh area corresponding to the makeup area in the initial face mesh, improving the makeup effect in the face mesh deformation result, and maintaining the overall visual effect of the makeup pattern.

[0065] Based on the above embodiments, the deformation operation includes a rotation operation and a scaling operation. The rotation operation is based on a rotation matrix representation, and the scaling operation is based on a scaling factor representation. The calculation steps for the first deformation energy include: For any template connection edge in each unit mesh cell within the template cell, a rotation operation is performed on any template connection edge based on the rotation matrix variable, and a scaling operation is performed on any template connection edge based on the scaling factor variable to obtain the deformed template connection edge. Determine the candidate connecting edge corresponding to any template connecting edge within the unit to be deformed, calculate the difference vector between the candidate connecting edge and the deformed template connecting edge, and calculate the second deformation energy of the candidate connecting edge relative to any template connecting edge based on the difference vector. The first deformation energy is calculated based on the second deformation energy corresponding to each candidate connection edge within the unit to be deformed.

[0066] Specifically, the connecting edge of each unit grid cell within the template unit is the template connecting edge, and the connecting edge of each unit grid cell within the unit to be deformed is the alternative connecting edge. The template connecting edge and the alternative connecting edge correspond one-to-one.

[0067] When calculating the first deformation energy, each template connection edge within the template element can be rotated and scaled to obtain each deformed template connection edge. For example, for any template connection edge in each unit mesh element within the template element, a rotation matrix variable is used to rotate any template connection edge, and a scaling factor variable is used to scale any template connection edge to obtain the deformed template connection edge.

[0068] Subsequently, alternative connecting edges corresponding to any template connecting edge within the unit to be deformed can be determined, the difference vector between the corresponding alternative connecting edge and the deformed template connecting edge can be calculated, and the second deformation energy of the alternative connecting edge relative to any template connecting edge can be calculated using this difference vector.

[0069] For example, assuming both the unit mesh elements within the template element and the unit mesh elements within the element to be deformed are triangular, the three vertices of the unit mesh element within the template element are... The three vertices of the unit mesh element within the element to be deformed are Then, alternative connection edges Relative to the template connection edge The second deformation energy can be denoted as Where R is the rotation matrix variable, used to define the in-situ rotation transformation, and s is the scaling factor variable, used to define the scaling ratio of the edges. Let be the Euclidean norm. Using the general formula, the second deformation energy can be denoted as . .in, and These are the two vertices of the candidate connecting edges. and These are the two vertices of the template connection edge.

[0070] Furthermore, the first deformation energy can be obtained by weighted summing of the second deformation energies corresponding to each candidate connection edge within the deformation unit. That is: ,in, The weight coefficient corresponding to each candidate connection edge. The scaling factor variable for each candidate connection edge. The rotation matrix variable corresponding to each candidate connection edge.

[0071] In this embodiment of the invention, the first deformation energy corresponding to the unit to be deformed is determined by the second deformation energy between each candidate connecting edge in the unit to be deformed and each template connecting edge in the template unit. The first deformation energy corresponding to the unit to be deformed is converted into the second deformation energy between the connecting edges, which can reduce the calculation difficulty of the first deformation energy.

[0072] Based on the above embodiments, the units to be deformed correspond one-to-one with the rotation matrix variables, and the units to be deformed correspond one-to-one with the scaling factor variables.

[0073] Specifically, in order to realize the basic units of data processing using the unit to be deformed and the template unit, the unit to be deformed can be mapped one-to-one with the rotation matrix variable and the scaling factor variable, that is, the rotation matrix variable and the scaling factor variable corresponding to each candidate connecting edge in the same unit to be deformed are all equal.

[0074] Meanwhile, to ensure that the face network to be deformed changes uniformly and continuously, the scaling factor variables corresponding to adjacent units to be deformed are close, that is, the difference between the scaling factor variables corresponding to adjacent units to be deformed is within a preset range.

[0075] Based on the above embodiments, the template makeup type corresponding to the template unit includes a preset makeup pattern, and the scaling factor variable corresponding to the unit to be deformed covered by the preset makeup pattern has a value of 1.

[0076] Specifically, such as Figure 5 The image shown is a schematic diagram of a human face mesh covered with preset makeup patterns such as freckles and moles. Figure 5 The yellow area represents the area covered by the preset makeup pattern.

[0077] from Figure 5 It can be seen that for preset makeup patterns such as freckles and moles, stretching will seriously affect the visual effect. Therefore, it is necessary to keep the shape and size of the deformable units covered by preset makeup patterns such as freckles and moles in the face network completely unchanged. That is, the scaling factor variable corresponding to the deformable unit covered by the preset makeup pattern is set to 1, which means that the deformable unit is rigidly deformed, and only rotation and translation operations are performed, without scaling operations.

[0078] For example, taking a template unit and a deformable unit where both the unit mesh cells are triangles, the second deformation energy corresponding to the candidate connecting edges in the deformable unit covered by the preset makeup pattern can be defined using a general formula as follows: The first deformation energy of the unit to be deformed, covered by the preset makeup pattern, can be denoted as... .

[0079] In this embodiment of the invention, for the unit to be deformed covered by the preset makeup pattern, by setting its scaling factor variable to 1, the shape and size of the unit to be deformed covered by the preset makeup pattern can remain completely unchanged when the face mesh to be deformed is deformed, thus avoiding changes in the effect when the preset makeup pattern is migrated to the target face.

[0080] Based on the above embodiments, the first deformation energy is calculated based on the second deformation energy corresponding to each candidate connection edge within the deformable unit, including: The first deformation energy is calculated by weighted summation of the second deformation energies corresponding to each candidate connection edge within the deformation unit. The weighting coefficients of each second deformation energy are determined based on the uniformity of the template face mesh.

[0081] Specifically, when performing a weighted summation of each second deformation energy, the weighting coefficients corresponding to each second deformation energy can be determined based on the uniformity of the template face mesh. The uniformity of the template face mesh, i.e., the uniformity of each unit mesh cell in the template face mesh, can be determined by one of the following uniformity indicators: the length of each template connecting edge in the template face mesh, the area of ​​each unit mesh cell, and the uniformity of all interior angles in the template face mesh.

[0082] For example, if the error between the side lengths of each template connection edge is within a first specified range, the template face mesh is determined to be uniform; otherwise, the template face mesh is determined to be non-uniform. If the error between the areas of each unit mesh cell is within a second specified range, the template face mesh is determined to be uniform; otherwise, the template face mesh is determined to be non-uniform. If the error between all interior angles in the template face mesh is within a third specified range, the template face mesh is determined to be uniform; otherwise, the template face mesh is determined to be non-uniform.

[0083] The first specified range, the second specified range, and the third specified range can all be set as needed. For example, the first specified range, the second specified range, and the third specified range can all be small ranges so that the uniformity indicators are as close as possible.

[0084] Subsequently, for different cases of whether the template face mesh is uniform, different weighting coefficients can be set for each second deformation energy.

[0085] In this embodiment of the invention, the uniformity of the template face mesh is considered when determining the weighting coefficients of each second deformation energy. This allows the calculated first deformation energy to also characterize the uniformity of the template face mesh, thereby improving the uniformity of each unit mesh cell in the face mesh deformation result.

[0086] Based on the above embodiments, the steps for determining the weighting coefficients of each second deformation energy include: Calculate the relevant angle within the unit grid cell where each candidate connection edge is located; the relevant angle is used to characterize the uniformity. Based on relevant perspectives, the weighting coefficient of the second deformation energy corresponding to each candidate connection edge is calculated.

[0087] Specifically, when determining the weighting coefficients for each second deformation energy, the relevant angle within the unit grid cell containing each candidate connection edge can be calculated. This relevant angle can be used to characterize the uniformity of the template face mesh. Subsequently, using the relevant angle, the weighting coefficient for the second deformation energy corresponding to each candidate connection edge is calculated. This weighting coefficient can be a cotangent weight.

[0088] For example, if the unit grid cell is a triangle, the relevant angles can be the opposite interior angles of the candidate connecting edges within the two triangles containing the candidate connecting edges. The weighting coefficient can be the sum of the cotangent values ​​of the two interior angles, or the average of the cotangent values ​​of the two interior angles. If the unit grid cell is a quadrilateral, it can be divided along the diagonal, transforming it into two triangles, and then the weighting coefficients can be calculated using the same method as for triangles. It should be noted that the same division method should be used for all quadrilaterals. For example, let's denote the four vertices of the quadrilateral as... Therefore, when dividing a quadrilateral, one can proceed along... To divide, and also along The face mesh is segmented, but to ensure consistency across the entire mesh, all quadrilaterals are segmented using the same method, such as segmenting them all along... The direction is used for segmentation. It should be understood that before segmentation, the candidate connecting edge is the common edge of two adjacent quadrilaterals. After segmentation, the candidate connecting edge becomes the common edge of two adjacent triangles. Correspondingly, the relevant angle can also be the interior angles of the candidate connecting edge relative to each other in the two triangles containing the candidate connecting edge. The weighting coefficient can be the sum of the cotangent values ​​of the two interior angles or the average of the cotangent values ​​of the two interior angles.

[0089] In this embodiment of the invention, a specific scheme for determining the weight coefficients of each second deformation energy is given. The uniformity of the template face mesh can be characterized by the relevant angles within the unit mesh cell where the candidate connecting edge is located, and then the corresponding weight coefficients can be calculated based on the relevant angles. This allows the weight coefficients to be related to each unit mesh cell, further improving the uniformity of each unit mesh cell in the face mesh deformation result.

[0090] Based on the above embodiments, and based on the deformation energy constraint of the face mesh, a deformation operation is performed on the face mesh to be deformed to obtain the face mesh deformation result, including: An optimization problem is constructed based on the energy constraint of face mesh deformation. The optimization problem is solved iteratively to obtain the optimal solution for the rotation matrix variable, scaling factor variable, and the position information of each vertex in the unit to be deformed. Based on the optimal solutions of the rotation matrix variables and scaling factor variables, the deformation operation is determined, and based on the optimal solutions of the position information of each vertex in each unit to be deformed, the deformation result of the face mesh is determined.

[0091] Specifically, when performing deformation operations on deformable face meshes, an optimization problem can be constructed first using the deformation energy constraints of the face mesh. For example, the deformation energy constraints of the face mesh can be directly used as the optimization problem, or a regularization term can be introduced into the optimization problem. This regularization term can be a sparsity constraint, a smoothness constraint, etc.

[0092] Since the formula for calculating the first deformation energy involves multiple variables and parameters such as the vertices of the unit mesh element within the element to be deformed, the process of solving the optimization problem is essentially finding a set of parameter combinations that minimizes the first deformation energy.

[0093] As can be seen from the formula for calculating the first deformation energy, once the position information of each vertex within the unit to be deformed is determined, the optimal energy can be calculated. and In other words, and Since the coordinates of each vertex are implicitly determined, the formula for calculating the first deformation energy can be uniquely determined by the position information of each vertex.

[0094] To solve for unknown parameter values, an iterative solution method can be used, first given an initial... and Calculate the optimal solution for the position information of each vertex, and then calculate the optimal solution based on the obtained position information of each vertex. and The optimal solution is obtained, and this process is repeated alternately until the first deformation energy reaches its minimum. At this point, the calculated position information of each vertex ensures that the size of each unit grid cell in the face mesh deformation result remains relatively consistent, thereby achieving uniform scaling of the initial face mesh.

[0095] When the first deformation energy is minimized, the optimal solutions for the rotation matrix variables, scaling factor variables, and each vertex in each unit to be deformed can be obtained. The deformation operation can be determined using the optimal solutions for the rotation matrix variables and scaling factor variables, and the face mesh deformation result can be determined using the optimal solutions for the position information of each vertex in each unit to be deformed.

[0096] In this embodiment of the invention, by iteratively solving the mesh region corresponding to the makeup area in the face mesh to be deformed, deformation operations that satisfy the face mesh deformation energy constraints are performed. This can improve the solution efficiency of the face mesh deformation results and ensure the deformation accuracy of the mesh region corresponding to the makeup area.

[0097] Based on the above embodiments, the target face and the template face are registered to obtain the registration result, including: The target face and the template face are coarsely registered to obtain the coarse registration result. Based on the coarse registration results, the iterative nearest point algorithm is applied to perform fine registration between the target face and the template face to obtain the registration result.

[0098] Specifically, when registering a target face with a template face, the registration process can include a coarse registration process and a fine registration process. The coarse registration process can employ one of the following methods: feature matching-based methods, global search-based methods, or deep learning-based methods. It involves determining the initial transformations of the target face and the template face, and the coarse registration result is obtained. This coarse registration result can be achieved by transforming the target face to a position close to the template face through the initial transformation.

[0099] Subsequently, the coarse registration result can be used as the initial state for the fine registration process. The Iterative Closest Point (ICP) algorithm is then used to finely register the target face and the template face, yielding the registration result. For example, the total distance between specified points on the target face and the template face can be calculated. This total distance is then used to adjust the target face's position in space. Through iterative optimization, the overlapping portions of the template and target faces are made to completely overlap, meaning the total distance between the specified points on the target and template faces meets the convergence condition, thus determining the target face's position in space. The specified points on the target face can be points obtained by sampling the target face; these can be feature points or contour points of the makeup area. Similarly, the specified points on the template face can be points obtained by sampling the template face; these can be feature points or contour points of the makeup area.

[0100] Since neither the coarse nor the fine registration process modifies the shape of the target face itself, but only continuously adjusts the position of the target face in space, that is, it performs a rigid transformation on the target face through rotation and translation, so as to obtain the registration result between the target face and the template face, that is, the transformation matrix between the target face and the template face.

[0101] In this embodiment of the invention, the registration process is divided into two stages: coarse registration and fine registration. This approach balances efficiency and accuracy while reducing algorithm complexity and addressing the issue of significant initial positional differences between the target face and the template face. The coarse registration process reduces the search space for the fine registration process, improving its convergence and accuracy.

[0102] Based on the above embodiments, coarse registration is performed between the target face and the template face to obtain the coarse registration result, including: The edge contour lines of the makeup area in the target face and the template face are determined respectively, and the feature points in the target face and the template face are determined respectively based on the facial shapes of the target face and the template face; Based on the distance between each edge contour line and / or the distance between each feature point, the spatial position of the target face is adjusted to obtain a coarse registration result.

[0103] Specifically, during the coarse registration process, the makeup area can be identified within the template face, and then the edge contour line of that makeup area can be determined. For example, the edge contour line of the makeup area in the template face can be determined at different projection angles by performing orthographic and oblique projections on the template face. The orthographic projection angle can be 90°, and the oblique projection angle can be 30° to the left and 30° to the right.

[0104] It is also possible to identify the makeup area within the target face, and then determine the edge contour line of that makeup area. For example, the edge contour line of the makeup area within the target face can be determined at different projection angles by performing orthographic and oblique projections on the target face.

[0105] Furthermore, the facial shape of the target face can be used to determine feature points within the target face, such as the tip of the nose and the alar points. Similarly, the facial shape of the template face can be used to determine feature points within the template face, such as the tip of the nose and the alar points.

[0106] Finally, the spatial position of the target face can be adjusted by using the distance between the edge contours of the target face and the template face, so that the edge contours of the target face and the template face coincide as much as possible, resulting in a coarse registration result. Alternatively, the spatial position of the target face can be adjusted by using the distance between corresponding feature points of the target face and the template face, so that the corresponding feature points of the target face and the template face coincide as much as possible, resulting in a coarse registration result.

[0107] In addition, the spatial position of the target face can be adjusted by simultaneously utilizing the distance between the edge contours of the makeup areas in the target face and the template face, as well as the distance between the corresponding feature points in the target face and the template face, so that the edge contours of the makeup areas in the target face and the corresponding feature points in the template face coincide as much as possible, thus obtaining a coarse registration result.

[0108] In this embodiment of the invention, the spatial position of the target face is adjusted by at least one of the distance between the edge contour line of the makeup area in the target face and the template face and the distance between the corresponding feature points, so as to achieve coarse alignment. This can improve the accuracy and efficiency of coarse alignment and reduce the difficulty of subsequent fine alignment.

[0109] Based on the above embodiments, the template makeup of the template face is retargeted based on the face mesh deformation result of the target face to obtain the target makeup of the target face, including: For any vertex in the face mesh deformation result, based on the first position information of any vertex in the face mesh deformation result and the second position information of each vertex in the template face mesh corresponding to the template face, calculate the first centroid coordinates of the face of the first unit mesh cell corresponding to any vertex in the template face mesh. Based on the first centroid coordinates and the third position information of each vertex in the first unit grid cell in the two-dimensional texture mapping coordinate system, calculate the fourth position information of any vertex in the two-dimensional texture mapping coordinate system; Based on the fourth position information of each vertex in the two-dimensional texture mapping coordinate system in the face mesh deformation result, the template makeup is mapped to the target face to obtain the target makeup.

[0110] Specifically, when retargeting the template makeup of the template face, the same operation can be performed on each vertex in the face mesh deformation result as follows: For any vertex in the face mesh deformation result, the first position information of any vertex in the face mesh deformation result and the second position information of each vertex in the template face mesh can be used to calculate the first centroid coordinates of the face of the first unit mesh cell corresponding to any vertex in the template face mesh.

[0111] Each location information is represented by coordinates. The first location information refers to the coordinates of the vertices in the face mesh deformation result, and the second location information refers to the coordinates of the vertices in the template face mesh.

[0112] It is understandable that the centroid coordinates can be represented by weights, which are the weights corresponding to the position information of each vertex in the unit grid cell when the position information of any point in the unit grid cell is expressed by a weighted sum of the position information of each vertex in the unit grid cell.

[0113] For example, if the unit grid cell is a spatial triangle For a spatial triangle any point inside There must be three unique numbers. ,satisfy: , ,Right now It can be done Represented by linear combinations of . At this time, for The coordinates of the centroid.

[0114] When calculating the first centroid coordinates, the vertical projection point of any vertex in the face mesh deformation result can be determined in the template face mesh based on the first position information of any vertex in the face mesh deformation result and the second position information of each vertex in the template face mesh corresponding to the template face. For example, the second unit mesh cell with the closest directed distance to any vertex in the template face mesh can be determined by calculating the first difference between the first position information and each second position information, and the vertical projection point of any vertex can be determined in this second unit mesh cell.

[0115] Subsequently, based on the coordinates of each vertex in the second unit grid cell, the second centroid coordinates of the vertical projection point in the second unit grid cell can be calculated, and these second centroid coordinates can be used as the first centroid coordinates of any vertex on the surface of the first unit grid cell.

[0116] It should be noted that, due to potential positional discrepancies between the vertices of the deformed face mesh and the corresponding unit cell in the template face mesh, the perpendicular projection point of any vertex in the deformed face mesh may not necessarily lie within the corresponding first unit cell in the template face mesh. Therefore, the calculation focuses on the first centroid coordinates of any vertex on the surface containing the first unit cell. These first centroid coordinates encompass all possibilities, including whether the perpendicular projection point of any vertex in the deformed face mesh lies within or does not lie within the corresponding first unit cell in the template face mesh.

[0117] Here, by calculating the coordinates of the first centroid, the relationship between the deformation result of the face mesh and the vertices of the corresponding unit cell mesh in the template face mesh can be established.

[0118] Subsequently, since the third position information of each vertex in the first unit mesh cell in the two-dimensional texture mapping coordinate system, that is, the UV coordinates of each vertex in the first unit mesh cell, can be used to calculate the fourth position information of any vertex in the face mesh deformation result in the two-dimensional texture mapping coordinate system, that is, the UV coordinates of any vertex in the face mesh deformation result, using the first centroid coordinates corresponding to any vertex in the face mesh deformation result and the UV coordinates of each vertex in the first unit mesh cell.

[0119] For example, if For any vertex in the face mesh deformation result Represented as , They are respectively represented as The vertices of the first unit grid cell corresponding to the template face mesh. for The first centroid coordinate can be obtained by... Replace with The UV coordinates of each vertex of the first unit grid cell corresponding to the template face mesh are calculated. Represented as UV coordinates.

[0120] Finally, using the UV coordinates of each vertex in the face mesh deformation result, and combining the correspondence between the UV coordinates and each pixel in the template 2D face texture image, the template makeup is mapped to the target face through interpolation calculation to obtain the target makeup on the target face.

[0121] In this embodiment of the invention, by calculating the first centroid coordinates, a correlation is established between the deformation result of the face mesh and the vertices of the corresponding unit cell mesh in the template face mesh. This correlation is then applied to the calculation of UV coordinates, which simplifies the calculation process of template makeup retargeting, reduces the amount of calculation for template makeup retargeting, and improves the efficiency of template makeup retargeting.

[0122] Based on the above embodiments, and based on the first position information of any vertex in the face mesh deformation result and the second position information of each vertex in the template face mesh corresponding to the template face, the first centroid coordinates of the face where the first unit mesh cell of any vertex is located in the template face mesh are calculated, including: Based on the first position information, each of the second position information, the initial position information of any vertex in the face mesh deformation result under the two-dimensional texture mapping coordinate system, and the fifth position information of each vertex in the template face mesh under the two-dimensional texture mapping coordinate system, determine the vertical projection point of any vertex in the template face mesh. The coordinates of the second centroid of the vertical projection point in the template face grid are determined as the coordinates of the first centroid.

[0123] Specifically, when determining the vertical projection point of any vertex in the template face mesh, in addition to directly using the first and second position information for calculation, the initial position information of any vertex in the face mesh deformation result under the two-dimensional texture mapping coordinate system and the fifth position information of each vertex in the template face mesh under the two-dimensional texture mapping coordinate system can also be introduced for calculation. The vertical projection point is determined based on the calculation results. Among them, the initial position information is the UV coordinates of any vertex in the face mesh deformation result in the initial state before template makeup retargeting, and the fifth position information refers to the UV coordinates of each vertex in the template face mesh, and part of the fifth position information is equal to the third position information.

[0124] Here, the vertical projection point of any vertex can be determined first using the first position information, the second position information, the initial position information, and the fifth position information. Then, the second centroid coordinates of the vertical projection point of any vertex in the template face mesh are calculated, and the second centroid coordinates are used as the first centroid coordinates corresponding to any vertex.

[0125] In this embodiment of the invention, the vertical projection point of any vertex is determined by using the first position information, each of the second position information, the initial position information, and the fifth position information. By calculating the second centroid coordinates of the vertical projection point of any vertex in the template face grid, and using the second centroid coordinates as the first centroid coordinates corresponding to any vertex, the accuracy of the first centroid coordinates can be further improved, thereby enhancing the effect of template makeup retargeting.

[0126] Based on the above embodiments, the vertical projection point of any vertex in the template face mesh is determined based on the first position information, each of the second position information, the initial position information of any vertex in the face mesh deformation result in the two-dimensional texture mapping coordinate system, and the fifth position information of each vertex in the template face mesh in the two-dimensional texture mapping coordinate system, including: Calculate the first difference between the first position information and each of the second position information, and calculate the second difference between the initial position information and each of the fifth position information; Based on the first difference and the second difference, determine the second unit grid cell in the template face grid that has the closest directed distance to any vertex, and determine the vertical projection point in the second unit grid cell.

[0127] Specifically, the first difference between the first position information and each of the second position information can be calculated separately, as well as the second difference between the initial position information and each of the fifth position information.

[0128] Subsequently, each first difference and its corresponding second difference can be weighted and summed, and the second unit grid cell formed by several vertices with the largest weighted sum in the template face grid can be selected as the second unit grid cell in the template face grid with the closest directed distance to any vertex.

[0129] Furthermore, the vertical projection point of any vertex is determined within the second unit grid cell, which is the vertical projection point of any vertex in the template face grid. The second centroid coordinates of the vertical projection point of any vertex in the second unit grid cell are then determined as the first centroid coordinates of any vertex.

[0130] In this embodiment of the invention, by calculating the first difference between the first position information and each second position information, and the second difference between the initial position information and each fifth position information, the vertical projection point of any vertex in the template face grid is determined. Then, the second centroid coordinate of the vertical projection point is used as the first centroid coordinate of any vertex. This can make full use of the differences between coordinates and the differences between UV coordinates, improve the accuracy of the vertical projection point, and thus improve the accuracy of the first centroid coordinate, thereby enhancing the effect of template makeup retargeting.

[0131] Based on the above embodiments, using the fourth position information of each vertex in the two-dimensional texture mapping coordinate system in the face mesh deformation result, the template makeup is mapped to the target face to obtain the target makeup, including: Based on the fourth position information, the first correspondence between each vertex in the face mesh deformation result and each pixel in the template two-dimensional face texture image of the template face is determined; Based on the first correspondence, a second correspondence is established between the surface of the face mesh deformation result and the template two-dimensional face texture image; Based on the second correspondence, the template makeup is mapped to the target face to obtain the target makeup.

[0132] Specifically, in the process of mapping the template makeup onto the target face, for any vertex in the face mesh deformation result, the UV coordinates of any vertex in the face mesh deformation result can be used to determine the first correspondence between each vertex in the face mesh deformation result and each pixel in the template face's two-dimensional face texture image. This first correspondence is a correspondence between discrete points.

[0133] Subsequently, using the first correspondence, a second correspondence is established between the surface of the deformed face mesh and the template 2D face texture image through interpolation calculation. This second correspondence is a continuous correspondence between the 3D mesh surface and the image.

[0134] By utilizing the second correspondence and combining the regional position of the template makeup in the template's two-dimensional face texture image, the template makeup can be mapped onto the target face to obtain the target makeup of the target face.

[0135] In this embodiment of the invention, by using UV coordinates as a medium to establish a second correspondence between the surface of the face mesh deformation result and the template two-dimensional face texture image, the mapping of the template makeup can be realized, which can ensure that the visual effect of the template makeup on the template face is the same as the visual effect of the target makeup on the target face.

[0136] Based on the above embodiments, such as Figure 6 The image shows a template makeup look and the makeup effects before and after retargeting the template makeup. Figure 6 The image on the left is a schematic diagram of the template makeup for the template face. Figure 6 The middle image is a schematic diagram of the target makeup of the target face obtained after retargeting the template makeup. Figure 6 The image on the right is a schematic diagram of a makeup look where the initial face mesh is mapped directly using the template's 2D face texture image before the makeup is redirected.

[0137] from Figure 6 The comparison in the middle shows that, Figure 6 The facial makeup in the right-hand image shows obvious stretching and distortion, while Figure 6 The visual effect of the target makeup in the middle image is similar to Figure 6 The template makeup in the left image has the same visual effect.

[0138] Based on the above embodiments, this invention also provides a method for generating a digital human, such as... Figure 7 As shown, the method includes: S21, Based on the digital face makeup retargeting method provided in the above embodiments, determine the target makeup of the target face; S22, based on the target face and the target body, determine the target digital person.

[0139] Specifically, the digital human generation method provided in this embodiment of the invention is executed by a digital human generation device, which can be configured in a 3D digital human creation platform or a 3D video generation platform. The 3D digital human creation platform or 3D video generation platform can be installed on a local computer or in the cloud. The local computer can be a computer, tablet, etc., and is not specifically limited here.

[0140] First, in step S21, the user can create a target face through the interactive interface and select the template makeup to be added to the target face. The template makeup can be configured on the template face so that the user can observe the makeup effect.

[0141] By using the target face and the template face, combined with the digital face makeup retargeting method provided in the above embodiments, the target makeup of the target face can be determined, that is, the makeup obtained by retargeting the template makeup to the target face.

[0142] Next, step S22 is executed, which stitches the target face and target body together to obtain the target digital human. The target body can be pre-defined or selected by the user. During the stitching process, key points of the target face and target body can be aligned first. For example, corresponding key points can be marked on the connecting areas of the target face and target body, such as the neck and shoulders. Then, an alignment tool is used to precisely align the key points of the target face and target body.

[0143] Then, a transition mesh is created at the junction of the target face and the target body to eliminate gaps.

[0144] In addition, a deformer modifier can be used to slightly conform the target face to the curved surface of the target body, enhancing the blending effect.

[0145] The digital human generation method provided in this embodiment of the invention first determines the target makeup of the target face based on the digital human face makeup retargeting method provided in the above embodiments; then, using the target face and the target body, the target digital human is determined. This method can achieve rapid generation of digital humans while ensuring that the display effect of the template makeup and the target makeup remains unchanged, and can also ensure the overall coordination of the obtained target digital human, thereby improving user satisfaction and user experience.

[0146] Based on the above embodiments, this invention also provides a method for generating digital human videos, such as... Figure 8 As shown, the method includes: S31, Based on the digital human generation methods provided in the above embodiments, determine the target digital human; S32, Generate a digital human video based on the target digital human.

[0147] Specifically, the digital human video generation method provided in this embodiment of the invention is executed by a digital human video generation device, which can be configured in a computer. The computer can be a local computer or a cloud computer. The local computer can be a computer, tablet, etc., and no specific limitation is made here.

[0148] First, step S31 is executed. Based on the digital human generation method provided in the above embodiments, and combined with the target face with the target makeup, the target digital human can be determined.

[0149] Next, step S32 is executed to generate a digital human video using the target digital human. For example, the target digital human can be used to replace the human figure in an existing video to obtain a digital human video.

[0150] The digital human video generation method provided in this embodiment of the invention first determines a target digital human based on the digital human generation methods provided in the above embodiments; then, it generates a digital human video using the target digital human. This method can achieve rapid generation of digital human videos while ensuring that the display effect of the target makeup and template makeup of the digital human image in the video remains unchanged. It can also ensure the overall coordination of the obtained digital human image, thereby improving user satisfaction and user experience.

[0151] Based on the above embodiments, this invention also provides a digital facial makeup repositioning device, such as... Figure 9 As shown, the device includes: Template face acquisition module 91 is used to acquire template faces corresponding to the target face; The makeup retargeting module 92 is used to retarget the template makeup of the template face based on the face mesh deformation result of the target face to obtain the target makeup of the target face.

[0152] Specifically, the functions of each module in the digital face makeup repositioning device provided in this embodiment of the invention correspond one-to-one with the operation flow of each step in the above-described digital face makeup repositioning method embodiment, and the achieved effects are also the same. For details, please refer to the above embodiments, and this will not be repeated in this embodiment of the invention.

[0153] Based on the above embodiments, this invention also provides a digital human generation device, such as... Figure 10 As shown, the device includes: The makeup determination module 101 is used to determine the target makeup of the target face based on the digital face makeup retargeting method provided in the above embodiments; The first digital human generation module 102 is used to determine the target digital human based on the target face and the target body.

[0154] Specifically, the functions of each module in the digital human generation device provided in this embodiment correspond one-to-one with the operation flow of each step in the above-described digital human generation method embodiment, and the achieved effects are also the same. Please refer to the above embodiments for details, and this will not be repeated in this embodiment.

[0155] Based on the above embodiments, this invention also provides a digital human video generation device, such as... Figure 11 As shown, the device includes: The second digital human generation module 111 is used to determine the target digital human based on the digital human generation methods provided in the above embodiments; The digital human video generation module 112 is used to generate digital human videos based on the target digital human.

[0156] Specifically, the functions of each module in the digital human video generation device provided in this embodiment of the invention correspond one-to-one with the operation flow of each step in the above-described digital human video generation method embodiment, and the achieved effects are also the same. For details, please refer to the above embodiments, and this will not be repeated in this embodiment of the invention.

[0157] Figure 12 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 12 As shown, the electronic device may include a processor 810, a communications interface 820, a memory 830, and a communication bus 840. The processor 810, communications interface 820, and memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions stored in the memory 830 to execute the digital face makeup reshaping method, digital human generation method, or digital human video generation method provided in the above embodiments.

[0158] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to related technologies, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0159] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the digital human face makeup retargeting method, the digital human generation method, or the digital human video generation method provided in the above embodiments.

[0160] In another aspect, the present invention also provides a computer-readable storage medium having a computer program stored thereon. When executed by a processor, the computer program is implemented to perform the digital face makeup reshaping method, the digital human generation method, or the digital human video generation method provided in the above embodiments. This computer-readable storage medium can be either a non-transitory computer-readable storage medium or a transient computer-readable storage medium; no specific limitation is made here.

[0161] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0162] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0163] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A digital facial makeup repositioning method, characterized in that, include: Obtain the template face corresponding to the target face; Based on the face mesh deformation result of the target face, the template makeup of the template face is retargeted to obtain the target makeup of the target face.

2. The digital facial makeup repositioning method according to claim 1, characterized in that, The process of retargeting the template makeup of the template face based on the face mesh deformation result of the target face to obtain the target makeup of the target face includes: The target face is registered with the template face to obtain the registration result; Based on the registration results, the initial face mesh of the target face is geometrically transformed by applying the face mesh deformation energy constraint to obtain the face mesh deformation result.

3. The digital facial makeup repositioning method according to claim 2, characterized in that, The registration result includes a transformation matrix; Based on the registration result, a face mesh deformation energy constraint is applied to perform a geometric transformation on the initial face mesh of the target face to obtain the face mesh deformation result, including: Based on the transformation matrix, the initial face mesh is transformed to obtain the face mesh to be deformed; Based on the deformation energy constraint of the face mesh, a deformation operation is performed on the face mesh to be deformed to obtain the deformation result of the face mesh.

4. The digital facial makeup repositioning method according to claim 3, characterized in that, The face mesh deformation energy constraint includes a constraint constructed based on the first deformation energy of the deformable unit in the face mesh to be deformed relative to the corresponding template unit in the template face mesh; Each of the deformable units and each of the template units includes a vertex, adjacent vertices of the vertex, connecting edges between the vertex and the adjacent vertices, and connecting edges between the adjacent vertices, or includes a unit mesh unit and adjacent unit mesh units of the unit mesh unit.

5. The digital facial makeup repositioning method according to claim 4, characterized in that, The deformation operation includes rotation operation and scaling operation. The rotation operation is based on rotation matrix representation, and the scaling operation is based on scaling factor representation. The calculation steps for the first deformation energy include: For any template connection edge of each unit grid cell within the template unit, a rotation operation is performed on the template connection edge based on the rotation matrix variable, and a scaling operation is performed on the template connection edge based on the scaling factor variable to obtain a deformed template connection edge. Determine the candidate connecting edge corresponding to any template connecting edge within the unit to be deformed, calculate the difference vector between the candidate connecting edge and the deformable template connecting edge, and calculate the second deformation energy of the candidate connecting edge relative to any template connecting edge based on the difference vector. The first deformation energy is calculated based on the second deformation energy corresponding to each of the candidate connection edges within the unit to be deformed.

6. The digital facial makeup repositioning method according to claim 5, characterized in that, The unit to be deformed corresponds one-to-one with the rotation matrix variable, and the unit to be deformed corresponds one-to-one with the scaling factor variable.

7. The digital facial makeup repositioning method according to claim 6, characterized in that, The template unit corresponds to a template makeup type including a preset makeup pattern, and the value of the scaling factor variable corresponding to the unit to be deformed covered by the preset makeup pattern is 1.

8. The digital facial makeup repositioning method according to claim 5, characterized in that, The calculation of the first deformation energy based on the second deformation energy corresponding to each of the candidate connection edges within the unit to be deformed includes: The first deformation energy is calculated by weighted summing of the second deformation energy corresponding to each of the candidate connection edges in the unit to be deformed. The weighting coefficients of each of the second deformation energies are determined based on the uniformity of the template face mesh.

9. The digital facial makeup repositioning method according to claim 8, characterized in that, The steps for determining the weighting coefficients for each of the second deformation energies include: Calculate the relevant angle within the unit grid cell where each of the candidate connection edges is located, and the relevant angle is used to characterize the uniformity; Based on the relevant angle, the weighting coefficient of the second deformation energy corresponding to each of the candidate connection edges is calculated.

10. The digital facial makeup repositioning method according to claim 5, characterized in that, The process of performing a deformation operation on the face mesh to be deformed based on the face mesh deformation energy constraint to obtain the face mesh deformation result includes: Based on the aforementioned face mesh deformation energy constraint, an optimization problem is constructed. The optimization problem is solved iteratively to obtain the optimal solution for the rotation matrix variable, the scaling factor variable, and the position information of each vertex in the unit to be deformed; Based on the optimal solutions of the rotation matrix variables and the scaling factor variables, the deformation operation is determined, and based on the optimal solutions of the position information of each vertex in each of the units to be deformed, the deformation result of the face mesh is determined.

11. The digital facial makeup repositioning method according to claim 2, characterized in that, The step of registering the target face with the template face to obtain the registration result includes: The target face and the template face are coarsely registered to obtain the coarse registration result. Based on the coarse registration result, the iterative nearest point algorithm is applied to perform fine registration between the target face and the template face to obtain the registration result.

12. The digital facial makeup repositioning method according to claim 11, characterized in that, The step of coarsely registering the target face with the template face to obtain the coarse registration result includes: The edge contour lines of the makeup area in the target face and the template face are determined respectively, and the feature points in the target face and the template face are determined respectively based on the facial shapes of the target face and the template face. Based on the distance between each of the edge contour lines and / or the distance between each of the feature points, the spatial position of the target face is adjusted to obtain the coarse registration result.

13. The digital facial makeup repositioning method according to any one of claims 1-12, characterized in that, The process of retargeting the template makeup of the template face based on the face mesh deformation result of the target face to obtain the target makeup of the target face includes: For any vertex in the face mesh deformation result, based on the first position information of any vertex in the face mesh deformation result and the second position information of each vertex in the template face mesh corresponding to the template face, calculate the first centroid coordinates of the face where the first unit mesh cell of any vertex is located in the template face mesh. Based on the first centroid coordinates and the third position information of each vertex in the first unit grid cell in the two-dimensional texture mapping coordinate system, calculate the fourth position information of any vertex in the two-dimensional texture mapping coordinate system; Based on the fourth position information of each vertex in the two-dimensional texture mapping coordinate system in the face mesh deformation result, the template makeup is mapped to the target face to obtain the target makeup.

14. The digital facial makeup repositioning method according to claim 13, characterized in that, The step of calculating the first centroid coordinates of the face containing the first unit mesh cell corresponding to any vertex in the face mesh, based on the first position information of any vertex in the face mesh deformation result and the second position information of each vertex in the template face mesh corresponding to the template face, includes: Based on the first position information, each of the second position information, the initial position information of any vertex in the two-dimensional texture mapping coordinate system in the face mesh deformation result, and the fifth position information of each vertex in the template face mesh in the two-dimensional texture mapping coordinate system, the vertical projection point of any vertex in the template face mesh is determined; The second centroid coordinates of the vertical projection point in the template face grid are determined as the first centroid coordinates.

15. The digital facial makeup repositioning method according to claim 14, characterized in that, The step of determining the vertical projection point of any vertex in the template face mesh based on the first position information, each of the second position information, the initial position information of any vertex in the two-dimensional texture mapping coordinate system in the face mesh deformation result, and the fifth position information of each vertex in the template face mesh in the two-dimensional texture mapping coordinate system includes: Calculate the first difference between the first location information and each of the second location information, and calculate the second difference between the initial location information and each of the fifth location information; Based on the first difference and the second difference, the second unit grid cell in the template face grid that has the closest directed distance to any vertex is determined, and the vertical projection point is determined in the second unit grid cell.

16. The digital facial makeup repositioning method according to claim 13, characterized in that, The step of mapping the template makeup onto the target face based on the fourth position information of each vertex in the two-dimensional texture mapping coordinate system in the face mesh deformation result to obtain the target makeup includes: Based on the fourth position information, a first correspondence is determined between each vertex in the face mesh deformation result and each pixel in the template two-dimensional face texture image of the template face; Based on the first correspondence, a second correspondence is established between the surface of the face mesh deformation result and the template two-dimensional face texture image; Based on the second correspondence, the template makeup is mapped to the target face to obtain the target makeup.

17. A method for generating a digital human, characterized in that, include: Based on the digital facial makeup retargeting method as described in any one of claims 1-16, the target makeup of the target face is determined; Based on the target face and target body, the target digital person is determined.

18. A method for generating digital human videos, characterized in that, include: The target digital human is determined based on the digital human generation method as described in claim 17; Based on the target digital human, a digital human video is generated.

19. A digital facial makeup repositioning device, characterized in that, include: The template face acquisition module is used to acquire the template face corresponding to the target face; The makeup retargeting module is used to retarget the template makeup of the template face based on the face mesh deformation result of the target face to obtain the target makeup of the target face.

20. A digital human generation device, characterized in that, include: A makeup determination module is used to determine the target makeup of the target face based on the digital face makeup retargeting method as described in any one of claims 1-16; The first digital human generation module is used to determine the target digital human based on the target face and the target body.

21. A digital human video generation device, characterized in that, include: The second digital human generation module is used to determine the target digital human based on the digital human generation method as described in claim 17; The digital human video generation module is used to generate digital human videos based on the target digital human.

22. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the digital human face makeup retargeting method as described in any one of claims 1-16, the digital human generation method as described in claim 17, or the digital human video generation method as described in claim 18.

23. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the digital human face makeup retargeting method as described in any one of claims 1-16, the digital human generation method as described in claim 17, or the digital human video generation method as described in claim 18.

24. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the digital human face makeup retargeting method as described in any one of claims 1-16, the digital human generation method as described in claim 17, or the digital human video generation method as described in claim 18.

Images