Five sense organ segmentation, sample generation and model training methods and devices, storage medium and program product

Abstract

The embodiment of the invention provides five sense organ segmentation, sample generation and model training methods and devices, a storage medium and a program product. In the embodiment of the invention, the topological invariance and the shape deformation of the digital human three-dimensional model are utilized, the three-dimensional rendering technology and the five-sense-organ segmentation technology are combined, the digital human and the training samples for five-sense-organ segmentation of the digital human are accurately generated, then the five-sense-organ segmentation model for the digital human is trained based on the training samples, and then the five-sense-organ segmentation model for the digital human is obtained. The five-sense-organ segmentation can be carried out on the digital person based on the trained five-sense-organ segmentation model for the digital person, the problem that the five-sense-organ segmentation model for a real human face is insufficient in generalization ability and cannot accurately carry out five-sense-organ segmentation on the digital person is solved, and the five-sense-organ segmentation precision of the digital person is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device, storage medium and program product for facial feature segmentation, sample generation and model training. Background Art

[0002] The facial features segmentation technology is of great significance in the fields of computer vision and human-computer interaction. For example, in face recognition, accurate facial features segmentation can help improve the accuracy and efficiency of recognition. For another example, in expression analysis, accurate segmentation of facial features can accurately identify different expressions, thereby achieving accurate perception of emotions. For another example, in virtual makeup applications, more natural and realistic virtual makeup effects can be achieved through fine segmentation of facial features.

[0003] With the development of digital human technology, live broadcasts and interactive games based on digital humans are becoming more and more widespread. If the facial features of digital humans can be accurately segmented, it will help improve the quality of applications based on digital humans. However, the facial features segmentation model used in existing facial features segmentation technology is trained on sample images containing real facial features, and is mainly used to segment real facial features. The generalization ability of this facial features segmentation model is insufficient. If it is directly applied to digital humans, there will be a problem of poor facial features segmentation accuracy. Summary of the invention

[0004] Multiple aspects of the present application provide a method, device, storage medium and program product for facial feature segmentation, sample generation and model training, which are used to train a facial feature segmentation model for digital humans, and then to more accurately segment the facial features of digital humans based on the model.

[0005] The embodiment of the present application provides a facial feature segmentation method, comprising: in response to a facial feature segmentation task for a target digital human, determining a three-dimensional model of the target digital human; rendering the three-dimensional model of the target digital human to obtain a target rendered image containing facial feature areas of the target digital human; performing facial feature segmentation on the target rendered image according to a first facial feature segmentation model for the digital human to obtain a target facial feature mask image corresponding to the target digital human; wherein the first facial feature segmentation model is obtained by training a second facial feature segmentation model for real human facial feature segmentation using sample rendered images and sample facial feature mask images corresponding to a plurality of sample digital humans as training samples.

[0006] The embodiment of the present application also provides a sample generation method, comprising: performing morphological changes on the three-dimensional model of a basic digital human according to the topological invariance and morphological variability of the basic digital human, so as to obtain three-dimensional models of multiple sample digital humans; rendering the three-dimensional models of the sample digital humans to obtain sample rendered images including facial feature areas of the sample digital humans, the sample rendered images also including facial feature contour information mapped from the positions of the facial feature vertices in the three-dimensional model of the basic digital human; performing facial feature segmentation on the sample rendered images according to the facial feature contour information included in the sample rendered images, so as to obtain sample facial feature mask images.

[0007] The embodiment of the present application also provides a model training method, including: generating sample rendering images and sample facial feature mask images corresponding to multiple sample digital humans as training samples; training a second facial feature segmentation model for real human facial features segmentation according to the training samples to obtain a first facial feature segmentation model for digital humans; wherein the sample rendering images and sample facial feature mask images corresponding to the multiple sample digital humans are generated using the sample generation method provided in the embodiment of the present application.

[0008] An embodiment of the present application also provides an electronic device, comprising: a memory and a processor; the memory is used to store one or more computer instructions; the processor is used to execute the one or more computer instructions to: execute the steps in the model training method, the sample generation method or the facial feature segmentation method provided in the embodiment of the present application.

[0009] An embodiment of the present application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor is enabled to implement the steps of the model training method, the sample generation method or the facial feature segmentation method provided in the embodiment of the present application.

[0010] An embodiment of the present application also provides a computer program product, including a computer program / instructions. When the computer program / instructions are executed by a processor, the processor is enabled to implement the steps of the model training method, the sample generation method or the facial feature segmentation method provided in the embodiment of the present application.

[0011] In this embodiment, the topological invariance and morphological deformability of the three-dimensional model of the digital human are utilized, combined with the three-dimensional rendering technology and the facial feature segmentation technology, to accurately generate training samples of the digital human and its facial features, and then train a facial feature segmentation model for the digital human based on these training samples. Afterwards, the facial features of the digital human can be segmented based on the trained facial feature segmentation model for the digital human, thereby solving the problem that the facial feature segmentation model for real human faces has insufficient generalization ability and cannot accurately segment the facial features of the digital human, thereby improving the accuracy of the facial feature segmentation of the digital human. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0013] Figure 1 A schematic diagram of a flow chart of a sample generation method provided by an exemplary embodiment of the present application;

[0014] Figure 2 A schematic diagram of a flow chart of a sample generation method in an actual scenario provided by an exemplary embodiment of the present application;

[0015] Figure 3 A flowchart of a model training method provided for an exemplary embodiment of the present application;

[0016] Figure 4 A schematic diagram of a model training process provided for an exemplary embodiment of the present application;

[0017] Figure 5 A schematic diagram of a flow chart of a facial features segmentation method provided by an exemplary embodiment of the present application;

[0018] Figure 6 A flowchart of a facial features segmentation method in a practical scenario provided by an exemplary embodiment of the present application;

[0019] Figure 7 A schematic diagram of the effect of facial features segmentation provided by an exemplary embodiment of the present application;

[0020] Figure 8 A schematic diagram of the structure of a facial features segmentation device provided by an exemplary embodiment of the present application;

[0021] Fig. 9 A schematic diagram of the structure of an electronic device provided as an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0023] It should be noted that, in the case where the embodiments of the present application involve user information, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse. In addition, the various models involved in this application (including but not limited to language models or large models) are in compliance with relevant laws and standards.

[0024] In the related field of facial feature segmentation, the facial feature segmentation model used is usually trained on sample images containing real facial features, and is mainly used to segment real facial features. The generalization ability of this facial feature segmentation model is insufficient. If it is directly applied to digital humans, there will be a problem of poor facial feature segmentation accuracy.

[0025] In the embodiment of the present application, firstly, by utilizing the topological invariance and morphological deformability of the three-dimensional model of the digital human, combined with the three-dimensional rendering technology and the facial features segmentation technology, a sample rendered image and a sample facial features mask image corresponding to the sample digital human are accurately generated as training samples, and the second facial features segmentation model used for real human facial features segmentation is trained to obtain the first facial features segmentation model for the digital human; then, the first facial features segmentation model is used to perform facial features segmentation on the target rendered image obtained by rendering the target digital human, so as to obtain the target facial features mask image corresponding to the target digital human. In this way, sample data of the digital human and its facial features segmentation are generated, and the first facial features segmentation model for the digital human is obtained based on the training sample training. The first facial features segmentation model can be used to perform relatively accurate facial features segmentation on the digital human, solving the problem that the facial features segmentation model used for real human faces is not generalized enough and cannot accurately perform facial features segmentation on the digital human, thereby improving the accuracy of facial features segmentation on the digital human.

[0026] Considering that the difficulty in training the first facial features segmentation model mainly lies in the generation of training samples, the embodiment of the present application provides a sample generation method to solve the problem of generating training samples and provide a basis for training the first facial features segmentation model.

[0027] Figure 1 A flow chart of a sample generation method provided by an exemplary embodiment of the present application is shown as follows: Figure 1 As shown, the method may include:

[0028] Step 11: Obtain a three-dimensional model of a basic digital human. The basic digital human has topological invariance and morphological variability. Topological invariance means that the positions of the facial features vertices in the three-dimensional model of the basic digital human are unchanged and known.

[0029] Step 12: According to the topological invariance and morphological variability of the basic digital human, the morphological changes are performed on the three-dimensional model of the basic digital human to obtain the three-dimensional models of multiple sample digital humans.

[0030] Step 13: Render the three-dimensional model of the sample digital human to obtain a sample rendered image including the facial features of the sample digital human. The sample rendered image also includes facial features contour information mapped from the positions of the facial features vertices in the three-dimensional model of the basic digital human.

[0031] Step 14: Segment the facial features of the sample rendered image according to the facial features contour information included in the sample rendered image to obtain a sample facial features mask image.

[0032] In this embodiment, the digital human is a virtual character created based on computer graphics, artificial intelligence (AI) or machine learning, etc. The basic digital human is used to derive multiple sample digital humans, which may have topological invariance and morphological variability.

[0033] Among them, topological invariance means that the positions of the facial features vertices in the three-dimensional model of the basic digital human are unchanged and known. Specifically, the three-dimensional model can have a topological structure, which refers to the basic geometric layout of the three-dimensional model, including the connection methods of vertices, edges and faces. The three-dimensional model can change its appearance through rotation, scaling or other deformation operations, but its topological structure remains unchanged. Among them, morphological variability means that the shape or appearance of the three-dimensional model of the basic digital human is changeable.

[0034] After obtaining the three-dimensional model of the basic digital human, the three-dimensional model of the basic digital human can be morphologically changed according to the topological invariance and morphological variability of the basic digital human to obtain three-dimensional models of multiple sample digital humans. The sample digital human refers to a digital human derived from the basic digital human that can be used as a training sample. The three-dimensional model of the basic digital human can be used as the three-dimensional model of the sample digital human or not, and this embodiment does not limit this.

[0035] Among them, the 3D model of the basic digital human can be changed in form by using 3D model making software without changing the topological structure of the 3D model. This embodiment does not limit the 3D model making software, and can be any software with model making function, such as Maya (a 3D animation software), Unreal (a real-time 3D creation tool) or Blender (a 3D graphics software), etc. This embodiment does not limit it. This embodiment also does not limit the way of morphological change, and can enlarge or reduce the overall 3D model, or adjust the adjustment items of the local 3D model to change the local form of the 3D model. For example, adjust the size of the nose or the width of the eyes in the local 3D model, etc.

[0036] After obtaining the 3D models of multiple sample digital humans, the 3D model of each sample digital human can be rendered, and a sample rendered image including the facial features of the sample digital human can be obtained. This embodiment does not limit the rendering method, and the 3D model of the sample digital human can be rendered using any engine with model rendering, such as Unreal Engine or Unity (a game development engine with real-time rendering function), etc. This embodiment does not limit it.

[0037] In the process of rendering the three-dimensional model of the sample digital human to obtain the sample rendered image, the positions of the facial features vertices in the three-dimensional model of the basic digital human can be mapped to the sample rendered image by using the 3D (3-Dimensional, three-dimensional)-2D (2-Dimensional, two-dimensional) mapping relationship between the three-dimensional model and the sample rendered image. Therefore, the sample rendered image can also include facial features contour information mapped from the positions of the facial features vertices in the three-dimensional model of the basic digital human; the facial features contour information is used to describe the external boundaries of the facial features.

[0038] After rendering to obtain the sample rendered image, the facial features of the sample rendered image can be segmented according to the facial features contour information included in the sample rendered image to obtain a sample facial features mask image. The sample digital human and its corresponding sample rendered image and sample facial features mask image are used as training samples to train the facial features segmentation model for the digital human. The accuracy of the training samples directly affects the accuracy of the trained facial features segmentation model.

[0039] Optionally, a binary mask (Mask) can be created for the internal area of ​​each facial feature contour according to the facial feature contour information included in the sample rendered image. The pixels of the area corresponding to the binary mask can be marked with a specified color, and the pixels of other areas are marked with other specified colors, so that the facial feature area is segmented from the sample rendered image to obtain a sample facial feature mask image. Alternatively, optionally, the sample rendered image can also be input into a pre-trained image segmentation model, so that the image segmentation model performs facial feature segmentation on the sample rendered image to obtain a sample facial feature mask image. This embodiment does not limit the image segmentation model, and it can be any model with image segmentation function, such as SAM (Segment Anything Model) or FCN (Fully Convolutional Networks), etc.

[0040] In this embodiment, based on the topological invariance and morphological variability of the basic digital human, multiple three-dimensional models of sample digital humans can be derived to render sample rendered images corresponding to the sample digital humans; facial features are segmented based on the sample rendered images to obtain sample facial features mask images. In this way, the training samples required for training the facial features segmentation model for digital humans can be generated more accurately and efficiently: sample rendered images and sample facial features mask images corresponding to each of the multiple sample digital humans.

[0041] In some optional embodiments, the aforementioned step 12 of "performing a morphological change on the three-dimensional model of the basic digital human according to the topological invariance and morphological variability of the basic digital human to obtain three-dimensional models of multiple sample digital humans" can be implemented based on the following steps:

[0042] Step 121: According to the topological invariance of the basic digital human, the facial feature vertices in the three-dimensional model of the basic digital human are marked to obtain the positions of the facial feature vertices.

[0043] Specifically, the three-dimensional model of the basic digital human can be displayed in the editing interface. The editing interface can be provided by a terminal device used for sample generation and displayed to the user. The terminal device can run the three-dimensional model making software to present the editing interface, and the user can view the three-dimensional model of the basic digital human displayed in the editing interface through other terminal devices.

[0044] The three-dimensional model of the basic digital human may include identification information of each vertex, and the identification information of the vertex may indicate the relative position of the vertex in the three-dimensional model of the digital human. Specifically, the three-dimensional model of the basic digital human has vertices and facets composed of vertices, such as triangular faces, and each vertex has unique identification information, and the identification information may indicate the relative position of the vertex. When displaying the three-dimensional model of the basic digital human, the identification information of each vertex may be displayed together. Among them, the identification information may be a digital number, such as 1, 2, 3 and 4, or an alphabetical number, such as a, b, c and d, etc., which is not limited in this embodiment. Afterwards, the user may perform a vertex marking operation to mark the vertices corresponding to the facial features and the facial features information to which they belong from multiple vertices, such as marking the vertices belonging to the nose, eyes or eyebrows, etc. Correspondingly, in response to the vertex marking operation, the identification information of the marked vertex and the facial features information to which the marked vertex belongs may be obtained as marking information for the position of the facial features vertex.

[0045] Since the 3D model of the basic digital human has topological invariance, the relative position of the vertex with other vertices can be determined by marking the identification information of any vertex in the 3D model. Based on the feature of topological invariance, the vertex corresponding to the facial features can be marked from multiple vertices through the vertex marking operation. The vertex marking operation can be a manual marking operation or it can be automatically completed by the 3D model making software according to the marking description information, and there is no limitation on this.

[0046] Step 122: According to the morphological variability of the basic digital human, the values ​​of the adjustment items for adjusting the morphology of the facial features of the basic digital human are generated multiple times.

[0047] The adjustment item refers to information items used to adjust the facial features of the basic digital person, such as eye size, eye corner position, eye shape, eyebrow type, mouth shape, etc. The facial features of the basic digital person will be different depending on the value of the adjustment item. The number of adjustment items can be one or more, and preferably, the number of adjustment items can be multiple.

[0048] In the case of multiple adjustment items, the target distribution that the values ​​of the multiple adjustment items obey can be determined first, and the target distribution can include at least one of uniform distribution and Gaussian distribution. In other words, the values ​​of the adjustment items used by the multiple sample digital persons can all obey Gaussian distribution, or the values ​​of the adjustment items used by the multiple sample digital persons can all obey uniform distribution, or, among the multiple sample digital persons, the values ​​of the adjustment items used by some sample digital persons obey Gaussian distribution, and the values ​​of the adjustment items used by another part of the sample digital persons obey uniform distribution.

[0049] For example, assuming that there are 100 adjustment items, 10,000 sample digital people can be generated by adjusting the basic digital person with the 100 adjustment items 10,000 times. Among these 10,000 sample digital people, the 100 adjustment items used by each sample digital person all obey Gaussian distribution, or the 100 adjustment items used by each sample digital person all obey uniform distribution, or the 100 adjustment items used by 6,000 digital people obey Gaussian distribution, and the 100 adjustment items used by the other 4,000 digital people obey uniform distribution.

[0050] Among them, uniform distribution and Gaussian distribution are two continuous probability distributions. In uniform distribution, the probability of all results occurring is equal, while in Gaussian distribution, they will obey a specific mean u and standard deviation σ. The characteristic of uniform sampling is that the probability of each value occurring is equal, but since the range of adjustment of the digital person is large, undesirable adjustment results will occur. The characteristic of Gaussian sampling is that the value of the adjustment item can appear more between 0 and the standard deviation σ, but it is not easy to have extreme values. Preferably, the target distribution may include uniform distribution and Gaussian distribution, that is, among multiple sample digital persons, the values ​​of multiple adjustment items used by some sample digital persons may obey Gaussian distribution, and the values ​​of multiple adjustment items used by another part of the sample digital persons may obey uniform distribution; for example, the values ​​of multiple adjustment items used by 60% of the sample digital persons conform to Gaussian distribution, and the values ​​of multiple adjustment items used by 40% of the sample digital persons conform to uniform distribution.

[0051] Based on the above analysis, when determining the target distribution that the values ​​of multiple adjustment items obey, the target distribution that the values ​​of multiple adjustment items obey can be determined according to the preset target distribution rules. The target distribution rules are used to determine the proportion of sample digital people whose values ​​of the adjustment items conform to the uniform distribution, and the proportion of sample digital people whose values ​​of the adjustment items conform to the Gaussian distribution. Further, after determining the target distribution that the values ​​of multiple adjustment items obey, the values ​​of multiple adjustment items can be generated each time according to the morphological variability and sample quantity of the basic digital person, with the goal of the values ​​of multiple adjustment items obeying the uniform distribution or Gaussian distribution. Among them, the number of samples of digital people determines the number of times generated. That is to say, each time multiple adjustment items are adjusted, a sample digital person can be generated, and each time the adjustment is made, the values ​​of multiple adjustment items either obey the uniform distribution or obey the Gaussian distribution. For example, assuming that the number of multiple adjustment items is 100, each time 100 adjustment items are adjusted, a sample digital person can be generated, and each time the 100 adjustment items are adjusted, the 100 adjustment items either obey the uniform distribution or obey the Gaussian distribution.

[0052] Among them, the number of samples determines the number of generations, which can be understood as: the number of samples is used to determine the number of generations. For example, when the required number of samples is 10,000, it means that 10,000 sample digital people need to be generated, and the corresponding number of generations is 10,000.

[0053] Specifically, when the target distribution includes only uniform distribution, the values ​​of multiple adjustment items can be generated based on the morphological variability of the basic digital human and the goal is that the values ​​of multiple adjustment items obey uniform distribution. When the target distribution includes only Gaussian distribution, the values ​​of multiple adjustment items can be generated based on the morphological variability of the basic digital human and the goal is that the values ​​of multiple adjustment items obey Gaussian distribution.

[0054] In the case where the target distribution includes uniform distribution and Gaussian distribution, the first number and the second number can be determined according to the number of samples, and the sum of the first number and the second number is equal to the number of times generated. That is, according to the number of training samples required, the first number and the second number can be set to any value while ensuring that the sum of the first number and the second number is equal to the number of times generated, and this embodiment does not impose any restrictions. For example, if the number of samples is 10,000, that is, the number of times generated can be 10,000, then the first number can be 4,000, and the second number can be 6,000.

[0055] After determining the first and second quantities based on the number of samples, based on the morphological variability of the basic digital human, in the first number of generation processes, the values ​​of the multiple adjustment items can be generated with the goal of making the values ​​of the multiple adjustment items obey a uniform distribution; in the second number of generation processes, the values ​​of the multiple adjustment items can be generated with the goal of making the values ​​of the multiple adjustment items obey a Gaussian distribution.

[0056] In this way, the values ​​of multiple adjustment items are generated with the goal of obeying uniform distribution or Gaussian distribution, which can effectively avoid the shortcomings of uniform distribution and Gaussian distribution, making the values ​​of multiple adjustment items generated finally more accurate and reasonable. After generating the values ​​of multiple adjustment items multiple times, the following steps 123 can be continued to generate the three-dimensional model of the sample digital human.

[0057] Step 123: According to the values ​​of the adjustment items generated multiple times, the 3D model of the basic digital human is repeatedly transformed to obtain the 3D models of multiple sample digital humans. That is, each sample digital human is obtained by adjusting the adjustment items on the basis of the basic digital human.

[0058] Among them, the target adjustment direction and the extreme value vertex matrix corresponding to the basic digital human in the target adjustment direction can be determined according to the value of the adjustment item generated each time, and the three-dimensional model of the basic digital human corresponds to the basic vertex matrix.

[0059] Among them, the adjustment item has an adjustable value range, for example, it can be but not limited to: [-1, +1]. According to the adjustable value range of the adjustment item, the adjustment direction of the adjustment item can be defined, including the positive adjustment direction of adjusting to the maximum positive value direction, which can be recorded as the +1 direction, and the negative adjustment direction of adjusting to the minimum negative value direction, which can be recorded as the -1 direction. Based on this, the target adjustment direction can be a positive adjustment direction or a negative adjustment direction. In this embodiment, when the value of each adjustment item is 0, the matrix formed by the position coordinates of each vertex in the basic digital human is called the basic vertex matrix; when the value of each adjustment item is 1 or -1, the position coordinates of each vertex in the basic digital human are adjusted to obtain new position coordinates, and the matrix formed by these new position coordinates is called the extreme value vertex matrix; further, the extreme value vertex matrix can be divided into a positive extreme value vertex matrix and a negative extreme value vertex matrix. Among them, the extreme value vertex matrix can be used to describe the vertex coordinates of the three-dimensional model when each adjustment item takes different extreme values ​​(for example, 1 or -1). The extreme value vertex matrix can be stored in the memory in a pre-exported manner. For example, the three-dimensional model corresponding to each extreme value vertex matrix can be pre-exported in the obj format model file in advance, and the pre-exported obj format model file can contain the extreme value vertex matrix. Among them, obj is a commonly used 3D model file format, mainly used to store geometric information. Among them, the basic vertex matrix corresponding to the three-dimensional model of the basic digital human can be used to describe the vertex coordinates of the three-dimensional model when each adjustment item is 0.

[0060] Afterwards, according to the linear change relationship between the adjustment item and the extreme vertex matrix and the basic vertex matrix, the morphological change of the three-dimensional model of the basic digital human is performed to obtain the three-dimensional model of the sample digital human. Specifically, the vertex matrix offset can be generated according to the value of the adjustment item, the extreme vertex matrix, the basic vertex matrix and the linear change relationship. This embodiment does not limit the linear change relationship. In some exemplary embodiments, the linear change relationship can be expressed based on the following formulas:

[0061] (N,3)1-(N,3)0=2*((N,3) 0.5 -(N,3)0) (Formula 1)

[0062] The coordinates of all vertices of the basic digital human model are represented as a matrix of (N, 3), where N is the number of vertices and 3 is the coordinate dimension of the vertex. (N, 3)1 is the extreme vertex matrix when the adjustment item is 1, (N, 3)0 is the basic vertex matrix when the adjustment item is 0, and (N, 3) 0.5 It is the intermediate extreme value vertex matrix when the adjustment item value is 0.5, that is, the vertex matrix after deformation.

[0063] The above formula 1 can be further transformed into the following formula 2:

[0064] (N, 3) p -(N,3)0=1 / p*((N,3) p -(N,3)0) (Formula 2)

[0065] Where p is the value of the adjustment item, (N, 3) p It refers to the intermediate vertex matrix when the adjustment item is p, that is, the vertex matrix after deformation. Among them, p*((N, 3) p -(N, 3)0) is the vertex matrix offset corresponding to the adjustment item value p.

[0066] When p=0.8, the above formula 2 can be further expressed as:

[0067] (N, 3) 0.8 -(N,3)0=1 / 0.8*((N,3) 0.8 -(N,3)0) (Formula 3)

[0068] Among them, (N, 3) 0.8 Refers to the intermediate vertex matrix when the adjustment item value p is 0.8, 0.8*((N, 3) 0.8 -(N, 3)0) is the vertex matrix offset corresponding to the adjustment item value of 0.8.

[0069] Through the above formulas, according to the value of the adjustment item, the extreme vertex matrix, the basic vertex matrix and the linear change relationship, after the vertex matrix offset is generated, the basic vertex matrix and the vertex matrix offset can be superimposed to obtain the deformed vertex matrix. The deformed vertex matrix can be used to describe the three-dimensional model of the sample digital human. Taking the adjustment item value p as 0.8 as an example, the deformed vertex matrix can be expressed as (N, 3)0+0.8*((N, 3) 0.8 -(N,3)0).

[0070] It should be noted that the vertex matrix involved above is in the world coordinate system. Therefore, the deformed vertex matrix can be transformed from the world coordinate system to the image coordinate system by projection, so that the transformed vertex matrix can more accurately describe the three-dimensional model of the sample digital human.

[0071] In this way, the three-dimensional model of the basic digital human can be morphologically changed multiple times according to the values ​​of the adjustment items generated multiple times, so as to obtain the three-dimensional models of multiple sample digital humans more accurately and efficiently.

[0072] In some optional embodiments, the aforementioned step 14 of "segmenting the sample rendered image according to the facial features contour information included in the sample rendered image to obtain a sample facial features mask image" can be implemented based on the following implementations:

[0073] Embodiment 1: Based on the facial features contour information included in the sample rendered image, the filling function in the computer vision library is called to perform facial features segmentation on the sample rendered image to obtain a sample facial features mask image. The filling function is used to perform region filling.

[0074] Embodiment 2: Based on the facial contour information included in the sample rendered image, the third facial features segmentation model in the computer vision library is called to perform facial features segmentation on the sample rendered image to obtain a sample facial features mask image. The third facial features segmentation model is a model used for image segmentation tasks, which is not limited to any model that can perform image segmentation tasks.

[0075] Implementation method three: based on the facial feature contour information included in the sample rendered image, call the filling function and the third facial feature segmentation model in the computer vision library to perform facial feature segmentation on the sample rendered image to obtain a sample facial feature mask image.

[0076] The facial features of the digital human can be divided into different categories, and different facial features segmentation methods are used to segment the facial features of different categories, so as to more accurately segment the facial features of the sample rendering image. This embodiment does not limit the specific method of segmenting the facial features. In an exemplary embodiment, the facial features can be divided into: a first category of facial features suitable for facial features segmentation using a filling function in a computer vision library and a second category of facial features suitable for facial features segmentation using a third facial features segmentation model. Exemplarily, the first category of facial features may include ears, nose, and mouth, and the second category of facial features may include eyes and eyebrows. The examples of the two categories of facial features here are only examples.

[0077] Specifically, according to the facial contour information of the first type of facial features in the sample rendered image, the filling function can be called to fill the image area surrounded by the facial contour information of the first type of facial features to obtain the mask area of ​​the first type of facial features. Among them, the filling function can be called to set the pixels in the image area surrounded by the facial contour information of the first type of facial features to a specific value to obtain the mask area of ​​the first type of facial features. Among them, the filling function can be called from the computer vision library. This embodiment does not limit the specific implementation of the computer vision library and the filling function. In an exemplary embodiment, the computer vision library can be OpenCV (Open Source Computer Vision Library, an open source computer vision library) or Dlib (a C++ toolkit containing machine learning algorithms), etc. The filling function can be FillPoly (a filling function for filling polygonal areas), CopyMakeBorder (a filling function) or Pillow (PIL, a filling function), etc., which are not limited in this embodiment.

[0078] Preferably, FillPoly can be called from the computer vision library OpenCV. FillPoly can fill the area surrounded by the facial feature contours according to the facial feature contour information of the first category of facial features in the sample rendered image. In this way, the mask area of ​​the first category of facial features can be obtained more efficiently, and the edge area of ​​the mask area is also relatively smooth.

[0079] In addition, according to the facial contour information of the second type of facial features in the sample rendered image, the third facial features segmentation model can be called to perform mask processing on the image area surrounded by the facial contour information of the second type of facial features to obtain the masked area of ​​the second type of facial features. Among them, the third facial features segmentation model can be any model with image segmentation function, such as SAM segmentation tool or FCN, etc., which is not limited in this embodiment. It should be noted that the embodiment of the present application does not limit the order of the above-mentioned two types of facial features segmentation process, which can be executed sequentially or in parallel.

[0080] After obtaining the mask areas of the first and second facial features based on the above content, a sample facial features mask image can be generated based on the mask areas of the first and second facial features. The mask areas of the first and second facial features can be overlaid on the sample rendered image to obtain the sample facial features mask image. In this way, facial features contour information can be divided into different categories, and different facial features segmentation methods can be used for facial features contour information of different categories, so as to more accurately segment the facial features of the sample rendered image.

[0081] In the aforementioned embodiment, the generation process of training samples for training the facial features segmentation model of digital humans is described in detail. A specific implementation method is given below. In this specific implementation method, auxiliary tools such as 3D model making tools (such as Maya or Blender or UE), 3D model rendering engines (such as Unreal rendering engines), filling functions in computer vision libraries (such as FillPoly) and image segmentation models (such as SAM) are pre-deployed. These tools can be deployed locally on the terminal device used to generate training samples, or can be deployed on the server side (such as a cloud server), without limitation.

[0082] The following will refer to Figure 2 , the sample generation method of the embodiment of the present application is further explained in combination with actual scenarios. Figure 2As shown, a 3D model of a basic digital human can be obtained and input into Maya. Maya can mark the vertices of the facial features in the 3D model based on the topological invariance of the 3D model to obtain the positions of the vertices of the facial features. Specifically, a development tool (such as a script text) can be used to run the script file in Maya to display the 3D model. The 3D model can carry identification information, that is, the number of each vertex, and the corresponding relationship between the vertex number and the facial features is preset. Based on this, it can be manually marked by the dotting tool provided by Maya. Dotting can be understood as the vertex marking operation in the aforementioned embodiment.

[0083] Correspondingly, in response to the vertex marking operation, the identification information of the marked vertex and the facial features information to which the marked vertex belongs can be obtained as marking information for the position of the facial features vertex. That is, the dotted vertices will be recorded, and the numbers of the vertices included in each facial features area such as the eye area and the nose area can be recorded.

[0084] Furthermore, the morphology of the three-dimensional model of the basic digital human can be changed according to the morphology variability of the basic digital human, so as to obtain the three-dimensional models of multiple sample digital humans. Specifically, the values ​​of multiple adjustment items can be randomly generated multiple times by uniform sampling or Gaussian sampling, and the morphology of the three-dimensional model of the basic digital human can be changed multiple times according to the values ​​of the adjustment items generated multiple times, so as to obtain the three-dimensional models of multiple sample digital humans.

[0085] When the three-dimensional model of the sample digital person is obtained, the three-dimensional model of the sample digital person can be rendered to obtain a sample rendered image including the facial features of the sample digital person. Specifically, the Unreal rendering engine can be called to load the three-dimensional model of the sample digital person, project the three-dimensional model of the sample digital person onto a two-dimensional image (i.e., Unreal screenshot), and record the projection result coordinates of the facial features vertices, thereby accurately generating mask data for facial features segmentation, i.e., the sample facial features mask image. Among them, when recording the projection result coordinates of the facial features vertices, the key point information of the three-dimensional model of the sample digital person can be extracted through the facial key point algorithm, and then the facial features contour information is determined based on the key points corresponding to the facial features, thereby obtaining the projection result coordinates of the facial features vertices. Among them, the facial key point algorithm can be Dlib (a facial key point detection algorithm), or DeepLearning Methods (a facial key point detection algorithm), etc., which are not limited in this embodiment.

[0086] It is explained here that each time a sample digital human is generated, the Unreal rendering engine may be called once to load the three-dimensional model of the sample digital human, and the three-dimensional model of the sample digital human may be projected onto a two-dimensional image; alternatively, after all sample digital humans are generated, the Unreal rendering engine may be called once to load the three-dimensional models of all sample digital humans, and the three-dimensional models of all sample digital humans may be projected onto a two-dimensional image; alternatively, after some sample digital humans are generated, the Unreal rendering engine may be called once to load the three-dimensional models of the said part of sample digital humans, and the three-dimensional models of the said part of sample digital humans may be projected onto a two-dimensional image; there is no limitation on this.

[0087] Furthermore, the filling function of the Opencv computer vision library can be combined to smooth the marked eyes and eyebrows and generate corresponding masks; combined with SAM, the eyebrows and eyes can be algorithmically segmented on the sample facial mask image and the corresponding masks can be generated, thereby reducing the labor cost caused by manual labeling.

[0088] Based on the above content, the generated sample digital human, sample rendering image and sample facial mask image can be used as training samples for training facial segmentation model. Figure 2 The architecture diagram shown is only an example and does not limit the tools, engines or software used in each step.

[0089] On the basis of generating training samples, the embodiment of the present application further provides a model training method, which may include: using the sample generation method in the above embodiment to generate a plurality of sample rendering images and sample facial mask images corresponding to sample digital humans as training samples. Then, the second facial segmentation model for real facial segmentation is trained according to the training samples to obtain the first facial segmentation model for digital humans. Specifically, Figure 3 As shown, the model training method includes:

[0090] Step 31: Obtain a three-dimensional model of a basic digital human. The basic digital human has topological invariance and morphological variability. The topological invariance means that the positions of the facial features vertices in the three-dimensional model of the basic digital human are unchanged and known.

[0091] Step 32: According to the topological invariance and morphological variability of the basic digital human, the morphological changes are performed on the three-dimensional model of the basic digital human to obtain the three-dimensional models of multiple sample digital humans.

[0092] Step 33: Render the three-dimensional model of the sample digital human to obtain a sample rendered image including the facial features of the sample digital human. The sample rendered image also includes facial features contour information mapped from the positions of the facial features vertices in the three-dimensional model of the basic digital human.

[0093] Step 34: Segment the facial features of the sample rendered image according to the facial features contour information included in the sample rendered image to obtain a sample facial features mask image.

[0094] Step 35: Use the sample rendering images and sample facial feature mask images corresponding to the multiple sample digital humans as training samples, and train the second facial feature segmentation model for real human facial features according to the training samples to obtain the first facial feature segmentation model for the digital humans.

[0095] The second facial features segmentation model is a model for segmenting facial features of real people. The second facial features segmentation model is trained in a vertical field by using training samples corresponding to the digital human, so that the trained model has the ability to segment facial features of the digital human, and the trained model is called the first facial features segmentation model for the digital human. The second facial features segmentation model performs facial features segmentation on the sample rendered image to obtain a facial features mask image (referred to as a predicted facial features mask image) in the model training process corresponding to the digital human; specifically, the rendered image can be input into the second facial features segmentation model, the image features corresponding to the rendered image can be extracted, and the facial features area analysis and facial features boundary detection can be performed on the image features respectively to obtain a facial features area segmentation map and a facial features boundary segmentation map; the facial features area segmentation map and the facial features boundary segmentation map are fused to obtain a predicted facial features mask image. The second facial features segmentation model can extract image features through its backbone network, and the backbone network can be ResNet (residual network) or VGG (Visual Geometry Group, a convolutional neural network), etc., which is not limited in this embodiment.

[0096] The extracted image features can be respectively passed through the two branch networks in the second facial feature segmentation model, namely the facial parsing branch network and the boundary detection branch network. The facial parsing branch network is used to perform facial feature region parsing on the image features to obtain a facial feature region segmentation map, and the boundary detection branch network is used to perform facial feature boundary detection on the image features to obtain a facial feature boundary segmentation map. Based on the above steps, the facial feature region segmentation map and the facial feature boundary segmentation map can be fused to obtain a predicted facial feature mask image. In other words, the second facial feature segmentation model separates the two tasks of facial parsing and boundary detection, but the output results of the two branch networks can be fused during reasoning to obtain a more accurate facial parsing effect. Among them, the facial feature boundary segmentation map output by the boundary detection branch network can more effectively improve the boundaries between facial regions, making the final facial feature segmentation result more accurate.

[0097] The detailed description of steps 31-34 can be found in the above embodiment, which will not be repeated here. This embodiment focuses on step 35, that is, the process of training the second facial features segmentation model for real facial features segmentation based on the training samples. Figure 4The above model training process is described in detail.

[0098] like Figure 4 As shown, when the second facial feature segmentation model for real human facial feature segmentation is trained according to the training samples, the training samples can be input into the second facial feature segmentation model, and under the supervision of the sample facial feature mask image, the second facial feature segmentation model is iteratively trained with the goal of converging the loss function between the predicted facial feature mask image output by the second facial feature segmentation model and the sample facial feature mask image within a specified range, that is, the model parameters of the second facial feature segmentation model are optimized in each training iteration round; until the loss function converges within the specified range, the training is stopped and the first facial feature segmentation model for the digital human is output.

[0099] Among them, the loss function can be used to evaluate the error between the predicted facial feature mask image and the sample facial feature mask image. The embodiment of the present application does not limit the specific implementation method of the loss function. It can be a mean square error (MSE) function or a mean absolute error (MAE) function, etc. This embodiment does not impose any restrictions.

[0100] In this embodiment, the sample generation method in the aforementioned embodiment can be adopted, taking advantage of the topological invariance and morphological deformability of the digital human three-dimensional model, combined with three-dimensional rendering technology and facial features segmentation technology, to obtain training samples more accurately and ensure the effect of model training.

[0101] The detailed implementation and beneficial effects of each step in the method of this embodiment have been described in detail in the aforementioned embodiments and will not be elaborated here.

[0102] Based on the first facial features segmentation model obtained through training, the embodiment of the present application further provides a facial features segmentation method, so as to use the first facial features segmentation model to segment the facial features of a digital human more accurately and efficiently, which will be described in detail below with reference to the accompanying drawings. Figure 5 As shown, the facial features segmentation method may include the following steps:

[0103] Step 51: In response to the facial features segmentation task for the target digital human, a three-dimensional model of the target digital human is determined.

[0104] Step 52: Render the three-dimensional model of the target digital human to obtain a target rendered image including the facial features of the target digital human.

[0105] Step 53: Perform facial feature segmentation on the target rendered image according to the first facial feature segmentation model for digital humans to obtain a target facial feature mask image corresponding to the target digital human; wherein the first facial feature segmentation model is obtained by training a second facial feature segmentation model for real human facial feature segmentation using sample rendered images and sample facial feature mask images corresponding to a plurality of sample digital humans as training samples.

[0106] In order to facilitate the understanding of the above steps, the above steps will be explained in combination with actual scenarios. Figure 6 As shown, the task issuing interface can be displayed to the user. The task issuing interface can include an upload window for the three-dimensional model of the target digital human and a model issuing control. The user can upload the three-dimensional model of the target digital human through the upload window and click the model issuing control to issue a facial feature segmentation task for the target digital human to the cloud server. Correspondingly, the cloud server can respond to the facial feature segmentation task, obtain the three-dimensional model of the target digital human corresponding to the facial feature segmentation task, and execute the above steps 51-53 to obtain the target facial feature mask image corresponding to the target digital human, and return it to the terminal device. Among them, the three-dimensional model of the target digital human refers to the three-dimensional model of the digital human who needs to perform facial feature segmentation. In this embodiment, the model rendering method of the target digital human is not limited. The three-dimensional model of the sample digital human can be rendered using any engine with model rendering, such as the Unreal engine or Unity, etc., which is not limited in this embodiment. It should be noted that Figure 6 The content shown is only an exemplary description and an optional solution, and does not constitute any limitation to the solution architecture of this embodiment.

[0107] In this way, the first facial feature segmentation model can be trained more accurately based on the model training method in the aforementioned embodiment, and then the first facial feature segmentation model can be used to more accurately segment the facial features of the target rendered image to obtain the target facial feature mask image corresponding to the target digital person.

[0108] In some optional embodiments, a three-dimensional model of a basic digital human can also be obtained. The basic digital human has topological invariance and morphological variability. The topological invariance means that the positions of the facial feature vertices in the three-dimensional model of the basic digital human are unchanged and known; based on the topological invariance and morphological variability of the basic digital human, the three-dimensional model of the basic digital human is morphologically changed to obtain three-dimensional models of multiple sample digital humans; the three-dimensional models of the sample digital humans are rendered to obtain sample rendered images including facial feature areas of the sample digital humans, the sample rendered images also including facial feature contour information mapped by the positions of the facial feature vertices in the three-dimensional model of the basic digital human; based on the facial feature contour information included in the sample rendered image, the sample rendered image is segmented to obtain a sample facial feature mask image.

[0109] In some optional embodiments, according to the topological invariance and morphological variability of the basic digital human, the three-dimensional model of the basic digital human is morphologically changed to obtain the three-dimensional models of multiple sample digital humans, including: according to the topological invariance of the basic digital human, the vertices of the facial features in the three-dimensional model of the basic digital human are marked to obtain the positions of the facial features vertices; according to the morphological variability of the basic digital human, the values ​​of the adjustment items for adjusting the facial features of the basic digital human are generated multiple times; according to the values ​​of the adjustment items generated multiple times, the three-dimensional model of the basic digital human is morphologically changed multiple times to obtain the three-dimensional models of multiple sample digital humans.

[0110] In some optional embodiments, based on the topological invariance of the basic digital human, the facial feature vertices in the three-dimensional model of the basic digital human are marked to obtain the positions of the facial feature vertices, including: displaying the three-dimensional model of the basic digital human in an editing interface, the three-dimensional model of the basic digital human including identification information of each vertex, the identification information of the vertex indicating the position of the vertex; in response to the vertex marking operation, obtaining the identification information of the marked vertex and the facial feature information to which the marked vertex belongs as marking information for the positions of the facial feature vertices.

[0111] In some optional embodiments, based on the morphological variability of the basic digital human, values ​​of adjustment items for adjusting the facial features of the basic digital human are generated multiple times, including: in the case of multiple adjustment items, determining the target distribution that the values ​​of the multiple adjustment items obey, the target distribution includes uniform distribution and / or Gaussian distribution; based on the morphological variability of the basic digital human and the number of samples, each time with the goal of the values ​​of the multiple adjustment items obeying uniform distribution or Gaussian distribution, generating multiple values ​​of the adjustment items, and the number of samples determines the number of generations.

[0112] In some optional embodiments, based on the morphological variability of the basic digital human and the number of samples, multiple values ​​of the adjustment items are generated each time with the goal of the values ​​of the multiple adjustment items obeying uniform distribution or Gaussian distribution, including: when the target distribution includes uniform distribution and Gaussian distribution, determining a first number and a second number based on the number of samples, the sum of the first number and the second number being equal to the number of generations; based on the morphological variability of the basic digital human, in a first number of generation processes, with the goal of the values ​​of the multiple adjustment items obeying uniform distribution, multiple values ​​of the adjustment items are generated; and in a second number of generation processes, with the goal of the values ​​of the multiple adjustment items obeying Gaussian distribution, multiple values ​​of the adjustment items are generated.

[0113] In some optional embodiments, the three-dimensional model of the basic digital human is morphologically changed multiple times according to the values ​​of the adjustment items generated multiple times to obtain the three-dimensional models of multiple sample digital humans, including: determining the target adjustment direction and the extreme vertex matrix corresponding to the basic digital human in the target adjustment direction for the value of the adjustment item generated each time, and the three-dimensional model of the basic digital human corresponds to the basic vertex matrix; according to the linear change relationship between the adjustment item and the extreme vertex matrix and the basic vertex matrix, the three-dimensional model of the basic digital human is morphologically changed to obtain the three-dimensional model of the sample digital human.

[0114] In some optional embodiments, the three-dimensional model of the basic digital human is morphologically changed according to the linear change relationship between the adjustment item and the extreme vertex matrix and the basic vertex matrix to obtain the three-dimensional model of the sample digital human, including: generating a vertex matrix offset according to the value of the adjustment item, the extreme vertex matrix, the basic vertex matrix and the linear change relationship; superimposing the basic vertex matrix and the vertex matrix offset to obtain a deformed vertex matrix, and the deformed vertex matrix is ​​used to describe the three-dimensional model of the sample digital human.

[0115] In some optional embodiments, the sample rendered image is segmented for facial features according to the facial feature contour information included in the sample rendered image to obtain a sample facial feature mask image, including: calling a filling function and / or a third facial feature segmentation model in a computer vision library according to the facial feature contour information included in the sample rendered image, and performing facial feature segmentation on the sample rendered image to obtain a sample facial feature mask image; wherein the third facial feature segmentation model is a model for image segmentation tasks, and the filling function is used for area filling.

[0116] In some optional embodiments, based on the facial feature contour information included in the sample rendered image, a filling function and a third facial feature segmentation model in a computer vision library are called to perform facial feature segmentation on the sample rendered image to obtain a sample facial feature mask image, including: based on the facial feature contour information of the first category of facial features in the sample rendered image, a filling function is called to fill the image area surrounded by the facial feature contour information of the first category of facial features to obtain the mask area of ​​the first category of facial features; based on the facial feature contour information of the second category of facial features in the sample rendered image, the third facial feature segmentation model is called to mask the image area surrounded by the facial feature contour information of the second category of facial features to obtain the mask area of ​​the second category of facial features; and a sample facial feature mask image is generated based on the mask area of ​​the first category of facial features and the mask area of ​​the second category of facial features.

[0117] In some optional embodiments, facial features of a target rendered image are segmented according to a first facial feature segmentation model for a digital human to obtain a target facial feature mask image corresponding to the target digital human, including: inputting the target rendered image into the first facial feature segmentation model, extracting image features corresponding to the target rendered image, performing facial feature region analysis and facial feature boundary detection on the image features respectively to obtain a facial feature region segmentation map and a facial feature boundary segmentation map; and fusing the facial feature region segmentation map and the facial feature boundary segmentation map to obtain a target facial feature mask image.

[0118] Among them, the first facial feature segmentation model can extract image features corresponding to the target rendered image, and extract image features through the backbone network in the first facial feature segmentation model. The backbone network can be ResNet (residual network) or VGG (Visual Geometry Group, a convolutional neural network), etc., which is not limited in this embodiment.

[0119] The extracted image features can be passed through two branch networks respectively, namely the facial parsing branch network and the boundary detection branch network. The facial parsing branch network is used to analyze the facial features of the image features to obtain a facial feature region segmentation map, and the boundary detection branch network is used to detect the facial features boundaries of the image features to obtain a facial feature boundary segmentation map.

[0120] Based on the above steps, the facial feature region segmentation map and the facial feature boundary segmentation map can be fused to obtain the target facial feature mask image. In other words, the first facial feature segmentation model separates the two tasks of facial analysis and boundary detection, but the output results of the two branch networks can be fused during inference to obtain a more accurate facial analysis effect. Among them, the facial feature boundary segmentation map output by the boundary detection branch network can effectively improve the boundaries between facial regions, making the final facial feature segmentation result more accurate.

[0121] Figure 7 This is the actual effect diagram of facial features segmentation, such as Figure 7 As shown, the above facial features segmentation method can be used to efficiently and accurately segment the target facial features mask image corresponding to the target digital human based on the three-dimensional model of the target digital human.

[0122] The detailed implementation and beneficial effects of each step in the method of this embodiment have been described in detail in the aforementioned embodiments and will not be elaborated here.

[0123] It should be noted that the execution subject of each step of the method provided in the above embodiment can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 11 to 13 can be device A; for another example, the execution subject of steps 11 and 12 can be device A, and the execution subject of step 13 can be device B; and so on.

[0124] In addition, in some of the processes described in the above embodiments and the accompanying drawings, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel, and the sequence numbers of the operations, such as 11, 12, etc., are only used to distinguish between different operations, and the sequence numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0125] Figure 8 FIG. 1 is a schematic diagram of a structure of a facial features segmentation device provided by another exemplary embodiment of the present application. Figure 8 As shown, the device comprises:

[0126] The determination module 801 is used to respond to the facial feature segmentation task of the target digital human and determine the three-dimensional model of the target digital human; the rendering module 802 is used to render the three-dimensional model of the target digital human to obtain a target rendered image containing the facial feature areas of the target digital human; the segmentation module 803 is used to perform facial feature segmentation on the target rendered image according to a first facial feature segmentation model for the digital human to obtain a target facial feature mask image corresponding to the target digital human; wherein the first facial feature segmentation model is obtained by training a second facial feature segmentation model for real human facial feature segmentation using sample rendered images and sample facial feature mask images corresponding to a plurality of sample digital humans as training samples.

[0127] Further optionally, the determination module 801 is also used to: obtain a three-dimensional model of a basic digital human, the basic digital human having topological invariance and morphological variability, the topological invariance indicating that the positions of the facial feature vertices in the three-dimensional model of the basic digital human are unchanged and known; perform morphological changes on the three-dimensional model of the basic digital human according to the topological invariance and morphological variability of the basic digital human, so as to obtain three-dimensional models of multiple sample digital humans; render the three-dimensional models of the sample digital humans to obtain sample rendered images including facial feature areas of the sample digital humans, the sample rendered images also including facial feature contour information mapped by the positions of the facial feature vertices in the three-dimensional model of the basic digital human; perform facial feature segmentation on the sample rendered image according to the facial feature contour information included in the sample rendered image, so as to obtain a sample facial feature mask image.

[0128] Further optionally, when the determination module 801 performs morphological changes on the three-dimensional model of the basic digital human according to the topological invariance and morphological variability of the basic digital human to obtain the three-dimensional models of multiple sample digital humans, it is specifically used to: mark the facial feature vertices in the three-dimensional model of the basic digital human according to the topological invariance of the basic digital human to obtain the positions of the facial feature vertices; generate values ​​of adjustment items for adjusting the facial feature morphology of the basic digital human for multiple times according to the morphological variability of the basic digital human; and perform morphological changes on the three-dimensional model of the basic digital human for multiple times according to the values ​​of the adjustment items generated for multiple times to obtain the three-dimensional models of multiple sample digital humans.

[0129] Further optionally, when the determination module 801 marks the facial feature vertices in the three-dimensional model of the basic digital human according to the topological invariance of the basic digital human to obtain the positions of the facial feature vertices, it is specifically used to: display the three-dimensional model of the basic digital human in an editing interface, the three-dimensional model of the basic digital human including identification information of each vertex, and the identification information of the vertex represents the position of the vertex; in response to the vertex marking operation, obtain the identification information of the marked vertex and the facial feature information to which the marked vertex belongs as marking information for the position of the facial feature vertices.

[0130] Further optionally, when the determination module 801 generates values ​​of adjustment items for adjusting the facial features of the basic digital human for multiple times according to the morphological variability of the basic digital human, it is specifically used to: when there are multiple adjustment items, determine the target distribution that the values ​​of the multiple adjustment items obey, and the target distribution includes uniform distribution and / or Gaussian distribution; according to the morphological variability of the basic digital human and the number of samples, generate the values ​​of the multiple adjustment items each time with the goal that the values ​​of the multiple adjustment items obey uniform distribution or Gaussian distribution, and the number of samples determines the number of generations.

[0131] Further optionally, when the determination module 801 generates the values ​​of the multiple adjustment items each time with the goal of the values ​​of the multiple adjustment items obeying uniform distribution or Gaussian distribution according to the morphological variability of the basic digital human and the number of samples, it is specifically used to: when the target distribution includes uniform distribution and Gaussian distribution, determine the first number and the second number according to the number of samples, and the sum of the first number and the second number is equal to the number of generation times; according to the morphological variability of the basic digital human, in the first number of generation processes, generate the values ​​of the multiple adjustment items with the goal of the values ​​of the multiple adjustment items obeying uniform distribution; and in the second number of generation processes, generate the values ​​of the multiple adjustment items with the goal of the values ​​of the multiple adjustment items obeying Gaussian distribution.

[0132] Further optionally, the determination module 801 performs morphological changes on the three-dimensional model of the basic digital human multiple times according to the values ​​of the adjustment items generated multiple times to obtain three-dimensional models of multiple sample digital humans, including: determining a target adjustment direction and an extreme vertex matrix corresponding to the basic digital human in the target adjustment direction for each value of the adjustment item generated, the three-dimensional model of the basic digital human corresponding to a basic vertex matrix; and performing morphological changes on the three-dimensional model of the basic digital human according to a linear change relationship between the adjustment item and the extreme vertex matrix and the basic vertex matrix to obtain the three-dimensional model of the sample digital human.

[0133] Further optionally, when determining that the module 801 performs morphological changes on the three-dimensional model of the basic digital human according to the linear change relationship between the adjustment item and the extreme vertex matrix and the basic vertex matrix to obtain the three-dimensional model of the sample digital human, it is specifically used to: generate a vertex matrix offset according to the value of the adjustment item, the extreme vertex matrix, the basic vertex matrix and the linear change relationship; superimpose the basic vertex matrix with the vertex matrix offset to obtain a deformed vertex matrix, and the deformed vertex matrix is ​​used to describe the three-dimensional model of the sample digital human.

[0134] Further optionally, when the segmentation module 803 performs facial feature segmentation on the sample rendered image according to the facial feature contour information included in the sample rendered image to obtain a sample facial feature mask image, it is specifically used to: call the filling function and / or the third facial feature segmentation model in the computer vision library according to the facial feature contour information included in the sample rendered image, and perform facial feature segmentation on the sample rendered image to obtain a sample facial feature mask image; wherein the third facial feature segmentation model is a model for image segmentation tasks, and the filling function is used for area filling.

[0135] Further optionally, the segmentation module 803 calls the filling function and the third facial feature segmentation model in the computer vision library according to the facial feature contour information included in the sample rendered image, and performs facial feature segmentation on the sample rendered image to obtain a sample facial feature mask image. The method is specifically used for: calling the filling function according to the facial feature contour information of the first category of facial features in the sample rendered image, filling the image area surrounded by the facial feature contour information of the first category of facial features, so as to obtain the mask area of ​​the first category of facial features; calling the third facial feature segmentation model according to the facial feature contour information of the second category of facial features in the sample rendered image, masking the image area surrounded by the facial feature contour information of the second category of facial features, so as to obtain the mask area of ​​the second category of facial features; generating the sample facial feature mask image according to the mask area of ​​the first category of facial features and the mask area of ​​the second category of facial features.

[0136] Further optionally, when the segmentation module 803 performs facial feature segmentation on the target rendered image according to the first facial feature segmentation model for the digital human to obtain a target facial feature mask image corresponding to the target digital human, it is specifically used to: input the target rendered image into the first facial feature segmentation model, extract image features corresponding to the target rendered image, perform facial feature region analysis and facial feature boundary detection on the image features respectively to obtain a facial feature region segmentation map and a facial feature boundary segmentation map; and fuse the facial feature region segmentation map and the facial feature boundary segmentation map to obtain the target facial feature mask image.

[0137] In this embodiment, the topological invariance and morphological deformability of the three-dimensional model of the digital human are utilized, combined with the three-dimensional rendering technology and the facial feature segmentation technology, to accurately generate training samples of the digital human and its facial features, and then train a facial feature segmentation model for the digital human based on these training samples. Afterwards, the facial features of the digital human can be segmented based on the trained facial feature segmentation model for the digital human, thereby solving the problem that the facial feature segmentation model for real human faces has insufficient generalization ability and cannot accurately segment the facial features of the digital human, thereby improving the accuracy of the facial feature segmentation of the digital human.

[0138] The above describes the internal functions and structure of the facial features segmentation device. Fig. 9 As shown, in practice, the facial features segmentation device can be implemented as an electronic device, including: a memory 901, a processor 902 and a communication component 903.

[0139] The memory 901 is used to store computer programs and can be configured to store various other data to support operations on the computing platform. Examples of such data include instructions for any application or method operating on the computing platform, contact data, phone book data, messages, pictures, videos, etc.

[0140] Memory 901 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0141] The processor 902 is coupled to the memory 901 and is used to execute the computer program in the memory 901, so as to: determine the three-dimensional model of the target digital human in response to the task of segmenting the facial features of the target digital human; render the three-dimensional model of the target digital human to obtain a target rendered image containing the facial feature areas of the target digital human; perform facial feature segmentation on the target rendered image according to a first facial feature segmentation model for the digital human to obtain a target facial feature mask image corresponding to the target digital human; wherein the first facial feature segmentation model is obtained by training a second facial feature segmentation model for real human facial feature segmentation using sample rendered images and sample facial feature mask images corresponding to a plurality of sample digital humans as training samples.

[0142] Further optionally, the processor 902 is also used to: obtain a three-dimensional model of a basic digital human, the basic digital human having topological invariance and morphological variability, the topological invariance indicating that the positions of the facial feature vertices in the three-dimensional model of the basic digital human are unchanged and known; perform morphological changes on the three-dimensional model of the basic digital human according to the topological invariance and morphological variability of the basic digital human, so as to obtain three-dimensional models of multiple sample digital humans; render the three-dimensional models of the sample digital humans to obtain sample rendered images including facial feature areas of the sample digital humans, the sample rendered images also including facial feature contour information mapped by the positions of the facial feature vertices in the three-dimensional model of the basic digital human; perform facial feature segmentation on the sample rendered image according to the facial feature contour information included in the sample rendered image, so as to obtain a sample facial feature mask image.

[0143] Further optionally, when the processor 902 performs morphological changes on the three-dimensional model of the basic digital human according to the topological invariance and morphological variability of the basic digital human to obtain three-dimensional models of multiple sample digital humans, the processor 902 is specifically used to: mark the facial feature vertices in the three-dimensional model of the basic digital human according to the topological invariance of the basic digital human to obtain the positions of the facial feature vertices; generate values ​​of adjustment items for adjusting the facial feature morphology of the basic digital human for multiple times according to the morphological variability of the basic digital human; and perform morphological changes on the three-dimensional model of the basic digital human for multiple times according to the values ​​of the adjustment items generated for multiple times to obtain three-dimensional models of multiple sample digital humans.

[0144] Further optionally, when the processor 902 marks the facial feature vertices in the three-dimensional model of the basic digital human according to the topological invariance of the basic digital human to obtain the positions of the facial feature vertices, it is specifically used to: display the three-dimensional model of the basic digital human in an editing interface, the three-dimensional model of the basic digital human including identification information of each vertex, and the identification information of the vertex represents the position of the vertex; in response to the vertex marking operation, obtain the identification information of the marked vertex and the facial feature information to which the marked vertex belongs as marking information for the position of the facial feature vertices.

[0145] Further optionally, when the processor 902 generates values ​​of adjustment items for adjusting the facial features of the basic digital human for multiple times based on the morphological variability of the basic digital human, it is specifically used to: when there are multiple adjustment items, determine the target distribution that the values ​​of the multiple adjustment items obey, and the target distribution includes uniform distribution and / or Gaussian distribution; based on the morphological variability of the basic digital human and the number of samples, generate the values ​​of the multiple adjustment items each time with the goal that the values ​​of the multiple adjustment items obey uniform distribution or Gaussian distribution, and the number of samples determines the number of generations.

[0146] Further optionally, when the processor 902 generates the values ​​of the multiple adjustment items each time with the goal of the values ​​of the multiple adjustment items obeying uniform distribution or Gaussian distribution according to the morphological variability of the basic digital human and the number of samples, it is specifically used to: when the target distribution includes uniform distribution and Gaussian distribution, determine a first number and a second number according to the number of samples, the sum of the first number and the second number being equal to the number of generation times; according to the morphological variability of the basic digital human, in the first number of generation processes, generate the values ​​of the multiple adjustment items with the goal of the values ​​of the multiple adjustment items obeying uniform distribution; and in the second number of generation processes, generate the values ​​of the multiple adjustment items with the goal of the values ​​of the multiple adjustment items obeying Gaussian distribution.

[0147] Further optionally, the processor 902 performs morphological changes on the three-dimensional model of the basic digital human multiple times according to the values ​​of the adjustment items generated multiple times to obtain three-dimensional models of multiple sample digital humans, including: determining a target adjustment direction and an extreme vertex matrix corresponding to the basic digital human in the target adjustment direction for each value of the adjustment item generated, the three-dimensional model of the basic digital human corresponding to a basic vertex matrix; and performing morphological changes on the three-dimensional model of the basic digital human according to a linear change relationship between the adjustment item and the extreme vertex matrix and the basic vertex matrix to obtain the three-dimensional model of the sample digital human.

[0148] Further optionally, when the processor 902 performs morphological changes on the three-dimensional model of the basic digital human according to the linear change relationship between the adjustment item and the extreme vertex matrix and the basic vertex matrix to obtain the three-dimensional model of the sample digital human, it is specifically used to: generate a vertex matrix offset according to the value of the adjustment item, the extreme vertex matrix, the basic vertex matrix and the linear change relationship; superimpose the basic vertex matrix with the vertex matrix offset to obtain a deformed vertex matrix, and the deformed vertex matrix is ​​used to describe the three-dimensional model of the sample digital human.

[0149] Further optionally, when the processor 902 performs facial feature segmentation on the sample rendered image according to the facial feature contour information included in the sample rendered image to obtain a sample facial feature mask image, it is specifically used to: call a filling function and / or a third facial feature segmentation model in a computer vision library according to the facial feature contour information included in the sample rendered image, and perform facial feature segmentation on the sample rendered image to obtain a sample facial feature mask image; wherein the third facial feature segmentation model is a model for image segmentation tasks, and the filling function is used for area filling.

[0150] Further optionally, the processor 902 calls the filling function and the third facial feature segmentation model in the computer vision library according to the facial feature contour information included in the sample rendered image, and performs facial feature segmentation on the sample rendered image to obtain a sample facial feature mask image. The method is specifically used for: calling the filling function according to the facial feature contour information of the first category of facial features in the sample rendered image, filling the image area surrounded by the facial feature contour information of the first category of facial features, so as to obtain the mask area of ​​the first category of facial features; calling the third facial feature segmentation model according to the facial feature contour information of the second category of facial features in the sample rendered image, masking the image area surrounded by the facial feature contour information of the second category of facial features, so as to obtain the mask area of ​​the second category of facial features; generating the sample facial feature mask image according to the mask area of ​​the first category of facial features and the mask area of ​​the second category of facial features.

[0151] Further optionally, when the processor 902 performs facial feature segmentation on the target rendered image according to the first facial feature segmentation model for the digital human to obtain a target facial feature mask image corresponding to the target digital human, the processor 902 is specifically used to: input the target rendered image into the first facial feature segmentation model, extract image features corresponding to the target rendered image, perform facial feature area analysis and facial feature boundary detection on the image features respectively to obtain a facial feature area segmentation map and a facial feature boundary segmentation map; and fuse the facial feature area segmentation map and the facial feature boundary segmentation map to obtain the target facial feature mask image.

[0152] Further, if Fig. 9As shown, the electronic device also includes: a display 904, a power component 905, an audio component 906 and other components. Fig. 9 Only some components are shown schematically, which does not mean that the electronic device only includes Fig. 9 In addition, Fig. 9 The components in the dashed box are optional components, not mandatory components, and the specific components depend on the product form of the electronic device. The electronic device of this embodiment can be implemented as a terminal device such as a desktop computer, a laptop computer, a smart phone, or an IOT device, or a server device such as a conventional server, a cloud server, or a server array. If the electronic device of this embodiment is implemented as a terminal device such as a desktop computer, a laptop computer, a smart phone, etc., it can include Fig. 9 If the electronic device of this embodiment is implemented as a server device such as a conventional server, a cloud server or a server array, it may not include Fig. 9 Components within the dashed box.

[0153] Accordingly, an embodiment of the present application further provides a computer program product, including a computer program / instruction, which, when executed by a processor, executes various steps that can be executed by an electronic device.

[0154] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed, can implement each step that can be executed by an electronic device in the above method embodiment.

[0155] In the embodiments of the present application, the topological invariance and morphological deformability of the three-dimensional model of the digital human are utilized, combined with three-dimensional rendering technology and facial feature segmentation technology, to accurately generate training samples of the digital human and its facial features, and then train a facial feature segmentation model for the digital human based on these training samples. Afterwards, the facial features of the digital human can be segmented based on the trained facial feature segmentation model for the digital human, thereby solving the problem that the facial feature segmentation model for real human faces has insufficient generalization ability and cannot accurately segment the facial features of the digital human, thereby improving the accuracy of the facial feature segmentation of the digital human.

[0156] The above-mentioned memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable programmable read only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0157] The above-mentioned communication component is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G / LTE, 5G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (BT) technology and other technologies.

[0158] The above-mentioned display includes a screen, and the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0159] The power supply assembly provides power to various components of the device where the power supply assembly is located. The power supply assembly may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply assembly is located.

[0160] The above-mentioned audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (Microphone, MIC), and when the device where the audio component is located is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker for outputting an audio signal.

[0161] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-readable storage media (including but not limited to disk storage, compact disc read-only memory (Compact Disc Read-Only Memory, CD-ROM), optical storage, etc.) containing computer-usable program code.

[0162] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0163] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0164] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0165] In a typical configuration, a computing device includes one or more processors (Central Processing Unit, CPU), input / output interface, network interface and memory.

[0166] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0167] Computer readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0168] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0169] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A facial features segmentation method, characterized in that: include: In response to a facial features segmentation task for a target digital human, a three-dimensional model of the target digital human is determined; Rendering the three-dimensional model of the target digital human to obtain a target rendered image including the facial features of the target digital human; According to the first facial feature segmentation model for the digital human, the facial features of the target rendered image are segmented to obtain a target facial feature mask image corresponding to the target digital human; The first facial feature segmentation model is obtained by training a second facial feature segmentation model for real human facial feature segmentation using sample rendered images and sample facial feature mask images corresponding to a plurality of sample digital humans as training samples.

2. The method according to claim 1, characterized in that Also includes: According to the topological invariance and morphological variability of the basic digital human, the morphological changes are performed on the three-dimensional model of the basic digital human to obtain three-dimensional models of multiple sample digital humans; Rendering the three-dimensional model of the sample digital human to obtain a sample rendered image including the facial features of the sample digital human, wherein the sample rendered image also includes facial features contour information mapped from the positions of the facial features vertices in the three-dimensional model of the basic digital human; According to the facial feature contour information included in the sample rendered image, the sample rendered image is segmented into facial features to obtain a sample facial feature mask image.

3. The method according to claim 2, characterized in that According to the topological invariance and morphological variability of the basic digital human, the morphological changes are performed on the three-dimensional model of the basic digital human to obtain the three-dimensional models of multiple sample digital humans, including: According to the topological invariance of the basic digital human, the vertices of the facial features in the three-dimensional model of the basic digital human are marked to obtain the positions of the vertices of the facial features; According to the morphological variability of the basic digital human, values ​​of adjustment items for adjusting the morphology of the facial features of the basic digital human are generated multiple times; According to the values ​​of the adjustment items generated multiple times, the three-dimensional model of the basic digital human is subjected to multiple morphological changes to obtain multiple three-dimensional models of sample digital humans.

4. The method according to claim 3, characterized in that According to the topological invariance of the basic digital human, the facial features vertices in the three-dimensional model of the basic digital human are marked to obtain the positions of the facial features vertices, including: Displaying the three-dimensional model of the basic digital human in the editing interface, the three-dimensional model of the basic digital human including identification information of each vertex, wherein the identification information of the vertex indicates the position of the vertex; In response to the vertex marking operation, identification information of the marked vertex and information of the facial features to which the marked vertex belongs are obtained as marking information for the position of the facial features vertex.

5. The method according to claim 3, characterized in that: According to the morphological variability of the basic digital human, values ​​of adjustment items for adjusting the morphological features of the basic digital human are generated multiple times, including: In the case where there are multiple adjustment items, determining a target distribution that the values ​​of the multiple adjustment items obey, wherein the target distribution includes a uniform distribution and / or a Gaussian distribution; According to the morphological variability and sample quantity of the basic digital human, the values ​​of the multiple adjustment items are generated each time with the goal of the values ​​of the multiple adjustment items obeying uniform distribution or Gaussian distribution, and the number of samples determines the number of generations.

6. The method according to claim 5, characterized in that According to the morphological variability of the basic digital human and the number of samples, the values ​​of the multiple adjustment items are generated with the goal of the values ​​of the multiple adjustment items obeying uniform distribution or Gaussian distribution each time, including: In a case where the target distribution includes a uniform distribution and a Gaussian distribution, determining a first number and a second number according to the number of samples, the sum of the first number and the second number being equal to the number of generation times; According to the morphological variability of the basic digital human, in the first number of generation processes, the values ​​of the multiple adjustment items are generated with the goal of making the values ​​of the multiple adjustment items obey a uniform distribution; and In the second number of generating processes, the values ​​of the multiple adjustment items are generated with the goal of the values ​​of the multiple adjustment items obeying Gaussian distribution.

7. The method according to claim 3, characterized in that According to the values ​​of the adjustment items generated multiple times, the three-dimensional model of the basic digital human is subjected to multiple morphological changes to obtain multiple three-dimensional models of sample digital humans, including: For each generated value of the adjustment item, determine a target adjustment direction and an extreme value vertex matrix corresponding to the basic digital human in the target adjustment direction, and the three-dimensional model of the basic digital human corresponds to a basic vertex matrix; According to the linear change relationship between the adjustment item and the extreme value vertex matrix and the basic vertex matrix, the three-dimensional model of the basic digital human is morphologically changed to obtain the three-dimensional model of the sample digital human.

8. The method according to claim 7, characterized in that According to the linear change relationship between the adjustment item and the extreme value vertex matrix and the basic vertex matrix, the morphological change is performed on the three-dimensional model of the basic digital human to obtain the three-dimensional model of the sample digital human, including: Generate a vertex matrix offset according to the value of the adjustment item, the extreme value vertex matrix, the basic vertex matrix and the linear change relationship; The basic vertex matrix and the vertex matrix offset are superimposed to obtain a deformed vertex matrix, and the deformed vertex matrix is ​​used to describe the three-dimensional model of the sample digital human.

9. The method according to any one of claims 2 to 8, characterized in that: According to the facial features contour information included in the sample rendered image, the sample rendered image is segmented into facial features to obtain a sample facial features mask image, including: According to the facial feature contour information included in the sample rendered image, calling a filling function and / or a third facial feature segmentation model in a computer vision library to perform facial feature segmentation on the sample rendered image to obtain a sample facial feature mask image; Among them, the third facial features segmentation model is a model used for image segmentation tasks, and the filling function is used for region filling.

10. The method according to claim 9, characterized in that According to the facial features contour information included in the sample rendered image, calling the filling function and the third facial features segmentation model in the computer vision library to perform facial features segmentation on the sample rendered image to obtain a sample facial features mask image, including: According to the facial feature contour information of the first category of facial features in the sample rendered image, calling the filling function to fill the image area surrounded by the facial feature contour information of the first category of facial features to obtain the mask area of ​​the first category of facial features; According to the facial feature contour information of the second type of facial features in the sample rendered image, calling the third facial feature segmentation model, and performing mask processing on the image area surrounded by the facial feature contour information of the second type of facial features to obtain the mask area of ​​the second type of facial features; The sample facial features mask image is generated according to the mask area of ​​the first category of facial features and the mask area of ​​the second category of facial features.

11. The method according to any one of claims 1 to 8, characterized in that: According to the first facial features segmentation model for the digital human, the facial features of the target rendered image are segmented to obtain a target facial features mask image corresponding to the target digital human, including: The target rendered image is input into the first facial feature segmentation model, image features corresponding to the target rendered image are extracted, facial feature region analysis and facial feature boundary detection are performed on the image features respectively to obtain a facial feature region segmentation map and a facial feature boundary segmentation map; the facial feature region segmentation map and the facial feature boundary segmentation map are fused to obtain the target facial feature mask image.

12. A sample generation method, characterized in that: include: According to the topological invariance and morphological variability of the basic digital human, the morphological changes are performed on the three-dimensional model of the basic digital human to obtain three-dimensional models of multiple sample digital humans; Rendering the three-dimensional model of the sample digital human to obtain a sample rendered image including the facial features of the sample digital human, wherein the sample rendered image also includes facial features contour information mapped from the positions of the facial features vertices in the three-dimensional model of the basic digital human; According to the facial feature contour information included in the sample rendered image, the sample rendered image is segmented into facial features to obtain a sample facial feature mask image.

13. The method according to claim 12, characterized in that According to the topological invariance and morphological variability of the basic digital human, the morphological change of the three-dimensional model of the basic digital human is performed to obtain the three-dimensional models of multiple sample digital humans, including: According to the topological invariance of the basic digital human, the vertices of the facial features in the three-dimensional model of the basic digital human are marked to obtain the positions of the vertices of the facial features; According to the morphological variability of the basic digital human, values ​​of adjustment items for adjusting the morphology of the facial features of the basic digital human are generated multiple times; According to the values ​​of the adjustment items generated multiple times, the three-dimensional model of the basic digital human is subjected to multiple morphological changes to obtain multiple three-dimensional models of sample digital humans.

14. The method according to claim 13, characterized in that According to the topological invariance of the basic digital human, the facial features vertices in the three-dimensional model of the basic digital human are marked to obtain the positions of the facial features vertices, including: Displaying the three-dimensional model of the basic digital human in the editing interface, the three-dimensional model of the basic digital human including identification information of each vertex, wherein the identification information of the vertex indicates the position of the vertex; In response to the vertex marking operation, identification information of the marked vertex and information of the facial features to which the marked vertex belongs are obtained as marking information for the position of the facial features vertex.

15. The method according to claim 13, characterized in that According to the morphological variability of the basic digital human, values ​​of adjustment items for adjusting the morphological features of the basic digital human are generated multiple times, including: In the case where there are multiple adjustment items, determining a target distribution that the values ​​of the multiple adjustment items obey, wherein the target distribution includes a uniform distribution and / or a Gaussian distribution; According to the morphological variability and sample quantity of the basic digital human, the values ​​of the multiple adjustment items are generated each time with the goal of the values ​​of the multiple adjustment items obeying uniform distribution or Gaussian distribution, and the number of samples determines the number of generations.

16. The method according to any one of claims 13 to 15, characterized in that: According to the facial features contour information included in the sample rendered image, the sample rendered image is segmented into facial features to obtain a sample facial features mask image, including: According to the facial feature contour information included in the sample rendered image, calling a filling function and / or a third facial feature segmentation model in a computer vision library to perform facial feature segmentation on the sample rendered image to obtain a sample facial feature mask image; Among them, the third facial features segmentation model is a model used for image segmentation tasks, and the filling function is used for region filling.

17. The method according to claim 16, characterized in that According to the facial features contour information included in the sample rendered image, calling the filling function and the third facial features segmentation model in the computer vision library to perform facial features segmentation on the sample rendered image to obtain a sample facial features mask image, including: According to the facial feature contour information of the first category of facial features in the sample rendered image, calling the filling function to fill the image area surrounded by the facial feature contour information of the first category of facial features to obtain the mask area of ​​the first category of facial features; According to the facial feature contour information of the second type of facial features in the sample rendered image, calling the third facial feature segmentation model, and performing mask processing on the image area surrounded by the facial feature contour information of the second type of facial features to obtain the mask area of ​​the second type of facial features; The sample facial features mask image is generated according to the mask area of ​​the first category of facial features and the mask area of ​​the second category of facial features.

18. A model training method, characterized in that: include: Generate sample rendering images and sample facial feature mask images corresponding to multiple sample digital humans as training samples; Training a second facial features segmentation model for real human facial features segmentation according to the training samples to obtain a first facial features segmentation model for digital human; Wherein, the sample rendering images and sample facial feature mask images corresponding to the multiple sample digital humans are generated by the method described in any one of claims 12-17.

19. An electronic device, characterized in that: include: A memory and a processor; the memory is used to store one or more computer instructions; the processor is used to execute the one or more computer instructions to: execute the steps in the method described in any one of claims 1-11, claims 12-17 or claim 18.

20. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the processor is enabled to implement the steps of the method according to any one of claims 1 to 11, claims 12 to 17 or claim 18.

21. A computer program product, characterized in that The method comprises a computer program / instruction, and when the computer program / instruction is executed by a processor, the steps in the method of any one of claims 1 to 11, claims 12 to 17 or claim 18 are performed.

Images