Three-dimensional face generation method and system based on local feature exchange

Abstract

The invention discloses a three-dimensional face generation method and system based on local feature exchange. The method comprises the steps of 1, dividing a face according to attributes to obtain a three-dimensional face point cloud data set; 2, dividing the three-dimensional face point cloud data set into a training set and a test set; step 3, mapping the three-dimensional face point cloud data set to a shared standard ball; step 4, serializing points of each area on the standard sphere; 5, grouping points in the point sequence corresponding to each region to obtain a shape combination; step 6, performing local feature replacement on the shape combination corresponding to the to-be-exchanged region to obtain a new shape combination; 7, constructing a three-dimensional face generation model; and step 8, training the three-dimensional face generation model. According to the method, a local feature exchange strategy is adopted to realize decoupling of potential variables, so that diversity and morphological consistency of three-dimensional face generation results can be kept, and the performance of a generation model in the aspect of generating local features can be remarkably improved.

Description

Technical Field

[0001] The invention belongs to the technical field of three-dimensional shape generation in computer vision, and relates to a three-dimensional face generation method and system based on local feature exchange. Background Art

[0002] With the development of 3D digital technology, 3D data has emerged. 3D data can provide more intuitive visual effects, such as rotation and scaling, making it easier for people to understand and perceive the shape and structure of objects, and more realistically simulate the appearance and characteristics of objects. At present, the cross-integration of related disciplines such as computer vision, artificial intelligence and graphics has set off a new revolution in 3D shape generation technology (represented by digital life).

[0003] Three-dimensional shape generation is the process of using computer programs to generate new three-dimensional shapes from given input conditions (such as pictures, text descriptions, etc.) through steps such as data preparation, model design, model training, model testing and optimization.

[0004] Traditional 3D shape generation methods include manual modeling and scanning generation. Manual modeling refers to modeling through computer-aided design software, which requires designers to have a high level of technical skills and creativity. This method is inefficient and its practicality and scalability are seriously limited by poor consistency. Scanning refers to scanning real objects through devices such as laser scanning or cameras to obtain their geometric shape and texture information, and then generate a 3D model. The scanning method greatly simplifies the design process, but due to the inherent constraints of its potential model, the generated model has poor flexibility. In the face of the application needs of 3D digital humans, the traditional modeling process is cumbersome and the cycle is lengthy. The use of generative artificial intelligence technology to generate highly realistic and diverse 3D shapes is gradually becoming a research hotspot.

[0005] At present, the most commonly used generative artificial intelligence technologies include 3D Morphable Models (3DMM), Variational Auto-Encoder (VAE), Generative Adversarial Networks (GAN) and Blend Shapes. Latent dissociation refers to the fact that in the latent space of the generative model, each dimension independently represents a specific attribute or feature in the data, without being affected by other dimensions. Such a latent representation helps to control the features and attributes of the data in a more fine-grained manner. This ability is particularly important in generative models because it allows the model to independently manipulate specific factors without affecting the performance of other factors, thereby achieving more accurate data generation. However, although latent dissociation has been widely studied in the field of deep learning, it is still an unsolved problem facing 3D shape generation. In particular, the structure of 3D human faces is complex and changeable. Although most generative models have successfully separated the control of identity and expression (or posture), they still cannot decouple the latent variables that control local attributes. Summary of the invention

[0006] The purpose of the present invention is to provide a three-dimensional face generation method and system based on local feature exchange, which enhances the local feature generation capability of the model by introducing a local feature exchange mechanism, and solves the problem that the existing technology of three-dimensional face generation cannot effectively control the latent variable decoupling of local attributes (features).

[0007] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0008] On the one hand, the present invention provides a method for generating a three-dimensional face based on local feature exchange, which specifically comprises the following steps:

[0009] Step 1: divide the face according to attributes to obtain a three-dimensional face point cloud dataset with multiple attributes, where each attribute corresponds to multiple latent variables;

[0010] Step 2, dividing the 3D face point cloud dataset with multiple attributes into a training set and a test set;

[0011] Step 3, mapping the 3D face point cloud dataset to a shared standard sphere, where each attribute of the 3D face corresponds to a color on the standard sphere, and each color represents an area;

[0012] Step 4, serialize the points of each region on the standard sphere η respectively to obtain the point sequence corresponding to each region;

[0013] Step 5: Use a self-supervised learning network to divide the points in the point sequence corresponding to each region on the standard sphere into G groups. These G groups serve as the shape combination s of their corresponding regions. G ;

[0014] Step 6: Select one or more regions to be exchanged as needed; and calculate the shape combination s corresponding to the region to be exchanged. G Perform local feature replacement to obtain a new shape combination S′ corresponding to the region G ; Shape combination s of the unexchanged region G remain unchanged;

[0015] Step 7, constructing a 3D face generation model based on the training set and test set obtained in step 2 and the exchanged result obtained in step 6;

[0016] Step 8, using the shape combination of all regions of the three-dimensional face obtained in step 6 to train the three-dimensional face generation model constructed in step 7, to obtain a trained three-dimensional face generation model.

[0017] Furthermore, in step 1, according to the three-dimensional face parts, F attributes corresponding to different parts are defined on the three-dimensional face point cloud, and each attribute corresponds to multiple latent variables. The latent variable refers to the feature variable corresponding to the three-dimensional face attribute in the feature space.

[0018] Furthermore, the operation of step 3 is as follows:

[0019] Given a 3D face point cloud dataset containing N points N represents the number of points in the 3D face point cloud, which is a natural number. First, DGCNN is used as an encoder, and the 3D face point cloud dataset P is input into the encoder for encoding to obtain a 512-dimensional global latent code z∈R 512 , replicate the global latent code z N times, connect the N replicated global latent codes with N points on the canonical sphere, and use the connection result as the input of a nonlinear function; at the output of the nonlinear function, the corresponding positions of the points on the 3D face on the canonical sphere are determined by calculating the chamfer loss.

[0020] Furthermore, in step 4, for each region on the canonical sphere, the serialization process is to use the Fibonacci spiral to traverse from the center point of each region, and serialize the points on the canonical sphere one by one in the order of the Fibonacci spiral path.

[0021] Furthermore, in step 5, for each region of the standard sphere, the grouping includes the following sub-steps:

[0022] Step 51, use a multilayer perceptron to predict each point in the point sequence corresponding to the current area on the typical sphere, and then apply the SoftMax activation function to determine the group to which it belongs, and obtain the probability matrix π (i,j) , π (i,j) represents the probability that the i-th point in each region is assigned to the j-th group;

[0023] Step 52, transfer the grouping probability of each point in each area on the standard sphere obtained in step 51 to the corresponding point of the corresponding area of ​​the original three-dimensional face; calculate and obtain the G key points K={K1, K2, ..., K G}, where the key point K j ∈R G×3 :

[0024]

[0025] in,

[0026] Step 53, calculate the Chamfer distance, which is used to measure the above G key points K = {K1, K2, ..., K G} and the corresponding region of the input 3D face P, and denote it as L CD (K,P h ), where h is the serial number of the current region; if the Chamfer distance is less than the preset threshold, it is considered to be consistent, and the current probability matrix π (i,j) Save and go to step 54; otherwise, it is considered that the consistency is poor and return to step 51;

[0027] Step 54, find the maximum probability value in each row of the probability matrix π obtained in step 53, assign each point in each region of the three-dimensional face to the group with the maximum probability as the final grouping result, and obtain G groups corresponding to each region. These G groups are used as the shape combination s of the current region. G .

[0028] Furthermore, the internal operation flow of the 3D face generation model constructed in step 7 is as follows:

[0029] Step 71: for all three-dimensional faces in the training set, the shape combination s corresponding to the area that has not been exchanged is G And the new feature combination S′ corresponding to the area obtained after the exchange G Feed into encoder together Perform feature aggregation and output a codebook of size K×D Codebook Corresponding to an index sequence S = {s1, s2, s3, ..., s K}, where the i-th index value si The corresponding one-dimensional vector e in the codebook i ;

[0030] Step 72, based on the index sequence S, use the Transformer autoregressive network for training to generate a new index sequence S';

[0031] Specifically, the input of Transformer is the index sequence S{e1,e2,e3,…,e K}, and the query position of the next element {i e2 ,i e3 ,...}, the output of the Transformer network is a new index sequence S', which corresponds to the new codebook

[0032] Step 73: Send a 3D face x in the test set to the encoder Output feature z e (x); calculate feature z e (x) and codebook {e1′,e2′,...,e′ K}, and the feature z e (x) Using codebook Medium distance feature z e (x) nearest e i ′ Update, get the updated feature z q (x);

[0033] Step 74, use the decoder Generate a 3D face by: q (x) is sent to the dual-branch network of SP-GAN; the first branch of the dual-branch network of SP-GAN sends the points in the canonical ball η to the graph attention module to extract point space features, and the second branch extracts the points from the feature z q (x) extracts style features; then uses adaptive instance normalization to fuse the point space features with the style features, and feeds the fused features into the decoder Generate a 3D face P′.

[0034] On the other hand, the present invention provides a method for generating a three-dimensional face based on local feature exchange, which specifically includes the following modules:

[0035] The attribute division module is used to divide the face according to the attributes to obtain a three-dimensional face point cloud data set with multiple attributes, where each attribute corresponds to multiple latent variables;

[0036] A data set partitioning module, used for partitioning a 3D face point cloud data set with multiple attributes into a training set and a test set;

[0037] A mapping module is used to map the 3D face point cloud dataset onto a shared standard sphere, where each attribute of the 3D face corresponds to a color on the standard sphere, and each color represents an area;

[0038] A serialization module is used to serialize the points of each region on the standard sphere η to obtain a point sequence corresponding to each region;

[0039] The grouping module is used to use a self-supervised learning network to divide the points in the point sequence corresponding to each area on the canonical sphere into G groups. These G groups are the shape combinations of their corresponding areas. G ;

[0040] The shape combination exchange module is used to select one or more areas to be exchanged according to the needs; the shape combination s corresponding to the exchange area G Perform local feature replacement to obtain a new shape combination S′ corresponding to the region G ; Shape combination s of the unexchanged region G remain unchanged;

[0041] A 3D face generation model building module is used to build a 3D face generation model based on the training set and test set obtained by the data set division module and the exchanged result obtained by the shape combination exchange module;

[0042] The model training module is used to train the 3D face generation model constructed by the 3D face generation model construction module using the shape combination of all regions of the 3D face obtained by the shape combination exchange module to obtain a trained 3D face generation model.

[0043] Compared with the existing technology, the beneficial effects of the present invention are as follows:

[0044] Although existing 3D face generation methods have successfully separated the control of identity and expression (or posture), they still cannot decouple the latent variables that control local attributes. The present invention adopts the strategy of local feature exchange to achieve the decoupling of latent variables, which can not only maintain the diversity and morphological consistency of 3D face generation results, but also significantly improve the performance of the generation model in generating local features. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a technical route schematic diagram of the method for generating three-dimensional human faces based on local feature exchange of the present invention;

[0046] Figure 2 It is a schematic diagram of point cloud-canonical sphere mapping;

[0047] Figure 3 It is a schematic diagram generated by shape combination;

[0048] Figure 4 It is a schematic diagram of feature exchange;

[0049] Figure 5 It is the latent variable decoupling effect on the UHM dataset;

[0050] Figure 6 is a random sample generated on the UHM dataset.

[0051] The present invention is further explained below in conjunction with the accompanying drawings and specific embodiments. DETAILED DESCRIPTION

[0052] Embodiment 1:

[0053] The framework of the 3D face generation method based on local feature exchange provided in this embodiment is as follows: Figure 1 As shown, this method is based on the vector quantized variational autoencoder architecture. Represents the encoder, which is used for feature aggregation; the appropriate quantization module VQ is used to quantize the aggregated features; Denotes a decoder that generates a new 3D face from the quantized features, specifically including the following steps:

[0054] Step 1: Divide the faces according to their attributes to obtain a three-dimensional face point cloud dataset with multiple attributes, where each attribute corresponds to multiple latent variables.

[0055] Specifically, according to the three-dimensional facial parts, F attributes corresponding to different parts are defined on the three-dimensional facial point cloud (the value of F can be set independently, and the larger the value of F, the more detailed the division of the three-dimensional face). For example, the value of F is set to 12, and these 12 attributes correspond to the eyes, ears, temporal bones, neck, back of the head, mouth, chin, cheeks, cheekbones, forehead, jawbone, and nose. Different colors are used for different attributes of the face, that is, a three-dimensional face template with 12 attributes is defined. Each attribute corresponds to multiple latent variables (latent variables refer to the feature variables corresponding to the three-dimensional facial attributes in the feature space), which are also arbitrarily set. The present invention sets the number of latent variables for each attribute to 5. For the convenience of the following description, the three-dimensional face point cloud data set obtained in step 1 is recorded as P.

[0056] Step 2: divide the three-dimensional face point cloud dataset with multiple attributes obtained in step 1 into a training set and a test set.

[0057] Step 3, mapping the 3D face point cloud dataset obtained in step 1 to a shared standard sphere, each attribute of the 3D face corresponds to a color on the standard sphere, and each color represents an area.

[0058] Since the 3D face point cloud is scattered and disordered, it cannot be directly fed into the variational autoencoder. Therefore, this step maps the 3D face to a canonical sphere in order to serialize the scattered point cloud in the subsequent steps.

[0059] The mapping process from the 3D face point cloud dataset P to the canonical sphere η is represented as f. i The corresponding point of ∈P on the orthogonal sphere η is denoted by f P→η (p i ), η i The corresponding point of ∈η on the 3D face is expressed as If the point is the t-th point on the canonical sphere, the t-th point on the point cloud will also be labeled accordingly.

[0060] Specifically, the operation of step 3 is as follows:

[0061] Given a 3D face point cloud dataset containing N points (N represents the number of points in the 3D face point cloud, and its value range is a natural number), such as Figure 2 First, DGCNN (Dynamic Graph Edge Convolutional Network) is used as an encoder, and the 3D face point cloud dataset P is input into the encoder for encoding to obtain a 512-dimensional global latent code z∈R 512 , the global latent code z is copied N times (if there are N points on the canonical sphere, N copies are made), the N copied global latent codes are connected with the N points on the canonical sphere, and the connection result is used as the input of a nonlinear function. At the output of the nonlinear function, the corresponding position of the points on the 3D face on the canonical sphere is determined by calculating the chamfer loss.

[0062] Through the above operations, we associate the point p∈P on the 3D face point cloud P with the corresponding point η on the canonical sphere η i ∈η is associated, that is, F different facial attributes are mapped to different regions on the canonical sphere (each region corresponds to a facial attribute). The above operation ensures that each point in the input point cloud has a corresponding point on the canonical sphere, thereby establishing a point-to-point mapping, which helps to simplify the point serialization process in subsequent steps.

[0063] Step 4, serialize the points of each region on the standard sphere η respectively to obtain the point sequence corresponding to each region.

[0064] Through the canonical mapping in step 3, all points on the 3D face point cloud P are mapped to the canonical sphere η. Serializing the points on the canonical sphere is equivalent to serializing the points on the original 3D face.

[0065] Preferably, for each region on the canonical sphere, the serialization process is to use the Fibonacci spiral to traverse from the center point of each region, and to serialize the points on the canonical sphere one by one in the order of the Fibonacci spiral path.

[0066] Step 5: Use a self-supervised learning network (such as Figure 3 As shown in FIG. 1 , the points in the point sequence corresponding to each region on the canonical sphere are divided into G groups (preferably, G=64), and these G groups are used as the shape combination s of the corresponding region. G .

[0067] Through step 4, each region on the canonical sphere is represented as a point sequence, but these sequences usually contain thousands of points, which are difficult to use for autoregressive modeling. Therefore, it is necessary to further convert the obtained 3D face point cloud sequence into a shape combination that can be used for autoregressive modeling. Since each face attribute corresponds to a region of the canonical sphere, decomposing the canonical sphere is equivalent to decomposing the 3D face point cloud.

[0068] In step 5, for each region of the canonical sphere, the grouping includes the following sub-steps:

[0069] Step 51, use a multi-layer perceptron (MLP) to predict each point in the point sequence corresponding to the current region on the typical sphere, and then apply the SoftMax activation function to determine the group to which it belongs, and obtain the probability matrix π (i,j) , π (i,j) represents the probability that the i-th point in each region is assigned to the j-th group.

[0070] Step 52, transferring the grouping probability of each point in each area on the standard sphere obtained in step 51 to the corresponding point in the corresponding area of ​​the original three-dimensional face;

[0071] In order to ensure that the learned grouping can capture the structure of the point cloud, the G key points K = {K1, K2, ..., K G}, where the key point K j ∈R G×3 :

[0072]

[0073] in

[0074] Step 53, calculate the Chamfer distance (chamfer distance), which is used to measure the above G key points K = {K1, K2, ..., K G} and the corresponding region of the input 3D face P, and denote it as L CD (K,P h), where h is the serial number of the current region; if the Chamfer distance is less than the preset threshold, it is considered to be consistent, and the current probability matrix π (i,j) Save and go to step 54; otherwise, it is considered that the consistency is poor and return to step 51;

[0075] Step 54, find the maximum probability value in each row of the probability matrix π obtained in step 53, assign each point in each region of the 3D face to the group with the maximum probability as the final grouping result, and obtain G groups (G=64) corresponding to each region. These G groups are used as the shape combination s of the current region. G .

[0076] Step 6: Select one or more regions to be exchanged as needed (each region corresponds to a part of a three-dimensional face, such as an eye or an ear); select the shape combination s corresponding to the region to be exchanged G Replace local features (local features refer to shape combinations G One or more groups in the region), and obtain the new shape combination S′ corresponding to the region G ; Shape combination s of the unexchanged region G remain unchanged;

[0077] Specifically, for example, given a 3D face A and a 3D face B, the features of the eye attributes Eye of the 3D faces A and B are exchanged. The eye attributes Eye of A and B correspond to a shape combination A_Eye_G and B_Eye_G of size G respectively. Figure 4 As shown, some groups in the shape combination are randomly selected and exchanged. Two random numbers are generated using the random number generation method: the first random number is used to determine the starting point, indicating which group to start the exchange; the second random number is used to determine the number of groups, indicating the number of groups to be exchanged starting from the starting point. The groups to be exchanged are further selected as needed, and the selected groups can be continuous or scattered. In this step, through random selection, while maintaining data integrity, the diversity of the data set samples is increased. Since the model will be exposed to features of different combinations during the training process, its adaptability to different data distributions is improved, which helps the model learn more robust and generalized feature representations.

[0078] Step 7: Based on the training set and test set obtained in step 2 and the exchanged result obtained in step 6, a 3D face generation model is constructed. The internal operation process of the model is as follows:

[0079] Step 71: for all three-dimensional faces in the training set, the shape combination s corresponding to the area that has not been exchanged is G And the new feature combination S′ corresponding to the area obtained after the exchange G Feed into encoder together (Preferably, DGCNN can be used, which contains 3 EdgeConv layers and a neighborhood size of 20. ) performs feature aggregation and outputs a codebook of size K×D Codebook Corresponding to an index sequence S = {s1, s2, s3, ..., s K}, where the i-th index value s i The corresponding one-dimensional vector e in the codebook i .

[0080] Step 72: Based on the index sequence S, a Transformer autoregressive network is used for training to generate a new index sequence S'.

[0081] Specifically, the input of Transformer is the index sequence S{e1,e2,e3,…,e K}, and the query position of the next element {i e2 ,i e3 ,...}, the output of the Transformer network is a new index sequence S', which corresponds to the new codebook

[0082] Step 73: Send a 3D face x in the test set to the encoder Output feature z e (x); calculate feature z e (x) and codebook {e1′,e2′,...,e′ K}, and the feature z e (x) Using codebook Medium distance feature z e (x) nearest e i ′ Update, get the updated feature z q (x).

[0083] Step 74, use the decoder Generate 3D faces.

[0084] Specifically, each point in the canonical sphere η is compared with the feature z q (x) is sent to the dual-branch network of SP-GAN. The first branch of the dual-branch network of SP-GAN sends the points in the normative ball η to the graph attention module to extract point-wise spatial features, and the second branch extracts the points from the feature z. q(x) extracts style features; then uses adaptive instance normalization to fuse the point space features with the style features, and feeds the fused features into the decoder (Preferably, the decoder DGCNN can be used, which contains 3 EdgeConv layers and a neighborhood size of 20. ) to generate a 3D face P′.

[0085] Step 8, using the shape combination of all regions of the three-dimensional face obtained in step 6 to train the three-dimensional face generation model constructed in step 7, to obtain a trained three-dimensional face generation model; the obtained model can be used to generate a three-dimensional face.

[0086] Specifically, during the training process, the loss function consists of reconstruction loss, vector quantization target and commitment loss.

[0087] Specifically, the loss function is expressed as:

[0088] L VQVAE =L R +L VQ +βL C Formula 1

[0089] Wherein: β is a weight constant, and in this embodiment, β=0.5.

[0090] L R is the reconstruction loss;

[0091] is the vector quantization loss;

[0092] L C It is a loss of commitment;

[0093] Reconstruction loss It makes the difference between the original input face P and the generated face P′ as small as possible by calculating the input (P n ∈P) and the corresponding output p′ n ∈P′ is the mean square error between the vertices. P refers to the original 3D face point cloud, n is the number of points, and P′ is the generated 3D face point cloud.

[0094] The vector quantization loss is defined as:

[0095]

[0096] Commitment loss is defined as:

[0097]

[0098] Where:

[0099] sg stands for the stopped gradient operator, which is defined as the identity function in the forward pass and has zero partial derivatives, thus effectively constraining its operand to be a non-updated constant.

[0100] Embodiment 2:

[0101] This embodiment provides a method for generating a 3D face based on local feature exchange, which specifically includes the following modules:

[0102] The attribute division module is used to divide the face according to the attributes to obtain a three-dimensional face point cloud data set with multiple attributes, where each attribute corresponds to multiple latent variables;

[0103] A data set partitioning module, used for partitioning a 3D face point cloud data set with multiple attributes into a training set and a test set;

[0104] A mapping module is used to map the 3D face point cloud dataset onto a shared standard sphere, where each attribute of the 3D face corresponds to a color on the standard sphere, and each color represents an area;

[0105] A serialization module is used to serialize the points of each region on the standard sphere η to obtain a point sequence corresponding to each region;

[0106] The grouping module is used to use a self-supervised learning network to divide the points in the point sequence corresponding to each area on the canonical sphere into G groups. These G groups are the shape combinations of their corresponding areas. G ;

[0107] The shape combination exchange module is used to select one or more areas to be exchanged according to the needs; the shape combination s corresponding to the exchange area G Perform local feature replacement to obtain a new shape combination S′ corresponding to the region G ; Shape combination s of the unexchanged region G remain unchanged;

[0108] A 3D face generation model building module is used to build a 3D face generation model based on the training set and test set obtained by the data set division module and the exchanged result obtained by the shape combination exchange module;

[0109] The model training module is used to train the 3D face generation model constructed by the 3D face generation model construction module using the shape combination of all regions of the 3D face obtained by the shape combination exchange module to obtain a trained 3D face generation model.

[0110] In order to prove the feasibility and beneficial effects of the present invention, the present invention conducts experiments on the UHM dataset. The results are as follows: Figure 4 , Figure 5 shown. Figure 4Indicates the decoupling effect of latent variables in different face attributes. For each latent variable (horizontally, 60 variables, 12 attributes, 5 variables for each attribute, different attributes correspond to different intervals), the change in the average distance between different attributes when the value of the latent variable changes from minimum to maximum is shown. On a specific attribute, any latent variable will show a higher average distance in its corresponding interval, while showing relatively low values ​​on other attributes. Figure 4 The results demonstrated the effectiveness of the method of the present invention.

[0111] Figure 5 The latent variable decoupling effect of this method and SD-VAE on the UHM dataset is demonstrated. It can be seen that our method can control local features more accurately than the SD-VAE method.

[0112] The proposed method is compared with other 3D face generation methods on the UHM dataset. The VP index is a criterion for measuring the decoupling performance of the method. From the experimental results in Table 1, it can be seen that the VP value of the proposed method is higher than that of other face generation methods, which indicates that the 3D face generation method in the present invention has excellent local feature decoupling performance.

[0113] Table 1 Quantitative comparison with other methods

[0114]

Claims

1. A method for generating three-dimensional faces based on local feature exchange, characterized in that: The specific steps include: Step 1: divide the face according to attributes to obtain a three-dimensional face point cloud dataset with multiple attributes, where each attribute corresponds to multiple latent variables; Step 2, dividing the 3D face point cloud dataset with multiple attributes into a training set and a test set; Step 3, mapping the 3D face point cloud dataset to a shared standard sphere, where each attribute of the 3D face corresponds to a color on the standard sphere, and each color represents an area; Step 4, serialize the points of each region on the standard sphere η respectively to obtain the point sequence corresponding to each region; Step 5: Use a self-supervised learning network to divide the points in the point sequence corresponding to each region on the standard sphere into G groups. These G groups serve as the shape combination s of their corresponding regions. G ; Step 6: Select one or more regions to be exchanged as needed; and calculate the shape combination s corresponding to the region to be exchanged. G Perform local feature replacement to obtain a new shape combination S′ corresponding to the region G ; Shape combination s of the unexchanged region G remain unchanged; Step 7, constructing a 3D face generation model based on the training set and test set obtained in step 2 and the exchanged result obtained in step 6; Step 8, using the shape combination of all regions of the three-dimensional face obtained in step 6 to train the three-dimensional face generation model constructed in step 7, to obtain a trained three-dimensional face generation model.

2. The method for generating a three-dimensional face based on local feature exchange according to claim 1, characterized in that: In step 1, according to the 3D face parts, F attributes corresponding to different parts are defined on the 3D face point cloud, and each attribute corresponds to multiple latent variables. The latent variable refers to the feature variable corresponding to the 3D face attribute in the feature space.

3. The method for generating a three-dimensional face based on local feature exchange according to claim 1, characterized in that: The operation of step 3 is as follows: Given a 3D face point cloud dataset containing N points N represents the number of points in the 3D face point cloud, which is a natural number. First, DGCNN is used as an encoder, and the 3D face point cloud dataset P is input into the encoder for encoding to obtain a 512-dimensional global latent code z∈R 512 , replicate the global latent code z N times, connect the N replicated global latent codes with N points on the canonical sphere, and use the connection result as the input of a nonlinear function; at the output of the nonlinear function, the corresponding positions of the points on the 3D face on the canonical sphere are determined by calculating the chamfer loss.

4. The method for generating a three-dimensional face based on local feature exchange according to claim 1, characterized in that: In step 4, for each region on the canonical sphere, the serialization process is to use the Fibonacci spiral to traverse from the center point of each region, and serialize the points on the canonical sphere one by one in the order of the Fibonacci spiral path.

5. The method for generating a three-dimensional face based on local feature exchange as claimed in claim 1, characterized in that: In step 5, for each region of the canonical sphere, the grouping includes the following sub-steps: Step 51, use a multilayer perceptron to predict each point in the point sequence corresponding to the current area on the typical sphere, and then apply the SoftMax activation function to determine the group to which it belongs, and obtain the probability matrix π (i,j) , π (i,j) represents the probability that the i-th point in each region is assigned to the j-th group; Step 52, transfer the grouping probability of each point in each area on the standard sphere obtained in step 51 to the corresponding point of the corresponding area of ​​the original three-dimensional face; calculate and obtain the G key points K={K1, K2, ..., K G }, where the key point K j ∈R G×3 : in, Step 53, calculate the Chamfer distance, which is used to measure the above G key points K = {K1, K2, ..., K G } and the corresponding region of the input 3D face P, and denote it as L CD (K,P h ), where h is the serial number of the current region; if the Chamfer distance is less than the preset threshold, it is considered to be consistent, and the current probability matrix π (i,j) Save and go to step 54; otherwise, it is considered that the consistency is poor and return to step 51; Step 54, find the maximum probability value in each row of the probability matrix π obtained in step 53, assign each point in each region of the three-dimensional face to the group with the maximum probability as the final grouping result, and obtain G groups corresponding to each region. These G groups are used as the shape combination s of the current region. G .

6. The method for generating a three-dimensional face based on local feature exchange according to claim 1, characterized in that: The internal operation process of the 3D face generation model constructed in step 7 is as follows: Step 71: for all three-dimensional faces in the training set, the shape combination s corresponding to the area that has not been exchanged is G And the new feature combination S′ corresponding to the area obtained after the exchange G Feed into encoder together Perform feature aggregation and output a codebook of size K×D Codebook Corresponding to an index sequence S = {s1, s2, s3, ..., s K }, where the i-th index value s i The corresponding one-dimensional vector e in the codebook i ; Step 72, based on the index sequence S, use the Transformer autoregressive network for training to generate a new index sequence S'; Specifically, the input of Transformer is the index sequence S{e1,e2,e3,…,e K }, and the query position of the next element {i e2 ,i e3 ,...}, the output of the Transformer network is a new index sequence S', which corresponds to the new codebook Step 73: Send a 3D face x in the test set to the encoder Output feature z e (x); calculate feature z e (x) and codebook {e1′,e2′,...,e′ K }, and the feature z e (x) Use the distance feature z in codebook z′ e (x) nearest e i ′ Update, get the updated feature z q (x); Step 74, use the decoder Generate a 3D face by: q (x) is sent to the dual-branch network of SP-GAN; the first branch of the dual-branch network of SP-GAN sends the points in the canonical ball η to the graph attention module to extract point space features, and the second branch extracts the points from the feature z q Extract style features from (x); Then adaptive instance normalization is used to fuse the point space features with the style features, and the fused features are fed into the decoder Generate a 3D face P′.

7. A 3D face generation system based on local feature exchange, characterized in that: The modules include: The attribute division module is used to divide the face according to the attributes to obtain a three-dimensional face point cloud data set with multiple attributes, where each attribute corresponds to multiple latent variables; A data set partitioning module, used for partitioning a 3D face point cloud data set with multiple attributes into a training set and a test set; A mapping module is used to map the 3D face point cloud dataset onto a shared standard sphere, where each attribute of the 3D face corresponds to a color on the standard sphere, and each color represents an area; A serialization module is used to serialize the points of each region on the standard sphere η to obtain a point sequence corresponding to each region; The grouping module is used to use a self-supervised learning network to divide the points in the point sequence corresponding to each area on the canonical sphere into G groups. These G groups are the shape combinations of their corresponding areas. G ; The shape combination exchange module is used to select one or more areas to be exchanged according to the needs; the shape combination s corresponding to the exchange area G Perform local feature replacement to obtain a new shape combination S′ corresponding to the region G ; Shape combination s of the unexchanged region G remain unchanged; A 3D face generation model building module is used to build a 3D face generation model based on the training set and test set obtained by the data set division module and the exchanged result obtained by the shape combination exchange module; The model training module is used to train the 3D face generation model constructed by the 3D face generation model construction module using the shape combination of all regions of the 3D face obtained by the shape combination exchange module to obtain a trained 3D face generation model.

Images