Editable facial three-dimensional reconstruction method, system and storage medium

Abstract

The present invention belongs to the field of image data processing technology, and specifically relates to an editable facial three-dimensional reconstruction method, system and storage medium. The method of the present invention mainly includes the following steps: obtaining the expressionless image data and the expression image data of the modeler; obtaining a neutral model and a real expression model; training the deformation field, the color field and the super-resolution neural network; training the expression encoder to construct the expression basis vector; constructing the modeler's expressive facial three-dimensional model through the expression basis vector and the finally generated model and outputting the rendered image. The present invention further provides a system for implementing the above method. The present invention performs three-dimensional reconstruction of the face based on the neural radiation field technology, can generate an accurate, natural, beautiful and real three-dimensional facial model, and can perform flexible and convenient personalized adjustments, which has a good application prospect in the fields of medical treatment and medical beauty.

Description

Technical Field

[0001] The invention belongs to the technical field of image data processing, and in particular relates to an editable facial three-dimensional reconstruction method, system and storage medium. Background Art

[0002] Facial scanning is a technology that uses facial images or videos to generate a three-dimensional model of a human face. It plays an important role in the medical field. It can provide doctors and patients with high-quality facial morphology information to help diagnose, treat and evaluate various face-related problems, such as plastic surgery, oral repair, maxillofacial deformities, etc. Facial scanning provides doctors and patients with high-quality facial morphology information, assists in diagnosis, treatment and evaluation, enhances confidence and satisfaction, and can also provide a large amount of facial data for medical research, promoting the development and innovation of medical knowledge.

[0003] The existing facial scanning solutions mainly include the following: direct anthropometry, radiology-based facial scanning, facial scanner, deep learning-based 3D facial reconstruction technology, and neural radiation field technology-based 3D facial reconstruction technology.

[0004] Among them, the deep learning-based 3D facial reconstruction technology is based on facial images, marks facial feature points, and generates a 3D model through a model with prior knowledge. The advantage of this method is that it can use a large amount of facial data for training, improve the efficiency and stability of reconstruction, and can handle a certain degree of occlusion, expression, posture and other changes. The disadvantage is that the accuracy and details of the reconstruction are limited by the complexity and expressiveness of the model, and it is difficult to capture individual differences and subtle features. In addition, a pre-trained facial shape and texture model, or a deep learning-based network, is required as input prior information.

[0005] The 3D facial reconstruction technology based on neural radiation field technology shoots the face from multiple perspectives, and learns the 3D geometry and texture information of the face through images, camera poses, and neural networks. This method does not require dedicated scanning equipment, and only requires an ordinary camera or mobile phone to scan the face, which reduces hardware costs and difficulty of use, and improves the popularity and convenience of face scanning technology. High-quality 3D facial images can be rendered from any perspective, unaffected by occlusion, expression, posture and other factors, which improves the accuracy and details of facial scanning.

[0006] It can be seen that there are many existing methods and research on facial scanning. However, these existing technologies still have the following problems:

[0007] 1. It needs to rely on pre-trained face models or networks. These models or networks often have certain limitations and deviations. They are difficult to adapt to different changes in face shapes, textures, expressions, postures, etc., and it is also difficult to capture individual differences and subtle features. As a result, the edited face model may appear unsuitable, uncoordinated, and unrealistic in different scenarios, affecting the flexibility and naturalness of face editing.

[0008] 2. Lack of face editing technology based on 3D models, that is, it is impossible to modify and adjust the shape, color, expression, teeth and other attributes of the patient's face according to the needs of the patient or doctor, generate a new 3D image, and thus realize personalized facial design and preview. It cannot meet the personalized needs in the medical field.

[0009] Therefore, there is an urgent need in this field to develop new technologies for constructing editable facial 3D models using facial scanning data. Summary of the invention

[0010] In view of the problems of the prior art, the present invention provides an editable facial three-dimensional reconstruction method, system and storage medium.

[0011] An editable facial 3D reconstruction method comprises the following steps:

[0012] Acquire the image data of the modeler without expression and the image data of the modeler with expression taken from multiple angles;

[0013] Generate a head point cloud using the expressionless image data to obtain a neutral model;

[0014] Generate a head point cloud using the expressive image data to obtain a real expressive model;

[0015] Use a real expression model to train deformation field and color field;

[0016] The neutral model is subjected to the color field and the deformation field to obtain the final output model;

[0017] Obtaining a rendered image using the final output model, and training a super-resolution neural network using the rendered image and the expression image data;

[0018] Use the expressionless image data and the expression image data to train the expression encoder;

[0019] After dragging the facial feature points of the patient in the expressionless image data, the feature points are sent to the expression encoder, and the expression encoder encodes the coordinates of the dragged feature points into expression basis vectors;

[0020] The neutral model is passed through the expression basis vector and the final output model to form a three-dimensional facial model of the modeler with expression;

[0021] The expressive facial three-dimensional model obtains a high-definition rendering through a super-resolution network.

[0022] Preferably, the expressionless image data is selected from at least one of pictures or videos; and the expression image data is selected from at least one of pictures or videos.

[0023] Preferably, the construction of the neutral model comprises the following steps:

[0024] Step a, preprocessing the expressionless image data;

[0025] Step b: Use Colmap to reconstruct the pose, perform feature extraction and feature matching, and generate a sparse point cloud;

[0026] Step c, using the sparse point cloud as the initial point cloud of the 3D Gaussian Splatting algorithm for training to generate a head point cloud;

[0027] Step d: input the head point cloud into DMTet to generate a 3D model, thereby obtaining a neutral model.

[0028] Preferably, the training process of the expression encoder is:

[0029] Input facial feature points in expressionless image data and expression image data, and obtain expression basis vectors through expression encoder;

[0030] Generate expression images and expression facial feature points through expression basis vectors, and then calculate the loss with real expression image data and facial feature points of real expression image data, so as to continuously optimize the expression encoder;

[0031] The specific process of constructing the expression basis vector is expressed as:

[0032] Z exp = E exp (I in , P0)

[0033] Among them, Z exp is the expression basis vector, E exp is the expression encoder, I in P0 is the 68 feature points of the expressionless image data;

[0034] In the process of training the expression encoder, the following loss function is used:

[0035] L exp = L gen + L f1 + αL f2

[0036] Among them, L exp To construct the loss function of the expression basis vector, L gen is the mean square error loss of each pixel of the newly generated image and the original image, L f1 is the MSE Loss of the facial feature points after feature restoration and the original facial features, L f2 is the loss between the feature points of the generated expression image and the original feature points, and α is the switch for generating expression feature loss.

[0037] Preferably, the training process of the deformation field, color field and super-resolution network is:

[0038] The neutral model and the expression basis vectors are subjected to deformation field and color field to obtain an expression sparse point cloud;

[0039] The low-resolution image with expression sparse point cloud rendering is used to obtain the corresponding high-resolution image through a super-resolution neural network;

[0040] The neutral model includes a neutral point cloud position vector and a neutral point cloud feature point color vector, wherein the neutral point cloud position vector is a vector composed of position coordinates of expression feature points that have undergone position changes; and the neutral point cloud feature point color vector is a vector composed of color values ​​of expression feature points that have undergone position changes.

[0041] In the deformation field, the neutral point cloud position vector and expression basis vector form a matrix through shape change, and are concatenated in the channel dimension to form a tensor with 4 channels. The tensor is subjected to feature extraction and fusion through ResNet, and finally the position offset relative to the corresponding point of the neutral model is mapped through three fully connected layers;

[0042] In the color field, the neutral point cloud position vector, the neutral point cloud feature point color vector and the expression basis vector form a matrix through shape changes and are concatenated in the channel dimension to form a tensor with 4 channels. The tensor is feature extracted and fused through ResNet, and finally mapped out the position offset relative to the corresponding point of the neutral model through three fully connected layers.

[0043] Preferably, the following three loss functions are used in the process of training the deformation field and the color field:

[0044] L RGB = ||I rgb – I gt ||1

[0045] L sil = 1- IOU(M, M gt )

[0046] L def= ||P - P gt ||2

[0047] Among them, L RGB is the RGB loss, L sil is the contour loss, L def is the loss of facial feature points; I rgb is the RGB value output by the color field, I gt is the RGB value of the corresponding point of the real expression model, M is the facial contour of the generated point cloud, and M gt is the facial contour in the real expression model, IOU is the intersection-over-union ratio of two contours, P gt is the position of the actual expression model point, P is the position of the corresponding point of the generated point cloud; L RGB In the calculation formula, subscript 1 represents the L1 Loss function; L def In the calculation formula, the subscript 2 represents the L2 Loss function;

[0048] P = P0 + f def (P0, z exp )

[0049] P0 is the position of the corresponding point in the neutral point cloud, Z exp is the expression basis vector, f def is the deformation field;

[0050] In the process of training deformation field and color field, the prior constraint L is introduced offset , the L offset To penalize all non-zero displacements,

[0051] L offset =λ1counter(f def (P0, z exp ) = 0)

[0052] Where λ1 is a weight used to scale the loss; counter(f def (P0, z exp ) = 0) means that all the offsets of the feature points are calculated to be zero;

[0053] During the training of the super-resolution neural network, the following loss function is used:

[0054] L =λ2||I hr − I gt ||1+ (1-λ2)||I lr − I gt ||1

[0055] Among them, I hrHD images after the super-resolution neural network has improved the resolution. lr Low-resolution image rendered for the point cloud, I gt is a real expression image, and λ2 is the weight.

[0056] The present invention also provides a system for implementing the above-mentioned editable facial 3D reconstruction method, comprising:

[0057] The input module is configured to obtain a series of expressionless image data and expression image data of the modeler;

[0058] A neutral model building module is configured to generate a head point cloud using the expressionless image data to obtain a neutral model;

[0059] A real expression model construction module is configured to generate a head point cloud using the expression image data to obtain a real expression model;

[0060] The deformation field and color field training module is configured to train the deformation field and color field using a real expression model;

[0061] The final output model training module is configured to obtain the final output model by subjecting the neutral model to the color field and the deformation field;

[0062] A super-resolution neural network module is configured to obtain a rendered image using the final output model, and train a super-resolution neural network using the rendered image and the expression image data;

[0063] The expression base vector construction module is configured to drag the facial feature points of the patient in the expressionless image data and send them to the expression encoder, and the expression encoder encodes the coordinates of the dragged feature points into the expression base vector;

[0064] A facial three-dimensional model construction module is configured to construct a facial three-dimensional model with expression of the modeler by passing the neutral model through the expression basis vector and the final output model;

[0065] The high-definition rendering module is configured to obtain a high-definition rendering image of the expressive facial three-dimensional model through a super-resolution network.

[0066] Preferably, it also includes:

[0067] The dynamic adjustment module is configured to change the positions of the feature points, adjust the facial three-dimensional model, and generate a new facial three-dimensional model.

[0068] Preferably, in the dynamic adjustment module, the process of generating a new facial three-dimensional model includes: generating a new expression base according to the new feature point positions, and generating a new facial three-dimensional model using the deformation field and the color field;

[0069] Or, in the dynamic adjustment module, feature points corresponding to at least one type of expression or face shape are generated by a GAN model, and the feature points corresponding to at least one type of expression or face shape are integrated into at least one option;

[0070] Or, in the dynamic adjustment module, a transformer model is used to generate changes in sequence feature points, and changes in the sequence facial three-dimensional model are generated according to the changes in the sequence feature points.

[0071] The present invention also provides a computer-readable storage medium, on which is stored: a computer program for implementing the above-mentioned editable facial three-dimensional reconstruction method, or a computer program for implementing the above-mentioned system.

[0072] The present invention performs 3D reconstruction of human face based on neural radiation field technology. By editing facial feature points, the edited high-definition 3D model of the human face can be output through color field, deformation field and super-resolution neural network. The technical solution of the present invention has achieved the following beneficial technical effects:

[0073] 1. By moving the facial feature points, the present invention can modify the facial 3D model, thereby finely editing the modeler's face, while ensuring that the edited model can still retain the modeler's basic facial features and identity features. It can meet the personalized needs of the medical field. In the preferred solution, the feature points corresponding to common expressions and face shapes can be made into options, and the corresponding facial 3D model can be generated by clicking, without complex manual operations, and the flexibility and convenience of adjusting the facial 3D model are improved.

[0074] 2. The present invention utilizes the combined action of multiple multi-layer perceptron (MLP) neural networks (including expression encoders, deformation fields, and color fields) to generate an edited three-dimensional facial model; further, a super-resolution network is used to improve the resolution of the image; in addition, the present invention also generates an expression-based feature vector based on feature points. Therefore, the three-dimensional facial model of the present invention ensures the naturalness and aesthetics of facial plastic surgery, and improves authenticity and personalization.

[0075] In summary, the facial three-dimensional model generated by the present invention is accurate, natural, beautiful, and realistic, and can be flexibly and conveniently adjusted for personalization, and has good application prospects in the fields of medicine and medical aesthetics.

[0076] Obviously, according to the above contents of the present invention, in accordance with common technical knowledge and customary means in the art, without departing from the above basic technical ideas of the present invention, other various forms of modification, replacement or change may be made.

[0077] The above contents of the present invention are further described in detail below through specific implementation methods in the form of embodiments. However, this should not be understood as the scope of the above subject matter of the present invention being limited to the following examples. All technologies realized based on the above contents of the present invention belong to the scope of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 Example diagram of the dataset used to construct expression basis vectors;

[0079] Figure 2 Schematic diagram of the network structure for constructing expression basis vectors;

[0080] Figure 3 Schematic diagram of the process of training deformation field, color field and super-resolution neural networks.

[0081] Figure 4 Schematic diagram of the specific structure of deformation field and color field. DETAILED DESCRIPTION

[0082] It should be noted that the algorithms of data collection, transmission, storage and processing steps not specifically described in the embodiments, as well as the hardware structure, circuit connection, etc. not specifically described can all be implemented through the contents disclosed in the prior art.

[0083] Embodiment 1 Editable facial 3D reconstruction method and system

[0084] This embodiment provides a system for facial 3D reconstruction, specifically comprising:

[0085] The input module is configured to obtain a series of expressionless image data and expression image data of the modeler;

[0086] A neutral model building module is configured to generate a head point cloud using the expressionless image data to obtain a neutral model;

[0087] A real expression model construction module is configured to generate a head point cloud using the expression image data to obtain a real expression model;

[0088] The deformation field and color field training module is configured to train the deformation field and color field using a real expression model;

[0089] The final output model training module is configured to obtain the final output model by subjecting the neutral model to the color field and the deformation field;

[0090] A super-resolution neural network module is configured to obtain a rendered image using the final output model, and train a super-resolution neural network using the rendered image and the expression image data;

[0091] The expression base vector construction module is configured to drag the facial feature points of the patient in the expressionless image data and send them to the expression encoder, and the expression encoder encodes the coordinates of the dragged feature points into the expression base vector;

[0092] A facial three-dimensional model construction module is configured to construct a facial three-dimensional model with expression of the modeler by passing the neutral model through the expression basis vector and the final output model;

[0093] A high-definition rendering module is configured to obtain a high-definition rendering image of the expressive facial three-dimensional model through a super-resolution network;

[0094] The dynamic adjustment module is configured to change the positions of the feature points, adjust the facial three-dimensional model, and generate a new facial three-dimensional model.

[0095] In the dynamic adjustment module, the process of generating a new facial 3D model includes: generating a new expression base according to the new feature point position, and generating a new facial 3D model using the deformation field and the color field; as a preferred method, in order to facilitate users to quickly select, generating feature points corresponding to at least one type of expression or face shape through the GAN model, and the feature points corresponding to at least one type of expression or face shape are integrated into at least one option; as a preferred method, in order to dynamically demonstrate the facial 3D model, generating sequential feature point changes through the transformer model, and generating sequential facial 3D model changes according to the sequential feature point changes.

[0096] The method for facial 3D reconstruction using the above system specifically includes the following steps:

[0097] Step 1, obtaining expressionless image data and expression image data of the modeler taken from multiple angles; the expressionless image data is selected from at least one of pictures or videos; the expression image data is selected from at least one of pictures or videos. The pictures or videos can be collected by ordinary mobile phones or camera equipment. As a preferred method, the video collection method is: 360 surround around the modeler's head or spiral surround recording aimed at the patient's face. The collection content of the expression image (video) data includes nodding, shaking head, blinking, smiling, opening mouth and other expressions.

[0098] Step 2, using the expressionless image data to generate a head point cloud to obtain a neutral model; using the expression image data to generate a head point cloud to obtain a real expression model;

[0099] Step 3, using a real expression model to train the deformation field and the color field; the neutral model is transformed through the color field and the deformation field to obtain the final output model;

[0100] Step 4, using the final output model to obtain a rendered image, and using the rendered image and the modeler's facial expression image data taken from multiple angles to train the super-resolution neural network. During the training, the image data output by the super-resolution neural network itself can be further added to the training data.

[0101] Step 5, using the expressionless image data and the expression image data to train the expression encoder;

[0102] Step 6, dragging the facial feature points of the patient in the expressionless image data and sending them to the expression encoder, and the expression encoder encodes the coordinates of the dragged feature points into expression basis vectors;

[0103] Step 7, the neutral model is passed through the expression basis vector and the final output model to form a three-dimensional facial model with expression of the modeler;

[0104] Step 8: The expressive facial 3D model is used to obtain a high-definition rendering through a super-resolution network.

[0105] In step 2, the construction of the neutral model includes the following steps:

[0106] Step 2.1, preprocess the expressionless image data; taking video data as an example, the preprocessing process includes:

[0107] Step 2.2, use Colmap to reconstruct the pose, perform feature extraction and feature matching, and generate a sparse point cloud;

[0108] Step 2.3, using the sparse point cloud as the initial point cloud of the 3D Gaussian Splatting algorithm for training to generate a head point cloud;

[0109] Step 2.4, input the head point cloud into DMTet to generate a 3D model, thereby obtaining a neutral model.

[0110] In step 5, we first need to construct such a dataset based on the video with table, where each data Figure 1 As shown in Figure 1, it includes: expressionless face pictures, expression face pictures, and expression face feature points. In order to be able to control the expression more finely and use the feature points to control the 3D model. This study uses an encoder to extract expression features and construct expression basis vectors. The training process of the expression encoder is as follows: Figure 2 As shown, the specific steps are:

[0111] Input facial feature points in expressionless image data and expression image data, and obtain expression basis vectors through expression encoder;

[0112] Expressive images and expressive facial feature points are generated through expression basis vectors, and then loss calculation is performed with real expressive image data and facial feature points of real expressive image data respectively, so as to continuously optimize the expression encoder.

[0113] The process of constructing the expression basis vector is expressed as:

[0114] Z exp = E exp (I in , P0)

[0115] Among them, Z exp is the expression basis vector, E exp is the expression encoder, I in P0 is the 68 feature points of the expressionless image data. exp The image will be compressed into a 1024 feature vector.

[0116] In the process of training the expression encoder, the following loss function is used:

[0117] L exp = L gen + L f1 + αL f2

[0118] Among them, L exp To construct the loss function of the expression basis vector, L gen is the mean square error loss of each pixel of the newly generated image and the original image, L f1 is the MSE Loss of the facial feature points after feature restoration and the original facial features, L f2 is the loss between the feature points of the generated expression image and the original feature points, and α is the switch for generating expression feature loss.

[0119] L exp = ||I in -I out ||2+ ||P0– P1||2+ α||P0– P2||2

[0120] α = 0.1*I(L gen <0.1)

[0121] P1 is the facial feature point generated by the feature recovery network (i.e. Figure 2 I out is the result after decoding by the decoder, and P2 is the facial features after generating expressions (i.e. Figure 2It should be noted that the generated expression images cannot be recognized at the beginning of network training, so this study sets α as the switch for generating expression feature loss, and I is an indicator function, indicating that only when L gen L will be turned on when it is less than a certain threshold f2 loss.

[0122] In step 4, the training process of the deformation field, color field and super-resolution network is as follows: Figure 3 As shown, the specific steps are:

[0123] The neutral model and the expression basis vectors are subjected to deformation field and color field to obtain an expression sparse point cloud;

[0124] The low-resolution image with expression sparse point cloud rendering is used to obtain the corresponding high-resolution image through a super-resolution neural network;

[0125] The neutral model includes a neutral point cloud position vector and a neutral point cloud feature point color vector. The neutral point cloud position vector is a vector composed of position coordinates of expression feature points that have undergone position changes; the neutral point cloud feature point color vector is a vector composed of color values ​​of expression feature points that have undergone position changes.

[0126] The structures of deformation field and color field are as follows Figure 4 shown.

[0127] In the deformation field, the neutral point cloud position vector and expression basis vector form a matrix through shape change, and are concatenated in the channel dimension to form a tensor with 4 channels. The tensor is subjected to feature extraction and fusion through ResNet, and finally the position offset relative to the corresponding point of the neutral model is mapped through three fully connected layers;

[0128] In the color field, the neutral point cloud position vector, the neutral point cloud feature point color vector and the expression basis vector form a matrix through shape changes and are concatenated in the channel dimension to form a tensor with 4 channels. The tensor is feature extracted and fused through ResNet, and finally mapped out the position offset relative to the corresponding point of the neutral model through three fully connected layers.

[0129] In the process of training deformation field and color field, the following three loss functions are used:

[0130] L RGB = ||I rgb – I gt ||1

[0131] L sil = 1- IOU(M, M gt )

[0132] Ldef = ||P - P gt ||2

[0133] Among them, L RGB is the RGB loss, L sil is the contour loss, L def is the loss of facial feature points; I rgb The RGB value output for the color field (i.e. Figure 3 RGB values ​​of "high resolution image" in I gt is the RGB value of the corresponding point of the real expression model (i.e. Figure 3 ), M is the facial contour of the expression sparse point cloud, and M gt is the facial contour in the real expression model, IOU is the intersection-over-union ratio of two contours, P gt is the position of the actual expression model point, P is the position of the corresponding point in the expression sparse point cloud; L RGB In the calculation formula, subscript 1 represents the L1 Loss function; L def In the calculation formula, the subscript 2 represents the L2 Loss function;

[0134] P = P0 + f def (P0, z exp )

[0135] P0 is the position of the corresponding point in the neutral point cloud, Z exp is the expression basis vector, f def is the deformation field;

[0136] In the process of training deformation field and color field, the prior constraint L is introduced offset , the L offset To penalize all non-zero displacements,

[0137] L offset =λ1counter(f def (P0, z exp ) = 0)

[0138] Where λ1 is a weight used to scale the loss; counter(f def (P0, z exp ) = 0) means that all the offsets of the feature points are calculated to be zero.

[0139] When training the super-resolution neural network, the following loss function is used:

[0140] L =λ2||I hr − I gt ||1+ (1-λ2)||I lr− I gt ||1

[0141] Among them, I hr HD images after the super-resolution neural network has improved the resolution. lr Low-resolution image rendered for the point cloud, I gt is a real expression image, and λ2 is the weight.

[0142] The method of using the model constructed by the above process through the dynamic adjustment module includes:

[0143] The user can drag the feature points on the expressionless picture of the modeler of the facial 3D model. The dynamic adjustment module can generate a new expression base according to the new feature point positions, thereby generating a new facial 3D model using the deformation field and color field control.

[0144] As a preferred method, for ease of use, the dynamic adjustment module can train and integrate GAN to generate feature points of specified expressions or face shapes, and make them into option buttons. When the user clicks the button, the corresponding feature points can be generated, and the user can then use the mouse to drag and slightly adjust the feature points, thereby performing rapid facial editing.

[0145] As a preferred method, in order to show the changes of feature points in the dynamic process, the dynamic adjustment module can integrate a transformer, and the transformer is used to generate sequential feature point changes, so that the dynamic three-dimensional facial model can be directly observed.

[0146] From the above embodiments, it can be seen that the present invention performs three-dimensional reconstruction of the face based on neural radiation field technology, can generate an accurate, natural, beautiful and realistic three-dimensional facial model, and can perform flexible and convenient personalized adjustments, and has good application prospects in the fields of medicine and medical aesthetics.

Claims

1. An editable facial 3D reconstruction method, characterized in that: The steps include: Acquire the image data of the modeler without expression and the image data of the modeler with expression taken from multiple angles; Generate a head point cloud using the expressionless image data to obtain a neutral model; Generate a head point cloud using the expressive image data to obtain a real expressive model; Use a real expression model to train deformation field and color field; The neutral model is subjected to the color field and the deformation field to obtain the final output model; Obtaining a rendered image using the final output model, and training a super-resolution neural network using the rendered image and the expression image data; Use the expressionless image data and the expression image data to train the expression encoder; After dragging the facial feature points of the patient in the expressionless image data, the feature points are sent to the expression encoder, and the expression encoder encodes the coordinates of the dragged feature points into expression basis vectors; The expression basis vectors and the final output model constitute a three-dimensional facial model of the modeler with expressions; The three-dimensional facial model with expression obtains a high-definition rendering through a super-resolution network; The training process of the super-resolution network is: The neutral model and the expression basis vectors are subjected to deformation field and color field to obtain an expression sparse point cloud; The low-resolution image with expression sparse point cloud rendering is used to obtain the corresponding high-resolution image through a super-resolution neural network; The training process of the deformation field and color field is as follows: The neutral model includes a neutral point cloud position vector and a neutral point cloud feature point color vector, wherein the neutral point cloud position vector is a vector composed of position coordinates of expression feature points that have undergone position changes; and the neutral point cloud feature point color vector is a vector composed of color values ​​of expression feature points that have undergone position changes. In the deformation field, the neutral point cloud position vector and expression basis vector form a matrix through shape change, and are concatenated in the channel dimension to form a tensor with 4 channels. The tensor is subjected to feature extraction and fusion through ResNet, and finally the position offset relative to the corresponding point of the neutral model is mapped through three fully connected layers; In the color field, the neutral point cloud position vector, the neutral point cloud feature point color vector and the expression basis vector are transformed into a matrix through shape changes, and are concatenated in the channel dimension to form a tensor with 4 channels. The tensor is subjected to feature extraction and fusion through ResNet, and finally the position offset relative to the corresponding point of the neutral model is mapped out through three fully connected layers; The following three loss functions are used in the process of training deformation field and color field: L RGB = ||I rgb – I gt ||1 L sil = 1- IOU(M, M gt ) L def = ||P - P gt ||2 Among them, L RGB is the RGB loss, L sil is the contour loss, L def is the loss of facial feature points; I rgb The RGB value output by the color field, I gt is the RGB value of the corresponding point of the real expression model, M is the facial contour of the generated point cloud, and M gt is the facial contour in the real expression model, IOU is the intersection-over-union ratio of two contours, P gt is the position of the actual expression model point, P is the position of the corresponding point of the generated point cloud; L RGB In the calculation formula, subscript 1 represents the L1 Loss function; L def In the calculation formula, the subscript 2 represents the L2 Loss function; P = P0 + f def (P0, z exp ) P0 is the position of the corresponding point in the neutral point cloud, Z exp is the expression basis vector, f def is the deformation field; In the process of training deformation field and color field, the prior constraint L is introduced offset , the L offset To penalize all non-zero displacements, L offset =λ1counter(f def (P0, z exp ) = 0) Where λ1 is a weight used to scale the loss; counter(f def (P0, z exp ) = 0) means that all the offsets of the feature points are calculated to be zero; During the training of the super-resolution neural network, the following loss function is used: L =λ2||I hr − In gt ||1+ (1-λ2)||I lr − In gt ||1 Among them, I hr HD images after the super-resolution neural network has improved the resolution. lr Low-resolution image rendered for the point cloud, I gt is a real expression image, λ2 is the weight; The specific process of constructing the expression basis vector is expressed as: Z exp = E exp (I in , P0) Among them, Z exp is the expression basis vector, E exp is the expression encoder, I in P0 is the expressionless image data, and P1 is the 68 feature points of the expression image data.

2. The editable facial 3D reconstruction method according to claim 1, characterized in that: The expressionless image data is selected from at least one of a picture or a video; the expression image data is selected from at least one of a picture or a video.

3. The editable facial 3D reconstruction method according to claim 1, characterized in that: The training process of the expression encoder is: Input facial feature points in expressionless image data and expression image data, and obtain expression basis vectors through expression encoder; Generate expression images and expression facial feature points through expression basis vectors, and then calculate the loss with real expression image data and facial feature points of real expression image data, so as to continuously optimize the expression encoder; In the process of training the expression encoder, the following loss function is used: L exp = L gen + L f1 + αL f2 Among them, L exp To construct the loss function of the expression basis vector, L gen is the mean square error loss of each pixel of the newly generated image and the original image, L f1 is the MSE Loss of the facial feature points after feature restoration and the original facial features, L f2 is the loss between the feature points of the generated expression image and the original feature points, and α is the switch for generating expression feature loss.

4. A system for implementing the editable facial 3D reconstruction method according to any one of claims 1 to 3, characterized in that: include: The input module is configured to obtain a series of expressionless image data and expression image data of the modeler; A neutral model building module is configured to generate a head point cloud using the expressionless image data to obtain a neutral model; A real expression model construction module is configured to generate a head point cloud using the expression image data to obtain a real expression model; The deformation field and color field training module is configured to train the deformation field and color field using a real expression model; The final output model training module is configured to obtain the final output model by subjecting the neutral model to the color field and the deformation field; A super-resolution neural network module is configured to obtain a rendered image using the final output model, and train a super-resolution neural network using the rendered image and the expression image data; The expression base vector construction module is configured to drag the facial feature points of the patient in the expressionless image data and send them to the expression encoder, and the expression encoder encodes the coordinates of the dragged feature points into the expression base vector; A facial three-dimensional model building module is configured such that the expression basis vectors and the final output model constitute a facial three-dimensional model with expression of the modeler; A high-definition rendering module is configured to obtain a high-definition rendering image of the expressive facial three-dimensional model through a super-resolution network; The training process of the deformation field, color field and super-resolution network is as follows: The neutral model and the expression basis vectors are subjected to deformation field and color field to obtain an expression sparse point cloud; The low-resolution image with expression sparse point cloud rendering is used to obtain the corresponding high-resolution image through a super-resolution neural network; The training process of the deformation field and color field is as follows: The neutral model includes a neutral point cloud position vector and a neutral point cloud feature point color vector, wherein the neutral point cloud position vector is a vector composed of position coordinates of expression feature points that have undergone position changes; and the neutral point cloud feature point color vector is a vector composed of color values ​​of expression feature points that have undergone position changes. In the deformation field, the neutral point cloud position vector and expression basis vector form a matrix through shape change, and are concatenated in the channel dimension to form a tensor with 4 channels. The tensor is subjected to feature extraction and fusion through ResNet, and finally the position offset relative to the corresponding point of the neutral model is mapped through three fully connected layers; In the color field, the neutral point cloud position vector, the neutral point cloud feature point color vector and the expression basis vector are transformed into a matrix through shape changes, and are concatenated in the channel dimension to form a tensor with 4 channels. The tensor is subjected to feature extraction and fusion through ResNet, and finally the position offset relative to the corresponding point of the neutral model is mapped out through three fully connected layers; The following three loss functions are used in the process of training deformation field and color field: L RGB = ||I rgb – I gt ||1 L sil = 1- IOU(M, M gt ) L def = ||P - P gt ||2 Among them, L RGB is the RGB loss, L sil is the contour loss, L def is the loss of facial feature points; I rgb The RGB value output by the color field, I gt is the RGB value of the corresponding point of the real expression model, M is the facial contour of the generated point cloud, and M gt is the facial contour in the real expression model, IOU is the intersection-over-union ratio of two contours, P gt is the position of the actual expression model point, P is the position of the corresponding point of the generated point cloud; L RGB In the calculation formula, subscript 1 represents the L1 Loss function; L def In the calculation formula, the subscript 2 represents the L2 Loss function; P = P0 + f def (P0, z exp ) P0 is the position of the corresponding point in the neutral point cloud, Z exp is the expression basis vector, f def is the deformation field; In the process of training deformation field and color field, the prior constraint L is introduced offset , the L offset To penalize all non-zero displacements, L offset =λ1counter(f def (P0, z exp ) = 0) Where λ1 is a weight used to scale the loss; counter(f def (P0, z exp ) = 0) means that all the offsets of the feature points are calculated to be zero; During the training of the super-resolution neural network, the following loss function is used: L =λ2||I hr − In gt ||1+ (1-λ2)||I lr − In gt ||1 Among them, I hr HD images after the super-resolution neural network has improved the resolution. lr Low-resolution image rendered for the point cloud, I gt is a real image with expression, λ2 is the weight; The specific process of constructing the expression basis vector is expressed as: Z exp = E exp (I in , P0) Among them, Z exp is the expression basis vector, E exp is the expression encoder, I in P0 is the expressionless image data, and P1 is the 68 feature points of the expression image data.

5. The system according to claim 4, characterized in that Also includes: The dynamic adjustment module is configured to change the positions of the feature points, adjust the facial three-dimensional model, and generate a new facial three-dimensional model.

6. The system according to claim 5, characterized in that: In the dynamic adjustment module, the process of generating a new facial three-dimensional model includes: generating a new expression base according to the new feature point positions, and generating a new facial three-dimensional model using the deformation field and the color field; Or, in the dynamic adjustment module, feature points corresponding to at least one type of expression or face shape are generated by a GAN model, and the feature points corresponding to at least one type of expression or face shape are integrated into at least one option; Or, in the dynamic adjustment module, a transformer model is used to generate changes in sequence feature points, and changes in the sequence facial three-dimensional model are generated according to the changes in the sequence feature points.

7. A computer-readable storage medium, characterized in that: Stored thereon is: a computer program for implementing the editable facial three-dimensional reconstruction method described in any one of claims 1-3, or a computer program for implementing the system described in any one of claims 4-6.

Images