A method, system, device and storage medium for three-dimensional reconstruction of human face
By transforming the face pictures and building continuous video frames, the three-dimensional reconstruction coefficient is determined, which solves the problem of unstable face three-dimensional reconstruction in the existing technology, improves the authenticity and stability of virtual faces, and improves the user experience .
Patent Information
- Application Number
- CN202310827554.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-07
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2043-07-07
AI Technical Summary
The existing three-dimensional face reconstruction technology lacks stability when reconstructing faces, and is prone to jitter and convulsions, resulting in poor user experience and ignoring the continuous frame information of the video.
By transforming the preset face pictures, multiple target face pictures are generated, and continuous video frames are constructed based on these pictures, the three-dimensional reconstruction coefficient is determined, and the virtual face is finally generated. The method includes rotating, translating, scaling, adding noise and blurring to the face picture, and using continuous video frames to train the initial prediction model of the three-dimensional reconstruction coefficients to generate realistic virtual faces.
It improves the quality and stability of three-dimensional reconstruction of human faces, makes virtual faces more realistic and natural, and improves user experience.
Smart Images

Figure CN116934963B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of virtual social interaction, and in particular to a method, system, device and storage medium for three-dimensional reconstruction of a human face. Background Art
[0002] In stranger social scenarios, users need to remain anonymous for video chats, so they need to use 3D virtual headgear instead of their real heads for interaction. To achieve this function, 3D facial reconstruction technology is needed to capture the user's facial information. Generally, 3D facial reconstruction technology uses convolutional neural networks to learn the mapping from facial images to 3D reconstruction coefficients to meet the real-time requirements in video scenarios.
[0003] However, existing face 3D reconstruction technology only uses a single face image and ignores the continuous frame information of the video. This results in the lack of stability of existing technology when reconstructing faces, and is prone to jitter and twitching problems, which brings a poor experience to users. Summary of the invention
[0004] The purpose of this application is to provide a method, system, device and storage medium for three-dimensional reconstruction of a face. By using multiple target face images to construct continuous video frames, more information from different angles and expressions can be captured, thereby improving the quality and stability of three-dimensional reconstruction of the face, improving the user experience, and making the virtual face more realistic and stable.
[0005] In order to solve the above technical problems, the present application provides a method for 3D reconstruction of a face, comprising:
[0006] Perform transformation processing on the preset face image to obtain multiple target face images;
[0007] Constructing continuous video frames according to a plurality of target face images;
[0008] A three-dimensional reconstruction coefficient is determined according to the continuous frames of the video, so as to generate a virtual face according to the three-dimensional reconstruction coefficient and the face picture to be processed.
[0009] In one embodiment, the preset face image is transformed to obtain multiple target face images, including:
[0010] The preset face image is rotated and / or translated and / or scaled to obtain a plurality of first target face images.
[0011] In one embodiment, the preset face image is transformed to obtain multiple target face images, including:
[0012] Preset noise and / or preset blur are added to the preset face image to obtain a plurality of second target face images.
[0013] In one embodiment, determining a 3D reconstruction coefficient according to the continuous frames of the video, and generating a virtual face according to the 3D reconstruction coefficient and a face picture to be processed, comprises:
[0014] Inputting the continuous frames of the video into an initial prediction model of 3D reconstruction coefficients to train the initial model of 3D reconstruction coefficients to obtain a target prediction model of 3D reconstruction coefficients;
[0015] The face image to be processed is input into the three-dimensional reconstruction coefficient target prediction model to obtain a virtual face corresponding to the face image to be processed.
[0016] In one embodiment, the continuous video frames are input into an initial prediction model of 3D reconstruction coefficients to train the initial model of 3D reconstruction coefficients to obtain a target prediction model of 3D reconstruction coefficients, including:
[0017] The continuous video frames and preset random face data are input into the initial prediction model of the 3D reconstruction coefficients to train the initial model of the 3D reconstruction coefficients and obtain a target prediction model of the 3D reconstruction coefficients.
[0018] In one embodiment, the continuous video frames are input into an initial prediction model of 3D reconstruction coefficients to train the initial model of 3D reconstruction coefficients to obtain a target prediction model of 3D reconstruction coefficients, including:
[0019] Inputting the continuous frames of the video into an initial prediction model of 3D reconstruction coefficients, constraining the initial prediction model of 3D reconstruction coefficients based on a preset function to train the initial prediction model of 3D reconstruction coefficients, and obtaining a target prediction model of 3D reconstruction coefficients;
[0020] The preset function is constructed based on the standard deviation of the facial information coefficients of n different frames of the facial images corresponding to the same person, where n is an integer greater than 2.
[0021] In one embodiment, the expression of the preset function is:
[0022] L=||w·(p_gt-p)|| 2 +||V(p_gt)-V(p)|| 2 +std(p_id_1,p_id_2,...,p_id_n);
[0023] Among them, w represents different weights, p represents the three-dimensional face reconstruction coefficient output by the initial prediction model of the three-dimensional reconstruction coefficient, p_gt represents the true value of the annotation result corresponding to p, V represents the process of three-dimensional reconstruction using the three-dimensional reconstruction parameter p, std represents the calculated standard deviation, and p_id_n represents the face information coefficient of n different frames of the face image corresponding to the same person.
[0024] In order to solve the above technical problems, the present application also provides a human face three-dimensional reconstruction system, comprising:
[0025] An image processing unit is used to transform a preset face image to obtain multiple target face images;
[0026] A video frame construction unit, used to construct continuous video frames according to a plurality of target face images;
[0027] The 3D reconstruction coefficient determination unit is used to determine the 3D reconstruction coefficient according to the continuous frames of the video, so as to generate a virtual face according to the 3D reconstruction coefficient and the face picture to be processed.
[0028] In order to solve the above technical problems, the present application also provides a three-dimensional face reconstruction device, comprising:
[0029] Memory for storing computer programs;
[0030] The processor is used to implement the steps of the above-mentioned method for three-dimensional reconstruction of the face when storing the computer program.
[0031] In order to solve the above technical problems, the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the three-dimensional face reconstruction method as described above are implemented.
[0032] The present application provides a method, system, device and storage medium for three-dimensional reconstruction of a face, which relates to the field of virtual social interaction and solves the problem of unstable three-dimensional reconstruction coefficients. In this solution, a preset face image is transformed to obtain multiple target face images; a video continuous frame is constructed based on the multiple target face images; a three-dimensional reconstruction coefficient is determined based on the video continuous frame, so as to generate a virtual face based on the three-dimensional reconstruction coefficient and the face image to be processed. It can be seen that in this application, by using multiple target face images to construct a video continuous frame, more information of different angles and expressions can be captured, thereby improving the quality and stability of three-dimensional reconstruction of the face, improving the user experience, and making the virtual face more realistic and stable. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the prior art and the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0034] Figure 1 A schematic diagram of a process for 3D reconstruction of a face provided in this application;
[0035] Figure 2 A structural block diagram of a 3D face reconstruction system provided in this application;
[0036] Figure 3 This is a structural block diagram of a three-dimensional face reconstruction device provided in this application. DETAILED DESCRIPTION
[0037] The core of this application is to provide a method, system, device and storage medium for three-dimensional reconstruction of a face. By using multiple target face images to construct continuous video frames, more information from different angles and expressions can be captured, thereby improving the quality and stability of three-dimensional reconstruction of the face, improving the user experience, and making the virtual face more realistic and stable.
[0038] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0039] Please refer to Figure 1 , Figure 1 A schematic diagram of a process for 3D reconstruction of a face provided in this application, the method comprising:
[0040] S11: transforming the preset face image to obtain multiple target face images;
[0041] In this embodiment, multiple target face images are obtained by transforming the preset face images. The idea of this embodiment is to improve the quality and stability of face 3D reconstruction by increasing the number of target face images. By transforming the preset face images, target face images with different expressions, different angles and different lighting conditions can be obtained, thereby capturing more face information.
[0042] This embodiment can use a variety of technical means to perform transformation processing, such as facial key point detection and alignment, posture estimation and transformation, expression recognition and conversion, etc. Through these technical means, the preset face image can be deformed and transformed to generate multiple target face images.
[0043] In one embodiment, transforming the preset face image to obtain a plurality of target face images includes: rotating and / or translating and / or scaling the preset face image to obtain a plurality of first target face images.
[0044] This embodiment mainly describes the steps of transforming the preset face image. These transformations include rotation, translation and scaling. By performing these transformations on the preset face image, multiple first target face images can be obtained.
[0045] Specifically, rotating the preset face image can rotate the face image on a plane, thereby changing the orientation of the face in the image (by gradually adding a small rotation angle to the preset face image, to simulate the rotational movement in the actual video). Translating the preset face image can move the face image horizontally and vertically on the image, thereby adjusting the position of the face in the image (by gradually adding a small zoom to the preset face image, to simulate the forward and backward movement in the actual video). Scaling the preset face image can change the size of the face image, enlarging or reducing it (by gradually adding a small displacement to the preset face image, to simulate the translational movement in the actual video).
[0046] Through these transformation processes, multiple first target face images can be generated. These images have different orientations, positions and sizes, providing more information for subsequent three-dimensional reconstruction. In this way, in subsequent steps, video continuous frames can be constructed based on these multiple target face images, and these continuous frames can be used to determine the three-dimensional reconstruction coefficients. Finally, a virtual face can be generated based on these coefficients and the face image to be processed.
[0047] In one embodiment, transforming the preset face picture to obtain a plurality of target face pictures includes: adding preset noise and / or preset blur to the preset face picture to obtain a plurality of second target face pictures.
[0048] In this embodiment, the process of transforming the preset face image to obtain multiple target face images is further concretized. Specifically, the transformation process includes two parts, namely, adding preset noise and / or preset blur. Adding preset noise means performing some noise processing on the preset face image to make it more blurred or unclear. This can simulate the poor image quality in the real world (that is, by adding various noises (such as Gaussian noise, salt and pepper noise) to the preset face image to simulate the noise problem in the actual video), so that the face three-dimensional reconstruction method can also produce accurate results for some blurred or noisy input images. Adding preset blur means blurring the preset face image to make it lose details and clarity. This can simulate some situations, for example, in dynamic videos, the face may be blurred due to movement. By performing preset blur processing on the image, this situation can be better dealt with and the accuracy of the face three-dimensional reconstruction method on these blurred images can be ensured (that is, by adding various blurs (such as Gaussian blur, motion blur) to the preset face image to simulate the blur phenomenon in the actual video).
[0049] In this embodiment, the preset face image is transformed to obtain multiple target face images. Specifically, the transformation process in this step includes adding preset noise and / or preset blur to better adapt to some situations where the image quality is poor or blurred.
[0050] In summary, step S11 can improve the quality and stability of face 3D reconstruction by using multiple target face images to construct continuous video frames. Compared with using only a single image for reconstruction, multiple target face images can capture more information from different angles and expressions, making the virtual face more realistic and stable. This helps to improve the user experience and make the virtual face more natural and realistic in the application.
[0051] S12: constructing video continuous frames based on multiple target face images;
[0052] This embodiment constructs continuous video frames based on multiple target face images to provide continuous data input for subsequent 3D reconstruction. The idea of this embodiment is to capture the dynamic changes of the face at different time points by constructing continuous video frames, thereby providing more complete and continuous face information.
[0053] Specifically, the present embodiment plays a plurality of target face pictures continuously in a specific order to construct a simulated video effect. The specific implementation method can be to arrange a plurality of target face pictures in chronological order, and then play these pictures continuously at a certain frame rate (such as how many frames are played per second). In this way, a continuous video can be obtained, in which each frame is composed of target face pictures. The purpose of doing so is to be able to perform three-dimensional reconstruction based on target face pictures of multiple different angles or expressions. By constructing continuous frames of video, more information and angles can be provided to more accurately determine the three-dimensional reconstruction coefficients. In subsequent steps, these three-dimensional reconstruction coefficients will be used to generate a virtual face similar to the face picture to be processed.
[0054] S13: Determine three-dimensional reconstruction coefficients according to the continuous frames of the video, so as to generate a virtual face according to the three-dimensional reconstruction coefficients and the face image to be processed.
[0055] In this embodiment, the coefficients required for three-dimensional reconstruction are determined by using the previously constructed continuous frames of video, and then the face picture to be processed is converted into a virtual face using these coefficients. Specifically, first, based on the face image information in the continuous frames of the video, the feature points and geometric shapes of the face in each frame, as well as the illumination and texture information are extracted. Then, this information is used to calculate the coefficients required for three-dimensional reconstruction. The three-dimensional reconstruction coefficients may include information on face information coefficients, face expression coefficients, and face posture coefficients. These coefficients can be calculated using a specific algorithm or model. By using the face image information in the continuous frames of the video, the three-dimensional reconstruction coefficients suitable for the face picture to be processed can be accurately obtained.
[0056] Through this method, the face to be processed can be reconstructed in three dimensions based on the facial information in the continuous frames of the video, and a virtual face with realistic effects can be generated. That is, in this embodiment, by combining the continuous frames of the video and the face picture to be processed, a virtual three-dimensional face model is generated using the three-dimensional reconstruction coefficients. A three-dimensional reconstruction technology based on deep learning is adopted, for example, a convolutional neural network is used for three-dimensional face reconstruction. By using the continuous frames of the video as prior information and using the face picture to be processed as input, a virtual three-dimensional face model can be generated by the generated three-dimensional reconstruction coefficients.
[0057] In summary, by using continuous video frames to determine the 3D reconstruction coefficients and combining the face images to be processed to generate virtual faces, this step can improve the precision and accuracy of face 3D reconstruction. This helps to generate more realistic and detailed virtual faces, making the virtual faces more realistic and natural in applications. At the same time, by using continuous video frames and face images to be processed, this step can make full use of diverse face information, thereby improving the diversity and plasticity of virtual faces.
[0058] In one embodiment, determining a 3D reconstruction coefficient according to continuous frames of a video to generate a virtual face according to the 3D reconstruction coefficient and a face image to be processed includes:
[0059] Inputting the continuous video frames into the initial prediction model of the 3D reconstruction coefficients to train the initial model of the 3D reconstruction coefficients and obtain a target prediction model of the 3D reconstruction coefficients;
[0060] The face image to be processed is input into the three-dimensional reconstruction coefficient target prediction model to obtain a virtual face corresponding to the face image to be processed.
[0061] This embodiment describes a method for determining three-dimensional reconstruction coefficients according to continuous frames of video in an embodiment to generate a virtual face corresponding to the face picture to be processed. In this embodiment, the continuous frames of video are first input into the initial prediction model of the three-dimensional reconstruction coefficients, and the model is trained to obtain a target prediction model of the three-dimensional reconstruction coefficients. This means that through training, the model can accurately predict the corresponding three-dimensional reconstruction coefficients based on the input continuous frames of video. Next, the face picture to be processed is input into the target prediction model of the three-dimensional reconstruction coefficients, and the virtual face corresponding to the face picture to be processed is obtained through the prediction ability of the model. In this way, by inputting the face picture to be processed, a virtual face can be obtained, and the virtual face is associated with the actual face.
[0062] In this embodiment, continuous video frames are used to determine 3D reconstruction coefficients, and high-quality virtual faces can be generated by training and predicting the 3D reconstruction coefficients. By corresponding to the face images to be processed, these virtual faces can be used in various application fields, such as virtual reality, game development, and face recognition.
[0063] In addition, the method in this embodiment is also scalable. The model can be trained by using more continuous frames of video in the training phase to improve the accuracy and stability of the model. In addition, the prediction ability of the model can be further improved to generate more realistic virtual faces. The ability of three-dimensional reconstruction and virtual face generation can also be enhanced by integrating other related technologies, such as deep learning or artificial intelligence algorithms.
[0064] In summary, this embodiment provides a method for determining 3D reconstruction coefficients according to continuous video frames in an embodiment to generate a virtual face corresponding to a face image to be processed. This method has high-quality reconstruction and generation capabilities and is scalable and can be used in various application fields.
[0065] In one embodiment, continuous video frames are input into an initial prediction model of 3D reconstruction coefficients to train the initial model of 3D reconstruction coefficients to obtain a target prediction model of 3D reconstruction coefficients, including:
[0066] The continuous video frames and preset random face data are input into the initial prediction model of the 3D reconstruction coefficients to train the initial model of the 3D reconstruction coefficients and obtain the target prediction model of the 3D reconstruction coefficients.
[0067] This embodiment further illustrates how to train a target prediction model for three-dimensional reconstruction coefficients. Specifically, in one embodiment, continuous video frames and preset random face data are input into an initial prediction model for three-dimensional reconstruction coefficients to train the initial model for three-dimensional reconstruction coefficients. In this way, the preset random face data can be used to help the model perform preliminary predictions and learning. By inputting continuous video frames and preset random face data, the initial prediction model for three-dimensional reconstruction coefficients can learn the initial values and associated information of the three-dimensional reconstruction coefficients. The target prediction model for three-dimensional reconstruction coefficients obtained in this way can more accurately predict the virtual face corresponding to the face image to be processed. The preset random face data can include samples of different faces, which helps to improve the adaptability and generalization ability of the model to different faces.
[0068] In this embodiment, by inputting the continuous video frames and the preset random face data into the model during the training process, the prediction ability and accuracy of the model can be optimized. Through training, the model can better understand and capture the association information between the continuous video frames and the preset random face data.
[0069] In one embodiment, continuous video frames are input into an initial prediction model of 3D reconstruction coefficients to train the initial model of 3D reconstruction coefficients to obtain a target prediction model of 3D reconstruction coefficients, including:
[0070] Inputting continuous video frames into an initial prediction model of 3D reconstruction coefficients, constraining the initial prediction model of 3D reconstruction coefficients based on a preset function, so as to train the initial prediction model of 3D reconstruction coefficients and obtain a target prediction model of 3D reconstruction coefficients;
[0071] The preset function is constructed based on the standard deviation of the facial information coefficients of n different frames of the facial images corresponding to the same person, where n is an integer greater than 2.
[0072] The design idea of this embodiment is that for the same person, the information coefficient of the face should be consistent. Therefore, in this embodiment, the standard deviation of the face information coefficient of n different frames of the same person in a batch of training data is calculated as a preset function to constrain the fluctuation of the information coefficient of the same person predicted by the model to be as small as possible, thereby improving the stability of the results.
[0073] Specifically, firstly, the initial model of the 3D reconstruction coefficients can be obtained by inputting the continuous frames of the video into the initial prediction model of the 3D reconstruction coefficients. Then, based on this initial model, the target prediction model of the 3D reconstruction coefficients is obtained by training through the preset function. The construction process of the preset function is as follows: n different frames (that is, several frames in the continuous frames of the above video) are selected from the face pictures corresponding to the same person. For each selected frame, the corresponding face information coefficient is generated using the 3D reconstruction algorithm. According to the face information coefficients of these n different frames, the standard deviation is calculated and used as part of the preset function.
[0074] In one embodiment, the expression of the preset function is:
[0075] L=||w·(p_gt-p)|| 2 +||V(p_gt)-V(p)|| 2 +std(p_id_1,p_id_2,...,p_id_n);
[0076] Among them, w represents different weights, p represents the 3D face reconstruction coefficient output by the initial prediction model of the 3D reconstruction coefficient, p_gt represents the true value of the annotation result corresponding to p, V represents the process of 3D reconstruction using the 3D reconstruction parameter p, std represents the calculated standard deviation, and p_id_n represents the face information coefficient of n different frames of the face image corresponding to the same person.
[0077] After the preset function is constructed, the preset function can be used to constrain the initial model of the 3D reconstruction coefficients during the training process to constrain the fluctuation of the information coefficient of the same person predicted by the model to be as small as possible, thereby improving the stability of the results and obtaining a more accurate 3D reconstruction coefficient target prediction model. After obtaining the 3D reconstruction coefficient target prediction model, the face image to be processed can be input into the model to obtain a virtual face corresponding to the face image to be processed.
[0078] To solve the above technical problems, this application also provides a 3D face reconstruction system, please refer to Figure 2 , Figure 2 This is a structural block diagram of a 3D face reconstruction system provided in this application, and the 3D face reconstruction system includes:
[0079] The image processing unit 21 is used to transform the preset face image to obtain multiple target face images;
[0080] A video frame construction unit 22, used to construct a video continuous frame according to a plurality of target face images;
[0081] The 3D reconstruction coefficient determination unit 23 is used to determine the 3D reconstruction coefficients according to the continuous frames of the video, so as to generate a virtual face according to the 3D reconstruction coefficients and the face picture to be processed.
[0082] For the introduction of the three-dimensional face reconstruction system, please refer to the above embodiment, and this application will not go into details here.
[0083] To solve the above technical problems, this application also provides a 3D face reconstruction device, please refer to Figure 3 , Figure 3 This is a structural block diagram of a 3D face reconstruction device provided in this application, and the 3D face reconstruction device includes:
[0084] A memory 31, used for storing computer programs;
[0085] The processor 32 is used to implement the steps of the above-mentioned three-dimensional face reconstruction method when storing the computer program.
[0086] For the introduction of the three-dimensional face reconstruction device, please refer to the above embodiment, and this application will not go into details here.
[0087] To solve the above technical problems, the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above-mentioned method for 3D reconstruction of a face are implemented. For the introduction of the computer-readable storage medium, please refer to the above-mentioned embodiment, and the present application will not repeat it here.
[0088] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises", "comprising" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.
[0089] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for three-dimensional reconstruction of a face, characterized in that: include: Perform transformation processing on the preset face image to obtain multiple target face images; Constructing continuous video frames according to a plurality of target face images; Inputting the continuous video frames into an initial prediction model for 3D reconstruction coefficients to train the initial prediction model for 3D reconstruction coefficients to obtain a target prediction model for 3D reconstruction coefficients; The face picture to be processed is input into the three-dimensional reconstruction coefficient target prediction model to obtain a virtual face corresponding to the face picture to be processed.
2. The method for three-dimensional reconstruction of a face according to claim 1, characterized in that: Transform the preset face image to obtain multiple target face images, including: The preset face image is rotated and / or translated and / or scaled to obtain a plurality of first target face images.
3. The method for 3D reconstruction of a face as claimed in claim 2, wherein: Transform the preset face image to obtain multiple target face images, including: Preset noise and / or preset blur are added to the preset face image to obtain a plurality of second target face images.
4. The method for three-dimensional reconstruction of a face according to any one of claims 1 to 3, characterized in that: Inputting the continuous video frames into an initial prediction model of 3D reconstruction coefficients to train the initial prediction model of 3D reconstruction coefficients to obtain a target prediction model of 3D reconstruction coefficients, including: The continuous video frames and preset random face data are input into the initial prediction model of the 3D reconstruction coefficients to train the initial prediction model of the 3D reconstruction coefficients to obtain a target prediction model of the 3D reconstruction coefficients.
5. The method for three-dimensional reconstruction of a face according to any one of claims 1 to 3, characterized in that: Inputting the continuous video frames into an initial prediction model of 3D reconstruction coefficients to train the initial prediction model of 3D reconstruction coefficients to obtain a target prediction model of 3D reconstruction coefficients, including: Inputting the continuous frames of the video into an initial prediction model of 3D reconstruction coefficients, constraining the initial prediction model of 3D reconstruction coefficients based on a preset function to train the initial prediction model of 3D reconstruction coefficients, and obtaining a target prediction model of 3D reconstruction coefficients; The preset function is constructed based on the standard deviation of the facial information coefficients of n different frames of the facial images corresponding to the same person, where n is an integer greater than 2.
6. The method for 3D reconstruction of a face as claimed in claim 5, characterized in that: The expression of the preset function is: ; Wherein, w represents different weights, p represents the 3D face reconstruction coefficient output by the 3D reconstruction coefficient initial prediction model, p_gt represents the true value of the annotation result corresponding to p, V represents the process of 3D reconstruction using the 3D reconstruction parameter p, std represents the calculation standard deviation, Represents the facial information coefficients of n different frames of face images corresponding to the same person.
7. A 3D face reconstruction system, characterized in that: include: An image processing unit is used to transform a preset face image to obtain multiple target face images; A video frame construction unit, used to construct continuous video frames according to a plurality of target face images; The 3D reconstruction coefficient determination unit is used to input the continuous frames of the video into an initial prediction model of the 3D reconstruction coefficient to train the initial prediction model of the 3D reconstruction coefficient to obtain a target prediction model of the 3D reconstruction coefficient; and input the face image to be processed into the target prediction model of the 3D reconstruction coefficient to obtain a virtual face corresponding to the face image to be processed.
8. A 3D face reconstruction device, characterized in that: include: Memory for storing computer programs; A processor, used to implement the steps of the method for three-dimensional reconstruction of a face as described in any one of claims 1 to 6 when storing a computer program.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the steps of the three-dimensional face reconstruction method according to any one of claims 1 to 6 are implemented.
Citation Information
Patent Citations
Single-image three-dimensional face reconstruction method based on video supervision
CN111950477A
Three-dimensional face reconstruction method and device, computer equipment and storage medium
CN114723884A
Method for generating virtual character video on the basis of neural network, and related device
WO2021073416A1
Method and apparatus for reconstructing face image by using video identity clarification network
WO2022225374A2
Network training method and apparatus for facial reconstruction, and device and storage medium
WO2023050992A1
