Face three-dimensional reconstruction method and system, electronic equipment and storage medium
The 3D facial reconstruction method addresses the limitations of 2D photos by generating 3D models from 2D images, improving surgical precision and reducing costs in plastic surgery.
Patent Information
- Application Number
- CN202411323779.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-23
- Publication Date
- 2024-12-27
AI Technical Summary
2D facial photos inadequately represent facial features, leading to incomplete documentation and hindering precise surgical planning in plastic surgery.
A method and system for 3D facial reconstruction using two-dimensional facial images at different angles, generating deformation and displacement maps through a neural network, and constructing a 3D facial model for improved surgical guidance and monitoring.
Enhances the efficiency and reduces the cost of 3D facial reconstruction, supporting preoperative evaluation, surgical guidance, and postoperative monitoring in plastic surgery.
Smart Images
Figure CN119206074A_ABST
Abstract
Description
Background Art
[0002] With the development of artificial intelligence technology, 3D reconstruction and virtual reality technology have been applied in the medical field, which helps to standardize and improve plastic surgery practices. In the field of plastic surgery, 2D photos have always been an important part of medical documentation. The limitations of 2D photos hinder the comprehensive description of patients' facial features and details. Since 2D facial photos only represent the projection of patients' facial features, some facial feature information will inevitably be lost, so plastic surgeons have always ignored the use of 2D photos as a reference for surgical operations.
[0003] Therefore, it is urgent to provide a technical solution to solve the above problems. Summary of the invention
[0004] In order to solve the above technical problems, the present invention provides a facial three-dimensional reconstruction method, system, electronic device and storage medium.
[0005] In a first aspect, the present invention provides a method for facial 3D reconstruction, the technical solution of the method is as follows:
[0006] Obtaining two-dimensional facial images of the target user at different viewing angles, and generating a target deformation map and a target displacement map corresponding to the two-dimensional facial image at each viewing angle;
[0007] Based on all the target deformation maps and all the target displacement maps, a three-dimensional facial model of the target user is generated.
[0008] The beneficial effects of a facial 3D reconstruction method of the present invention are as follows:
[0009] The method of the present invention can improve the efficiency of facial three-dimensional reconstruction and reduce the cost of facial three-dimensional reconstruction, providing support for preoperative evaluation, surgical guidance and postoperative monitoring in the field of plastic surgery.
[0010] Based on the above solution, the facial 3D reconstruction method of the present invention can also be improved as follows.
[0011] In an optional manner, the step of generating a target deformation map and a target displacement map corresponding to the two-dimensional facial image at any viewing angle includes:
[0012] Inputting the two-dimensional facial image at any viewing angle into the trained target neural network to obtain an original deformation map and an original displacement map at any viewing angle;
[0013] The original deformation map and the original displacement map under any viewing angle are subjected to image de-modification processing to obtain a target deformation map and a target displacement map respectively corresponding to the two-dimensional facial image under any viewing angle.
[0014] In an optional manner, the target neural network includes: a generator and a discriminator;
[0015] The generator is used to: generate an original deformation map and an original displacement map according to a two-dimensional facial image;
[0016] The discriminator is used to make a true or false judgment on the original deformation image and the original displacement image.
[0017] In an optional manner, the two-dimensional facial images at different viewing angles include at least a front two-dimensional facial image and a side two-dimensional facial image.
[0018] In a second aspect, the present invention provides a facial 3D reconstruction system, the technical solution of which is as follows:
[0019] Includes: generation module and reconstruction module;
[0020] The generating module is used to: obtain the two-dimensional facial images of the target user at different viewing angles, and generate a target deformation map and a target displacement map corresponding to the two-dimensional facial image at each viewing angle;
[0021] The reconstruction module is used to generate a three-dimensional facial model of the target user based on all target deformation maps and all target displacement maps.
[0022] The beneficial effects of a facial 3D reconstruction system of the present invention are as follows:
[0023] The system of the present invention can improve the efficiency of facial three-dimensional reconstruction and reduce the cost of facial three-dimensional reconstruction, providing support for preoperative evaluation, surgical guidance and postoperative monitoring in the field of plastic surgery.
[0024] Based on the above solution, the facial 3D reconstruction system of the present invention can also be improved as follows.
[0025] In an optional manner, the generating module is specifically used for:
[0026] Inputting the two-dimensional facial image at any viewing angle into the trained target neural network to obtain an original deformation map and an original displacement map at any viewing angle;
[0027] The original deformation map and the original displacement map under any viewing angle are subjected to image de-modification processing to obtain a target deformation map and a target displacement map respectively corresponding to the two-dimensional facial image under any viewing angle.
[0028] In an optional manner, the target neural network includes: a generator and a discriminator;
[0029] The generator is used to: generate an original deformation map and an original displacement map according to a two-dimensional facial image;
[0030] The discriminator is used to make a true or false judgment on the original deformation image and the original displacement image.
[0031] In an optional manner, the two-dimensional facial images at different viewing angles include at least a front two-dimensional facial image and a side two-dimensional facial image.
[0032] In a third aspect, a technical solution of an electronic device of the present invention is as follows:
[0033] The invention comprises a memory, a processor and a program stored in the memory and running on the processor. When the processor executes the program, the steps of the facial three-dimensional reconstruction method of the present invention are implemented.
[0034] In a fourth aspect, the present invention provides a computer-readable storage medium having the following technical solution:
[0035] The computer-readable storage medium stores instructions, and when the computer-readable storage medium reads the instructions, the computer-readable storage medium executes the steps of the facial three-dimensional reconstruction method of the present invention.
[0036] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] The accompanying drawings are only used to illustrate the embodiments and are not to be considered as limiting the present invention. In addition, the same reference symbols are used to represent the same components throughout the accompanying drawings. In the accompanying drawings:
[0038] Figure 1 A schematic diagram of a flow chart of an embodiment of a facial three-dimensional reconstruction method of the present invention;
[0039] Figure 2 is a schematic diagram of the structure of the target neural network;
[0040] Figure 3 A schematic diagram of the structure of an embodiment of a facial three-dimensional reconstruction system of the present invention;
[0041] Figure 4 The figure is a schematic structural diagram of an embodiment of an electronic device of the present invention. DETAILED DESCRIPTION
[0042] Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein.
[0043] Figure 1 A flow chart of an embodiment of a facial 3D reconstruction method provided by the present invention is shown, and the facial 3D reconstruction method can be executed by an electronic device such as a terminal device or a server. The terminal device can be any fixed or mobile terminal such as a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The server can be a single server or a server cluster composed of multiple servers. Any electronic device can implement the facial 3D reconstruction method by means of a processor calling computer-readable instructions stored in a memory. Figure 1 As shown, the following steps are included:
[0044] S1. Obtain two-dimensional facial images of the target user at different viewing angles, and generate a target deformation map and a target displacement map corresponding to the two-dimensional facial image at each viewing angle.
[0045] The target user may be a subject or a patient undergoing plastic surgery, and there is no limitation here. The two-dimensional facial images under different viewing angles at least include: a front two-dimensional facial image and a side two-dimensional facial image.
[0046] In S1, the step of generating a target deformation map and a target displacement map corresponding to the two-dimensional facial image at any viewing angle includes:
[0047] The two-dimensional facial image at any viewing angle is input into a trained target neural network to obtain an original deformation map and an original displacement map at any viewing angle.
[0048] The target neural network is used to generate the original deformation map and original displacement map corresponding to the two-dimensional facial image. Each two-dimensional facial image under each viewing angle corresponds to an original deformation map and an original displacement map.
[0049] The original deformation map and the original displacement map under any viewing angle are subjected to image de-modification processing to obtain a target deformation map and a target displacement map respectively corresponding to the two-dimensional facial image under any viewing angle.
[0050] The de-modification process includes: aligning the facial texture, removing high-frequency details in the facial skin (such as moles, scars, freckles, etc.), generating the facial albedo of high-frequency details, and obtaining a clearer basic texture.
[0051] S2. Generate a three-dimensional facial model of the target user based on all target deformation maps and all target displacement maps.
[0052] Specifically: ①Convert the target deformation map into a grayscale image and obtain the grayscale value, and convert the target displacement map into an RGB image and obtain the RGB value. ②Create a basic 3D mesh, such as a plane mesh, and the vertices of the mesh can be initialized as points on the plane. ③Traverse each pixel of the target deformation map and map its grayscale value to the height value of the mesh vertex. Since the grayscale value is usually between 0 and 255, its grayscale value can also be normalized to between 0 and 1, and then mapped to an appropriate height range. ④Traverse each pixel of the target displacement map and convert the RGB value of the target displacement map into a normal direction. Since the RGB value is usually between 0 and 255, it can also be normalized to between 0 and 1, and then mapped to between -1 and 1, and the converted normal direction is applied to the normal attribute of the mesh vertex. ⑤Use the vertex and normal data generated by all target deformation maps and all target displacement maps to build a three-dimensional facial model. Among them, 3D modeling libraries (such as Blender, Maya) or programming libraries (such as OpenGL, Three.js) can be used for rendering and saving.
[0053] In an alternative approach, Figure 2 As shown, the target neural network includes: a generator and a discriminator.
[0054] The generator is used to generate an original deformation map and an original displacement map according to a two-dimensional facial image.
[0055] The generator includes: a first input layer, an encoder, a first feature extraction layer and a decoder. The first input layer is used to receive a two-dimensional facial image. The encoder is used to encode the two-dimensional facial image into low-dimensional features. The first feature extraction layer is used to extract key information from the low-dimensional features. The decoder is used to decode the key information into an original deformation map and an original displacement map.
[0056] The discriminator is used to make a true or false judgment on the original deformation image and the original displacement image.
[0057] The discriminator includes: a second input layer, a second feature extraction layer and a classifier. The second input layer is used to receive the original deformation map and the original displacement map, as well as the corresponding real deformation map and the real displacement map. The second feature extraction layer is used to extract the features of the input image of the second input layer through the convolution layer. The classifier is used to map the extracted features to a binary classification output (true or false).
[0058] It should be noted that the training of the target neural network is guided by designing the target loss function, specifically:
[0059] 1) The target loss function includes: ① Adversarial Loss: used to measure the similarity between the image generated by the generator and the real image. ② Reconstruction Loss: used to measure the difference between the generated deformation map and displacement map and the real image. The loss function usually uses L1 or L2 loss.
[0060] 2) The generator and discriminator are optimized through adversarial training. The goal of the generator is to generate realistic deformation and displacement maps so that the discriminator cannot distinguish whether these maps are real or generated. The goal of the discriminator is to correctly distinguish between real deformation and displacement maps and generated maps.
[0061] 3) The training process is as follows: ① Initialization: Randomly initialize the parameters of the generator and discriminator. ② In iterative training: In the generation phase, the generator is used to generate deformation maps and displacement maps. In the discrimination phase, the discriminator is used to classify the generated maps and calculate the adversarial loss. In the optimization phase, the parameters of the generator and discriminator are updated according to the loss function. Repeat the iterative training until the generated deformation maps and displacement maps are realistic enough.
[0062] 4) Multi-view Figure 1 ① In order to ensure the consistency of the generated deformation map and displacement map between different views, a multi-view Figure 1 ② Geometric consistency: ensure that the deformation and displacement maps generated by the front and side photos are consistent in geometry. ③ Texture consistency: ensure that the textures generated by different views are consistent in high-frequency details.
[0063] 5) The training data consists of two parts: ① 2D in-the-wild images: These images are collected from multiple sources and contain images of faces in various natural environments. ② 3D facial scans and corresponding multi-view images: The data comes from multiple public and proprietary datasets, including FaceScape, ESRC, and FaceHD-100. The dataset contains subjects of different races and age groups, as well as 3D scans of multiple expressions. A total of about 9,000 3D scan data were collected from nearly 500 subjects. Finally, about 260K in-the-wild images were collected for self-supervised training, and about 150K laboratory images and their corresponding ground-truth detail maps were used for supervised training.
[0064] 6) Data Processing. All scans and corresponding multi-view images are processed to generate ground truth deformation and displacement maps for each image. Input images are pre-processed to ensure consistency and quality.
[0065] 7) Training strategy. ① Pre-training: Use the pre-trained facial analyzer to predict the 3DMM coefficients, position map, and texture map. ② De-modification module training: Use the paired texture map to train the de-modification module to generate facial albedo with high-frequency details. ③ Whole network training: Fix the parameters of the facial analyzer and de-modification module, and train the entire network using facial images, position maps, and texture maps as input. ④ Training settings: The model alternates between self-supervised and supervised training using in-the-wild images and laboratory images.
[0066] The technical solution of this embodiment can improve the efficiency of facial three-dimensional reconstruction and reduce the cost of facial three-dimensional reconstruction, providing support for preoperative evaluation, surgical guidance and postoperative monitoring in the field of plastic surgery.
[0067] Figure 3 FIG. 2 is a schematic diagram showing a structure of an embodiment of a facial 3D reconstruction system 200 provided by the present invention. Figure 3 As shown, the system 200 includes: a generation module 210 and a reconstruction module 220;
[0068] The generating module 210 is used to: obtain the two-dimensional facial images of the target user at different viewing angles, and generate a target deformation map and a target displacement map corresponding to the two-dimensional facial image at each viewing angle;
[0069] The reconstruction module 220 is used to generate a three-dimensional facial model of the target user based on all target deformation maps and all target displacement maps.
[0070] In an optional manner, the generating module 210 is specifically used for:
[0071] Inputting the two-dimensional facial image at any viewing angle into the trained target neural network to obtain an original deformation map and an original displacement map at any viewing angle;
[0072] The original deformation map and the original displacement map under any viewing angle are subjected to image de-modification processing to obtain a target deformation map and a target displacement map respectively corresponding to the two-dimensional facial image under any viewing angle.
[0073] In an optional manner, the target neural network includes: a generator and a discriminator;
[0074] The generator is used to: generate an original deformation map and an original displacement map according to a two-dimensional facial image;
[0075] The discriminator is used to make a true or false judgment on the original deformation image and the original displacement image.
[0076] In an optional manner, the two-dimensional facial images at different viewing angles include at least a front two-dimensional facial image and a side two-dimensional facial image.
[0077] The technical solution of this embodiment can improve the efficiency of facial three-dimensional reconstruction and reduce the cost of facial three-dimensional reconstruction, providing support for preoperative evaluation, surgical guidance and postoperative monitoring in the field of plastic surgery.
[0078] The above parameters and steps for each module to implement corresponding functions in the facial 3D reconstruction system 200 of this embodiment can refer to the parameters and steps in the embodiment of the facial 3D reconstruction method above, and will not be repeated here.
[0079] like Figure 4 As shown, an electronic device 300 according to an embodiment of the present invention includes a processor 320, the processor 320 is coupled to a memory 310, and the memory 310 stores at least one computer program 330, and the at least one computer program 330 is loaded and executed by the processor 320, so that the electronic device 300 implements any of the above-mentioned facial three-dimensional reconstruction methods, specifically:
[0080] The electronic device 300 may have relatively large differences due to different configurations or performances, and may include one or more processors 320 (Central Processing Units, CPU) and one or more memories 310, wherein the one or more memories 310 store at least one computer program 330, and the at least one computer program 330 is loaded and executed by the one or more processors 320, so that the electronic device 300 implements any of the facial three-dimensional reconstruction methods provided in the above embodiments. Of course, the electronic device 300 may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output, and the electronic device 300 may also include other components for implementing device functions, which will not be described in detail here.
[0081] A computer-readable storage medium according to an embodiment of the present invention stores at least one computer program, and the at least one computer program is loaded and executed by a processor so that a computer implements any of the above-mentioned facial three-dimensional reconstruction methods.
[0082] Optionally, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, and the like.
[0083] In an exemplary embodiment, a computer program product or a computer program is also provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device performs any of the above-mentioned facial three-dimensional reconstruction methods.
[0084] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects and represent the definition of a specific order or sequence. The order of use of similar objects can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than the order shown or described.
[0085] Those skilled in the art know that the present invention can be implemented as a system, method or computer program product. Therefore, the present disclosure can be specifically implemented in the following forms, namely: it can be complete hardware, it can be complete software (including firmware, resident software, microcode, etc.), or it can be a combination of hardware and software, which is generally referred to as "circuit", "module" or "system" herein. In addition, in some embodiments, the present invention can also be implemented in the form of a computer program product in one or more computer-readable media, and the computer-readable medium contains computer-readable program code.
[0086] Any combination of one or more computer-readable media may be used. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, device, or device.
[0087] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.
Claims
1. A facial 3D reconstruction method, characterized in that: include: Obtaining two-dimensional facial images of the target user at different viewing angles, and generating a target deformation map and a target displacement map corresponding to the two-dimensional facial image at each viewing angle; A three-dimensional facial model of the target user is generated based on all target deformation maps and all target displacement maps.
2. The facial 3D reconstruction method according to claim 1, characterized in that: The steps of generating a target deformation map and a target displacement map corresponding to the two-dimensional facial image at any viewing angle respectively include: Inputting the two-dimensional facial image at any viewing angle into the trained target neural network to obtain an original deformation map and an original displacement map at any viewing angle; The original deformation map and the original displacement map under any viewing angle are subjected to image de-modification processing to obtain a target deformation map and a target displacement map respectively corresponding to the two-dimensional facial image under any viewing angle.
3. The facial 3D reconstruction method according to claim 2, characterized in that: The target neural network includes: generator and discriminator; The generator is used to: generate an original deformation map and an original displacement map according to a two-dimensional facial image; The discriminator is used to make a true or false judgment on the original deformation image and the original displacement image.
4. The facial three-dimensional reconstruction method according to any one of claims 1 to 3, characterized in that: The two-dimensional facial images under different viewing angles include at least a front two-dimensional facial image and a side two-dimensional facial image.
5. A facial 3D reconstruction system, characterized in that: include: Generate modules and rebuild modules; The generating module is used to: obtain the two-dimensional facial images of the target user at different viewing angles, and generate a target deformation map and a target displacement map corresponding to the two-dimensional facial image at each viewing angle; The reconstruction module is used to generate a three-dimensional facial model of the target user based on all target deformation maps and all target displacement maps.
6. The facial 3D reconstruction system according to claim 5, characterized in that: The generation module is specifically used for: Inputting the two-dimensional facial image at any viewing angle into the trained target neural network to obtain an original deformation map and an original displacement map at any viewing angle; The original deformation map and the original displacement map under any viewing angle are subjected to image de-modification processing to obtain a target deformation map and a target displacement map respectively corresponding to the two-dimensional facial image under any viewing angle.
7. The facial 3D reconstruction system according to claim 6, characterized in that: The target neural network includes: generator and discriminator; The generator is used to: generate an original deformation map and an original displacement map according to a two-dimensional facial image; The discriminator is used to make a true or false judgment on the original deformation image and the original displacement image.
8. The facial 3D reconstruction system according to any one of claims 5 to 7, characterized in that: The two-dimensional facial images under different viewing angles include at least a front two-dimensional facial image and a side two-dimensional facial image.
9. An electronic device, characterized in that: The electronic device includes a processor, the processor is coupled to a memory, the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor so that the electronic device implements the facial three-dimensional reconstruction method as described in any one of claims 1 to 4.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer program, and the at least one computer program is loaded and executed by the processor so that the computer-readable storage medium implements the facial three-dimensional reconstruction method according to any one of claims 1 to 4.