Face three-dimensional reconstruction method, system and device based on portable device and medium

Abstract

The invention relates to the technical field of three-dimensional reconstruction, and discloses a face three-dimensional reconstruction method, system and device based on a portable device and a medium, and the method comprises the steps: collecting a face image frame sequence of a target user through the portable device; performing face key point detection on each frame of face image in the face image frame sequence, and performing face geometric reconstruction based on a face key point result in combination with a multi-stage fusion algorithm to obtain a face geometric reconstruction model; determining a plurality of key face images in the face image frame sequence based on a frame selection strategy, performing de-illumination processing on each key face image, and fusing each de-illuminated face texture image through a Laplacian pyramid network to generate a face texture reconstruction model; and mapping the face texture reconstruction model to the face geometric reconstruction model to obtain a complete face three-dimensional reconstruction result. According to the invention, the hardware cost can be reduced, the convenience is improved, remote medical treatment is realized, and the face three-dimensional reconstruction precision and quality are improved.

Description

Technical Field

[0001] This invention relates to the field of three-dimensional reconstruction technology, and in particular to a method, system, device and medium for three-dimensional face reconstruction based on a portable device. Background Technology

[0002] 3D facial reconstruction technology has wide applications in plastic surgery, assisted facial recognition, virtual digital humans, and film and animation. In plastic surgery, 3D facial reconstruction can aid in medical diagnosis and pre- and post-operative outcome evaluation. Currently, 3D facial reconstruction mainly relies on specialized equipment such as 3DMD and CT scanners, requiring the patient's presence and data collection under the guidance of a doctor. With the widespread adoption of depth sensors, more and more portable devices possess the hardware foundation for detailed facial reconstruction, making telemedicine a possibility. However, existing specialized equipment suffers from high costs and inconvenience, limiting the further promotion and application of this technology.

[0003] Therefore, there is an urgent need to provide a technical solution to address the above problems. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a method, system, device, and medium for three-dimensional facial reconstruction based on a portable device.

[0005] Firstly, the present invention provides a method for three-dimensional face reconstruction based on a portable device, the technical solution of which is as follows: Using portable devices, capture a sequence of facial image frames of the target user; Facial key points are detected for each frame of the face image in the face image frame sequence. Based on the facial key point results of each frame of the face image in the face image frame sequence, and combined with a multi-stage fusion algorithm, face geometry is reconstructed to obtain a face geometry reconstruction model. Based on the frame selection strategy, multiple key face images in the face image frame sequence are determined, each key face image is de-illuminated, and each de-illuminated face texture image is fused through the Laplacian pyramid network to generate a face texture reconstruction model. The face texture reconstruction model is mapped to the face geometry reconstruction model to obtain the complete 3D face reconstruction result of the target user.

[0006] The beneficial effects of the three-dimensional face reconstruction method based on a portable device of the present invention are as follows: The method of this invention can reduce hardware costs, improve convenience, realize telemedicine, improve the accuracy and quality of 3D facial reconstruction, and promote the widespread application of 3D facial reconstruction technology in fields such as plastic surgery.

[0007] Based on the above scheme, the method for three-dimensional face reconstruction based on portable devices of the present invention can be further improved as follows.

[0008] In one optional approach, the multi-stage fusion algorithm includes: a 3DMM algorithm, a nodal deformation field algorithm, and a Laplacian deformation algorithm; the steps of performing face geometric reconstruction based on the facial key point results of each frame of the face image in the face image frame sequence, combined with the multi-stage fusion algorithm, to obtain a face geometric reconstruction model include: Based on the 3DMM algorithm, global shape fitting is performed on the facial key point results of each frame of the face image in the face image frame sequence to generate a first face geometric model; Based on the nodal deformation field algorithm, the first face geometric model is locally structurally optimized to generate the second face geometric model; Based on the Laplacian deformation algorithm, high-frequency detail enhancement is performed on the second face geometric model to generate the face geometric reconstruction model.

[0009] In one alternative approach, the step of determining multiple key face images in the face image frame sequence based on a frame selection strategy includes: Based on the rotation matrix of each face image in the face image frame sequence, key face images with multiple preset face angles are determined in the face image frame sequence.

[0010] In one alternative approach, the steps of removing illumination from each key face image and fusing each removed face texture image using a Laplacian pyramid network to generate a face texture reconstruction model include: Using a de-illumination network, each key face image is de-illuminated to obtain a de-illumination face texture image. The Laplacian pyramid network is used to perform multi-scale decomposition and weighted fusion on each illuminated face texture image to obtain fused pyramid features. The fused pyramid features are then reconstructed using Laplacian pyramids to generate the face texture reconstruction model.

[0011] In one alternative approach, the step of mapping the facial texture reconstruction model to the facial geometry reconstruction model to obtain the complete 3D facial reconstruction result of the target user includes: Based on the UV mapping coordinates of each vertex of the face geometry model, the texture pixels of the face texture reconstruction model are mapped to the corresponding vertices of the face geometry reconstruction model to obtain the mapped three-dimensional face geometry model. Color interpolation calculations are performed on the three vertices of each triangular facet of the mapped 3D face geometry model to generate a continuous color distribution within each triangular facet, which is then rendered to form a textured complete 3D face reconstruction result.

[0012] Secondly, this invention provides a three-dimensional face reconstruction system based on a portable device, the technical solution of which is as follows: It includes: an acquisition module, a geometry reconstruction module, a texture reconstruction module, and a fusion module; The acquisition module is used to: acquire a sequence of facial image frames of a target user using a portable device; The geometric reconstruction module is used to: perform facial key point detection on each frame of the face image in the face image frame sequence, and perform face geometric reconstruction based on the facial key point results of each frame of the face image in the face image frame sequence and in combination with a multi-stage fusion algorithm to obtain a face geometric reconstruction model. The texture reconstruction module is used to: determine multiple key face images in the face image frame sequence based on a frame selection strategy, perform illumination removal processing on each key face image, and fuse each illumination-removed face texture image through a Laplacian pyramid network to generate a face texture reconstruction model. The fusion module is used to: map the face texture reconstruction model to the face geometry reconstruction model to obtain the complete three-dimensional face reconstruction result of the target user.

[0013] The beneficial effects of the portable device-based 3D face reconstruction system of the present invention are as follows: The system of this invention can reduce hardware costs, improve convenience, realize remote medical care, improve the accuracy and quality of 3D facial reconstruction, and promote the widespread application of 3D facial reconstruction technology in fields such as plastic surgery.

[0014] Based on the above solution, the 3D face reconstruction system based on a portable device of the present invention can be further improved as follows.

[0015] In one alternative approach, the multi-stage fusion algorithm includes: a 3DMM algorithm, a nodal deformation field algorithm, and a Laplacian deformation algorithm; the geometric reconstruction module is specifically used for: Based on the 3DMM algorithm, global shape fitting is performed on the facial key point results of each frame of the face image in the face image frame sequence to generate a first face geometric model; Based on the nodal deformation field algorithm, the first face geometric model is locally structurally optimized to generate the second face geometric model; Based on the Laplacian deformation algorithm, high-frequency detail enhancement is performed on the second face geometric model to generate the face geometric reconstruction model.

[0016] In one alternative approach, the texture reconstruction module is specifically used for: Based on the rotation matrix of each face image in the face image frame sequence, key face images with multiple preset face angles are determined in the face image frame sequence.

[0017] Thirdly, the technical solution of an electronic device according to the present invention is as follows: It includes a memory, a processor, and a program stored in the memory and running on the processor, wherein the processor executes the program to implement the steps of the portable device-based three-dimensional face reconstruction method of the present invention.

[0018] Fourthly, the technical solution of a computer-readable storage medium provided by the present invention is as follows: The computer-readable storage medium stores instructions that, when read, cause the computer-readable storage medium to perform the steps of the portable device-based three-dimensional face reconstruction method of the present invention.

[0019] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0020] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 This is a flowchart illustrating an embodiment of a three-dimensional face reconstruction method based on a portable device according to the present invention. Figure 2 A schematic diagram of the face geometry reconstruction process; Figure 3 A flowchart illustrating the process of facial texture reconstruction; Figure 4 This is one of the schematic diagrams of a software system architecture; Figure 5 This is the second schematic diagram of the software system architecture; Figure 6 This is a diagram simulating a human face before and after surgery; Figure 7 A schematic diagram for facial symmetry analysis; Figure 8 A diagram illustrating the comparison of the effects of face reconstruction algorithms; Figure 9This is a schematic diagram of an embodiment of a three-dimensional face reconstruction system based on a portable device according to the present invention; Figure 10 This is a schematic diagram of an embodiment of an electronic device according to the present invention. Detailed Implementation

[0021] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0022] Figure 1 This diagram illustrates a flowchart of an embodiment of a portable device-based 3D face reconstruction method provided by the present invention. This portable device-based 3D face reconstruction method can be executed by electronic devices such as terminal devices or servers. The terminal device can be any fixed or mobile terminal, such as user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, or wearable device. The server can be a single server or a server cluster consisting of multiple servers. Any electronic device can implement the portable device-based 3D face reconstruction method by having its processor call computer-readable instructions stored in its memory. Figure 1 As shown, it includes the following steps: S1. Use a portable device to collect a sequence of facial image frames of the target user.

[0023] Here, "target user" refers to the individual who needs to undergo 3D facial reconstruction. "Portable device" refers to a mobile electronic device with a built-in depth sensor and camera, such as the smartphone used by target user A. "Face image frame sequence" refers to a collection of multiple frames containing RGB facial images and depth data, continuously acquired by the portable device, such as a 30-second video stream (containing 500 frames of RGBD data) captured by target user A using a smartphone.

[0024] S2. Perform facial landmark detection on each frame of the face image in the face image frame sequence. Based on the facial landmark results of each frame of the face image in the face image frame sequence, and combined with a multi-stage fusion algorithm, perform face geometric reconstruction to obtain a face geometric reconstruction model.

[0025] Here, "face image" refers to a single frame image in a face image frame sequence, such as the RGB image and corresponding depth map of frame 15 in a video of target user A. "Facial landmark detection" refers to the process of locating facial feature points from a single face image, such as detecting the coordinates of multiple landmarks like the center of the left eye, the tip of the nose, and the corners of the mouth in frame 15 of target user A's image. "Facial landmark result" refers to the set of landmark coordinates obtained after facial landmark detection, such as a two-dimensional coordinate data matrix of multiple landmarks in frame 15 of target user A's image. "Multi-stage fusion algorithm" refers to the process of geometric reconstruction using the 3DMM algorithm, the nodal deformation field algorithm, and the Laplacian deformation algorithm sequentially. For example, first, the 3DMM algorithm is used to fit the global shape of target user A's face; then, the nodal deformation field algorithm is used to optimize the local structure of the bridge of the nose; finally, the Laplacian deformation algorithm is used to enhance the details of the nasolabial folds. "Facial geometric reconstruction model" refers to a three-dimensional mesh model generated by the multi-stage fusion algorithm, such as the three-dimensional mesh of target user A's face (containing 50,000 vertices and 100,000 triangular faces).

[0026] S3. Based on the frame selection strategy, determine multiple key face images in the face image frame sequence, perform illumination removal processing on each key face image, and fuse each illumination-removed face texture image through a Laplacian pyramid network to generate a face texture reconstruction model.

[0027] The frame selection strategy refers to a rule for filtering facial images at preset angles based on a rotation matrix. For example, selecting three key images—left face (-40°), frontal face (0°), and right face (40°)—from 500 frames of data from target user A. Key facial images refer to facial images selected by the frame selection strategy that conform to the preset angles, such as frames 102 (left face), 255 (frontal face), and 378 (right face) in target user A's video. The de-illuminated facial texture image refers to a two-dimensional facial texture map after eliminating the effects of illumination, such as a shadow-free facial image obtained by de-illuminating the key facial images of target user A. The Laplacian pyramid network refers to a network structure that performs multi-scale decomposition and weighted fusion of multiple frames of images. For example, decomposing the three de-illuminated facial images of target user A into four pyramid layers and performing weighted fusion at each layer. A face texture reconstruction model refers to a two-dimensional texture map generated by fusing three images of target user A with a Laplacian pyramid network, such as a 2048×2048 resolution face texture map generated by fusing three images of target user A after removing illumination.

[0028] S4. Map the face texture reconstruction model to the face geometry reconstruction model to obtain the complete three-dimensional face reconstruction result of the target user.

[0029] Among them, the complete 3D face reconstruction result refers to a textured 3D face model, such as a renderable 3D model generated by mapping the texture reconstruction model of target user A's face to its geometric reconstruction model.

[0030] The technical solution of this embodiment can reduce hardware costs, improve convenience, realize remote medical care, improve the accuracy and quality of 3D facial reconstruction, and promote the widespread application of 3D facial reconstruction technology in fields such as plastic surgery.

[0031] In one alternative approach, the multi-stage fusion algorithm includes: a 3DMM algorithm, a nodal deformation field algorithm, and a Laplacian deformation algorithm. The 3DMM algorithm refers to an algorithm that fits the global shape of a face based on a three-dimensional deformation model, such as using the BFM2019 model to fit the basic shape of target user A's face. The nodal deformation field algorithm refers to an algorithm that drives local mesh deformation by controlling node displacement, such as moving 12 control points at the bridge of target user A's nose to optimize the bridge height. The Laplacian deformation algorithm refers to a deformation algorithm that preserves details by maintaining differential coordinates, such as enhancing the geometric details of the nasolabial folds while maintaining the overall topology of target user A's face.

[0032] like Figure 2 As shown, the steps for obtaining a face geometric reconstruction model based on the facial key point results of each frame of the face image frame sequence and combined with a multi-stage fusion algorithm to perform face geometric reconstruction include: Based on the 3DMM algorithm, global shape fitting is performed on the facial key point results of each frame of the face image in the face image frame sequence to generate the first face geometric model.

[0033] The first face geometric model refers to the initial three-dimensional face model generated by the 3DMM algorithm, such as the coarse-grained face mesh (containing 10,000 vertices) of target user A.

[0034] Specifically: 1) Select a predefined set of base face models, which contains multiple base face models; 2) Calculate shape coefficients based on the facial key point results of each face image in the face image frame sequence, where the shape coefficients represent the weighted weights of each base face model; 3) Adjust the shape coefficients through an iterative optimization algorithm to minimize the spatial position error between the initial 3D face model generated by the weighted combination of base face models and the facial key point results; 4) Substitute the optimized shape coefficients into the weighted combination formula of the base face model set to generate a topologically complete first face geometric model.

[0035] Based on the nodal deformation field algorithm, the first face geometric model is locally structurally optimized to generate the second face geometric model.

[0036] The second face geometric model refers to the 3D face model optimized by the node deformation field algorithm, such as the face mesh (containing 30,000 vertices) optimized for the nose and chin of target user A.

[0037] Specifically: 1) Select a set of control nodes (key points) on the surface of the first face geometric model; 2) Calculate the target displacement vector of the control nodes based on the local geometric features of the depth data of multiple frames in the face image frame sequence; 3) Calculate the weighted displacement vector of all non-control nodes in the first face geometric model affected by the displacement of the control nodes based on radial basis function interpolation; 4) Superimpose the weighted displacement vector onto the corresponding node coordinates of the first face geometric model to generate a second face geometric model with local structure optimization.

[0038] Based on the Laplacian deformation algorithm, high-frequency detail enhancement is performed on the second face geometric model to generate the face geometric reconstruction model.

[0039] Specifically: 1) Calculate the Laplacian coordinates of each vertex in the second face geometric model to characterize local geometric features; 2) Define a set of geometric constraint points on the surface of the second face geometric model based on the high-frequency detail information of the depth data of multiple frames in the face image frame sequence; 3) Construct a system of linear equations with Laplacian coordinate preservation as the optimization objective and the spatial position of the geometric constraint points as the equality constraint; 4) Solve the system of linear equations to obtain the displacement vectors of all vertices of the second face geometric model; 5) Superimpose the displacement vectors of all vertices onto the vertex coordinates of the second face geometric model to generate a face geometric reconstruction model that preserves local detail features.

[0040] Among the above-mentioned optional methods, a multi-stage fusion of 3DMM, nodal deformation field and Laplacian deformation algorithm is further adopted to gradually optimize the face geometric model, enhance the reconstruction details, and improve the realism and accuracy of face 3D reconstruction.

[0041] In one alternative approach, such as Figure 3 As shown, the steps for determining multiple key face images in the face image frame sequence based on a frame selection strategy include: Based on the rotation matrix of each face image in the face image frame sequence, key face images with multiple preset face angles are determined in the face image frame sequence.

[0042] The rotation matrix refers to a 3×3 matrix representing the facial pose. For example, the matrix describing the rotation of user A's face in frame 102 relative to a frontal view by -40° is represented as: [[0.77,-0.64,0],[0.64,0.77,0],[0,0,1]]. The preset face angle refers to a pre-set standard face orientation angle, such as -40° for the left face, 0° for the frontal face, and 40° for the right face.

[0043] Specifically: 1) Decompose the left - right rotation angle and the up - down rotation angle from the rotation matrix of each frame of face image; 2) Screen candidate frames that meet the frontal face angle range, left face angle range, and right face angle range according to the preset face angle range; 3) If there are multiple candidate frames, select the frame with the smallest absolute value of the up - down rotation angle as the key frame; 4) If there are no frames that meet the corresponding angle range, relax the preset angle range threshold and re - screen until the key face images corresponding to each preset angle are obtained.

[0044] It should be noted that based on the rotation matrix, the corresponding left - right rotation angle x and up - down rotation angle y can be calculated. Then the preset face angle range is as follows: ① The judgment criterion for a frontal face is: the absolute value of x is less than 5 and the absolute value of y is less than 12. ② The judgment criterion for a left face is: - 55 < x < - 25. ③ The judgment criterion for a right face is: 25 < x < 55. ④ The relaxed frontal face angle range is: when there are no frames that meet the strict frontal face range, select the frame with the smallest absolute value of the left - right rotation angle x; ⑤ The relaxed side face angle range is: when there are no frames that meet the strict side face range, for the left face, select frames with the left - right rotation angle x between - 90 degrees and - 15 degrees, and for the right face, select frames with the left - right rotation angle x between 15 degrees and 90 degrees.

[0045] In the above - mentioned optional method, further based on the frame selection strategy of the rotation matrix, accurately locate the key face images, ensure that the angles selected in the reconstruction process are diverse and representative, and improve the comprehensiveness of the reconstruction model.

[0046] In an optional way, as Figure 3 shown, the steps of performing de - illumination processing on each key face image and fusing each de - illuminated face texture image through a Laplacian pyramid network to generate a face texture reconstruction model include: Use a de - illumination network to perform de - illumination processing on each key face image respectively to obtain each de - illuminated face texture image.

[0047] Among them, the de - illumination network refers to a neural network model that eliminates the influence of illumination in face images based on deep learning. For example, process the key face image of target user A to remove the facial shadows and highlight areas caused by indoor lighting and generate a face texture image with normalized illumination.

[0048] Through the Laplacian pyramid network, perform multi - scale decomposition and weighted fusion on each de - illuminated face texture image to obtain the fused pyramid features, and perform Laplacian pyramid reconstruction on the fused pyramid features to generate the face texture reconstruction model.

[0049] Here, the fused pyramid features refer to the feature representation of the Laplacian pyramid network after weighted fusion at each scale layer. For example, the 512×512 feature map of the target user A after fusing three de-illuminated images at the second layer of the pyramid. Laplacian pyramid reconstruction refers to the process of reconstructing the complete image from the fused pyramid features. For example, reconstructing the fused 4-layer feature map of the target user A into a 2048×2048 texture map.

[0050] In the above-mentioned alternative methods, a de-illumination network is further utilized to eliminate the influence of illumination, and a Laplacian pyramid network is combined with multi-scale fusion of texture images to generate a high-quality, detailed texture reconstruction model.

[0051] In one alternative approach, the step of mapping the facial texture reconstruction model to the facial geometry reconstruction model to obtain the complete 3D facial reconstruction result of the target user includes: Based on the UV mapping coordinates of each vertex of the face geometry model, the texture pixels of the face texture reconstruction model are mapped to the corresponding vertices of the face geometry reconstruction model to obtain the mapped three-dimensional face geometry model.

[0052] Here, UV mapping coordinates refer to the coordinate correspondence between 3D mesh vertices and 2D texture maps. For example, the 3D mesh nose tip vertex of target user A corresponds to the coordinate position (0.35, 0.62) in the texture map. The mapped 3D face geometry model refers to the 3D mesh model after texture binding. For example, the geometry model of target user A after associating texture colors with UV coordinates.

[0053] Color interpolation calculations are performed on the three vertices of each triangular facet of the mapped 3D face geometry model to generate a continuous color distribution within each triangular facet, which is then rendered to form a textured complete 3D face reconstruction result.

[0054] Here, a triangular facet refers to a facet in a 3D mesh consisting of three vertices. A continuous color distribution refers to the continuous color within a facet generated by interpolating the vertex colors of the triangular facets, such as the gradient transition from light red to dark red within the triangular facet of target user A's left cheek.

[0055] In the above-mentioned optional methods, UV mapping and color interpolation techniques are further used to accurately map the texture reconstruction model to the geometric model, thereby achieving efficient and high-quality rendering of complete 3D face reconstruction results.

[0056] It should be noted that, as Figure 4 and Figure 5As shown, the technical solution of this embodiment supports the entire process of assisted diagnosis, including voice and image-assisted shooting guidance, data uploading, cloud computing, and web-based result display, achieving complete remote 3D face-assisted diagnosis and evaluation. The topologically consistent high-precision 3D face model of this embodiment can closely fit the CT bone model, facilitating preoperative simulation and planning. Doctors can observe changes on the face surface by moving a small piece of bone, such as... Figure 6 As shown. Furthermore, topologically consistent high-precision 3D face models used for face symmetry analysis can be used to measure the thickness of facial soft tissue, for example, by defining the principal axis of the face for face symmetry analysis (e.g. Figure 7 As shown in the figure, this is of great significance for predicting the effects of plastic surgery and planning surgical methods.

[0057] Furthermore, the 3D face reconstruction method of this embodiment is analyzed from both qualitative and quantitative perspectives. The 3dMD scanning device, currently the most accurate and widely used face scanning and reconstruction device in the field of cosmetic surgery, is used as a benchmark model for comparison. Coarse alignment of the face model is performed by marking five key points on it. Then, fine alignment of the face model is performed by algebraically solving for the distance from points to the plane, and the face region error is statistically analyzed. By visualizing the face region error using color, the alignment result of the face can be seen more clearly, such as... Figure 8 As shown. In Figure 8 From left to right, the images show: the baseline model acquired by the 3DMD scanning device, the Emotion-Preserving Blendshape reconstruction result, the High-Fidelity 3D Digital Human Head reconstruction result, the 3D face reconstruction result of this embodiment, the error graph between the Emotion-Preserving Blendshape reconstruction result and the baseline result, the error graph between the High-Fidelity 3D Digital Human Head reconstruction result and the baseline result, and the error graph between the 3D face reconstruction result of this embodiment and the baseline result. Through quantitative and qualitative accuracy verification, the 3D face reconstruction result of this embodiment surpasses some existing reconstruction methods using RGBD video in terms of geometric accuracy.

[0058] Figure 9 This diagram illustrates a structural schematic of an embodiment of a portable device-based 3D face reconstruction system 200 provided by the present invention. Figure 9 As shown, the system 200 includes: an acquisition module 210, a geometry reconstruction module 220, a texture reconstruction module 230, and a fusion module 240; The acquisition module 210 is used to: acquire a sequence of facial image frames of a target user using a portable device; The geometric reconstruction module 220 is used to: perform facial key point detection on each frame of the face image in the face image frame sequence, and perform face geometric reconstruction based on the facial key point results of each frame of the face image in the face image frame sequence and in combination with a multi-stage fusion algorithm to obtain a face geometric reconstruction model. The texture reconstruction module 230 is used to: determine multiple key face images in the face image frame sequence based on a frame selection strategy, perform illumination removal processing on each key face image, and fuse each illumination-removed face texture image through a Laplacian pyramid network to generate a face texture reconstruction model. The fusion module 240 is used to: map the face texture reconstruction model to the face geometry reconstruction model to obtain the complete three-dimensional reconstruction result of the target user's face.

[0059] In one alternative approach, the multi-stage fusion algorithm includes: a 3DMM algorithm, a nodal deformation field algorithm, and a Laplacian deformation algorithm; the geometric reconstruction module 220 is specifically used for: Based on the 3DMM algorithm, global shape fitting is performed on the facial key point results of each frame of the face image in the face image frame sequence to generate a first face geometric model; Based on the nodal deformation field algorithm, the first face geometric model is locally structurally optimized to generate the second face geometric model; Based on the Laplacian deformation algorithm, high-frequency detail enhancement is performed on the second face geometric model to generate the face geometric reconstruction model.

[0060] In an alternative embodiment, the texture reconstruction module 230 is specifically used for: Based on the rotation matrix of each face image in the face image frame sequence, key face images with multiple preset face angles are determined in the face image frame sequence.

[0061] In an alternative embodiment, the fusion module 240 is specifically used for: Using a de-illumination network, each key face image is de-illuminated to obtain a de-illumination face texture image. The Laplacian pyramid network is used to perform multi-scale decomposition and weighted fusion on each illuminated face texture image to obtain fused pyramid features. The fused pyramid features are then reconstructed using Laplacian pyramids to generate the face texture reconstruction model.

[0062] In one alternative approach, the step of mapping the facial texture reconstruction model to the facial geometry reconstruction model to obtain the complete 3D facial reconstruction result of the target user includes: Based on the UV mapping coordinates of each vertex of the face geometry model, the texture pixels of the face texture reconstruction model are mapped to the corresponding vertices of the face geometry reconstruction model to obtain the mapped three-dimensional face geometry model. Color interpolation calculations are performed on the three vertices of each triangular facet of the mapped 3D face geometry model to generate a continuous color distribution within each triangular facet, which is then rendered to form a textured complete 3D face reconstruction result.

[0063] It should be noted that the beneficial effects of the portable device-based 3D face reconstruction system 200 provided in the above embodiments are the same as those of the portable device-based 3D face reconstruction method described above, and will not be repeated here. Furthermore, the system provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the system can be divided into different functional modules according to the actual situation to complete all or part of the functions described above. In addition, the system and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process is detailed in the method embodiments, and will not be repeated here.

[0064] The portable device-based 3D face reconstruction system 200 of the present invention can be a computer program (including program code) running on a computer device. For example, the portable device-based 3D face reconstruction system 200 of the present invention is an application software that can be used to execute the corresponding steps in the portable device-based 3D face reconstruction method of the present invention.

[0065] In some embodiments, the portable device-based 3D face reconstruction system 200 of the present invention can be implemented in a combination of hardware and software. As an example, the portable device-based 3D face reconstruction system 200 of the present invention can be a processor in the form of a hardware decoding processor, which is programmed to execute the portable device-based 3D face reconstruction method of the present invention. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0066] The modules described in the embodiments of this invention can be implemented in software or hardware. The names of the modules are not, in some cases, limiting the scope of the module itself.

[0067] An electronic device according to an embodiment of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any of the above-mentioned methods for three-dimensional face reconstruction based on a portable device. That is, an electronic device according to an embodiment of the present invention may include, but is not limited to: a processor and a memory; the memory is used to store the computer program; the processor is used to execute the method for three-dimensional face reconstruction based on a portable device as shown in any embodiment of the present invention by calling the computer program.

[0068] In one alternative embodiment, an electronic device is provided, such as Figure 10 As shown, Figure 10 The illustrated electronic device 4000 includes a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present invention.

[0069] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this invention. Processor 4001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0070] Bus 4002 may include a path for transmitting information between the aforementioned components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 4002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 10 The bus 4002 is represented by only one thick line, but this does not mean that there is only one bus or one type of bus.

[0071] The memory 4003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0072] The memory 4003 stores application code (computer program) for executing the present invention, and its execution is controlled by the processor 4001. The processor 4001 executes the application code stored in the memory 4003 to implement the content shown in the foregoing method embodiments.

[0073] Among them, electronic devices can also be terminal devices. A terminal device can be any terminal device that can install applications and access web pages through applications, including at least one of smartphones, tablets, laptops, desktop computers, smart speakers, smartwatches, smart TVs, and smart in-vehicle devices.

[0074] It should be noted that, Figure 10 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of the present invention.

[0075] An embodiment of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the above-described methods for three-dimensional face reconstruction based on a portable device.

[0076] Alternatively, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, a floppy disk, and an optical data storage device, etc.

[0077] In an exemplary embodiment, a computer program product or computer program is also provided, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the aforementioned three-dimensional face reconstruction method based on a portable device.

[0078] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0079] It should be understood that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0080] The computer-readable storage medium provided in this invention can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0081] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the method shown in the above embodiments.

[0082] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this invention is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-disclosed concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this invention.

[0083] It should be noted that the terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and represent a limitation on a specific order or sequence. Where appropriate, the order of use for similar objects can be interchanged so that the embodiments of this application described herein can be implemented in an order other than that shown or described.

[0084] Those skilled in the art will recognize that this invention can be implemented as a system, method, or computer program product. Therefore, this invention can be specifically implemented in the following forms: it can be entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, generally referred to herein as a "circuit," "module," or "system." Furthermore, in some embodiments, this invention can also be implemented as a computer program product contained in one or more computer-readable media, which includes computer-readable program code.

[0085] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A face three-dimensional reconstruction method based on a portable device, characterized in that, The method comprises the following steps: Collecting a face image frame sequence of a target user by using a portable device; Performing face key point detection on each face image in the face image frame sequence respectively, performing face geometry reconstruction based on the face key point results of each face image in the face image frame sequence, and combining a multi-stage fusion algorithm to obtain a face geometry reconstruction model; Determining a plurality of key face images in the face image frame sequence based on a frame selection strategy, performing de-illumination processing on each key face image, and fusing each de-illumination face texture image through a Laplacian pyramid network to generate a face texture reconstruction model; Mapping the face texture reconstruction model to the face geometry reconstruction model to obtain a complete face three-dimensional reconstruction result of the target user. 2.The face three-dimensional reconstruction method based on a portable device according to claim 1, characterized in that, The multi-stage fusion algorithm comprises a 3DMM algorithm, a node deformation field algorithm and a Laplacian deformation algorithm; the step of performing face geometry reconstruction based on the face key point results of each face image in the face image frame sequence and combining the multi-stage fusion algorithm to obtain the face geometry reconstruction model comprises: Performing global shape fitting on the face key point results of each face image in the face image frame sequence based on the 3DMM algorithm to generate a first face geometry model; Performing local structure optimization on the first face geometry model based on the node deformation field algorithm to generate a second face geometry model; Performing high-frequency detail enhancement on the second face geometry model based on the Laplacian deformation algorithm to generate the face geometry reconstruction model. 3.The face three-dimensional reconstruction method based on a portable device according to claim 1, characterized in that, The step of determining a plurality of key face images in the face image frame sequence based on a frame selection strategy comprises: Determining key face images of a plurality of preset face angles in the face image frame sequence based on the rotation matrix of each face image in the face image frame sequence. 4.The method of claim 1, wherein, The step of performing de-illumination processing on each key face image and fusing each de-illumination face texture image through a Laplacian pyramid network to generate a face texture reconstruction model comprises: Performing de-illumination processing on each key face image respectively by using a de-illumination network to obtain each de-illumination face texture image; Performing multi-scale decomposition and weighted fusion on each de-illumination face texture image through the Laplacian pyramid network to obtain fused pyramid features, and performing Laplacian pyramid reconstruction on the fused pyramid features to generate the face texture reconstruction model.

5. The face three-dimensional reconstruction method based on a portable device according to any one of claims 1 to 4, characterized in that, The step of mapping the face texture reconstruction model to the face geometry reconstruction model to obtain a complete face three-dimensional reconstruction result of the target user comprises: Mapping the texture pixels of the face texture reconstruction model to the corresponding vertices of the face geometry reconstruction model based on the UV mapping coordinates of each vertex of the face geometry model to obtain a mapped three-dimensional face geometry model; Performing color interpolation calculation on the three vertices of each triangular facet of the mapped three-dimensional face geometry model to generate continuous color distribution inside each triangular facet to render the complete face three-dimensional reconstruction result with texture.

6. A face three-dimensional reconstruction system based on a portable device, characterized by, The method comprises the following steps: The acquisition module, the geometry reconstruction module, the texture reconstruction module, and the fusion module; The acquisition module is configured to acquire a sequence of face image frames of a target user by using a portable device. The geometry reconstruction module is configured to perform face key point detection on each face image in the sequence of face image frames respectively, perform face geometry reconstruction based on face key point results of each face image in the sequence of face image frames and in combination with a multi-stage fusion algorithm, and obtain a face geometry reconstruction model. The texture reconstruction module is configured to determine a plurality of key face images in the sequence of face image frames based on a frame selection strategy, perform de-illumination processing on each key face image, and fuse each de-illuminated face texture image through a Laplacian pyramid network to generate a face texture reconstruction model. The fusion module is configured to map the face texture reconstruction model to the face geometry reconstruction model to obtain a complete face three-dimensional reconstruction result of the target user. 7.The portable device based face 3D reconstruction system of claim 6, wherein, The multi-stage fusion algorithm includes a 3DMM algorithm, a node deformation field algorithm, and a Laplacian morphing algorithm. The geometry reconstruction module is specifically configured to: perform global shape fitting on face key point results of each face image in the sequence of face image frames based on the 3DMM algorithm to generate a first face geometry model; perform local structure optimization on the first face geometry model based on the node deformation field algorithm to generate a second face geometry model; and 8.The portable device based face 3D reconstruction system of claim 7, wherein, perform high-frequency detail enhancement on the second face geometry model based on the Laplacian morphing algorithm to generate the face geometry reconstruction model. The texture reconstruction module is specifically configured to:

9. An electronic device, comprising: determine key face images of a plurality of preset face angles in the sequence of face image frames based on a rotation matrix of each face image in the sequence of face image frames.

10. A computer-readable storage medium, characterized in that, The electronic device includes a processor coupled with a memory, and the memory stores at least one computer program, which is loaded and executed by the processor to enable the electronic device to implement the face three-dimensional reconstruction method based on a portable device according to any one of claims 1 to 5. The computer readable storage medium stores at least one computer program, which is loaded and executed by the processor to enable the computer readable storage medium to implement the face three-dimensional reconstruction method based on a portable device according to any one of claims 1 to 5.

Images