A method and system for repairing and 3D reconstruction based on contaminated face pictures

Abstract

The application discloses a kind of repair and 3D reconstruction method and system based on contaminated face picture, it is related to artificial intelligence technical field, including: obtaining and preprocessing the contaminated image containing real prototype face image;Through the generator-discriminator network of preestablished, the face image after pre-processing is handled;Initial generation prototype face image is merged with the public face dataset after pre-processing as input image, and the input image is processed according to the parallel encoder network, to obtain initial 3D face model, depth map and reflectivity map;Depth reflection loss function is established based on depth map and reflectivity map;And generator and discriminator are trained based on the generator loss function after updating and the preset discriminator loss function respectively, to output target 3D face reconstruction model.The application solves the problem that the detail fidelity and realism of 3D reconstruction result of single contaminated picture are poor.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of artificial intelligence, and particularly relates to a repair and 3D reconstruction method and system based on a polluted face picture. BACKGROUND

[0002] In today's information globalization, the demand for high-precision and high-efficiency capture and restoration of real face information is becoming increasingly urgent. 3D face reconstruction technology, with its ability to construct a three-dimensional face model with depth, texture and topological structure, has shown great application value in identity authentication, entertainment creativity, medical health, human-computer interaction and many other fields. Traditional 3D face capture mainly relies on complex multi-camera arrays, structured light scanners or laser scanning devices. Although these methods have high precision, they are high in cost, cumbersome to operate and strict in environmental requirements, and are difficult to meet the needs of real-time, convenient and large-scale applications. With the rapid development of computer vision and deep learning technology, 3D face reconstruction methods based on monocular or multi-view images have emerged and developed rapidly. Early algorithms focus on solving reconstruction problems under specific angles, specific lighting or specific expressions, such as using feature point positioning for basic modeling or generating approximate models through template fitting. However, these methods often have limited generalization ability and are difficult to robustly handle complex and variable face poses, expression changes, occlusions and different lighting conditions. The detail fidelity and realism of the reconstruction results are often not satisfactory. How to handle 3D face reconstruction of polluted pictures due to different lighting conditions, expression changes, face poses, etc. is a major issue in 3D face reconstruction tasks.

[0003] In recent years, generative models have developed rapidly in the field of artificial intelligence, and their core goal is to create pre-trained models that can efficiently generate diverse content. Such models learn the internal distribution rules and generation patterns of data through training on massive data, thereby possessing strong content creation capabilities. In the application of 3D face reconstruction models, generative models focus on pre-training using a large amount of 3D face data and related multi-modal information, thereby learning the fine geometric structure, texture features and variation rules under different lighting and pose conditions of the face. Such a basic model can be quickly adapted and optimized for various specific application scenarios such as face recognition, expression analysis and virtual avatar creation, which is expected to significantly improve the accuracy, efficiency and robustness of 3D face reconstruction, provide strong technical support for intelligent applications in related fields, and greatly promote the innovation and development process of computer vision and human-computer interaction fields.

[0004] However, this process faces several challenges. First, generative models often need to handle complex geometric and textural information and perform intricate optimization and sampling in high-dimensional space when processing 3D face data. This significantly increases computational complexity, and in practical applications, this high computational cost may limit its use in real-time or resource-constrained scenarios. Second, 3D face reconstruction has extremely high requirements for data quality and diversity. Although generative models can learn from massive amounts of data, acquiring high-quality 3D face data is costly, and the data annotation and preprocessing processes are complex, making it easy for generative models to produce distortions and unnatural results when processing 3D face data. Third, some existing works separate 2D generative models from 3D face reconstruction tasks, but these generally suffer from poor interoperability and compatibility between the two models.

[0005] Therefore, there is an urgent need to provide a solution to improve the above problems. Summary of the Invention

[0006] The purpose of this invention is to provide a method and system for the restoration and 3D reconstruction of contaminated facial images, which can solve the problem of poor detail fidelity and realism in the 3D reconstruction results of a single contaminated image in the existing technology.

[0007] To solve the above-mentioned technical problems, the present invention is implemented as follows:

[0008] In a first aspect, embodiments of the present invention provide a method for the restoration and 3D reconstruction of contaminated facial images, the method comprising:

[0009] S1. Obtain and preprocess the contaminated image containing the real prototype face image to obtain the preprocessed face image;

[0010] S2. The preprocessed face image is processed through a preset generator-discriminator network to obtain an initial generated prototype face image and a confidence map. The preset generator-discriminator network includes a generator and a discriminator.

[0011] S3. The initial generated prototype face image is merged with the preprocessed public face dataset as the input image. The input image is processed according to the parallel encoder network to obtain the initial 3D face model, depth map and reflectance map.

[0012] S4. Repeat steps S2 to S3 to establish a depth reflection loss function based on the depth map and reflectance map;

[0013] S5. Obtain the updated generator loss function based on the depth reflection loss function and the preset generator loss function, and train the generator and discriminator based on the updated generator loss function and the preset discriminator loss function respectively, so as to output the target 3D face reconstruction model.

[0014] Optionally, the process of processing the preprocessed face image using a preset generator-discriminator network includes:

[0015] The generator and discriminator are trained based on preset generator loss functions and preset discriminator loss functions, respectively, to obtain the trained generator and trained discriminator. The mathematical expression of the preset generator loss function is as follows:

[0016]

[0017]

[0018]

[0019]

[0020]

[0021] in, For preprocessed, potentially contaminated input face images, An uncontaminated, authentic human face image. It is random noise. , , These are the discriminators corresponding to the prototype, identity, and mutation, respectively. , This represents the weights of the generator loss function; the default values ​​are 5, 0.5, and 0.1. , , , These represent the prototype authenticity loss function, identity classification loss function, mutation judgment loss function, and consistency constraint reconstruction loss function, respectively. , , This indicates the expected calculation. Represents the loss function of the generator. This indicates the generation of a prototype, and satisfies , A pixel-by-pixel confidence map representing the model's predictions;

[0022] The mathematical expression for the preset discriminator loss function is:

[0023]

[0024]

[0025]

[0026] in, This represents the loss function of the discriminator, and the loss function based on prototype authenticity. Identity discrimination loss function Pollution discrimination loss function The components are used to distinguish uncontaminated, real-life prototype facial images. and generating prototypes ,enter Identity tags and input Does it contain pollution? , Indicates weight, and This indicates that the discriminator detects potentially contaminated input face images. The result of the processing.

[0027] Optionally, the process of merging the initially generated prototype face image with a preprocessed public face dataset as the input image, and processing the input image according to a parallel encoder network to obtain the initial 3D face model, depth map, and reflectance map includes:

[0028] S31. Using a parallel encoder, the initially generated prototype face image and the preprocessed CelebA face dataset are decomposed to obtain a reconstructed view. The mathematical expression for obtaining the reconstructed view is:

[0029]

[0030] in, To reconstruct the view, Given the lighting function, the normal vector is derived from the depth map, and the illuminated texture image is obtained using the Lambertian lighting model. The reprojection function projects a 3D model onto a 2D viewpoint through 3D point reconstruction, viewpoint transformation, and pixel resampling to obtain a reconstructed view. , Indicates reflectivity, Indicates perspective, Indicates illumination. This represents the depth map generated by the parallel encoder network;

[0031] S32. Generate a symmetrical reconstructed view based on the object's symmetry, and establish a reconstruction loss function based on the confidence map, the input image, and the reconstructed view. The mathematical expression of the reconstruction loss function is as follows:

[0032]

[0033] in, Represents the reconstruction loss function. , This is a horizontal flip operation;

[0034] S33. Train the parallel encoder network using the reconstruction function so that both the reconstructed view and the symmetrical reconstructed view approximate the input image, and obtain the depth map, reflectance map and 3D face reconstruction model corresponding to each generated prototype face image.

[0035] Optionally, the specific process of obtaining the updated generator loss function based on the depth reflection loss function and the preset generator loss function includes: constructing a depth reflection loss function based on the depth map and reflectance map, and adding the depth reflection loss function to the preset generator loss function to obtain the updated generator loss function, wherein the mathematical expression of the updated generator loss function is:

[0036]

[0037] in, This represents the generator loss function before the update. This represents the updated generator loss function. and This indicates a pre-processed but potentially contaminated input face image. , These represent the weights of the loss function, with default values ​​of 10 and 5. This represents the depth map loss function. This represents the reflectivity loss function.

[0038] Secondly, embodiments of the present invention provide a system for the restoration and 3D reconstruction of contaminated facial images, the system comprising:

[0039] The image preprocessing module is used to acquire and preprocess contaminated images containing real-world prototype face images to obtain preprocessed face images;

[0040] The generator-discriminator network module is used to process the preprocessed face image to obtain the initial generated prototype face image and confidence map. The preset generator-discriminator network includes a generator and a discriminator.

[0041] The parallel encoder network module is used to merge the initially generated prototype face image with the preprocessed public face dataset as the input image, and process the input image according to the parallel encoder network to obtain the initial 3D face model, depth map and reflectance map.

[0042] The depth reflection loss function construction module is used to build a depth reflection loss function based on the depth map and reflectance map.

[0043] The 3D face reconstruction module is used to obtain an updated generator loss function based on the depth reflection loss function and a preset generator loss function, and to train the generator and discriminator based on the updated generator loss function and the preset discriminator loss function respectively, so as to output the target 3D face reconstruction model.

[0044] Thirdly, embodiments of this application provide an electronic device including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the method as described in the first aspect.

[0045] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, and when the program or instructions are executed by a processor, they implement the steps of the method as described in the first aspect.

[0046] Compared with existing technologies, the beneficial effects of the method for repairing and 3D reconstructing polluted facial images provided by this invention are as follows:

[0047] First, the present invention cyclically trains the generator discriminator network and the encoder network, enabling the former to incorporate 3D information such as reflectivity and depth maps during training, while the latter incorporates 2D information such as confidence. This allows the two networks to interact and promote each other.

[0048] Secondly, this invention proposes an end-to-end model that can obtain a restored 2D face image and its 3D face reconstruction model from a single 2D contaminated image as input. This effectively restores a single contaminated face image to its original form and obtains its 3D face model.

[0049] In summary, this invention effectively utilizes 2D face images contaminated by different lighting conditions, facial expressions, and poses, along with publicly available face datasets, to obtain decontaminated, realistic prototype face images and their 3D face reconstruction models. This invention solves the problem of poor detail fidelity and realism in 3D reconstruction results from single contaminated images. Attached Figure Description

[0050] Figure 1 This is a flowchart of a method for repairing and 3D reconstructing contaminated facial images provided in the first embodiment of the present invention;

[0051] Figure 2 This is a flowchart of the generator-discriminator network provided in the first embodiment of the present invention;

[0052] Figure 3 This is a flowchart of the parallel encoder network provided in the first embodiment of the present invention;

[0053] Figure 4 This is a qualitative example diagram of the reconstruction model provided in the first embodiment of the present invention;

[0054] Figure 5 This is a structural diagram of a system for repairing and 3D reconstructing polluted facial images provided in the second embodiment of the present invention. Detailed Implementation

[0055] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] The terms "first," "second," etc., used in the specification and claims of this invention are used to distinguish similar objects and are not used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0057] The following detailed description, in conjunction with the accompanying drawings, of a method and system for repairing and 3D reconstructing contaminated facial images provided by the present invention, through specific embodiments and application scenarios, will be provided in detail.

[0058] Example 1

[0059] First, it is worth noting that the technical solution of this invention can be divided into the following six parts:

[0060] I. Data Preprocessing. A dataset of 2D contaminated face images under various lighting conditions, facial expressions, and poses is collected to ensure that each contaminated face image with a unique identifier contains an uncontaminated, original face image. Each face image undergoes standardization processing, including image cropping and normalization, to ensure that face images from different sources and shooting conditions have a uniform size, which is then used as input to the generator and discriminator networks.

[0061] II. Generator-Discriminator Network Structure Construction. Two independent encoder-decoder structures are designed to form the generator G. One encoder-decoder structure is used to extract identity features and concatenate these features with random noise to generate the prototype; the other encoder-decoder structure is used to decode the confidence map of the image, which measures the probability of symmetry for each pixel in the image. The discriminator D is designed as a multi-task discriminator to distinguish image identity labels, whether the image is contaminated, and whether the image was generated by the generator.

[0062] III. Coarse-to-Fine Training Process. Through iterative training of the generator and discriminator, the generator eventually generates a real prototype face image. The discriminator has only about a 1 / 2 probability of distinguishing whether the image is a real image or a generated image. Subsequently, the face image generated by the generator is used as the initial real prototype face image.

[0063] IV. Constructing a Parallel Encoder Network. A four-branch prediction network is constructed. Inputting preliminary prototype face images and images from a publicly available face dataset into the prediction network yields the corresponding depth map, reflectance map, viewpoint parameters, and illumination parameters. The depth map is used to standardize the 3D geometry at a specific viewpoint; the reflectance map is used to obtain the face's surface color independent of illumination; and the viewpoint and illumination parameters are used to simulate the viewing angle of the face and the corresponding lighting conditions. Using the depth map, reflectance map, viewpoint parameters, and illumination parameters, a preliminary 3D face reconstruction model can be obtained.

[0064] V. Rendering. Using the depth map, reflectivity map, viewpoint parameters, and lighting parameters obtained above, the 3D face reconstruction model is projected onto a 2D plane using a renderer. The loss function between the 2D face and the real face is calculated using the confidence map, photometric reconstruction loss function, and perceptual loss function. The parameters of the parallel encoder network are then optimized and trained. During training, the model parameters are continuously adjusted to minimize the loss function value.

[0065] VI. Fine-grained Training Process. Repeat steps two, three, four, and five. By introducing depth maps and reflectivity maps containing 3D information into the generator-discriminator network, a new generator-discriminator loss function is designed. The generator-discriminator network is iteratively trained to obtain fine-grained, realistic prototype face images. The final 3D face reconstruction model is obtained by iteratively training the parallel encoder network.

[0066] Specifically, please refer to Figure 1 This is a flowchart of the overall process of a method for repairing and 3D reconstructing contaminated facial images provided by the present invention, including steps S1 to S5.

[0067] S1: Acquire and preprocess the contaminated image containing the real prototype face image to obtain the preprocessed face image.

[0068] Specifically, the process of acquiring and preprocessing contaminated images containing real-world prototype face images includes: collecting a dataset of 2D contaminated face images under different lighting conditions, with varying facial expressions and poses, ensuring that each contaminated face image with a unique identifier contains an uncontaminated real-world prototype face image. Each face image undergoes standardization processing, including image cropping and normalization, to ensure that face images from different sources and shooting conditions have a uniform size, which is then used as input to the generator and discriminator networks.

[0069] S2: The preprocessed face image is processed through a preset generator-discriminator network to obtain an initial prototype face image and a confidence map. The preset generator-discriminator network includes a generator and a discriminator.

[0070] Please see Figure 2 The diagram illustrates the structure of the generator-discriminator network provided in this invention. This invention designs two independent encoder-decoder structures to form the generator G. One encoder-decoder structure extracts identity features and concatenates these features with random noise to generate a prototype; the other encoder-decoder structure decodes the confidence map of an image, which measures the probability of symmetry for each pixel in the image. The discriminator D is designed as a multi-task discriminator to distinguish image identity labels, whether the image is contaminated, and whether the image was generated by the generator. Through iterative training of the generator and discriminator, the discriminator has only approximately a 1 / 2 probability of distinguishing whether the final prototype face image generated by the generator is a real image or a generated image. The face image generated by the generator is then used as the initial prototype face image.

[0071] Specifically, the steps to obtain the initial prototype face image and confidence map include the following:

[0072] S21. Obtain the corresponding confidence level from the confidence encoder-decoder and identity feature encoder under the generator network. , 'and facial image identity features Identification features through facial images With random noise The images are concatenated and the result is input into the identity feature decoder to generate the prototype face image. .

[0073] S22. Using the input face image, generate a prototype face image. After iteratively training the generator and discriminator with real prototype face images for several rounds, the generator can generate relatively realistic prototype face images. The generator G loss function is as follows:

[0074]

[0075]

[0076]

[0077]

[0078]

[0079] in, For preprocessed, potentially contaminated input face images, An uncontaminated, authentic human face image. It is random noise. , , These are the discriminators corresponding to the prototype, identity, and mutation, respectively. , This represents the weights of the generator loss function; the default values ​​are 5, 0.5, and 0.1. , , , These represent the prototype authenticity loss function, identity classification loss function, mutation judgment loss function, and consistency constraint reconstruction loss function, respectively. , , This indicates the expected calculation. Represents the loss function of the generator. This indicates the generation of a prototype, and satisfies , A pixel-by-pixel confidence map representing the model's predictions;

[0080] The loss function of discriminator D is as follows:

[0081]

[0082]

[0083]

[0084] in, This represents the loss function of the discriminator, and the loss function based on prototype authenticity. Identity discrimination loss function Pollution discrimination loss function The components are used to distinguish uncontaminated, real-life prototype facial images. and generating prototypes ,enter Identity tags and input Does it contain pollution? , Indicates weight, and This indicates that the discriminator detects potentially contaminated input face images. The result of the processing.

[0085] S3. The initial generated prototype face image is merged with the preprocessed public face dataset as the input image. The input image is processed according to the parallel encoder network to obtain the initial 3D face model, depth map and reflectance map.

[0086] It is worth noting that, please refer to Figure 3 , represents the parallel encoder network diagram provided by the present invention.

[0087] Specifically, the process of processing the input image using a parallel encoder network to obtain an initial 3D face model, depth map, and reflectance map includes:

[0088] S31, Generating a Prototype The preprocessed CelebA face data is decomposed into depth map d, reflectivity a, viewpoint w, and illumination l by a parallel encoder.

[0089]

[0090] in, To reconstruct the view, Given the lighting function, the normal vector is derived from the depth map, and the illuminated texture image is obtained using the Lambertian lighting model. The reprojection function projects a 3D model onto a 2D viewpoint through 3D point reconstruction, viewpoint transformation, and pixel resampling to obtain a reconstructed view. , Indicates reflectivity, Indicates perspective, Indicates illumination. This represents a depth map.

[0091] S32. Generating a virtual reconstructed view using object symmetry. And utilize the confidence map, input image I, and reconstructed image Construct a loss function that enables the reconstruction of the view. and symmetrical reconstruction view Both approximate the input image I. Reconstruct the view. The mathematical expression is as follows:

[0092]

[0093] in, Represents the reconstruction loss function. , This is a horizontal flip operation.

[0094] S4. Repeat steps S2 to S3 to establish a depth reflection loss function based on the depth map and reflectance map.

[0095] Specifically, in step S4, this invention introduces the 3D information depth map and reflectance map of the image into the generator-discriminator loss function, adding a depth map loss function to the generator loss function. With reflectivity loss function as follows:

[0096]

[0097] in, This represents the generator loss function before the update. This represents the updated generator loss function. and This indicates a pre-processed but potentially contaminated input face image. , These represent the weights of the loss function, with default values ​​of 10 and 5. This represents the depth map loss function. This represents the reflectivity loss function.

[0098] S5. Obtain the updated generator loss function based on the depth reflection loss function and the preset generator loss function, and train the generator and discriminator based on the updated generator loss function and the preset discriminator loss function respectively, so as to output the target 3D face reconstruction model.

[0099] Specifically, this invention iteratively trains the generator and discriminator for several rounds to obtain the final generated prototype image. The obtained confidence map is then fed into a parallel encoder network to participate in the loss function calculation, and after iterative training, the final 3D face reconstruction model is obtained.

[0100] To demonstrate the feasibility of the present invention, simulation experiments were also conducted. Based on the method provided by the present invention, this example uses the publicly available face dataset CelebA for pre-training. The contaminated images are obtained by applying lighting, pose, expression, and occlusion contamination to a subset of samples in the CelebA dataset. The CelebA dataset contains 202,599 celebrity facial images. We adjusted the image size to 128×128 for easier subsequent operations.

[0101] This invention compares two parts based on objective evaluation metrics: image restoration metrics and 3D reconstruction metrics. FID, PSNR, and SSIM are used as image restoration quality evaluation metrics, and comparative experiments are conducted with two common image restoration algorithms. The experimental results are shown in Table 1.

[0102] Table 1 Comparison of 2D Repair Experiment Results

[0103]

[0104] As shown in Table 1, the FID of the present invention is 20.92, the PSNR is 25.33, and the SSIM is 0.945. This indicates that the image set restored by the present invention is not only very similar to the set of real images in terms of visual feature distribution, but also has a high degree of similarity in pixel values ​​and edge and texture structure.

[0105] Furthermore, this invention uses depth correlation score (DepthCorr) as the evaluation metric for the 3D reconstruction portion. The experiment used the 3DFAW dataset to calculate the Pearson correlation coefficient between the depth corresponding to 66 key points of the real face image and the predicted depth, and normalized the result to between 0 and 66 to obtain the final DepthCorr. The experimental results are shown in Table 2. As shown in Table 2, the DepthCorr value of this invention is the highest, at 55.62. This indicates that the 3D face reconstructed by this invention has high accuracy in geometric structure while avoiding scale ambiguity.

[0106] Table 2. Comparison of 3D Reconstruction Experiment Results

[0107]

[0108] Figure 4 The text provides a qualitative example of generating a 2D realistic prototype face image and a 3D face reconstruction model from a contaminated image, where the first row on the left... Figure 1 The input image is corrupted by facial expressions; the first row is on the left. Figure 2 This is a restored image of the original face, on the right. Figure 1 The first row shows its 3D face model, and the second row shows the reflectance and depth maps of the real-world prototype face image.

[0109] Example 2

[0110] Please see Figure 5 The second embodiment of the present invention also provides a system for the restoration and 3D reconstruction of contaminated facial images, the system comprising:

[0111] Image preprocessing module 100 is used to acquire and preprocess contaminated images containing real prototype human face images to obtain preprocessed face images;

[0112] The generator-discriminator network module 200 is used to process the preprocessed face image to obtain an initial generated prototype face image and a confidence map;

[0113] The parallel encoder network module 300 is used to merge the initially generated prototype face image with the preprocessed public face dataset as the input image, and process the input image according to the parallel encoder network to obtain the initial 3D face model, depth map and reflectance map.

[0114] The depth reflection loss function construction module 400 is used to build a depth reflection loss function based on the depth map and reflectance map.

[0115] The 3D face reconstruction module 500 is used to obtain an updated generator loss function based on the depth reflection loss function and a preset generator loss function, and to train the generator and discriminator based on the updated generator loss function and the preset discriminator loss function respectively, so as to output the target 3D face reconstruction model.

[0116] The beneficial effects of the proposed system for restoring and reconstructing contaminated facial images are as follows: This invention utilizes a generator-discriminator network module 200 and a parallel encoder network module 300. The generator-discriminator network module 200 incorporates 3D information such as reflectivity and depth maps during training, while the parallel encoder network module 300 incorporates 2D information such as confidence levels. This allows the two networks to interact and mutually reinforce each other. Inputting a single contaminated 2D image yields a restored 2D facial image and its corresponding 3D facial reconstruction model, effectively restoring a single contaminated facial image to its original, realistic form and obtaining its 3D facial model.

[0117] The system for repairing and 3D reconstructing contaminated facial images in this embodiment of the invention can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, a mobile electronic device can refer to a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc., while a non-mobile electronic device can refer to a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This embodiment of the invention does not impose specific limitations.

[0118] The system for repairing and 3D reconstructing contaminated facial images in this embodiment of the invention can refer to a device with an operating system. This operating system can refer to Android, iOS, or other possible operating systems; this embodiment of the invention does not impose specific limitations.

[0119] The present invention provides a system for the restoration and 3D reconstruction of contaminated facial images, which can achieve... Figures 1 to 4 The various processes implemented in the method embodiment of a method for repairing and 3D reconstructing a polluted face image will not be described again here to avoid repetition.

[0120] Optionally, embodiments of the present invention also provide an electronic device, including a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the various processes of the above-described embodiment of a method for repairing and 3D reconstructing polluted facial images and achieve the same technical effect. To avoid repetition, further details are omitted here.

[0121] This invention also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described embodiment of a method for repairing and 3D reconstructing polluted facial images, and achieve the same technical effect. To avoid repetition, this will not be elaborated further here.

[0122] The processor refers to the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0123] It should be noted that, in this invention, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatus in the embodiments of this invention is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0124] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0125] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of the present invention.

Claims

1. A method for restoring and 3D reconstructing contaminated facial images, characterized in that, include: S1. Obtain and preprocess the contaminated image containing the real prototype face image to obtain the preprocessed face image; S2. The preprocessed face image is processed through a preset generator-discriminator network to obtain an initial generated prototype face image and a confidence map. The preset generator-discriminator network includes a generator and a discriminator. The process of processing the preprocessed face image using a preset generator-discriminator network includes: The generator and the discriminator are trained based on preset generator loss functions and preset discriminator loss functions, respectively, to obtain trained generators and trained discriminators. The mathematical expression of the preset generator loss function is as follows: in, The input face image to be determined as contaminated after preprocessing. An uncontaminated, authentic human face image. It is random noise. , , These are the discriminators corresponding to the prototype, identity, and mutation, respectively. , This represents the weights of the generator loss function; the default values ​​are 5, 0.5, and 0.

1. , , , These represent the prototype authenticity loss function, identity classification loss function, mutation judgment loss function, and consistency constraint reconstruction loss function, respectively. , , This indicates the expected calculation. Represents the loss function of the generator. This indicates the generation of a prototype, and satisfies... , A pixel-by-pixel confidence map representing the model's predictions; The mathematical expression for the preset discriminator loss function is: in, This represents the loss function of the discriminator, and the loss function based on prototype authenticity. Identity discrimination loss function Pollution discrimination loss function The components are used to distinguish uncontaminated, real-life prototype facial images. and generating prototypes ,enter Identity tags and input Does it contain pollution? , Indicates weight, and This indicates that the discriminator is determining whether the input face image is contaminated. The result of the processing; S3. The initial generated prototype face image is merged with the preprocessed public face dataset as the input image. The input image is processed according to the parallel encoder network to obtain the initial 3D face model, depth map and reflectance map. S4. Repeat steps S2 to S3 to establish a depth reflection loss function based on the depth map and the reflectance map; S5. Obtain an updated generator loss function based on the depth reflection loss function and the preset generator loss function, and train the generator and the discriminator based on the updated generator loss function and the preset discriminator loss function respectively, so as to output the target 3D face reconstruction model.

2. The method for restoration and 3D reconstruction of contaminated facial images according to claim 1, characterized in that, The process of merging the initially generated prototype face image with a preprocessed public face dataset as the input image, and processing the input image using a parallel encoder network to obtain the initial 3D face model, depth map, and reflectance map includes: S31. Using a parallel encoder, the initially generated prototype face image and the preprocessed CelebA face dataset are decomposed to obtain a reconstructed view, wherein the mathematical expression for obtaining the reconstructed view is: in, To reconstruct the view, Given the lighting function, the normal vector is derived from the depth map, and the illuminated texture image is obtained using the Lambertian lighting model. The reprojection function projects a 3D model onto a 2D viewpoint through 3D point reconstruction, viewpoint variation, and pixel resampling to obtain a reconstructed view. , Indicates reflectivity, Indicates perspective, Indicates illumination. This represents the depth map generated by the parallel encoder network; S32. Generate a symmetrical reconstructed view based on the object's symmetry, and establish a reconstruction loss function based on the confidence map, the input image, and the reconstructed view, wherein the mathematical expression of the reconstruction loss function is: in, Represents the reconstruction loss function. , This is a horizontal flip operation; S33. The parallel encoder network is trained using the reconstruction loss function so that both the reconstructed view and the symmetrical reconstructed view approximate the input image, thereby obtaining the depth map, reflectance map, and 3D face reconstruction model corresponding to each generated prototype face image.

3. The method for restoration and 3D reconstruction of contaminated facial images according to claim 1, characterized in that, The specific process of obtaining the updated generator loss function based on the depth reflection loss function and the preset generator loss function includes: constructing a depth reflection loss function based on the depth map and the reflectance map, and adding the depth reflection loss function to the preset generator loss function to obtain the updated generator loss function, wherein the mathematical expression of the updated generator loss function is: in, This represents the generator loss function before the update. This represents the updated generator loss function. and This indicates the preprocessed input face image to be determined as contaminated. , These represent the weights of the loss function, with default values ​​of 10 and 5. This represents the depth map loss function. This represents the reflectivity loss function.

4. A system for the restoration and 3D reconstruction of contaminated facial images, characterized in that, A system for implementing a method for repairing and 3D reconstructing contaminated facial images as described in any one of claims 1-3; the system comprises: The image preprocessing module is used to acquire and preprocess contaminated images containing real-world prototype face images to obtain preprocessed face images; The generator-discriminator network module is used to process the preprocessed face image to obtain an initial generated prototype face image and a confidence map; The parallel encoder network module is used to merge the initial generated prototype face image with the preprocessed public face dataset as the input image, and process the input image according to the parallel encoder network to obtain the initial 3D face model, depth map and reflectance map. A depth reflection loss function construction module is used to establish a depth reflection loss function based on the depth map and the reflectance map; The 3D face reconstruction module is used to obtain an updated generator loss function based on the depth reflection loss function and a preset generator loss function, and to train the generator and the discriminator based on the updated generator loss function and the preset discriminator loss function respectively, so as to output a target 3D face reconstruction model.

5. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of a method for repairing and 3D reconstructing a contaminated face image as described in any one of claims 1-3.

6. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the method for repairing and 3D reconstructing a polluted face image as described in any one of claims 1-3.

Images