Three-dimensional reconstruction method, device and storage medium based on image set

Abstract

The application relates to a three-dimensional reconstruction method, device and storage medium based on an image set, wherein the three-dimensional reconstruction method based on the image set comprises the following steps: acquiring a plurality of to-be-processed human face images under a dark light scene; inputting the to-be-processed human face images into a preset encoder architecture to obtain a feature vector comprising spherical harmonic illumination coefficients, facial shadow parameters and non-human face illumination characteristic parameters; adjusting the spherical harmonic illumination coefficients and the facial shadow parameters based on the to-be-processed human face images and a preset target brightness, and then combining the non-human face illumination characteristic parameters to obtain an optimized feature vector; inputting the optimized feature vector and a preset random noise image into a preset decoder architecture to generate a normal brightness human face image feature; and inputting the normal brightness human face image feature into a preset three-dimensional Gaussian splatting reconstruction network to obtain a human face three-dimensional model. Through the application, the problem that the quality of human face three-dimensional modeling is low when the input images are a plurality of dimly lit human face images is solved.

Description

Technical Field

[0001] This application relates to the fields of computer vision and 3D modeling technology, and in particular to 3D reconstruction methods, apparatus and storage media based on image sets. Background Technology

[0002] In the fields of computer vision and 3D modeling, generating high-precision 3D models based on 2D image sets has always been a core research topic. Among these, 3D face reconstruction has become a research hotspot due to its significant applications in identity recognition, virtual reality, and film and television production. In traditional 3D reconstruction methods, if the input image is a face image with dim lighting, the quality of the 3D face model will be low.

[0003] Currently, in related technologies, if the input image consists of multiple dimly lit facial images, the quality of 3D facial modeling is low, and no effective solution has been proposed. Summary of the Invention

[0004] This application provides a three-dimensional reconstruction method, apparatus, and storage medium based on an image set, to at least solve the problem in the related art where the quality of three-dimensional face modeling is low when the input image is a face image under dim lighting.

[0005] In a first aspect, embodiments of this application provide a three-dimensional reconstruction method based on an image set, the method comprising:

[0006] Acquire multiple facial images of the same person from different angles in a low-light scene;

[0007] The face image to be processed is input into a preset encoder architecture to obtain a feature vector; the feature vector includes spherical harmonic illumination coefficients, facial shadow parameters, and non-face illumination feature parameters;

[0008] Based on the face image to be processed and the preset target brightness, the spherical harmonic illumination coefficient is adjusted to obtain the optimized spherical harmonic illumination coefficient; the facial shadow parameter is lowered to a preset value to obtain the optimized facial shadow parameter; and an optimized feature vector is obtained based on the optimized spherical harmonic illumination coefficient, the optimized facial shadow parameter, and the non-face illumination feature parameter.

[0009] The optimized feature vector and the preset random noise image are input into the preset decoder architecture to generate normal brightness face image features;

[0010] The features of the normal brightness face image are input into a preset three-dimensional Gaussian splash reconstruction network to obtain a three-dimensional face model.

[0011] In some embodiments, the encoder architecture includes a bilateral segmentation network, a face reconstruction network, an identity feature extraction network, and a diffusion model network; the non-face illumination feature parameters include face geometric parameters, projection parameters, background region, and identity feature parameters; the step of inputting the face image to be processed into the preset encoder architecture to obtain a feature vector includes:

[0012] The face image to be processed is input into the face reconstruction network to obtain the spherical harmonic illumination coefficient, the face geometric parameters, and the projection parameters;

[0013] The face image to be processed is input into the bilateral segmentation network to obtain the background region;

[0014] The face image to be processed is input into the identity feature extraction network to obtain the identity feature parameters;

[0015] The face image to be processed is input into the diffusion model network to obtain the facial shadow parameters;

[0016] The feature vector is generated based on the spherical harmonic illumination coefficient, the facial shadow parameters, the facial geometric parameters, the projection parameters, the background region, and the identity feature parameters.

[0017] In some embodiments, the diffusion model network includes an image semantic diffusion encoder and a logistic regression classifier; the step of inputting the face image to be processed into the diffusion model network to obtain the facial shadow parameters includes:

[0018] The face image to be processed is input into the image semantic diffusion encoder to obtain the target low-dimensional latent vector;

[0019] The target low-dimensional latent vector is input into the logistic regression classifier, and the original score of the logistic regression classifier on the decision boundary is extracted as the facial shadow parameter.

[0020] In some embodiments, the training process of the diffusion model network includes:

[0021] Obtain a first face image training set; the first face image training set carries preset shadow region coordinates;

[0022] The first face image training set is input into the initial image semantic diffusion encoder in the initial diffusion model network, which outputs low-dimensional latent variables. The low-dimensional latent variables are then input into the initial logistic regression classifier in the initial diffusion model network, which outputs shadow prediction classification results.

[0023] Based on the difference between the shadow prediction classification result and the shadow region coordinates, the loss function result is calculated, and the gradient of the loss function result is backpropagated to the initial image semantic diffusion encoder and the initial logistic regression classifier for iterative training to generate the diffusion model network.

[0024] In some embodiments, the training process of the identity feature extraction network includes:

[0025] The acquired second face image training set is input into the initial identity feature extraction network to obtain the identity feature vector; the second face image training set carries pre-labeled real category labels;

[0026] Based on the identity feature vector and the weights of multiple classifiers, multiple cosine similarities are obtained; each of the multiple cosine similarities corresponds one-to-one with the true category label.

[0027] For the cosine similarity of the true category label being the correct class, a preset corner margin parameter is added to obtain the corner margin category score; for the cosine similarity of the true category label being the incorrect class, the cosine similarity corresponding to the incorrect class is used as the category score.

[0028] The class assignment probability is calculated based on the corner margin category score corresponding to each correct class, the category score corresponding to each incorrect class, and a preset scaling factor.

[0029] Based on the class attribution probability, the additive angle margin loss is calculated, and the gradient of the additive angle margin loss is backpropagated to the initial identity feature extraction network for iterative training to generate the identity feature extraction network.

[0030] In some embodiments, adjusting the spherical harmonic illumination coefficient based on the face image to be processed and a preset target brightness to obtain an optimized spherical harmonic illumination coefficient includes:

[0031] Based on the face image to be processed, calculate the average brightness of the image;

[0032] Based on the average brightness of the image and the preset target brightness, the lower-order coefficients of the spherical harmonic illumination coefficient are adjusted to obtain optimized lower-order coefficients.

[0033] Based on the average brightness of the image, the higher-order coefficients of the spherical harmonic illumination coefficients are adjusted to obtain optimized higher-order coefficients.

[0034] The optimized spherical harmonic illumination coefficients are obtained based on the optimized low-order coefficients and the optimized high-order coefficients.

[0035] In some embodiments, the step of inputting the features of the normal brightness face image into a preset three-dimensional Gaussian splash reconstruction network to obtain a three-dimensional face model includes:

[0036] Extract the pose parameters of the face image to be processed; generate an initial 3D reconstruction unit based on multiple face images to be processed;

[0037] The pose parameters, the normal brightness face image features, and the initial 3D reconstruction unit are input into a preset 3D Gaussian splash reconstruction network;

[0038] Based on the pose parameters and the features of the normal brightness face image, the initial 3D reconstruction unit is iteratively calculated to obtain the target 3D reconstruction unit;

[0039] The three-dimensional model of the face is obtained based on the target three-dimensional reconstruction unit.

[0040] In some embodiments, after obtaining the target 3D reconstruction unit, the method further includes:

[0041] The target 3D reconstruction unit is projected onto a 2D plane using a microrasterization method to generate a set of relit face images.

[0042] Secondly, embodiments of this application provide a three-dimensional reconstruction apparatus based on an image set, the apparatus comprising:

[0043] The acquisition module is used to acquire multiple face images of the same person from different angles in low-light scenes.

[0044] The feature encoding module is used to input the face image to be processed into a preset encoder architecture to obtain a feature vector; the feature vector includes spherical harmonic illumination coefficients, facial shadow parameters, and non-face illumination feature parameters;

[0045] The parameter optimization module is used to adjust the spherical harmonic illumination coefficient based on the face image to be processed and the preset target brightness to obtain the optimized spherical harmonic illumination coefficient; to lower the facial shadow parameter to a preset value to obtain the optimized facial shadow parameter; and to obtain the optimized feature vector based on the optimized spherical harmonic illumination coefficient, the optimized facial shadow parameter, and the non-face illumination feature parameter.

[0046] The face image feature generation module is used to input the optimized feature vector and the preset random noise image into the preset decoder architecture to generate normal brightness face image features;

[0047] The face 3D reconstruction module is used to input the features of the normal brightness face image into a preset 3D Gaussian splash reconstruction network to obtain a face 3D model.

[0048] Thirdly, embodiments of this application provide a storage medium storing a computer program that, when executed by a processor, implements the image set-based three-dimensional reconstruction method as described in the first aspect above.

[0049] Compared to related technologies, the image set-based 3D reconstruction method, apparatus, and storage medium provided in this application embodiment acquire multiple face images of the same person from different angles in a low-light scene; input the face images to be processed into a preset encoder architecture to obtain feature vectors; the feature vectors include spherical harmonic illumination coefficients, facial shadow parameters, and non-face illumination feature parameters; adjust the spherical harmonic illumination coefficients based on the face images to be processed and a preset target brightness to obtain optimized spherical harmonic illumination coefficients; lower the facial shadow parameters to a preset value to obtain optimized facial shadow parameters; obtain optimized feature vectors based on optimized spherical harmonic illumination coefficients, optimized facial shadow parameters, and non-face illumination feature parameters; input the optimized feature vectors and a preset random noise image into a preset decoder architecture to generate normal brightness face image features; input the normal brightness face image features into a preset 3D Gaussian splash reconstruction network to obtain a 3D face model, thus solving the problem of low quality of 3D face modeling when the input image is multiple face images in dim lighting.

[0050] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0051] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0052] Figure 1 This is a hardware structure block diagram of a terminal for a three-dimensional reconstruction method based on an image set according to an embodiment of this application;

[0053] Figure 2 This is a flowchart of a three-dimensional reconstruction method based on an image set according to an embodiment of this application;

[0054] Figure 3 This is a roadmap of a face image relighting method according to an embodiment of this application;

[0055] Figure 4 This is a roadmap of a method for 3D face reconstruction in low-light scenes according to an embodiment of this application;

[0056] Figure 5 This is a structural block diagram of an image set-based 3D reconstruction apparatus according to an embodiment of this application. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application. Furthermore, it is understood that although the efforts made in such a development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, modifications to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0058] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0059] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application means two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The terms “first,” “second,” “third,” etc., used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0060] The method embodiments provided in this example can be executed on a terminal, computer, or similar computing device. Taking running on a terminal as an example, Figure 1 This is a hardware structure block diagram of a terminal for a three-dimensional reconstruction method based on an image set, according to an embodiment of this application. For example... Figure 1 As shown, a terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. Optionally, the terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the terminal described above. For example, the terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0061] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the image set-based 3D reconstruction method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thus implementing the aforementioned method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0062] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0063] High-fidelity scene reconstruction based on a set of face images captured from specific sequential angles has been a long-standing research topic in computer vision. Neural Radiation Field (NeRF)-based face reconstruction methods are excellent 3D representation techniques, achieving some progress in presenting realistic new views of static face models. However, depicting sufficiently realistic and detailed 3D faces requires dense image input sets, which is costly. NeRF-based reconstruction methods typically consume long training times. Furthermore, NeRF methods have high requirements for image sources, hindering their application in some extreme face reconstruction scenarios, such as those with few samples, low lighting, and high noise. 3D Gaussian Splashing (3DGS) has emerged as a groundbreaking new perspective synthesis technique. Through explicit, point-based scene representation, it makes the rendering process easier to control and optimize, and can be used for 3D scene editing. 3DGS is widely adopted in many extreme scenarios, such as excessively dark scenes, input images with out-of-focus or motion blur, and large-scale 3D scene reconstruction.

[0064] When the source image contains noise or is poorly lit, causing a decline in image quality, the 3D reconstruction results will inevitably exhibit noticeable artifacts. When shooting facial images in dim environments, inconsistent brightness is common. If the brightness of facial images differs from different viewpoints, 3DGS may incorrectly estimate the color attributes of these Gaussian points during optimization. The "dim environment" referred to here is a scene where the face, as a 3D object, has a complex geometry and is often obscured by numerous objects, resulting in insufficient illumination. This type of dim environment is characterized by bright areas near the light source, while areas far from the light source or obscured by it are noticeably dark or shadowed. Due to limitations in the automatic exposure (AE) mechanism of intelligent cameras, photos taken in dim environments are prone to inconsistent brightness between viewpoints. When the camera focuses on a brighter area of ​​the scene, it automatically reduces exposure to preserve highlight details, leading to underexposure in dark areas; conversely, when focusing on a darker area, it increases exposure to improve the visibility of dark details, leading to overexposure in bright areas. This automatic exposure strategy produces significant brightness differences at different viewpoints.

[0065] In the process of 3D facial reconstruction or rendering, challenges arise from extreme scenarios such as low-light conditions, which may cause traditional methods to fail or experience significant performance degradation. These scenarios typically include the following six types:

[0066] (1) Extreme lighting conditions: such as extremely dark lighting, strong backlighting, high dynamic range lighting, or rapidly changing lighting environments.

[0067] (2) Complex occlusion: The face is severely occluded by objects, hair or other elements, resulting in the loss of key facial features.

[0068] (3) Extreme postures: The face is presented at an unconventional angle (such as a side profile close to 90 degrees or an extreme tilt angle), making it difficult to capture facial geometric information.

[0069] (4) Dynamic interference: such as rapid movement, drastic changes in facial expression, or dynamic interference from external objects.

[0070] (5) Low quality input: The input data has extremely low resolution, severe noise, or incomplete sensor data (such as missing depth map).

[0071] (6) Environmental complexity: cluttered backgrounds, reflective surfaces, or scenes with multiple people can cause confusion in depth or color information.

[0072] To address the aforementioned issues, this embodiment provides a 3D reconstruction method based on an image set. Figure 2 This is a flowchart of a three-dimensional reconstruction method based on an image set according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps:

[0073] Step S201: Obtain multiple face images of the same person from different angles in a low-light scene;

[0074] Specifically, acquiring multiple facial images of the same person from different angles in low-light scenes requires, on the one hand, diverse poses (including multiple discrete views taken at 15-degree intervals from -90 degrees to +90 degrees, covering frontal, 30° side profile, 60° side profile, upward angle, downward angle, etc.), diverse lighting conditions (multiple lighting conditions at different locations), and diverse expressions (including neutral, smiling, surprised, angry, etc.) to provide a foundation for acquiring facial images in different poses; on the other hand, the acquired dark... The light images need to cover typical interference situations in real low-light scenes, including inconsistent brightness due to the camera's automatic exposure mechanism (such as underexposure of dark areas due to focusing on bright areas of the face, and overexposure of bright areas due to focusing on dark areas), as well as local dark areas or shadows caused by complex facial geometry (such as nose and brow bone occlusion). This ensures that the multiple facial images to be processed from different angles can not only meet the needs of subsequent 3D Gaussian splash reconstruction for multi-view geometric and texture information, but also provide original data support that fits the real scene for subsequent facial relighting processing.

[0075] Step S202: Input the face image to be processed into the preset encoder architecture to obtain the feature vector; the feature vector includes spherical harmonic illumination coefficients, facial shadow parameters and non-face illumination feature parameters;

[0076] Specifically, when the face image to be processed is input into the preset encoder architecture to extract feature vectors, the encoder architecture needs to integrate a bilateral segmentation network, a face reconstruction network, an identity feature extraction network, and a diffusion model network, and complete feature parsing according to their functional divisions to obtain harmonic illumination coefficients, facial shadow parameters, and non-face illumination feature parameters, providing structured input for subsequent illumination adjustment and relight generation. Among them, the non-face illumination feature parameters include face geometric parameters, projection parameters, background region, and identity feature parameters. These non-face illumination feature parameters need to remain stable in subsequent processing to ensure that the face geometry, identity information, and viewpoint consistency are not affected by illumination adjustment.

[0077] Step S203: Based on the face image to be processed and the preset target brightness, adjust the spherical harmonic illumination coefficient to obtain the optimized spherical harmonic illumination coefficient; lower the facial shadow parameter to the preset value to obtain the optimized facial shadow parameter; and obtain the optimized feature vector based on the optimized spherical harmonic illumination coefficient, the optimized facial shadow parameter, and the non-face illumination feature parameter.

[0078] Specifically, when adjusting the spherical harmonic illumination coefficient based on the face image to be processed and the preset target brightness to obtain the optimized spherical harmonic illumination coefficient, the average brightness is first calculated from the dark face image to be processed. Then, based on the average brightness and the preset target brightness, the low-order enhancement coefficient and the high-order suppression coefficient are calculated. The low-order part of the spherical harmonic illumination coefficient is amplified, and the high-order part is reduced or suppressed. Finally, the optimized low-order coefficient and the high-order coefficient are integrated to obtain the optimized spherical harmonic illumination coefficient.

[0079] When processing facial shadow parameters, they are lowered to a preset value (usually set to 0) to minimize the effect of projected shadows and remove hard shadows, thereby optimizing facial shadow parameters and ensuring that shadow interference is effectively eliminated in the subsequently generated normal brightness face images.

[0080] Then, the optimized spherical harmonic illumination coefficients and facial shadow parameters are integrated with the non-face illumination feature parameters to ensure that the core non-illumination features such as face geometric parameters, identity feature parameters, projection parameters, and background regions remain unchanged, ultimately forming an optimized feature vector that meets the input requirements of the subsequent decoder.

[0081] Step S204: Input the optimized feature vector and the preset random noise image into the preset decoder architecture to generate normal brightness face image features;

[0082] Specifically, when the optimized feature vector and the preset random noise image are input into a preset decoder architecture (such as a conditional denoising diffusion implicit model, DDIM) to generate normal brightness face image features: First, the preset random noise image is an initial noise map conforming to a Gaussian distribution, and its dimension is consistent with the normal brightness face image to be generated, serving as the initial input for the decoding process; Second, the optimized feature vector needs to guide the decoding process as conditional information throughout, where the optimized spherical harmonic illumination coefficient (adjusted through low-order enhancement and high-order suppression) and optimized facial shadow parameters (usually set to 0) are responsible for defining normal illumination conditions, and non-face illumination feature parameters (including...) The facial geometric parameters, projection parameters, background region, and identity feature parameters ensure that the facial geometry, viewpoint, and identity of the generated image are consistent with the original image to be processed, avoiding facial feature distortion caused by lighting adjustments. During the decoding process, the decoder architecture uses a reverse denoising process, combining optimized feature vectors with conditional constraints at each step. At the same time, it uses high-frequency details extracted by the encoder (encoded from the noise map generated from the input dark-light image) to preserve facial texture information. The final output is a facial image feature with natural lighting, clear details, and normal brightness standards, providing a high-quality two-dimensional image input foundation for subsequent 3D Gaussian splash reconstruction.

[0083] Step S205: Input the features of a normal brightness face image into a preset three-dimensional Gaussian splash reconstruction network to obtain a three-dimensional face model.

[0084] Specifically, it is necessary to first extract the pose parameters of each face image in the face image set to be processed. The geometric structure of the three-dimensional scene can be recovered from the two-dimensional image set based on the Camera Localization and Mapping (COLMAP) algorithm, thereby extracting the pose parameters of the person object.

[0085] Furthermore, the geometric structure of the person is determined based on the face images to be processed from different angles in the face image set. The specific methods for calculating the geometric structure include, but are not limited to, extracting feature points from each image in the face image set (or target face image set), matching these feature points between different images, calculating the three-dimensional coordinates of each feature point based on the matching results, and combining the camera parameters (intrinsic and extrinsic parameters) of each image to obtain a preliminary three-dimensional point cloud. This three-dimensional point cloud represents the geometric structure of the aforementioned person. An initial three-dimensional reconstruction unit is calculated based on the geometric structure. This initial three-dimensional reconstruction unit is a three-dimensional Gaussian function reconstruction unit. Specifically, the initial 3D reconstruction unit represents each point or group of points in the 3D point cloud of the face geometry as a 3D Gaussian distribution. This can be achieved by calculating the density distribution of the surrounding point cloud for each point or group of points. For example, a calculation radius can be preset, and then the average position and variance of all points within this radius can be calculated to obtain the parameters of the 3D Gaussian distribution. Many points in the point cloud correspond to a series of 3D Gaussian function reconstruction units. This series of 3D Gaussian function reconstruction units is the aforementioned initial 3D reconstruction unit. The initial 3D reconstruction unit is the basic building block used to represent the face scene, and its essence is a 3D Gaussian function. The initial 3D reconstruction unit is a preliminary result of modeling point cloud data based on the 3D Gaussian distribution. Each point or group of points in the point cloud can be regarded as a Gaussian distribution in 3D space, and each such Gaussian distribution can be regarded as a reconstruction unit. All reconstruction units together constitute the reconstruction of the face scene.

[0086] The obtained pose parameters, normal brightness face image features, and initial 3D reconstruction unit are input into a preset 3D Gaussian splash reconstruction network. Based on the pose parameters and face feature parameters, the trainable parameters in the initial 3D reconstruction unit are iteratively calculated to obtain the target 3D reconstruction unit. Based on the target 3D reconstruction unit, a 3D face model is obtained.

[0087] In the above steps, multi-view low-light images of the same person are collected to provide multi-dimensional raw data support for subsequent modeling. Then, the encoder architecture is used to accurately decouple the spherical harmonic illumination coefficients, facial shadow parameters, and non-facial illumination feature parameters (including facial geometry, identity, camera parameters, background areas, etc.) in the image, achieving effective separation of illumination and non-illumination information. Subsequently, the spherical harmonic illumination coefficients are dynamically optimized based on the brightness of the image to be processed and the preset target brightness (amplifying low-order coefficients to improve overall brightness and suppressing high-order coefficients to reduce hard shadows), and the facial shadow parameters are lowered to a preset value (usually 0) to eliminate shadow interference. At the same time, the optimized parameters and non-illumination features are integrated to form an optimized feature vector. This process ensures that the core structure and identity information of the face remain unchanged. Then, by combining random noise with a decoder architecture, normal brightness face image features with natural lighting and clear details are generated from the optimized feature vectors, solving the problems of excessive image noise and loss of details in low-light environments. Finally, a 3D Gaussian splash reconstruction network is used to process the normal brightness image features, ultimately outputting a geometrically accurate and highly realistic 3D face model. This approach breaks through the limitations of traditional 3D reconstruction methods, which are prone to artifacts and have low modeling reliability in low-light scenes. It expands the application scenarios of the 3D Gaussian splash modeling method, reduces the implementation cost of 3D face reconstruction, and ensures high reliability and high realism of the reconstructed model.

[0088] In some embodiments, the encoder architecture includes a bilateral segmentation network, a face reconstruction network, an identity feature extraction network, and a diffusion model network; non-face illumination feature parameters include face geometric parameters, projection parameters, background region, and identity feature parameters; the face image to be processed is input into the preset encoder architecture to obtain a feature vector, including:

[0089] The face image to be processed is input into the face reconstruction network to obtain the spherical harmonic illumination coefficient, face geometric parameters and projection parameters;

[0090] The face image to be processed is input into a bilateral segmentation network to obtain the background region;

[0091] The face image to be processed is input into the identity feature extraction network to obtain identity feature parameters;

[0092] The face image to be processed is input into a diffusion model network to obtain facial shadow parameters;

[0093] Feature vectors are generated based on spherical harmonic illumination coefficients, facial shadow parameters, facial geometric parameters, projection parameters, background regions, and identity feature parameters.

[0094] Specifically, the encoder architecture integrates four sub-networks: a bilateral segmentation network, a face reconstruction network, an identity feature extraction network, and a diffusion model network. This enables the multi-dimensional feature decomposition of the face image to be processed in low-light scenes, ultimately generating a complete feature vector containing both illumination-related and non-illumination-related features. Among these, the non-face illumination feature parameters specifically cover face geometric parameters, projection parameters, background region, and identity feature parameters.

[0095] Among them, the face reconstruction network is a Detailed Expression Capture and Animation (DECA) network, a method for 3D face reconstruction based on images, used to extract spherical harmonic illumination coefficients, facial geometric parameters, and projection parameters. On one hand, the face reconstruction network predicts spherical harmonic illumination coefficients (9×3 dimension, 9 coefficients per RGB channel) from the image. These coefficients can be further divided into low-order coefficients controlling overall brightness and high-order coefficients controlling directional shadows, providing a basis for subsequent illumination optimization. Combined with the FLAME model, it outputs facial geometric parameters and projection parameters. The facial geometric parameters consist of identity shape parameter β, pose parameter θ, and expression parameter ψ, comprehensively representing the three-dimensional structural features of the face. The projection parameters follow orthogonal projection rules, integrating one scaling parameter and two translation parameters to describe the geometric relationship of the three-dimensional face projected onto the two-dimensional image plane, ensuring the consistency of the viewpoint in subsequent 3D reconstruction.

[0096] Bilateral segmentation networks, such as BiSeNet (a real-time semantic segmentation model), are used to extract background regions. This network employs a bilateral structure combining spatial and contextual paths to perform pixel-level semantic classification of images. Its core task is to distinguish between facial foreground and non-facial background regions. Specifically, it accurately segments foreground regions such as the face, hair, ears, and neck using pre-trained category labels, while simultaneously labeling background pixel regions and generating background region parameters. Notably, hair and hats are considered part of the face, while the background region refers only to environmental areas unrelated to the face. Furthermore, the background region parameters output by this network are used in subsequent processing only to preserve the texture and color of the original image background, without participating in illumination adjustments, thereby improving the overall processing efficiency and accuracy of the network.

[0097] The identity feature extraction network is based on the pre-trained ArcFace model (a face recognition model based on the original additive corner margin loss). By optimizing the additive corner margin loss, the input image is transformed into 512-dimensional identity feature parameters (Id). The residual connection design solves the gradient vanishing problem of deep networks, while the corner margin loss forces the feature vectors to cluster on the hypersphere, making the feature embedding of the same identity more concentrated and the feature embedding of different identities more dispersed. Ultimately, this ensures that the identity feature parameters have high discriminativeness and identity preservation ability, avoiding face identity distortion in subsequent relighting and 3D reconstruction processes.

[0098] The diffusion model network first uses a pre-trained encoder (DiffAE, a type of diffusion autoencoder) to map the image into a low-dimensional latent space and extract a semantic feature vector z containing shadow information. Then, a logistic regression classifier is trained based on z, and combined with multiple manually annotated face images, it outputs a logit value representing the intensity of shadows, i.e., the facial shadow parameter (scalar form). This parameter quantifies the intensity of shadows projected onto the face by facial components (such as the nose and brow bone) or external objects (such as a sun hat), providing a quantitative basis for subsequent shadow suppression.

[0099] After obtaining the six core parameters—spherical harmonic illumination coefficient, facial shadow parameters, face geometric parameters, projection parameters, background region, and identity feature parameters—and following the standard form of feature vector construction, v = (f bg ,l,S,C,I d S ha ), where f bg Let l be the background region parameter, l be the spherical harmonic illumination coefficient, S be the face geometric parameters, C be the projection parameter, and I be the background region parameter. d For identity feature parameters, S ha The parameters (including facial shadow parameters) are structured and integrated to generate a complete feature vector. This vector includes both lighting-related parameters that need further optimization (spherical harmonic illumination coefficients, facial shadow parameters) and non-facial lighting feature parameters that need to be kept stable (facial geometric parameters, projection parameters, background region, identity feature parameters), laying the foundation for the next step of lighting optimization and normal brightness image generation.

[0100] In some embodiments, the diffusion model network includes an image semantic diffusion encoder and a logistic regression classifier; the face image to be processed is input into the diffusion model network to obtain facial shadow parameters, including:

[0101] The face image to be processed is input into the image semantic diffusion encoder to obtain the target low-dimensional latent vector;

[0102] The target low-dimensional latent vector is input into a logistic regression classifier, and the original score of the logistic regression classifier on the decision boundary is extracted as the facial shadow parameter.

[0103] Specifically, the diffusion model network achieves accurate extraction of facial shadow parameters through the collaborative action of an image semantic diffusion encoder and a logistic regression classifier. First, the face image to be processed is input into the image semantic diffusion encoder, which maps the image to a semantically meaningful low-dimensional latent space, generating a target low-dimensional latent vector. This vector does not simply compress image pixel information but focuses on capturing semantic features related to facial shadows (such as shadow location, intensity, and boundary characteristics), while filtering out irrelevant noise interference. Then, the target low-dimensional latent vector is input into the logistic regression classifier (trained from multiple face images with manually labeled shadow region coordinates). The classifier makes judgments based on the shadow semantic features in the vector and extracts the original score (i.e., the logit value) at the decision boundary before activation by the sigmoid function (a non-linear activation function). This score is used as the facial shadow parameter, a continuous scalar that quantifies the degree of shadow projected onto the face by facial components (such as the nose and brow bone) or external objects (such as a sun hat), providing accurate quantitative basis for subsequent shadow suppression optimization.

[0104] In the above steps, by inputting the face image to be processed into the image semantic diffusion encoder, the image can be mapped to a semantically meaningful low-dimensional latent space, generating a target low-dimensional latent vector focusing on semantic features related to facial shadows. Then, this vector is input into the logistic regression classifier, and the original score on its decision boundary that has not been activated by sigmoid (a non-linear activation function) is extracted as the facial shadow parameter. This parameter can accurately quantify the degree of shadows cast on the face by facial components (such as nose and brow bone) or external objects (such as sun hats) in the image to be processed. Finally, it realizes the accurate extraction and quantification of facial shadow information in face images in low-light scenes, providing a reliable parameter basis for targeted adjustments (such as reducing or removing shadows) and optimizing lighting conditions during subsequent relighting, ensuring that the face image is uniformly and realistically lit after relighting, while avoiding the impact of shadow interference on the quality of subsequent 3D Gaussian splash reconstruction.

[0105] In some embodiments, the training process of the diffusion model network includes:

[0106] Obtain the first face image training set; the first face image training set carries the preset shadow region coordinates;

[0107] The first face image training set is input into the initial image semantic diffusion encoder in the initial diffusion model network, which outputs low-dimensional latent variables. The low-dimensional latent variables are then input into the initial logistic regression classifier in the initial diffusion model network, which outputs the shadow prediction classification result.

[0108] Based on the difference between the shadow prediction classification result and the shadow region coordinates, the loss function result is calculated, and the gradient of the loss function result is backpropagated to the initial image semantic diffusion encoder and the initial logistic regression classifier for iterative training to generate the diffusion model network.

[0109] Specifically, during the training process of the diffusion model network, the first step is to obtain a training set of first face images. This dataset must cover samples of different shadow types (such as natural projection shadows formed by the geometric structure of the face itself and unnatural shadows caused by external objects occlusion), different lighting directions, and face poses in low-light scenes. Each image carries the coordinates of the shadow area that has been accurately bounded by manual or semi-automatic tools, which serve as the true labels for subsequent training.

[0110] Next, the training set is input into the initial diffusion model network. The image is first processed by the initial image semantic diffusion encoder (refer to the principle of diffusion autoencoder). This encoder maps the high-dimensional image pixel information to the low-dimensional latent space through a multi-step diffusion and de-diffusion process, generating low-dimensional latent variables that focus on capturing the semantic features of shadows (such as shadow gray-scale distribution and boundary gradient). These low-dimensional latent variables are then input into the initial logistic regression classifier. The classifier outputs the shadow prediction classification results for each region in the image through linear transformation and an adapted activation function (such as the sigmoid function).

[0111] Finally, based on the difference between the shadow prediction classification result and the preset shadow region coordinates, a loss function (such as cross-entropy loss) is used to calculate the loss function result. Then, the gradient of the loss result is propagated to the trainable parameters (such as the convolution kernel weights of the encoder and the linear transformation coefficients of the classifier) ​​of the initial image semantic diffusion encoder and the initial logistic regression classifier through the backpropagation algorithm. Through multiple rounds of iterative training of "forward propagation-loss calculation-backward optimization", the loss value converges to a stable range, and finally a diffusion model network that can accurately extract facial shadow features is generated.

[0112] The training steps of the above diffusion model network provide a reliable basis for reducing or removing hard shadows during the subsequent relighting process of low-light face images, thereby ensuring the authenticity of face images with normal brightness after relighting, avoiding shadow interference from affecting the model quality of subsequent 3D Gaussian splash reconstruction, and ultimately improving the robustness and accuracy of the entire low-light scene face 3D reconstruction scheme.

[0113] In some embodiments, the training process of the identity feature extraction network includes:

[0114] The acquired second face image training set is input into the initial identity feature extraction network to obtain the identity feature vector; the second face image training set carries pre-labeled real category labels;

[0115] Based on the identity feature vector and the weights of multiple classifiers, multiple cosine similarities are obtained; each cosine similarity corresponds one-to-one with the true class label.

[0116] For cosine similarity scores where the true category label is the correct class, a preset corner margin parameter is added to obtain the corner margin category score; for cosine similarity scores where the true category label is the incorrect class, the cosine similarity score corresponding to the incorrect class is used as the category score.

[0117] Based on the corner margin category score corresponding to each of the correct classes, the category score corresponding to each of the incorrect classes, and a preset scaling factor, the category attribution probability is calculated.

[0118] Based on the category attribution probability, the additive angle margin loss is calculated, and the gradient of the additive angle margin loss is backpropagated to the initial identity feature extraction network for iterative training to generate the identity feature extraction network.

[0119] Specifically, the first step is to obtain a second face image training set. This training set can be the same as or different from the first face image training set. It covers samples of different face identities, poses (such as frontal, side, and tilt angles), expressions (such as neutral, smiling, and surprised) and lighting conditions in low-light scenes. Each image carries a pre-labeled real category label, thus providing clear identity supervision for model training.

[0120] Next, the training set is input into an initial identity feature extraction network (such as the ArcFace model architecture, with ResNet as the backbone network). Deep image features are extracted through residual connections and bottleneck structures, outputting an identity feature vector x (such as a 512-dimensional deep embedding feature vector) with identity discriminative capabilities. Subsequently, this identity feature vector x and the weights w_j (corresponding to the j-th class) of multiple classifiers in the network are normalized to unit vectors, i.e., ||x||=1, ||w_j||=1. Then, based on the normalized identity feature vector and the multiple classifier weights, multiple cosine similarities are calculated through vector dot products (each similarity represents the degree of matching between the current feature vector and the corresponding class weight), and these cosine similarities correspond one-to-one with the true class labels of the training set. In the score adjustment stage, a preset additive angle margin parameter m (such as 0.5) is introduced to constrain the cosine similarities for the true class labels that are correct. The calculation method yields the corner distance category score, thereby increasing the angular difference between the correct class and other categories. For the cosine similarity of the true category label as incorrect, it is directly retained as the category score. (j≠y i ).

[0121] Next, the corner margin category scores of all correct classes and the category scores of incorrect classes are amplified using a preset scaling factor s (typically 64). Then, the amplified scores are mapped to category belonging probabilities in the 0-1 interval using the softmax function. This probability directly reflects the likelihood that the current identity feature vector belongs to each identity category. The calculation formula for the softmax function is as follows:

[0122] ;

[0123] Where, p i The final output probability value represents the probability that the current sample belongs to the i-th class. Is it the real category y? i Exponential score after corner margin adjustment; For other categories (j≠y) i The sum of the indexed scores of ) ; s is the scaling factor; The feature vector of the current sample and the true class y i The angle between the corresponding category weight vectors; m is the additive angle margin.

[0124] Finally, the additive angle margin loss (i.e., the negative log-likelihood of the batch average) is calculated based on the class assignment probability and the true class label. The loss function formula is as follows:

[0125] ;

[0126] Where N is the batch size; Is it the real category y? i Exponential score after corner margin adjustment; For other categories (j≠y) i The sum of the indexed scores of ) ; s is the scaling factor; The feature vector of the current sample and the true class y i The angle between the corresponding category weight vectors; m is the additive angle margin.

[0127] The gradient of the loss result is then passed to the trainable parameters of the initial identity feature extraction network (such as the convolutional kernel weights and classifier weights of the backbone network) through the backpropagation algorithm. The parameters are optimized through multiple rounds of iterative training until the loss value converges, and finally an identity feature extraction network with high identity discriminativeness and robustness is generated.

[0128] The training steps of the aforementioned identity feature extraction network enable the trained network to stably output highly robust identity feature vectors. This avoids identity feature distortion caused by low light and pose changes, and provides accurate identity information support for subsequent face relighting and 3D Gaussian splash reconstruction. It ensures that the reconstruction model maintains the consistency of face identity when adjusting lighting conditions, thereby improving the credibility and realism of the entire low-light scene face 3D reconstruction scheme.

[0129] In some embodiments, based on the face image to be processed and a preset target brightness, the spherical harmonic illumination coefficient is adjusted to obtain an optimized spherical harmonic illumination coefficient, including:

[0130] Calculate the average brightness of the face image to be processed;

[0131] Based on the average brightness of the image and the preset target brightness, the lower-order coefficients of the spherical harmonic illumination coefficient are adjusted to obtain the optimized lower-order coefficients.

[0132] Based on the average brightness of the image, the higher-order coefficients of the spherical harmonic illumination coefficient are adjusted to obtain the optimized higher-order coefficients;

[0133] The optimized spherical harmonic illumination coefficient is obtained by optimizing the low-order and high-order coefficients.

[0134] First, the average brightness of the face image to be processed (a face image taken in a low-light scene) needs to be calculated. Specifically, the image pixel values ​​are normalized to the range of [0,1], and the average value of all pixels is calculated to quantify the darkness of the current image (the lower the average brightness, the darker the image), providing an objective basis for subsequent coefficient adjustments.

[0135] Next, the lower-order coefficients of the spherical harmonic illumination coefficients (such as the first 3 orders, controlling the overall brightness and low-frequency illumination components of soft lighting) are adjusted based on the calculated average image brightness and the preset target brightness (usually set to 0.5, which can be adjusted according to actual needs). If the average image brightness is lower than the target brightness, the lower-order enhancement coefficient α is calculated using the formula "target brightness / max(average brightness, 0.01)". α is multiplied by the original lower-order coefficient to achieve dynamic magnification, thereby improving the overall brightness of the face image and simulating natural and soft ambient light. At the same time, the higher-order coefficients of the spherical harmonic illumination coefficients (such as the 4th order and above, controlling the high-frequency illumination components of directionality and shadow boundaries) are adjusted based on the average image brightness. The higher-order suppression coefficient β is calculated using the formula "0.5×(1-average brightness)". β is multiplied by the original higher-order coefficient to achieve dynamic reduction (the lower the average brightness, the stronger the suppression), thereby reducing hard-edge shadows caused by directional lighting.

[0136] Finally, the adjusted optimized low-order coefficients and optimized high-order coefficients are integrated according to the structure of the original spherical harmonic illumination coefficients to obtain the optimized spherical harmonic illumination coefficients. This ensures that the coefficients can both enable the face image after subsequent relighting to reach the target brightness and have uniform illumination, and effectively weaken shadow interference.

[0137] By adjusting the spherical harmonic illumination coefficients as described above to obtain optimized spherical harmonic illumination coefficients, and integrating the optimized low-order and high-order coefficients, the optimized spherical harmonic illumination coefficients can not only ensure that the face image after subsequent relighting reaches the target brightness and has uniform illumination, but also accurately weaken hard shadows in low-light scenes, avoiding interference from uneven illumination on the 3D Gaussian splash reconstruction. This provides key illumination parameter support for generating high-quality, normal-brightness face image features and improving the realism and credibility of the final 3D face model.

[0138] In some embodiments, features of a normal-brightness facial image are input into a preset three-dimensional Gaussian splash reconstruction network to obtain a three-dimensional facial model, including:

[0139] Extract the pose parameters of the face images to be processed; generate initial 3D reconstruction units based on multiple face images to be processed;

[0140] The pose parameters, normal brightness face image features, and initial 3D reconstruction units are input into a preset 3D Gaussian splash reconstruction network;

[0141] The initial 3D reconstruction unit is iteratively calculated based on the pose parameters and features of a face image with normal brightness to obtain the target 3D reconstruction unit.

[0142] A 3D model of the face is obtained based on the target 3D reconstruction unit.

[0143] Specifically, the pose parameters of each face image in the face image set to be processed are first extracted. The geometric structure of the three-dimensional scene can be recovered from the two-dimensional image set based on the Camera Localization and Mapping (COLMAP) algorithm, thereby extracting the pose parameters of the human object.

[0144] Subsequently, the geometric structure of the person is determined based on the face images to be processed from different angles in the face image set. The specific methods for calculating the geometric structure include, but are not limited to, extracting feature points from each image in the face image set (or target face image set), matching these feature points between different images, calculating the three-dimensional coordinates of each feature point based on the matching results, and combining the camera parameters (intrinsic and extrinsic parameters) of each image to obtain a preliminary three-dimensional point cloud. This three-dimensional point cloud represents the geometric structure of the aforementioned person. The initial 3D reconstruction unit is calculated based on the geometric structure. This initial 3D reconstruction unit is a 3D Gaussian function reconstruction unit. Specifically, the initial 3D reconstruction unit represents each point or group of points in the 3D point cloud of the face geometry as a 3D Gaussian distribution. This can be achieved by calculating the density distribution of the surrounding point cloud for each point or group of points. For example, a calculation radius can be preset, and then the average position and variance of all points within this radius can be calculated to obtain the parameters of the 3D Gaussian distribution. Furthermore, many points in the point cloud correspond to a series of 3D Gaussian function reconstruction units. At this time, this series of 3D Gaussian function reconstruction units is the aforementioned initial 3D reconstruction unit. The initial 3D reconstruction unit is the basic building block used to represent the face scene. Its essence is a 3D Gaussian function. In summary, the initial 3D reconstruction unit is a preliminary result of modeling point cloud data based on the 3D Gaussian distribution. Each point or group of points in the point cloud can be regarded as a Gaussian distribution in 3D space. Each such Gaussian distribution can be regarded as a reconstruction unit. All reconstruction units together constitute the reconstruction of the face scene.

[0145] Next, the obtained pose parameters, normal brightness face image features, and initial 3D reconstruction units are input into the 3D reconstruction model. Based on the pose parameters and face feature parameters, the trainable parameters in the initial 3D reconstruction unit are iteratively calculated to obtain the target 3D reconstruction unit. The 3D reconstruction model can be selected by relevant technicians according to actual needs, preferably a multilayer perceptron. The above three parameters are input into the multilayer perceptron, and the trainable parameters in the initial 3D reconstruction unit are iteratively calculated to output a target 3D reconstruction unit conforming to a 3D Gaussian distribution. This target 3D reconstruction unit is an iteratively optimized set of 3D Gaussian distribution parameters, representing the geometric structure and feature distribution of the face object in 3D space. Finally, a 3D face model of the person is established based on the target 3D reconstruction unit. Specifically, normal brightness face image feature parameters represent certain features of a face under normal brightness, such as two-dimensional keypoint coordinates, extracted high-dimensional feature vectors, texture or appearance information; the pose parameters mentioned above are generally camera pose parameters, which characterize the camera projection matrix of each image, including intrinsic parameter matrix K, rotation matrix Ri, and translation vector Ti. By inputting the pose parameters, face feature parameters, and initial 3D reconstruction units into the 3D reconstruction model, the trainable parameters in the initial 3D reconstruction units can be iteratively optimized so that the final generated target 3D reconstruction unit conforms to the two-dimensional observation of the input image, is consistent with the face features, and maintains the mathematical properties of a three-dimensional Gaussian distribution.

[0146] Furthermore, the trainable parameters in the initial 3D reconstruction unit include, but are not limited to, the mean vector μj, which represents the 3D point position of the human object; the covariance matrix ∑j, which represents the distribution shape of the human object; and the weight ωj, a parameter used in calculation for weight updates in the hybrid model. Moreover, the pose parameters provide a projection relationship from 3D to 2D, ensuring that the trainable parameters in the initial 3D reconstruction unit obtained after iterative calculation are consistent with observations from all viewpoints. The facial feature parameters serve as constraints, guiding the adjustment of the trainable parameters in the initial 3D reconstruction unit. For example, if the facial feature parameters are keypoint coordinates, the mean vector μj should be close to the 3D position of these points; if the facial feature parameters are texture information, the covariance matrix ∑j needs to reflect the local geometric features of the human object.

[0147] Through the above steps, pose parameters are used to ensure that the Gaussian volume projection is consistent with the image observation under different viewpoints. Based on normal brightness characteristics, color and texture attributes that are prone to distortion in low light environments are corrected, and finally, a target 3D reconstruction unit that conforms to the three-dimensional Gaussian distribution is generated. The 3D face model constructed based on this target unit not only avoids the geometric holes and texture artifacts caused by low light input in traditional methods, but also achieves high-fidelity rendering through explicit Gaussian volume representation, while maintaining the accuracy of facial identity features and geometric structure. This provides high-quality 3D assets for subsequent applications such as virtual reality and face recognition, and effectively expands the applicability of 3D Gaussian splashing technology in low light scenes.

[0148] In some embodiments, after obtaining the target 3D reconstruction unit, the method further includes:

[0149] The target 3D reconstruction unit is projected onto a 2D plane using a microrasterization method to generate a set of relit face images.

[0150] Specifically, in order to display and observe the three-dimensional model, this embodiment can generate a set of face images based on the above-mentioned three-dimensional face model. That is, the target three-dimensional reconstruction unit is projected onto a two-dimensional plane through microrasterization, thereby generating a set of face images after relighting. Through this embodiment, it is not only convenient for relevant technicians to observe and analyze the model, but also to further reduce the requirements for the set of face images to be processed. Even if the set of face images to be processed is a face image in a low-light scene, this application can also realize the relighting processing of the set of face images to be processed, and finally obtain a set of face images with normal brightness.

[0151] Figure 3 This is a roadmap of the face image relighting method according to an embodiment of this application, such as... Figure 3 As shown, in the encoding stage, key features are extracted through multiple sub-networks: the bilateral segmentation network is responsible for obtaining the background region, the face reconstruction network outputs the spherical harmonic illumination coefficient, face geometric parameters, and projection parameters, the identity feature extraction network generates identity feature parameters, and the diffusion model network obtains facial shadow parameters. Before entering the decoding stage, the spherical harmonic illumination coefficient is adjusted to the optimized spherical harmonic illumination coefficient, and the facial shadow parameters are lowered to a preset value to form optimized facial shadow parameters. Finally, these optimized illumination-related parameters and the unmodified non-face illumination feature parameters (face geometric parameters, projection parameters, background region, and identity feature parameters) participate in the subsequent decoding process to provide support for generating normal brightness face image features.

[0152] Figure 4 This is a roadmap of a method for 3D face reconstruction in low-light scenes according to embodiments of this application, such as... Figure 4As shown, the input set of dark-light face images is first subjected to relighting processing, and then the face scene point cloud is combined to generate the target 3D reconstruction unit. Then, appropriate rendering parameters are set, and finally, it is rendered into a 2D image set.

[0153] This embodiment also provides a three-dimensional reconstruction device based on an image set, which is used to implement the above embodiments and preferred embodiments, and will not be repeated as already described. As used below, the terms "module," "unit," "subunit," etc., can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0154] Figure 5 This is a structural block diagram of an image set-based 3D reconstruction apparatus according to an embodiment of this application, such as... Figure 5 As shown, the device includes:

[0155] The acquisition module 51 is used to acquire multiple face images of the same person from different angles in a low-light scene.

[0156] The feature encoding module 52 is used to input the face image to be processed into a preset encoder architecture to obtain a feature vector; the feature vector includes spherical harmonic illumination coefficients, facial shadow parameters, and non-face illumination feature parameters;

[0157] The parameter optimization module 53 is used to adjust the spherical harmonic illumination coefficient based on the face image to be processed and the preset target brightness to obtain the optimized spherical harmonic illumination coefficient; to lower the facial shadow parameter to a preset value to obtain the optimized facial shadow parameter; and to obtain the optimized feature vector based on the optimized spherical harmonic illumination coefficient, the optimized facial shadow parameter and the non-face illumination feature parameter.

[0158] The face image feature generation module 54 is used to input the optimized feature vector and the preset random noise image into the preset decoder architecture to generate normal brightness face image features.

[0159] The face 3D reconstruction module 55 is used to input the features of the normal brightness face image into a preset 3D Gaussian splash reconstruction network to obtain a face 3D model.

[0160] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination. Specific examples in this embodiment can be found in the examples described in the above embodiments and optional implementations, and will not be repeated in this embodiment.

[0161] Furthermore, in conjunction with the image set-based 3D reconstruction methods in the above embodiments, this application embodiment can provide a storage medium for implementation. This storage medium stores a computer program; when executed by a processor, the computer program implements any of the image set-based 3D reconstruction methods in the above embodiments.

[0162] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0163] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0164] Those skilled in the art should understand that the technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0165] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A three-dimensional reconstruction method based on an image set, characterized in that, include: Acquire multiple facial images of the same person from different angles in a low-light scene; The face image to be processed is input into a preset encoder architecture to obtain a feature vector; The feature vector includes spherical harmonic illumination coefficients, facial shadow parameters, and non-face illumination feature parameters; Based on the face image to be processed and the preset target brightness, the spherical harmonic illumination coefficient is adjusted to obtain the optimized spherical harmonic illumination coefficient; the facial shadow parameter is lowered to a preset value to obtain the optimized facial shadow parameter; and an optimized feature vector is obtained based on the optimized spherical harmonic illumination coefficient, the optimized facial shadow parameter, and the non-face illumination feature parameter. The optimized feature vector and the preset random noise image are input into the preset decoder architecture to generate normal brightness face image features; The features of the normal brightness face image are input into a preset three-dimensional Gaussian splash reconstruction network to obtain a three-dimensional face model.

2. The three-dimensional reconstruction method based on an image set according to claim 1, characterized in that, The encoder architecture includes a bilateral segmentation network, a face reconstruction network, an identity feature extraction network, and a diffusion model network; the non-face illumination feature parameters include face geometric parameters, projection parameters, background region, and identity feature parameters; The process involves inputting the face image to be processed into a preset encoder architecture. The feature vector is obtained, including: The face image to be processed is input into the face reconstruction network to obtain the spherical harmonic illumination coefficient, the face geometric parameters, and the projection parameters; The face image to be processed is input into the bilateral segmentation network to obtain the background region; The face image to be processed is input into the identity feature extraction network to obtain the identity feature parameters; The face image to be processed is input into the diffusion model network to obtain the facial shadow parameters; The feature vector is generated based on the spherical harmonic illumination coefficient, the facial shadow parameters, the facial geometric parameters, the projection parameters, the background region, and the identity feature parameters.

3. The three-dimensional reconstruction method based on an image set according to claim 2, characterized in that, The diffusion model network includes an image semantic diffusion encoder and a logistic regression classifier; the step of inputting the face image to be processed into the diffusion model network to obtain the facial shadow parameters includes: The face image to be processed is input into the image semantic diffusion encoder to obtain the target low-dimensional latent vector; The target low-dimensional latent vector is input into the logistic regression classifier, and the original score of the logistic regression classifier on the decision boundary is extracted as the facial shadow parameter.

4. The three-dimensional reconstruction method based on an image set according to claim 2, characterized in that, The training process of the diffusion model network includes: Obtain a first face image training set; the first face image training set carries preset shadow region coordinates; The first face image training set is input into the initial image semantic diffusion encoder in the initial diffusion model network, which outputs low-dimensional latent variables. The low-dimensional latent variables are then input into the initial logistic regression classifier in the initial diffusion model network, which outputs shadow prediction classification results. Based on the difference between the shadow prediction classification result and the shadow region coordinates, the loss function result is calculated, and the gradient of the loss function result is backpropagated to the initial image semantic diffusion encoder and the initial logistic regression classifier for iterative training to generate the diffusion model network.

5. The three-dimensional reconstruction method based on an image set according to claim 2, characterized in that, The training process of the identity feature extraction network includes: The acquired second face image training set is input into the initial identity feature extraction network to obtain the identity feature vector; the second face image training set carries pre-labeled real category labels; Based on the identity feature vector and the weights of multiple classifiers, multiple cosine similarities are obtained; each of the multiple cosine similarities corresponds one-to-one with the true category label. For the cosine similarity of the true category label being the correct class, a preset corner margin parameter is added to obtain the corner margin category score; for the cosine similarity of the true category label being the incorrect class, the cosine similarity corresponding to the incorrect class is used as the category score. Based on the corner margin category score corresponding to each of the correct classes, the category score corresponding to each of the incorrect classes, and a preset scaling factor, the category attribution probability is calculated. Based on the class attribution probability, the additive angle margin loss is calculated, and the gradient of the additive angle margin loss is backpropagated to the initial identity feature extraction network for iterative training to generate the identity feature extraction network.

6. The three-dimensional reconstruction method based on an image set according to claim 1, characterized in that, The step of adjusting the spherical harmonic illumination coefficient based on the face image to be processed and the preset target brightness to obtain the optimized spherical harmonic illumination coefficient includes: Based on the face image to be processed, calculate the average brightness of the image; Based on the average brightness of the image and the preset target brightness, the lower-order coefficients of the spherical harmonic illumination coefficient are adjusted to obtain optimized lower-order coefficients. Based on the average brightness of the image, the higher-order coefficients of the spherical harmonic illumination coefficients are adjusted to obtain optimized higher-order coefficients. The optimized spherical harmonic illumination coefficients are obtained based on the optimized low-order coefficients and the optimized high-order coefficients.

7. The three-dimensional reconstruction method based on an image set according to claim 1, characterized in that, The step of inputting the features of the normal brightness face image into a preset three-dimensional Gaussian splash reconstruction network to obtain a three-dimensional face model includes: Extract the pose parameters of the face image to be processed; generate an initial 3D reconstruction unit based on multiple face images to be processed; The pose parameters, the normal brightness face image features, and the initial 3D reconstruction unit are input into a preset 3D Gaussian splash reconstruction network; Based on the pose parameters and the features of the normal brightness face image, the initial 3D reconstruction unit is iteratively calculated to obtain the target 3D reconstruction unit; The three-dimensional model of the face is obtained based on the target three-dimensional reconstruction unit.

8. The three-dimensional reconstruction method based on an image set according to claim 7, characterized in that, After obtaining the target 3D reconstruction unit, the method further includes: The target 3D reconstruction unit is projected onto a 2D plane using a microrasterization method to generate a set of relit face images.

9. A three-dimensional reconstruction device based on an image set, characterized in that, include: The acquisition module is used to acquire multiple face images of the same person from different angles in low-light scenes. The feature encoding module is used to input the face image to be processed into a preset encoder architecture to obtain a feature vector; The feature vector includes spherical harmonic illumination coefficients, facial shadow parameters, and non-face illumination feature parameters; The parameter optimization module is used to adjust the spherical harmonic illumination coefficient based on the face image to be processed and the preset target brightness to obtain the optimized spherical harmonic illumination coefficient; to lower the facial shadow parameter to a preset value to obtain the optimized facial shadow parameter; and to obtain the optimized feature vector based on the optimized spherical harmonic illumination coefficient, the optimized facial shadow parameter, and the non-face illumination feature parameter. The face image feature generation module is used to input the optimized feature vector and the preset random noise image into the preset decoder architecture to generate normal brightness face image features; The face 3D reconstruction module is used to input the features of the normal brightness face image into a preset 3D Gaussian splash reconstruction network to obtain a face 3D model.

10. A storage medium, characterized in that, The storage medium stores a computer program, wherein the computer program is configured to execute the three-dimensional reconstruction method based on an image set as described in any one of claims 1 to 8 when it is run.

Images