A three-dimensional reconstruction method, device, equipment and medium based on image set

Abstract

The present application relates to a method, apparatus, device, and medium for three-dimensional reconstruction based on an image set. The method comprises: obtaining a set of facial images to be processed, wherein the set includes multiple facial images of the same person taken from different angles; extracting pose parameters from the set of facial images to be processed, upsampling the set of facial images to obtain a target facial image set, and determining facial feature parameters based on the target facial image set; and calculating an initial three-dimensional reconstruction unit based on the geometric structure of the person determined from the facial images taken from different angles; inputting the pose parameters, facial feature parameters, and the initial three-dimensional reconstruction unit into a three-dimensional reconstruction model to obtain a target three-dimensional reconstruction unit, and establishing a three-dimensional facial model corresponding to the person based on the target three-dimensional reconstruction unit. This method can efficiently complete three-dimensional facial modeling of a person.

Description

Technical Field

[0001] The present application relates to the technical field of computer vision and three-dimensional modeling, and in particular to a three-dimensional reconstruction method, apparatus, device, and medium based on an image set. Background Art

[0002] With the advancement of computer information technology, especially artificial intelligence (AI), AI is increasingly solving traditional problems. For example, generating a 3D model from a set of 2D images has been a research topic in many fields. Due to the importance and richness of facial features, accurately generating a reliable 3D face model from 2D facial images is a worthy research problem. This technology has applications in areas such as 3D facial recognition, film post-production, and gaming.

[0003] In the existing technology, there are two methods for establishing a three-dimensional face model: manual three-dimensional modeling, but this method requires the user to have professional skills, is time-consuming, inefficient, and has unstable modeling results; or three-dimensional modeling is performed using a captured face image, but this method requires a high resolution of the captured face image. If the captured image is blurry, three-dimensional modeling cannot be performed.

[0004] Currently, in the existing technology, if the captured face image set is blurred, no effective solution has been proposed on how to complete the high-quality reconstruction of the three-dimensional face scene. Summary of the Invention

[0005] Based on this, it is necessary to provide a three-dimensional reconstruction method, device, equipment, and medium based on an image set to address the above technical problems.

[0006] In a first aspect, the present application provides a three-dimensional reconstruction method based on an image set. The method comprises:

[0007] Obtaining a set of face images to be processed, wherein the set of face images to be processed includes a plurality of face images to be processed of the same person object at different angles;

[0008] Extracting pose parameters of the face image set to be processed, upsampling the face image set to be processed to obtain a target face image set, determining facial feature parameters based on the target face image set; and calculating an initial 3D reconstruction unit based on the geometric structure of the human object determined from the face images to be processed at different angles;

[0009] The posture parameters, facial feature parameters and initial 3D reconstruction unit are input into the 3D reconstruction model, and the initial 3D reconstruction unit is iteratively calculated based on the posture parameters and facial feature parameters to obtain the target 3D reconstruction unit. A 3D facial model corresponding to the human object is established based on the target 3D reconstruction unit.

[0010] In one embodiment, after obtaining the target 3D reconstruction unit, the method further includes:

[0011] The 3D face model is projected onto a 2D plane through differentiable rasterization to generate a sequence set of deblurred face images.

[0012] In one embodiment, upsampling the face image set to be processed to obtain the target face image set includes:

[0013] Taking each of the to-be-processed face images in the to-be-processed face image set as the first face image in turn;

[0014] Mapping the first face image into latent space variables;

[0015] Inputting the latent space variables and the preset constant tensor into a preset normalization module for normalization processing to obtain a target face image corresponding to the first face image;

[0016] Based on the target face image corresponding to each image in the face image set to be processed, a target face image set is obtained.

[0017] In one embodiment, the normalization module includes multiple processing sub-modules, and the data transmission relationship between the multiple processing sub-modules is one-way sequential transmission. Each processing sub-module includes a normalization sub-module, a convolution module and an upsampling module, wherein the input data of the first normalization sub-module is the latent space variable and the constant tensor, and the input data of all normalization sub-modules except the first normalization sub-module in all processing sub-modules are the output data and latent space variables of the previous module, the input data of the convolution module is the output data of the previous module, and the input data of the upsampling module is the output data of the previous module.

[0018] In one embodiment, the latent space variables and the constant tensor are input into a first normalization submodule for normalization to obtain a first normalization result, including:

[0019] Mapping latent space variables into scaling factors and bias factors;

[0020] The first normalization result is obtained by combining the scaling factor, the deviation factor and the sub-feature map corresponding to the target face image for weighted summation, where the scaling factor represents the contrast and texture detail intensity of the target face image, and the deviation factor represents the tone trend of the target face image.

[0021] In one embodiment, iterative calculation of the initial 3D reconstruction unit based on the posture parameters and the facial feature parameters to obtain the target 3D reconstruction unit includes:

[0022] The initial 3D reconstruction unit is calculated based on the posture parameters and the facial feature parameters to obtain a basic 3D reconstruction unit, and a preliminary deblurred face image set is obtained based on the basic 3D reconstruction unit;

[0023] The face image set after preliminary deblurring is blurred to obtain multiple blurred face images. The loss function results between the blurred face images and the corresponding face images to be processed are calculated, and the gradient of the loss function results is reversely transmitted to the 3D reconstruction model. The initial 3D reconstruction unit is iteratively calculated until the 3D reconstruction model including the initial 3D reconstruction unit converges to generate the target 3D reconstruction unit, wherein the loss function results include pixel-by-pixel loss terms and structural similarity loss terms between the blurred face images and the face images to be processed.

[0024] In one embodiment, obtaining an initial 3D reconstruction unit includes:

[0025] Extract key feature points from the face image set to be processed, and determine the matching relationship between the key feature points of each face image to be processed;

[0026] Based on the matching relationship, the three-dimensional coordinates of each key feature point are determined to obtain an initial three-dimensional point cloud, wherein the initial three-dimensional point cloud represents the geometric structure of the face of the human object;

[0027] An initial 3D reconstruction unit is established based on the initial 3D point cloud.

[0028] In a second aspect, the present application also provides a three-dimensional reconstruction device based on an image set. The device includes:

[0029] An acquisition module is used to acquire a set of face images to be processed, wherein the set of face images to be processed includes multiple face images to be processed of the same person object at different angles;

[0030] A calculation module is used to extract the pose parameters of the face image set to be processed, upsample the face image set to be processed to obtain a target face image set, determine the facial feature parameters based on the target face image set; and calculate an initial 3D reconstruction unit based on the geometric structure of the human object determined by the face images to be processed at different angles;

[0031] A generation module is used to input posture parameters, facial feature parameters and an initial 3D reconstruction unit into a 3D reconstruction model, iteratively calculate the initial 3D reconstruction unit based on the posture parameters and facial feature parameters to obtain a target 3D reconstruction unit, and establish a 3D facial model corresponding to the human object based on the target 3D reconstruction unit.

[0032] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are performed:

[0033] Obtaining a set of face images to be processed, wherein the set of face images to be processed includes a plurality of face images to be processed of the same person object at different angles;

[0034] Extracting pose parameters of the face image set to be processed, upsampling the face image set to be processed to obtain a target face image set, determining facial feature parameters based on the target face image set; and calculating an initial 3D reconstruction unit based on the geometric structure of the human object determined from the face images to be processed at different angles;

[0035] The posture parameters, facial feature parameters and initial 3D reconstruction unit are input into the 3D reconstruction model, and the initial 3D reconstruction unit is iteratively calculated based on the posture parameters and facial feature parameters to obtain the target 3D reconstruction unit. A 3D facial model corresponding to the human object is established based on the target 3D reconstruction unit.

[0036] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the following steps:

[0037] Obtaining a set of face images to be processed, wherein the set of face images to be processed includes a plurality of face images to be processed of the same person object at different angles;

[0038] Extracting pose parameters of the face image set to be processed, upsampling the face image set to be processed to obtain a target face image set, determining facial feature parameters based on the target face image set; and calculating an initial 3D reconstruction unit based on the geometric structure of the human object determined from the face images to be processed at different angles;

[0039] The posture parameters, facial feature parameters and initial 3D reconstruction unit are input into the 3D reconstruction model, and the initial 3D reconstruction unit is iteratively calculated based on the posture parameters and facial feature parameters to obtain the target 3D reconstruction unit. A 3D facial model corresponding to the human object is established based on the target 3D reconstruction unit.

[0040] The above-mentioned three-dimensional reconstruction method, device, equipment, and medium based on an image set first obtains a face image set to be processed, extracts the posture parameters of the face image set to be processed, performs upsampling processing on the face image set to be processed to obtain a target face image set, determines facial feature parameters based on the target face image set, and calculates an initial three-dimensional reconstruction unit based on the geometric structure of the human object determined by the face images to be processed at different angles; finally, inputs the posture parameters, facial feature parameters, and the initial three-dimensional reconstruction unit into a three-dimensional reconstruction model, iteratively calculates the initial three-dimensional reconstruction unit based on the posture parameters and facial feature parameters to obtain a target three-dimensional reconstruction unit, and establishes a three-dimensional face model corresponding to the face object based on the target three-dimensional reconstruction unit. Through the three-dimensional reconstruction method based on face images provided in this application, a face image set to be processed composed of multiple blurred face images can be realized, scene reconstruction can be completed using relatively blurred face images to be processed, and three-dimensional face modeling of human objects can also be completed efficiently. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A diagram showing an application environment of a three-dimensional reconstruction method according to an embodiment;

[0042] Figure 2 1 is a flow chart of a three-dimensional reconstruction method according to an embodiment;

[0043] Figure 3 A schematic diagram of a normalization module structure in one embodiment;

[0044] Figure 4 Schematic diagram of the process of building a three-dimensional face model in a preferred embodiment;

[0045] Figure 5 1 is a schematic diagram of a process for obtaining a deblurred face image set based on a three-dimensional model in one embodiment;

[0046] Figure 6 is a structural block diagram of a three-dimensional reconstruction device in one embodiment;

[0047] Figure 7 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0049] The three-dimensional reconstruction method of the image set provided in the embodiment of the present application can be applied to Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store data that server 104 needs to process. The data storage system can be integrated with server 104, or placed on a cloud or other network server. First, a set of facial images to be processed is obtained, wherein the set of facial images to be processed includes multiple facial images to be processed of the same person from different angles. Then, pose parameters of the set of facial images to be processed are extracted, and the set of facial images to be processed is upsampled to obtain a set of target facial images. Facial feature parameters are determined based on the target facial image set, and the geometric structure of the person is determined based on the facial images to be processed. An initial 3D reconstruction unit is calculated. Finally, the pose parameters, facial feature parameters, and the initial 3D reconstruction unit are input into a fully trained 3D reconstruction model to obtain a target 3D reconstruction unit. A 3D facial model corresponding to the person is established based on the target 3D reconstruction unit. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, Internet of Things devices, and portable wearable devices. Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, etc. The portable wearable device may be a smart watch, a smart bracelet, a head-mounted device, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0050] In one embodiment, Figure 2 As shown, a three-dimensional reconstruction method based on an image set is provided, which is applied to Figure 1 The following steps are used as an example to illustrate the server in the example:

[0051] Step S210 , obtaining a set of face images to be processed, wherein the set of face images to be processed includes a plurality of face images to be processed taken from different angles of the same person object.

[0052] Specifically, the face image set to be processed includes a plurality of face image sets to be processed (15-50 images) shot at 15° intervals for the same person object. ori , wherein the face image to be processed is a relatively blurred image. In practical applications, the face image to be processed may be a blurred image obtained by shooting errors or in special scenes. The judgment of the blurred image includes but is not limited to the judgment based on the clarity of the edges of the graphics in the image. If the edges of the graphics are sharp and the outlines and details of the objects can be clearly distinguished, the image is considered to be relatively clear, otherwise the image is considered to be relatively blurred; or, judgment based on the grayscale changes in the image. If the grayscale changes in the image are obvious and there are rich layered details, the image is considered to be relatively clear, otherwise the image is considered to be relatively blurred, and so on.

[0053] Step S220: extract the posture parameters of the face image set to be processed, upsample the face image set to be processed to obtain a target face image set, determine the facial feature parameters based on the target face image set; and calculate the initial three-dimensional reconstruction unit based on the geometric structure of the human object determined by the face images to be processed at different angles.

[0054] Specifically, pose parameters are extracted from each face image in the set of face images to be processed. The geometric structure of a three-dimensional scene can be restored from the two-dimensional image set based on a Camera Localization and Mapping (COLMAP) algorithm, thereby extracting pose parameters of the human object. Each face image in the set of face images to be processed is then upsampled. In some preferred embodiments, the resolution of the face images to be processed can be increased to 1024 pixels × 1024 pixels, thereby obtaining a high-resolution target face image set. Facial feature parameters are then determined from the target face image set. These facial feature parameters can be determined from the target face image set using deep learning methods or calculated using principal component analysis or other methods. Facial feature parameters include various parameters that can reflect facial information, such as geometric features and texture features.

[0055] Furthermore, the geometric structure of the human object is determined based on the facial images to be processed at different angles in the facial image set. The specific method of calculating the geometric structure includes but is not limited to first extracting feature points from each image in the facial image set to be processed (or the target facial image set), and matching these feature points between different images. The three-dimensional coordinates of each feature point are calculated according to the matching results, and the camera parameters (intrinsic parameters and external parameters) of each image are combined to obtain a preliminary three-dimensional point cloud, which represents the geometric structure of the above-mentioned human object. An initial 3D reconstruction unit is obtained based on the geometric structure calculation. The initial 3D reconstruction unit is a 3D Gaussian function reconstruction unit. Specifically, the initial 3D reconstruction unit characterizes each point or a group of points in the 3D point cloud of the face geometric structure as a 3D Gaussian distribution, which can be achieved by calculating the density distribution of the surrounding point cloud for each point or a group of points. For example, a calculation radius can be preset, and then the average position and variance of all points within this radius can be calculated to obtain the parameters of the 3D Gaussian distribution. Furthermore, many points contained in the point cloud correspond to a series of 3D Gaussian function reconstruction units. At this time, the series of 3D Gaussian function reconstruction units is the above-mentioned initial 3D reconstruction unit. The initial 3D reconstruction unit is the basic building block for representing the face scene. Its essence is a 3D Gaussian function. For any 3D variable, its multidimensional Gaussian distribution probability function can be defined as:

[0056] ;

[0057] In summary, the initial 3D reconstruction unit is the preliminary result of modeling the point cloud data based on the 3D Gaussian distribution. Each point or a group of points in the point cloud can be regarded as a Gaussian distribution in a 3D space. Each such Gaussian distribution can be regarded as a reconstruction unit, and each reconstruction unit together constitutes the reconstruction of the face scene.

[0058] In step S230, the posture parameters, facial feature parameters, and the initial three-dimensional reconstruction unit are input into the three-dimensional reconstruction model, and the initial three-dimensional reconstruction unit is iteratively calculated based on the posture parameters and the facial feature parameters to obtain a target three-dimensional reconstruction unit, and a three-dimensional facial model corresponding to the human object is established based on the target three-dimensional reconstruction unit.

[0059] Specifically, the posture parameters, facial feature parameters and initial three-dimensional reconstruction unit obtained above are input into the three-dimensional reconstruction model, and the trainable parameters in the initial three-dimensional reconstruction unit are iteratively calculated based on the posture parameters and facial feature parameters to obtain the target three-dimensional reconstruction unit, wherein the three-dimensional reconstruction model can be selected by relevant technical personnel according to actual needs, and preferably can be a multi-layer perceptron. The above three parameters are input into the multi-layer perceptron, and the trainable parameters in the initial three-dimensional reconstruction unit are iteratively calculated to output a target three-dimensional reconstruction unit that conforms to the three-dimensional Gaussian distribution. The target three-dimensional reconstruction unit is a set of three-dimensional Gaussian distribution parameters after iterative optimization, which characterizes the geometric structure and feature distribution of the facial object in three-dimensional space. Finally, a three-dimensional facial model of the human object is established based on the target three-dimensional reconstruction unit. Specifically, the facial feature parameters represent certain features of the face, such as two-dimensional key point coordinates, extracted high-dimensional feature vectors, texture or appearance information; the above posture parameters are generally camera posture parameters, which characterize the camera projection matrix of each image, including the intrinsic parameter matrix K, the rotation matrix R i , translation vector T i , the pose parameters, facial feature parameters and the initial 3D reconstruction unit are input into the 3D reconstruction model. The trainable parameters in the initial 3D reconstruction unit can be iteratively optimized so that the final generated target 3D reconstruction unit conforms to the 2D observation of the input image, is consistent with the facial features, and maintains the mathematical properties of the 3D Gaussian distribution.

[0060] Furthermore, the trainable parameters in the initial 3D reconstruction unit include but are not limited to the mean vector μ j, represents the three-dimensional point position of the human object; the covariance matrix ∑j, represents the distribution shape of the human object; the weight ωj, is a parameter in the calculation, used to update the weight in the hybrid model. In addition, the above-mentioned pose parameters are used to provide a projection relationship from three dimensions to two dimensions, ensuring that the trainable parameters in the initial three-dimensional reconstruction unit obtained after iterative calculation are consistent with observations from all perspectives; the above-mentioned facial feature parameters are constraints that can guide the adjustment of the trainable parameters in the initial three-dimensional reconstruction unit. For example, if the above-mentioned facial feature parameters are key point coordinates, then the above-mentioned mean vector μ j It should be close to the three-dimensional positions of these points. If the above-mentioned facial feature parameters are texture information, then the above-mentioned covariance matrix ∑j needs to reflect the local geometric features of the human object.

[0061] Through steps S210 to S230, a 3D facial model of a person can be reconstructed based on a set of 15-50 blurred, low-quality facial images of the same person taken from different angles. Furthermore, the present application performs 3D reconstruction based on pose parameters, facial feature parameters, and an initial 3D reconstruction unit, which is more efficient and conserves computing resources.

[0062] In some embodiments, after obtaining the target 3D reconstruction unit, the method further includes:

[0063] The 3D face model is projected onto a 2D plane through differentiable rasterization to generate a sequence set of deblurred face images.

[0064] Specifically, in order to display the observed three-dimensional model, in this embodiment, a face image sequence set can be generated based on the above-mentioned three-dimensional face model, that is, the target three-dimensional reconstruction unit is projected onto a two-dimensional plane through differentiable rasterization, so as to generate a deblurred face image sequence set. Through this embodiment, it is not only convenient for relevant technical personnel to observe and analyze the model, but also the requirements for the face image set to be processed can be further reduced. Even if the face image set to be processed is a blurred face image, the face image set to be processed can be deblurred through this application, and finally a clear face image sequence set is obtained.

[0065] In some embodiments, upsampling the face image set to be processed to obtain the target face image set includes:

[0066] Taking each of the to-be-processed face images in the to-be-processed face image set as the first face image in turn;

[0067] Mapping the first face image into latent space variables;

[0068] Inputting the latent space variables and the preset constant tensor into a preset normalization module for normalization processing to obtain a target face image corresponding to the first face image;

[0069] Based on the target face image corresponding to each image in the face image set to be processed, a target face image set is obtained.

[0070] Specifically, this embodiment provides a method for upsampling a set of facial images to be processed, taking each facial image to be processed in the set of facial images to be processed as a first facial image in turn, and mapping the first facial image to a latent space variable w. Specifically, the first facial image can be input into a preset upsampling reconstruction module, and passed through multiple (preferably set to 8) fully connected layers, so as to achieve mapping the image information of the first facial image to the latent space variable w.

[0071] Then, the latent space variables and the preset constant tensor are input into the preset normalization module for normalization processing to obtain the target face image corresponding to the first face image, wherein the normalization module includes a normalization submodule and a convolution submodule. The normalization submodule can reduce the deviation of the data distribution, making the model easier to learn and generalize. The convolution submodule can extract features in the image, and generate a high-resolution image through upsampling and further convolution operations. In summary, the first face image is input into the normalization module to obtain the corresponding target face image, and the target face image set is obtained based on the processed target face image corresponding to each image in the face image set to be processed. In summary, this embodiment provides a method for upsampling the face image to be processed, which can improve the resolution of the face image to be processed and facilitate the processing of the face image in subsequent steps.

[0072] In one embodiment, the normalization module includes multiple processing sub-modules, and the data transmission relationship between the multiple processing sub-modules is one-way sequential transmission. Each processing sub-module includes a normalization sub-module, a convolution module and an upsampling module, wherein the input data of the first normalization sub-module is the latent space variable and the constant tensor, and the input data of all normalization sub-modules except the first normalization sub-module in all processing sub-modules are the output data and latent space variables of the previous module, the input data of the convolution module is the output data of the previous module, and the input data of the upsampling module is the output data of the previous module.

[0073] Specifically, Figure 3 FIG is a structural diagram of a normalization module in an embodiment. The normalization module includes multiple processing submodules, and the data transmission relationship between the multiple processing submodules is one-way sequential transmission. Each processing submodule includes a normalization module, a convolution module and an upsampling module. Figure 3In a preferred embodiment as an example, the arrangement order of each module in each processing sub-module is normalization module - convolution module - normalization module - upsampling module - convolution module, wherein the convolution module is preferably a 3×3 convolution module, and multiple processing sub-modules can be set in the normalization module. In practical applications, the number of processing sub-modules to be set can be determined according to actual needs. For example, 9 processing sub-modules can increase the resolution of the facial image to be processed to 1024×1024 pixels.

[0074] The above constant tensor can be set by relevant technical personnel according to actual needs. The introduction of the constant tensor is used to provide the initial feature basis and control the global consistency of the generation process. In some preferred embodiments, the constant tensor can be set to 4×4×512. Figure 3 As can be seen, the input data for the first normalization submodule is the latent space variables and a constant tensor, and the input data for each convolution module is the output data of the previous module. In addition, the input data for all processing submodules, except the first normalization submodule, is the output data of the previous module and the latent space variables. For the nine processing submodules, the initial input first face image is upgraded to resolutions of 4×4 pixels, 8×8 pixels, 16×16 pixels, 32×32 pixels, 64×64 pixels, 128×128 pixels, 256×256 pixels, 512×512 pixels, and 1024×1024 pixels, respectively. These normalization submodules can be configured as AdaIN (Adaptive Instance Normalization) modules.

[0075] In one embodiment, the latent space variables and the constant tensor are input into a first normalization submodule for normalization to obtain a first normalization result, including:

[0076] Mapping latent space variables into scaling factors and bias factors;

[0077] The first normalization result is obtained by combining the scaling factor, the deviation factor and the sub-feature map corresponding to the target face image for weighted summation, where the scaling factor represents the contrast and texture detail intensity of the target face image, and the deviation factor represents the tone trend of the target face image.

[0078] Specifically, each first face image includes i feature maps, and each feature map x i is normalized separately, first mapping the latent space variables to the scaling factor y through a learnable affine transformation s,i and the deviation factor y b,i , and then the scaling factor and the bias factor will be weighted summed with the features of the above feature map. This process is regarded as the normalization of the feature map x through the above normalization submodule. iAfter a normalization process, a first normalization result is obtained. The first normalization result is the output of the normalization submodule, that is, the input of the convolution module in the same processing submodule. The normalization process of the normalization submodule can be expressed as:

[0079] ;

[0080] in, Indicates the calculated x i The mean of Indicates the calculated x i The standard deviation of .

[0081] In one embodiment, iterative calculation of the initial 3D reconstruction unit based on the posture parameters and the facial feature parameters to obtain the target 3D reconstruction unit includes:

[0082] The initial 3D reconstruction unit is calculated based on the posture parameters and the facial feature parameters to obtain a basic 3D reconstruction unit, and a preliminary deblurred face image set is obtained based on the basic 3D reconstruction unit;

[0083] The face image set after preliminary deblurring is blurred to obtain multiple blurred face images. The loss function results between the blurred face images and the corresponding face images to be processed are calculated, and the gradient of the loss function results is reversely transmitted to the 3D reconstruction model. The initial 3D reconstruction unit is iteratively calculated until the 3D reconstruction model including the initial 3D reconstruction unit converges to generate the target 3D reconstruction unit, wherein the loss function results include pixel-by-pixel loss terms and structural similarity loss terms between the blurred face images and the face images to be processed.

[0084] Specifically, this embodiment provides a method for iteratively calculating trainable parameters in a 3D reconstruction unit until a target 3D reconstruction unit is obtained. First, after obtaining a set of facial images to be processed, pose parameters, facial feature parameters, and an initial 3D reconstruction unit are obtained based on the set of facial images to be processed. Then, based on the pose parameters and facial feature parameters, the parameters of the initial 3D reconstruction unit are optimized and calculated to obtain an optimized basic 3D reconstruction unit. After obtaining the basic 3D reconstruction unit, a basic 3D facial model is obtained based on the basic 3D reconstruction unit. The basic 3D facial model is then projected onto a 2D plane using differentiable rasterization to generate the aforementioned set of preliminarily deblurred facial images.

[0085] Each of the preliminarily deblurred face images in the preliminarily deblurred face image set is blurred again based on the motion blur formula, which is as follows:

[0086] ;

[0087] The blurred image here comes from the average value of n corresponding clear images. In the formula, B(x) represents the blurred face image after further blurring, and C l (x) represents the face image after the above preliminary deblurring.

[0088] In summary, multiple blurred face images after re-blurring can be obtained. Calculate the loss function result between the blurred face image and the corresponding face image to be processed, where each blurred face image has a corresponding face image to be processed, the two are shot at the same angle, and both are blurred images. In this embodiment, the loss function consists of a pixel-by-pixel loss term and a structural similarity loss term, that is, the loss function L all have:

[0089] L all =(1-λ)L pixel +λL SSIM , where λ=0.2, L pixel is the pixel-by-pixel loss term, L SSIM is the structural similarity loss term.

[0090] Specifically, the above pixel-by-pixel loss term L pixel It can be expressed as:

[0091] ;

[0092] Where M represents the total number of pixels in each image, B(x) represents the blurred face image, C(x) represents the corresponding face image to be processed, and || || represents the L1 norm.

[0093] The above structural similarity loss term L SSIM It can be expressed as:

[0094] ;

[0095] Among them, q represents the total number of pixel blocks used to compare difference values ​​in each image, B i Represents the pixel block of the blurred face image, C i A pixel block representing the face image to be processed. SSIM (Structural Similarity Index) is a metric used to measure the similarity between a blurred face image and its corresponding face image to be processed. The similarity is typically compared based on several aspects, such as brightness, contrast, and structure.

[0096] The loss function result between the blurred face image and the face image to be processed is calculated by the above method, and the gradient of the loss function result is reversely transmitted to the 3D reconstruction model to iteratively calculate the trainable parameters in the initial 3D reconstruction unit. That is, in the 3D modeling stage, it is necessary to iteratively calculate the above initial 3D reconstruction unit based on the posture parameters, facial feature parameters and the reversely transmitted gradient until the 3D reconstruction unit converges. The specific convergence conditions include but are not limited to: the iteration reaches a preset number of iterations (such as 100 times, 300 times, etc.), the change in the loss function result is less than a preset threshold ε, that is, L (t+1) -L (t) <ε, etc. In some preferred embodiments, the 3D reconstruction model may utilize a multilayer perceptron (MLP). In summary, during the modeling phase, this embodiment iteratively calculates the initial 3D reconstruction unit based on the aforementioned pose parameters and facial feature parameters, ultimately outputting a fully trained target 3D reconstruction unit that conforms to the 3D features of the face.

[0097] In one embodiment, obtaining an initial 3D reconstruction unit includes:

[0098] Extract key feature points from the face image set to be processed, and determine the matching relationship between the key feature points of each face image to be processed;

[0099] Based on the matching relationship, the three-dimensional coordinates of each key feature point are determined to obtain an initial three-dimensional point cloud, wherein the initial three-dimensional point cloud represents the geometric structure of the face of the human object;

[0100] An initial 3D reconstruction unit is established based on the initial 3D point cloud.

[0101] Specifically, a Structure from Motion (SFM) algorithm can be used to extract key feature points from each image in the aforementioned facial image set. Image feature points can be extracted based on the SIFT (Scale-Invariant Feature Transform) key point detection algorithm. These feature points are then matched across different facial images to establish a correspondence between the key feature points in each image. Based on this correspondence, a matching relationship between the key feature points is then derived. The matching relationship represents the matching results between the key feature points in each image.

[0102] Then, based on the matching relationship, the three-dimensional coordinates of each key feature point are calculated, wherein the three-dimensional coordinates of each feature point can be calculated by triangulation. Furthermore, the camera parameters corresponding to each image (including intrinsic and extrinsic parameters) are combined to calculate the initial three-dimensional point cloud, which represents the geometric structure of the face of the human object. Finally, each point or a group of points in the point cloud is modeled as a three-dimensional Gaussian distribution. The many points contained in the point cloud correspond to a series of three-dimensional Gaussian function reconstruction units. At this time, the series of three-dimensional Gaussian function reconstruction units is the above-mentioned initial three-dimensional reconstruction unit.

[0103] The present application also provides a preferred embodiment of a three-dimensional reconstruction method based on a face image set.

[0104] In the modeling phase of the 3D facial model, a set of facial images to be processed is first obtained. Based on the set of facial images to be processed, pose parameters, facial feature parameters, and an initial 3D reconstruction unit are obtained. The parameters of the initial 3D reconstruction unit are then optimized and calculated based on the pose parameters and facial feature parameters to obtain a basic 3D reconstruction unit. After obtaining the basic 3D reconstruction unit, a basic 3D facial model is obtained based on the basic 3D reconstruction unit. The basic 3D facial model is projected onto a two-dimensional plane to obtain a set of preliminarily deblurred facial images. Each preliminarily deblurred facial image in the preliminarily deblurred facial image set is then reblurred to obtain a set of reblurred blurred facial images. The loss function is then calculated between the blurred facial images and the corresponding preliminarily processed facial images. Based on the loss function result, the trainable parameters in the initial 3D reconstruction unit are iteratively trained until the initial 3D reconstruction unit converges, thereby obtaining the target 3D reconstruction unit. Figure 4 This is the modeling stage of the three-dimensional face model in an embodiment. It should be noted that this embodiment mainly trains the trainable parameters in the initial three-dimensional reconstruction unit. Figure 4 The other parts passed by the red arrow do not contain trainable parameters and only serve to propagate gradients.

[0105] After the 3D face model is built, Figure 5 As shown, Figure 5 1. The 3D face image sequence is a flowchart of obtaining a two-dimensional deblurred face image sequence set based on a three-dimensional face model in one embodiment. After obtaining the above-mentioned three-dimensional face model, in order to display the observed 3D model, the target three-dimensional reconstruction unit can be projected onto a two-dimensional plane through a differentiable rasterization process to generate a clear face image sequence set. In summary, it can be seen that the present application can not only achieve efficient three-dimensional scene reconstruction, but also achieve deblurring of multiple blurred face images to obtain a clear face image set, i.e., a deblurred face image set I. Out .

[0106] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0107] Based on the same inventive concept, embodiments of the present application also provide a 3D facial image reconstruction device for implementing the aforementioned 3D facial image reconstruction method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of the embodiments of the 3D facial image reconstruction device provided below can be found in the above-mentioned limitations of the 3D facial image reconstruction method and will not be further elaborated here.

[0108] In one embodiment, Figure 6 As shown, a three-dimensional reconstruction device based on an image set is provided, comprising: an acquisition module 61, a calculation module 62 and a generation module 63, wherein:

[0109] An acquisition module 61 is configured to acquire a set of face images to be processed, wherein the set of face images to be processed includes a plurality of face images to be processed of the same person object at different angles;

[0110] A calculation module 62 is configured to extract pose parameters of a to-be-processed facial image set, perform upsampling on the to-be-processed facial image set to obtain a target facial image set, determine facial feature parameters based on the target facial image set, and calculate an initial 3D reconstruction unit based on the geometric structure of the human object determined from the to-be-processed facial images at different angles.

[0111] A generation module 63 is used to input the posture parameters, facial feature parameters and the initial three-dimensional reconstruction unit into a fully trained three-dimensional reconstruction model, iteratively calculate the initial three-dimensional reconstruction unit based on the posture parameters and facial feature parameters to obtain a target three-dimensional reconstruction unit, and establish a three-dimensional facial model corresponding to the human object based on the target three-dimensional reconstruction unit.

[0112] Each module in the apparatus for 3D reconstruction of an image set described above may be implemented in whole or in part through software, hardware, or a combination thereof. Each module may be embedded in or independent of a processor in a computer device in the form of hardware, or may be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0113] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 7 As shown. The computer device includes a processor, a memory and a network interface connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data related to the three-dimensional reconstruction of facial images. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a three-dimensional reconstruction method based on a set of facial images is implemented.

[0114] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0115] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0116] Obtaining a set of face images to be processed, wherein the set of face images to be processed includes a plurality of face images to be processed of the same person object at different angles;

[0117] Extracting pose parameters of the face image set to be processed, upsampling the face image set to be processed to obtain a target face image set, determining facial feature parameters based on the target face image set; and calculating an initial 3D reconstruction unit based on the geometric structure of the human object determined from the face images to be processed at different angles;

[0118] The posture parameters, facial feature parameters and initial 3D reconstruction unit are input into the 3D reconstruction model, and the initial 3D reconstruction unit is iteratively calculated based on the posture parameters and facial feature parameters to obtain the target 3D reconstruction unit. A 3D facial model corresponding to the human object is established based on the target 3D reconstruction unit.

[0119] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0120] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processors (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), data processing logic devices based on quantum computing, and the like.

[0121] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0122] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A three-dimensional reconstruction method based on an image set, characterized in that: The method comprises: Acquire a set of face images to be processed, wherein the set of face images to be processed includes a plurality of face images to be processed taken from different angles of the same person object; Extracting pose parameters of the face image set to be processed, upsampling the face image set to be processed to obtain a target face image set, determining facial feature parameters based on the target face image set; and calculating an initial three-dimensional reconstruction unit based on the geometric structure of the human object determined from the face images to be processed at different angles; Inputting the posture parameters, facial feature parameters, and the initial 3D reconstruction unit into a 3D reconstruction model, iteratively calculating the initial 3D reconstruction unit based on the posture parameters and the facial feature parameters to obtain a target 3D reconstruction unit, and establishing a 3D face model corresponding to the human object based on the target 3D reconstruction unit, wherein the iteratively calculating the initial 3D reconstruction unit based on the posture parameters and the facial feature parameters to obtain the target 3D reconstruction unit includes: The initial 3D reconstruction unit is calculated based on the posture parameters and the facial feature parameters to obtain a basic 3D reconstruction unit, and a preliminary deblurred facial image set is obtained based on the basic 3D reconstruction unit; the preliminary deblurred facial image set is blurred to obtain multiple blurred facial images, and the loss function results between the blurred facial images and the one-to-one corresponding facial images to be processed are calculated, and the gradient of the loss function results is reversely transmitted to the 3D reconstruction model, and the initial 3D reconstruction unit is iteratively calculated until the 3D reconstruction model including the initial 3D reconstruction unit converges to generate the target 3D reconstruction unit, wherein the loss function result includes a pixel-by-pixel loss term and a structural similarity loss term between the blurred facial image and the facial image to be processed.

2. The method according to claim 1, characterized in that After obtaining the target 3D reconstruction unit, the method further includes: The three-dimensional face model is projected onto a two-dimensional plane through differentiable rasterization to generate a deblurred face image sequence set.

3. The method according to claim 1, characterized in that The upsampling of the face image set to be processed to obtain a target face image set includes: sequentially taking each of the to-be-processed facial images in the to-be-processed facial image set as a first facial image; Mapping the first face image into a latent space variable; Inputting the latent space variables and the preset constant tensor into a preset normalization module for normalization processing to obtain a target facial image corresponding to the first facial image; The target face image set is obtained based on the target face image corresponding to each image in the face image set to be processed.

4. The method according to claim 3, characterized in that The normalization module includes multiple processing sub-modules, and the data transmission relationship between the multiple processing sub-modules is one-way sequential transmission. Each of the processing sub-modules includes a normalization sub-module, a convolution module and an upsampling module, wherein the input data of the first normalization sub-module is the latent space variable and the constant tensor, and the input data of all normalization sub-modules except the first normalization sub-module in all the processing sub-modules are the output data of the previous module and the latent space variable, the input data of the convolution module is the output data of the previous module, and the input data of the upsampling module is the output data of the previous module.

5. The method according to claim 4, characterized in that The step of inputting the latent space variables and the constant tensor into a first normalization submodule for normalization to obtain a first normalization result includes: Mapping the latent space variables into scaling factors and bias factors; The first normalization result is obtained by combining the scaling factor, the deviation factor and the sub-feature map corresponding to the target facial image and performing weighted summation, wherein the scaling factor represents the contrast and texture detail intensity of the target facial image, and the deviation factor represents the tone trend of the target facial image.

6. The method according to claim 1, characterized in that Acquiring the initial three-dimensional reconstruction unit includes: Extracting key feature points from the set of face images to be processed, and determining matching relationships between the key feature points between the face images to be processed; Based on the matching relationship, determining the three-dimensional coordinates of each key feature point to obtain an initial three-dimensional point cloud, wherein the initial three-dimensional point cloud represents the geometric structure of the face of the human object; The initial three-dimensional reconstruction unit is established based on the initial three-dimensional point cloud.

7. A three-dimensional reconstruction device based on an image set, characterized in that: The device comprises: An acquisition module is used to acquire a set of face images to be processed, wherein the set of face images to be processed includes a plurality of face images to be processed of the same person object at different angles; a calculation module, configured to extract pose parameters of the face image set to be processed, perform upsampling on the face image set to obtain a target face image set, determine facial feature parameters based on the target face image set; and calculate an initial 3D reconstruction unit based on the geometric structure of the human object determined from the face images to be processed at different angles; A generation module is configured to input the posture parameters, facial feature parameters, and the initial 3D reconstruction unit into a 3D reconstruction model, perform iterative calculations on the initial 3D reconstruction unit based on the posture parameters and the facial feature parameters to obtain a target 3D reconstruction unit, and establish a 3D face model corresponding to the human object based on the target 3D reconstruction unit, wherein the iterative calculations on the initial 3D reconstruction unit based on the posture parameters and the facial feature parameters to obtain the target 3D reconstruction unit include: The initial 3D reconstruction unit is calculated based on the posture parameters and the facial feature parameters to obtain a basic 3D reconstruction unit, and a preliminary deblurred facial image set is obtained based on the basic 3D reconstruction unit; the preliminary deblurred facial image set is blurred to obtain multiple blurred facial images, and the loss function results between the blurred facial images and the one-to-one corresponding facial images to be processed are calculated, and the gradient of the loss function results is reversely transmitted to the 3D reconstruction model, and the initial 3D reconstruction unit is iteratively calculated until the 3D reconstruction model including the initial 3D reconstruction unit converges to generate the target 3D reconstruction unit, wherein the loss function result includes a pixel-by-pixel loss term and a structural similarity loss term between the blurred facial image and the facial image to be processed.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images