A method, system, device, and medium for multi-view face surface reconstruction

Abstract

The application discloses a multi-view face surface reconstruction method, system, device and medium, and relates to the fields of computer vision and three-dimensional reconstruction; through segmentation and scaling processing on obtained multiple face images at t time, a main-view scaling image at t time and multiple target scaling images are obtained; a deformable convolution is used, the main-view scaling image and each target scaling image are matched respectively, a corresponding cosine angle is calculated, and selection of a set interval is performed, and finally an optimized offset is obtained; then, multiple matching coordinates at t time are obtained according to three-dimensional coordinates determined by a camera external parameter at t time; after a face point cloud at t time is finally determined, a three-dimensional face surface at t time is constructed; the three-dimensional reconstruction is performed after the multi-view image matching, and information is truly restored.

Description

Technical Field

[0001] This invention relates to the field of computer vision and 3D reconstruction, and in particular to a method, system, device and medium for multi-view face surface reconstruction. Background Technology

[0002] 3D face reconstruction has always been a popular research area in computer vision. Especially with the development of AR / VR, gaming, security, and other fields, the demand for 3D face models is increasing.

[0003] Traditional 3D face reconstruction methods are mostly based on parametric face models. They typically use one or more input face images and employ optimization or deep learning to regress the geometric and expression parameters of the parametric face model, thereby obtaining the corresponding 3D face model.

[0004] However, these methods are affected by the model and the data, making it difficult to reconstruct high-precision 3D face models. Firstly, the parametric face model is obtained by performing PCA dimensionality reduction on the large-scale 3D face data collected. This statistically based model often has limited expressive power, performing poorly on high-frequency information such as wrinkles and exaggerated expressions. Secondly, to regress the parameters of the face model, most methods use facial landmark information. However, facial landmark annotation is prone to significant errors, especially in cases of significant facial occlusion. Large annotation errors affect the final calculation results, causing distortion in the reconstructed 3D face. Summary of the Invention

[0005] The purpose of this invention is to provide a method, system, device and medium for multi-view face surface reconstruction, which performs three-dimensional reconstruction after multi-view image matching to realistically restore information.

[0006] To achieve the above objectives, the present invention provides the following solution:

[0007] A method for multi-view face surface reconstruction, the method comprising:

[0008] Multiple face images are acquired at time t; the multiple face images are obtained by taking pictures of the same target person with a camera from different perspectives; wherein, the image obtained by the camera from the main perspective is the main perspective face image; the images other than the main perspective face image among the multiple face images are the target face images; the main perspective is the perspective corresponding to the minimum rotation angle of the camera along a set axis; t≥1;

[0009] The main view face image at time t is segmented and scaled to obtain the main view scaled image at time t. The target face images at time t are segmented and scaled separately to obtain multiple target scaled images at time t.

[0010] Deformable convolution is used to match the main view scaled image at time t with the target scaled images at time t respectively, to obtain multiple initial offsets at time t.

[0011] The cosine angle is calculated based on the initial offsets at time t; the cosine angle is used to characterize the feature similarity between the pixels of the main view zoomed image and the corresponding pixels of the target zoomed image.

[0012] The 3D coordinates of each pixel in the main view scaled image at time t are determined based on the camera extrinsic parameters at time t; when t > 1, the camera extrinsic parameters at time t are determined based on the face point cloud at time t-1 and the optimized offset at time t-1; when t = 1, the camera extrinsic parameters at time t are determined based on multiple face images at time t.

[0013] Select a set interval for multiple cosine angles at time t to obtain the preferred angle at time t, and determine the initial offset corresponding to the preferred angle at time t as the optimized offset at time t.

[0014] Based on the optimized offset at time t, the three-dimensional coordinates are projected onto the scaled images of each target at time t to obtain multiple matching coordinates at time t.

[0015] The face point cloud at time t is determined based on the three-dimensional coordinates and multiple matching coordinates at time t.

[0016] Based on the face point cloud at time t, the 3D surface of the face at time t is constructed using the isosurface extraction method.

[0017] Optionally, when t > 1, the method for determining the camera extrinsic parameters at time t specifically includes:

[0018] Based on the face point cloud at time t-1 and the optimized offset at time t-1, the viewpoint deformation of each target scaling image at time t-1 is performed so that each target scaling image at time t-1 is projected onto the main viewpoint scaling image at time t-1, thus obtaining the reprojection matching image at time t-1.

[0019] The camera extrinsic parameters at time t are determined based on the reprojection matching coordinates in the reprojection matching image at time t-1 and the three-dimensional coordinates at time t-1.

[0020] Optionally, the main view face image at time t is segmented and scaled to obtain a scaled main view image at time t. Then, each target face image at time t is segmented and scaled separately to obtain multiple scaled target images at time t. Specifically, this includes:

[0021] Background segmentation is performed on the front-view face image at time t to obtain the segmented front-view face image at time t;

[0022] Background segmentation is performed on each target face image at time t to obtain the segmented target face images at time t;

[0023] The face regions in the segmented main view face image at time t are identified to obtain the main view scaled image at time t. The face regions in each target face image segmented at time t are also identified to obtain multiple target scaled images at time t.

[0024] A system for multi-view face surface reconstruction, the system comprising:

[0025] An image acquisition module is used to acquire multiple face images at time t; the multiple face images are obtained by taking pictures of the same target person from different perspectives using a camera; wherein, the image obtained by the camera from the main perspective is the main perspective face image; the images other than the main perspective face image among the multiple face images are the target face images; the main perspective is the perspective corresponding to the minimum rotation angle of the camera along a set axis; t≥1;

[0026] The processing module is used to segment and scale the main view face image at time t to obtain the main view scaled image at time t, and to segment and scale each target face image at time t to obtain multiple target scaled images at time t.

[0027] The matching module is used to match the main view scaled image at time t with each target scaled image at time t using deformable convolution to obtain multiple initial offsets at time t.

[0028] The calculation module is used to calculate the corresponding cosine angle based on each initial offset at time t; the cosine angle is used to characterize the feature similarity between the pixels of the main view zoomed image and the corresponding target zoomed image.

[0029] The 3D coordinate determination module is used to determine the 3D coordinates of each pixel in the main view scaled image at time t based on the camera extrinsic parameters at time t. When t > 1, the camera extrinsic parameters at time t are determined based on the face point cloud at time t-1 and the optimized offset at time t-1. When t = 1, the camera extrinsic parameters at time t are determined based on multiple face images at time t.

[0030] The optimization module is used to select a set interval for multiple cosine angles at time t, obtain the preferred angle at time t, and determine the initial offset corresponding to the preferred angle at time t as the optimized offset at time t.

[0031] The matching coordinate determination module is used to project the three-dimensional coordinates onto the target scaled images at time t based on the optimized offset at time t, so as to obtain multiple matching coordinates at time t.

[0032] The point cloud determination module is used to determine the face point cloud at time t based on the three-dimensional coordinates and multiple matching coordinates at time t.

[0033] The reconstruction module is used to construct the 3D surface of the face at time t based on the face point cloud at time t using the isosurface extraction method.

[0034] Optionally, when t > 1, the three-dimensional coordinate determination module includes:

[0035] The reprojection matching submodule is used to perform viewpoint deformation on each target scaled image at time t-1 based on the face point cloud at time t-1 and the optimized offset at time t-1, so that each target scaled image at time t-1 is projected onto the main view scaled image at time t-1, thus obtaining the reprojection matching image at time t-1.

[0036] The extrinsic parameter determination submodule is used to determine the camera extrinsic parameters at time t based on the reprojection matching coordinates in the reprojection matching image at time t-1 and the three-dimensional coordinates at time t-1.

[0037] The coordinate determination submodule is used to determine the three-dimensional coordinates of each pixel in the main view scaled image at time t based on the camera extrinsic parameters at time t.

[0038] Optionally, the processing module includes:

[0039] The first segmentation submodule is used to perform background segmentation on the main view face image at time t to obtain the segmented main view face image at time t.

[0040] The second segmentation submodule is used to perform background segmentation on each target face image at time t, so as to obtain the segmented target face images at time t.

[0041] The recognition submodule is used to recognize the face regions in the segmented main view face image at time t to obtain the main view zoomed image at time t, and to recognize the face regions in each target face image segmented at time t to obtain multiple target zoomed images at time t.

[0042] An electronic device includes a memory and a processor, the memory storing a computer program, and the processor running the computer program to enable the electronic device to perform the multi-view face surface reconstruction method described above.

[0043] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the multi-view face surface reconstruction method described above.

[0044] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0045] This invention provides a method, system, device, and medium for multi-view face surface reconstruction. It involves segmenting and scaling multiple 3D face images acquired at time t to obtain a main-view scaled image and multiple target scaled images at time t. Using deformable convolution, the main-view scaled image and each target scaled image are matched separately, and the corresponding cosine angles are calculated and a set interval is selected to obtain an optimized offset. Then, based on the 3D coordinates determined by the camera extrinsic parameters at time t, multiple matching coordinates at time t are obtained. Finally, after determining the face point cloud at time t, the 3D face surface at time t is constructed using isosurface extraction. This invention performs 3D reconstruction after multi-view image matching, eliminating reliance on face parameters for model construction, avoiding face distortion problems, and accurately restoring face information. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 A flowchart of a multi-view face surface reconstruction method provided in an embodiment of the present invention;

[0048] Figure 2 This is an actual flowchart of the multi-view face surface reconstruction method provided in an embodiment of the present invention;

[0049] Figure 3 This is a schematic diagram of the multi-view face surface reconstruction process provided in an embodiment of the present invention;

[0050] Figure 4 This is a multi-view matching structure diagram provided in an embodiment of the present invention;

[0051] Figure 5 An optimized 3D point cloud structure diagram with external camera involvement provided in an embodiment of the present invention. Detailed Implementation

[0052] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0053] The purpose of this invention is to provide a method, system, device and medium for multi-view face surface reconstruction, which performs three-dimensional reconstruction after multi-view image matching to realistically restore information.

[0054] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0055] Example 1

[0056] like Figure 1 As shown, this embodiment of the invention provides a method for multi-view face surface reconstruction, the method comprising:

[0057] Step 100: Obtain multiple face images at time t; these multiple face images are obtained by taking pictures of the same target person from different perspectives using a camera; among them, the image taken by the camera from the main perspective is the main perspective face image; the images other than the main perspective face image are the target face images; the main perspective is the perspective corresponding to the minimum rotation angle of the camera along a set axis; t≥1. Where t is an integer.

[0058] Step 200: Perform segmentation and scaling on the main view face image at time t to obtain the main view scaled image at time t. Perform segmentation and scaling on each target face image at time t to obtain multiple target scaled images at time t.

[0059] Specifically, background segmentation is performed on the main view face image at time t to obtain the segmented main view face image at time t.

[0060] Background segmentation is performed on each target face image at time t to obtain the segmented target face images at time t.

[0061] The face regions in the segmented main view face image at time t are identified to obtain the main view scaled image at time t. The face regions in each target face image segmented at time t are also identified to obtain multiple target scaled images at time t.

[0062] In practical applications, the acquired images can be segmented into background and face using a real-time semantic segmentation model (BiseNet), and the face region can be scaled and shifted to the center of the image.

[0063] Each viewpoint is input into a 3D DFA network to estimate the camera's extrinsic parameters. The viewpoint with the smallest rotation angle around the y-axis is selected as the primary viewpoint, and the other viewpoints are selected as the target viewpoints.

[0064] Multi-view color images of faces are captured using a camera. After preprocessing, the BiseNet network is used to extract the face region from the background, and the area occupied by the face is shifted to the center of the image and scaled to occupy the total image area. The size of the camera's extrinsic parameters is determined. A 3D DFA network is used to estimate the pose offset of the face from each viewpoint, serving as the initial values ​​for the camera's extrinsic parameters. The viewpoint with the smallest rotation angle around the y-axis is selected as the primary viewpoint I0, and the other viewpoints are designated as target viewpoints I1, I2, I3, etc.

[0065] Step 300: Using deformable convolution, the main view zoomed image at time t is matched with each target zoomed image at time t to obtain multiple initial offsets at time t.

[0066] Step 400: Calculate the corresponding cosine angle based on each initial offset at time t; the cosine angle is used to characterize the feature similarity between the pixels of the main view zoomed image and the corresponding pixels of the target zoomed image.

[0067] Step 500: Determine the 3D coordinates of each pixel in the main-view scaled image at time t based on the camera extrinsic parameters at time t; when t > 1, the camera extrinsic parameters at time t are determined based on the face point cloud at time t-1 and the optimized offset at time t-1. When t = 1, the camera extrinsic parameters at time t are determined based on multiple face images at time t.

[0068] When t > 1, the specific methods for determining the camera extrinsic parameters at time t include:

[0069] Based on the face point cloud at time t-1 and the optimized offset at time t-1, the viewpoint deformation of each target scaling image at time t-1 is performed so that each target scaling image at time t-1 is projected onto the main viewpoint scaling image at time t-1, thus obtaining the reprojection matching image at time t-1.

[0070] The camera extrinsic parameters at time t are determined based on the reprojection matching coordinates in the reprojection matching image at time t-1 and the three-dimensional coordinates at time t-1.

[0071] Specifically, the 9x9 neighborhood space centered on the coarse matching point from the main viewpoint to the target viewpoint can be called the search space. The features of the main viewpoint are multiplied by the corresponding features in the search space to calculate the correlation between features across multiple viewpoints. The result is then input into DenseNet to obtain a fine-grained match between the main viewpoint and the target viewpoint.

[0072] In practical applications, the initial extrinsic parameters of the camera are used to calculate the 3D point coordinates based on the matching results between the main viewpoint and other viewpoints. The 3D point coordinates are then reprojected onto the target viewpoint using the camera extrinsic parameters to obtain a reprojection match. Features are extracted from the target viewpoint using both the reprojection match and refined matching results. These are then input into the ResNet neural network module to update the 3D point coordinates and camera extrinsic parameters.

[0073] Step 600: Select a set interval for the multiple cosine angles at time t to obtain the preferred angle at time t, and determine the initial offset corresponding to the preferred angle at time t as the optimized offset at time t.

[0074] Specifically, in practical applications, the PWC-Net model calculates the matching relationship between the main viewpoint and the target viewpoint, serving as the initial value for matching, i.e., the initial offset. Deformable convolutions are then used in conjunction with this initial matching relationship to optimize the matching result between the main viewpoint and the target viewpoint. When calculating the target viewpoint features, deformable convolutions are employed, using the matching result between viewpoints as the initial value for the deformable convolution offset. The convolution module then processes the viewpoint features to further optimize the offset.

[0075] The cosine angle between the features of the main viewpoint and the target viewpoint is calculated as a measure of feature similarity. The feature similarity is input into a densely connected convolutional network (DenseNet) module to calculate the refined matching result. When extracting features from the main viewpoint image, the offset of the deformable convolution is fixed at 0. For the target viewpoint image, a deformable convolution with an initial offset is used; the specific calculation method for the initial offset is as follows: using the matching point from the target viewpoint to the main viewpoint as the center, find a 3*3 neighborhood pixel in the main viewpoint, then find the matching point of the neighborhood pixel in the main viewpoint in the target viewpoint and use it as the initial offset of the deformable convolution. The main viewpoint features and target viewpoint features from the previous layer are input into the CNN convolutional layer, and the updated offset is output. The updated offset is added to the initial offset to obtain the final offset value of the deformable convolution.

[0076] Step 700: Project the 3D coordinates onto the scaled images of each target at time t based on the optimized offset at time t, to obtain multiple matching coordinates at time t.

[0077] Step 800: Determine the face point cloud at time t based on the three-dimensional coordinates and multiple matching coordinates at time t.

[0078] Step 900: Based on the face point cloud at time t, construct the 3D surface of the face at time t using the isosurface extraction method.

[0079] In other words, the above operating steps can also be Figure 2 The process is as follows: First, acquire multi-view face images and preprocess them; then, optimize the matching from the main viewpoint to the target viewpoint based on deformable convolution and coarse matching; calculate the 3D point cloud of the face and reproject it to the target viewpoint using camera extrinsic parameters to obtain reprojection matching; update the camera extrinsic parameters and face point cloud with the optimization objective of minimizing the gap between fine matching and reprojection matching; iterate continuously for 3 iterations, constructing the 3D surface of the human body based on the marching-cubes algorithm; and render and visualize the reconstructed 3D face surface and texture from any viewpoint.

[0080] Specifically, the main viewpoint I0 is paired one-to-one with target viewpoints I1, I2, I3, etc. PwcNet is used to calculate the coarse matching relationships between them. The coarse matching results are then used with deformable convolutions to adjust the feature extraction method of the target viewpoints. Finally, the features of both the main viewpoint and the target viewpoints are input into the matching network to optimize the coarse matching results. Specifically, PwcNet calculates the coarse matching results using multi-view images as input.

[0081] The image is then input into two 3x3 deformable convolution layers, where the deformation of the fixed main viewpoint is kept constant at 0. The deformation of the target viewpoint is calculated as follows: based on I... i The matching relationship from viewpoint to I0 viewpoint, find i i The matching point of pixel p in view I0, and the 3x3 neighborhood of the matching point in view I0. Then, from view I0 to I... i The matching relationship of perspectives, finding the neighborhood under I0 in i i Matching points from the perspective of I, and as I i The initial value δ of the deformation of the view convolution {0→1} Then connect I0 and I i The input is fed into two 3x3 convolutional layers, and the offset is finely adjusted. The adjustment value is conv(I0, I...). i ). w i This represents the convolution weights.

[0082]

[0083] From I0 to I iThe 9*9 neighborhood space centered on the coarse matching point is used as the search space. Within the search space, the feature correlation is calculated by multiplying the features of the main viewpoint and the feature of the target viewpoint. The result is then input into the DenseNet module to obtain the fine matching results between the main viewpoint and other viewpoints.

[0084] The coordinates of three-dimensional points are calculated using the fine matching results and the initial extrinsic parameters of the camera. The coordinates of the three-dimensional points are then reprojected onto each viewpoint using the camera extrinsic parameters to obtain the reprojection matching.

[0085] The specific process involves drawing two rays, l1 and l2, from the camera centers O1 and O2 to the corresponding pixel points P1 and P2 in the 2D plane. A perpendicular line is drawn perpendicular to the two rays l1 and l2, with the feet of the perpendiculars at x1 and x2. Following this process, a system of equations is established, and x1 and x2 are solved:

[0086] x1=O1+λ1l1 x2=O2+λ2l2;

[0087] (x1-x2)*l1=0 (x1-x2)*l2=0;

[0088] Ideally, x1 and x2 are two overlapping 3D points. However, due to matching errors and camera extrinsic errors, the two points are separated. Here, the point on the ray corresponding to the main viewpoint is taken as the 3D point. If there are multiple target viewpoints, the average value of all 3D points is taken.

[0089] The features from other perspectives are deformed into the main perspective using both fine-grained matching and reprojection matching results, and then merged with the features from the main perspective in the feature dimension. These are then input into a ResNet network to regress the camera extrinsic parameters and the 3D coordinates of the face.

[0090] The 3D point coordinates are projected onto other viewpoints using updated camera extrinsic parameters to obtain new reprojection matches. The process then jumps back to "using the fine matching results and reprojection matching results respectively to deform the features of other viewpoints onto the main viewpoint and merge them with the features of the main viewpoint in the feature dimension," iterating a total of three times.

[0091] A 3D human body surface is constructed based on the marching-cubes algorithm; the reconstructed 3D human face surface and texture are then rendered and visualized from any viewpoint.

[0092] Specifically, the 3D human body surface constructed using the marching-cubes algorithm is divided into a mesh. All meshes are traversed, and based on the reconstructed 3D face point cloud and camera extrinsic parameters, the TSDF algorithm is used to obtain the mesh occupancy field. Based on the mesh occupancy field results, meshes crossed by isosurfaces are extracted. The intersection points of isosurfaces and meshes, and their corresponding triangular faces, are calculated according to the mesh state index table. For a detailed operational process and structural diagrams between each step, see [link to documentation]. Figures 3-5 .

[0093] Furthermore, to obtain the dataset for model training and testing, this invention requires calculating a realistic 3D face model. Specifically, the processed image is input into 3D modeling software to obtain the 3D geometric model of the face and camera extrinsic parameters. Using rasterization, the 3D model is reprojected onto the main viewpoint to determine the 3D spatial coordinates of each pixel. Then, the corresponding 3D points are reprojected onto other viewpoints to determine the matching relationship between the main viewpoint and other viewpoints. The matching relationship between multiple viewpoints, the 3D geometric model, and the camera extrinsic parameters can be used to construct model supervision errors. A constructed 3D face model is obtained by building the 3D surface of the face. The training error of the model is used to supervise the 3D point coordinates, the matching results from the main viewpoint to the target viewpoint, and the normal vector of the 3D model. The specific formula is as follows: where f′ represents the true match; P′ represents the 3D point coordinates; and N′ represents the normal vector. f, P, and N all represent the estimated results, and λ1, λ2, and λ3 are hyperparameters used to adjust the weight of different supervision modules.

[0094]

[0095] Example 2

[0096] This invention provides a system for multi-view face surface reconstruction, which includes: an image acquisition module, a processing module, a matching module, a calculation module, a three-dimensional coordinate determination module, an optimization module, a matching coordinate determination module, a point cloud determination module, and a reconstruction module.

[0097] The image acquisition module is used to acquire multiple face images at time t. The multiple face images are obtained by taking pictures of the same target person from different perspectives using a camera. Among them, the image taken by the camera from the main perspective is the main perspective face image. The images other than the main perspective face image among the multiple face images are the target face images. The main perspective is the perspective corresponding to the minimum rotation angle of the camera along the set axis. t≥1.

[0098] The camera in the image acquisition module can be a color camera; the number of color cameras can be at least 3, the color cameras can completely cover the face area and the angle between the cameras is not less than 15° and not more than 45°.

[0099] The processing module is used to segment and scale the main view face image at time t to obtain the main view scaled image at time t, and to segment and scale each target face image at time t to obtain multiple target scaled images at time t.

[0100] The processing module includes: a first segmentation submodule, a second segmentation submodule, and a recognition submodule.

[0101] The first segmentation submodule is used to perform background segmentation on the main view face image at time t, so as to obtain the segmented main view face image at time t.

[0102] The second segmentation submodule is used to perform background segmentation on each target face image at time t, so as to obtain the segmented target face images at time t.

[0103] The recognition submodule is used to recognize the face regions in the segmented main view face image at time t to obtain the main view zoomed image at time t, and to recognize the face regions in each target face image segmented at time t to obtain multiple target zoomed images at time t.

[0104] The matching module is used to match the main view scaled image at time t with each target scaled image at time t using deformable convolution, so as to obtain multiple initial offsets at time t.

[0105] The calculation module is used to calculate the corresponding cosine angle based on each initial offset at time t; the cosine angle is used to characterize the feature similarity between the pixels of the main view zoomed image and the corresponding target zoomed image.

[0106] The 3D coordinate determination module is used to determine the 3D coordinates of each pixel in the main-view scaled image at time t based on the camera extrinsic parameters at time t. When t > 1, the camera extrinsic parameters at time t are determined based on the face point cloud at time t-1 and the optimized offset at time t-1. When t = 1, the camera extrinsic parameters at time t are determined based on multiple face images at time t.

[0107] When t > 1, the three-dimensional coordinate determination module includes: reprojection matching submodule, extrinsic parameter determination submodule, and coordinate determination submodule.

[0108] The reprojection matching submodule is used to perform viewpoint deformation on each target scaled image at time t-1 based on the face point cloud at time t-1 and the optimized offset at time t-1, so that each target scaled image at time t-1 is projected onto the main view scaled image at time t-1, thus obtaining the reprojection matching image at time t-1.

[0109] The extrinsic parameter determination submodule is used to determine the camera extrinsic parameters at time t based on the reprojection matching coordinates in the reprojection matching image at time t-1 and the three-dimensional coordinates at time t-1.

[0110] The coordinate determination submodule is used to determine the three-dimensional coordinates of each pixel in the main view scaled image at time t based on the camera extrinsic parameters at time t.

[0111] The optimization module is used to select a set interval for multiple cosine angles at time t, obtain the preferred angle at time t, and determine the initial offset corresponding to the preferred angle at time t as the optimized offset at time t.

[0112] The matching coordinate determination module is used to project the three-dimensional coordinates onto the scaled images of each target at time t based on the optimized offset at time t, thereby obtaining multiple matching coordinates at time t.

[0113] The point cloud determination module is used to determine the face point cloud at time t based on the three-dimensional coordinates and multiple matching coordinates at time t.

[0114] The reconstruction module is used to construct the 3D surface of the face at time t based on the face point cloud at time t using the isosurface extraction method.

[0115] Example 3

[0116] This invention provides an electronic device, including a memory and a processor. The memory stores a computer program, and the processor runs the computer program to enable the electronic device to perform the multi-view face surface reconstruction method in Embodiment 1.

[0117] As an alternative implementation, the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method for multi-view face surface reconstruction in Embodiment 1.

[0118] This invention innovatively proposes a face 3D reconstruction method that is not based on a parametric model, but rather a method for 3D reconstruction of face images from multiple perspectives based on deep learning. This realizes an end-to-end face 3D reconstruction system, allowing users to quickly obtain face reconstruction results. Compared with reconstruction methods based on parametric models, it can more realistically restore the high-frequency information of the face.

[0119] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0120] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method of multi-view face surface reconstruction, characterized in that, The method comprises: obtaining multiple face images at time t; the multiple face images are obtained by photographing a same target person at different angles by using a camera; wherein, an image obtained by photographing by the camera at a main angle is a main angle face image; images other than the main angle face image in the multiple face images are target face images; the main angle is an angle corresponding to a minimum rotation angle of the camera along a set axis; t≥1; performing segmentation and scaling processing on the main angle face image at time t to obtain a main angle scaling image at time t, and performing segmentation and scaling processing on each target face image at time t to obtain multiple target scaling images at time t; using a deformable convolution, matching the main angle scaling image at time t with each target scaling image at time t to obtain multiple initial offsets at time t; calculating a corresponding cosine angle according to each initial offset at time t; the cosine angle is used to represent a feature similarity between a pixel point of the main angle scaling image and a pixel point of a corresponding target scaling image; determining three-dimensional coordinates of each pixel point in the main angle scaling image at time t according to a camera extrinsic parameter at time t; when t>1, the camera extrinsic parameter at time t is determined according to a face point cloud at time t-1 and an optimized offset at time t-1; when t=1, the camera extrinsic parameter at time t is determined according to the multiple face images at time t; performing selection in a set interval on the multiple cosine angles at time t to obtain an optimized angle at time t, and determining an initial offset corresponding to the optimized angle at time t as an optimized offset at time t; projecting the three-dimensional coordinates onto each target scaling image at time t according to the optimized offset at time t to obtain multiple matching coordinates at time t; determining a face point cloud at time t according to the three-dimensional coordinates and the multiple matching coordinates at time t; constructing a face three-dimensional surface at time t by using an isosurface extraction method according to the face point cloud at time t.

2. The method of multi-view face surface reconstruction of claim 1, wherein, When t>1, the method for determining the camera extrinsic parameter at time t comprises: performing angle deformation on each target scaling image at time t-1 according to the face point cloud at time t-1 and the optimized offset at time t-1, so that each target scaling image at time t-1 is projected onto the main angle scaling image at time t-1 to obtain a re-projection matching image at time t-1; determining the camera extrinsic parameter at time t according to a re-projection matching coordinate in the re-projection matching image at time t-1 and the three-dimensional coordinates at time t-1.

3. The method of multi-view face surface reconstruction of claim 1, wherein, performing segmentation and scaling processing on the main angle face image at time t to obtain a main angle scaling image at time t, and performing segmentation and scaling processing on each target face image at time t to obtain multiple target scaling images at time t, specifically comprising: performing background segmentation on the main angle face image at time t to obtain a segmented main angle face image at time t; performing background segmentation on each target face image at time t to obtain segmented target face images at time t; The face region in the main-view face image segmented at the t-th moment is recognized to obtain a main-view zoomed image at the t-th moment, and the face region in each target face image segmented at the t-th moment is recognized to obtain multiple target zoomed images at the t-th moment.

4. A system for multi-view face surface reconstruction, characterized in that, The system comprises: An image acquisition module is configured to acquire multiple face images at the t-th moment; the multiple face images are obtained by photographing a same target person at different angles by using a camera; an image obtained by photographing by the camera at a main view is a main-view face image; an image other than the main-view face image in the multiple face images is a target face image; the main view is an angle corresponding to a minimum rotation angle of the camera along a set axis; t≥1; A processing module is configured to perform segmentation and zooming processing on the main-view face image at the t-th moment to obtain a main-view zoomed image at the t-th moment, and perform segmentation and zooming processing on each target face image at the t-th moment to obtain multiple target zoomed images at the t-th moment; A matching module is configured to perform matching on the main-view zoomed image at the t-th moment and each target zoomed image at the t-th moment by using a deformable convolution to obtain multiple initial offsets at the t-th moment; A calculation module is configured to calculate a corresponding cosine angle according to the multiple initial offsets at the t-th moment; the cosine angle is used to represent a feature similarity between a pixel point of the main-view zoomed image and a pixel point of a corresponding target zoomed image; A three-dimensional coordinate determination module is configured to determine three-dimensional coordinates of each pixel point in the main-view zoomed image at the t-th moment according to a camera extrinsic parameter at the t-th moment; when t>1, the camera extrinsic parameter at the t-th moment is determined according to a face point cloud at the (t-1)-th moment and an optimized offset at the (t-1)-th moment; when t=1, the camera extrinsic parameter at the t-th moment is determined according to the multiple face images at the t-th moment; An optimization module is configured to select a set interval for the multiple cosine angles at the t-th moment to obtain an optimized angle at the t-th moment, and determine an initial offset corresponding to the optimized angle at the t-th moment as an optimized offset at the t-th moment; A matching coordinate determination module is configured to project the three-dimensional coordinates to each target zoomed image at the t-th moment according to the optimized offset at the t-th moment to obtain multiple matching coordinates at the t-th moment; A point cloud determination module is configured to determine a face point cloud at the t-th moment according to the three-dimensional coordinates and the multiple matching coordinates at the t-th moment; A reconstruction module is configured to construct a three-dimensional face surface at the t-th moment by using an isosurface extraction method according to the face point cloud at the t-th moment.

5. The system for multi-view face surface reconstruction of claim 4, wherein, When t>1, the three-dimensional coordinate determination module comprises: A re-projection matching sub-module is configured to perform angle deformation on each target zoomed image at the (t-1)-th moment according to the face point cloud at the (t-1)-th moment and the optimized offset at the (t-1)-th moment, so that each target zoomed image at the (t-1)-th moment is projected onto the main-view zoomed image at the (t-1)-th moment to obtain a re-projection matching image at the (t-1)-th moment; An extrinsic parameter determination sub-module is configured to determine the camera extrinsic parameter at the t-th moment according to a re-projection matching coordinate in the re-projection matching image at the (t-1)-th moment and the three-dimensional coordinates at the (t-1)-th moment. A coordinate determination submodule is configured to determine the three-dimensional coordinates of each pixel point in the main-view zoomed image at the t time according to the camera extrinsic parameter at the t time.

6. The system for multi-view face surface reconstruction of claim 4, wherein, The processing module comprises: A first segmentation submodule is configured to perform background segmentation on the main-view face image at the t time to obtain the segmented main-view face image at the t time. A second segmentation submodule is configured to perform background segmentation on each target face image at the t time to obtain the segmented target face image at the t time. An identification submodule is configured to identify the face region in the segmented main-view face image at the t time to obtain the main-view zoomed image at the t time, and identify the face region in each segmented target face image at the t time to obtain the multiple target zoomed images at the t time.

7. An electronic device, comprising: The electronic device comprises a memory and a processor, the memory is configured to store a computer program, and the processor is configured to run the computer program to enable the electronic device to perform the method for multi-view face surface reconstruction according to any one of claims 1 to 3.

8. A computer-readable storage medium, characterized in that, The computer program is stored in the memory and is executed by the processor to implement the method for multi-view face surface reconstruction according to any one of claims 1 to 3.

Images