Three-dimensional face reconstruction method for removing invalid points based on coding structured light
Through the method based on encoded structured light technology and multi-sensor data fusion, combined with deep learning and computational geometric algorithms, the problem of traditional three-dimensional face reconstruction being invalid in complex environments and different lighting conditions is solved, achieving high precision and high efficiency Three-dimensional face reconstruction ensures the authenticity and consistency of the three-dimensional model.
Patent Information
- Application Number
- CN202410927007.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-11
- Publication Date
- 2024-11-08
AI Technical Summary
Traditional three-dimensional face reconstruction methods are prone to generate a large number of invalid points in complex environments and different lighting conditions, affecting the accuracy and quality of reconstruction.
Using coded structured light technology to project adaptive structured light patterns to the face surface, combined with multi-sensor data fusion and improved deep learning algorithms, identify and remove invalid points, and generate them through adaptive convolutional neural networks and multi-level attention mechanisms. High-precision three-dimensional point cloud data and computational geometry algorithm optimization are carried out to ensure that the texture information of the three-dimensional model is consistent with the geometric structure.
The impact of noise and occlusion areas is significantly reduced under various lighting conditions, improve the accuracy and detailed performance of the three-dimensional face model, and achieve fast and reliable three-dimensional reconstruction. The output three-dimensional model has a high sense of reality and consistency.
Smart Images

Figure CN118918288A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of three-dimensional face reconstruction, and in particular to a three-dimensional face reconstruction method based on coded structured light to remove invalid points. Background Art
[0002] With the development of computer vision, image processing and 3D reconstruction technology, 3D face reconstruction has become a key technology in the fields of virtual reality, augmented reality, biometrics and animation production. However, traditional 3D face reconstruction methods are prone to generate a large number of invalid points (such as noise and occlusion areas) when facing complex environments and different lighting conditions. These invalid points will affect the accuracy and quality of reconstruction.
[0003] Existing 3D reconstruction methods mainly include technologies based on stereo vision, structured light, and time of flight (ToF). Among them, the method based on stereo vision relies on the matching of multi-view images and reconstructs the 3D structure through parallax calculation, but this method is not effective when dealing with smooth surfaces and repeated textures, and is prone to matching errors and invalid points. The 3D reconstruction method based on ToF obtains depth information by measuring the flight time of light pulses. Although it improves the accuracy of depth measurement to a certain extent, it is easily disturbed in strong light environments and has limited depth resolution.
[0004] Structured light technology is a common 3D reconstruction technology that obtains the 3D shape of an object by projecting a specific grating pattern onto the target object and analyzing the changes in the reflected light. However, traditional structured light technology is easily interfered with under complex lighting conditions, causing the reflected light signal of the projected pattern to weaken or distort, thereby introducing a large number of invalid points. In addition, traditional structured light technology usually uses a fixed grating pattern, which is difficult to adapt to different environmental changes and target object characteristics, further affecting the accuracy and robustness of 3D reconstruction.
[0005] Therefore, how to provide a three-dimensional face reconstruction method based on coded structured light to remove invalid points is a problem that technicians in this field urgently need to solve. Summary of the invention
[0006] One object of the present invention is to propose a three-dimensional face reconstruction method based on coded structured light to remove invalid points. The present invention ensures that the texture information and geometric structure of the three-dimensional face model are highly consistent, and the final output three-dimensional model has a high degree of realism and consistency.
[0007] A 3D face reconstruction method based on coded structured light to remove invalid points according to an embodiment of the present invention comprises the following steps:
[0008] S1. Use coded structured light technology to project a specific structured light pattern onto the surface of the face;
[0009] S2, capturing the reflected structured light pattern using an image sensor and generating initial image data;
[0010] S3, modulating and decoding the initial image data, extracting the structured light coding information, and generating preliminary three-dimensional point cloud data;
[0011] S4. Analyze the preliminary 3D point cloud data by improving the adaptive convolutional neural network based on deep learning, adaptively learn the features of invalid points in the image, and identify and mark the invalid points;
[0012] S5. Use an improved multi-level attention mechanism to perform secondary filtering on the marked invalid points, combine global and local information, improve the accuracy of invalid point removal, and generate optimized 3D point cloud data;
[0013] S6. Based on the optimized three-dimensional point cloud data, a computational geometry algorithm is applied to reconstruct and optimize the optimized three-dimensional point cloud data to generate a high-precision three-dimensional face model;
[0014] S7. Post-process the high-precision 3D face model to meet application requirements, including smoothing, texture mapping, and model calibration.
[0015] Optionally, the S1 includes the following steps:
[0016] S11, providing a coded structured light projection device including a projector and a coded pattern generator, wherein the projector is used to project an adaptive coded structured light pattern onto a face surface, and the coded pattern generator is used to generate and adjust the structured light pattern in real time according to a shape, posture and surface features of the face;
[0017] S12, calibrating the projection angle and distance of the projector so that the structured light pattern can evenly cover the surface of the face, and the projection angle θ and the projection distance D are adjusted according to the size and position of the face;
[0018] S13, through the intelligent projection control system, using machine learning algorithms to analyze the ambient light, facial posture changes and reflection intensity in real time, and dynamically adjust the projection parameters, including brightness adjustment L(t), contrast adjustment C(t) and color temperature adjustment T c ;
[0019] S14. Select a structured light pattern, where the structured light pattern includes a stripe pattern, a chessboard pattern, or other regular patterns.
[0020] Optionally, S2 includes the following steps:
[0021] S21, setting a multi-sensor capture device including an RGB camera, an infrared camera and a depth sensor, wherein the RGB camera is used to capture a color image of the structured light pattern, the infrared camera is used to capture the structured light pattern under low light conditions, and the depth sensor is used to obtain depth information;
[0022] S22. Calibrate the multi-sensor to capture the viewing angle and position of the RGB camera, infrared camera, and depth sensor, so that the RGB camera, infrared camera, and depth sensor work in the same reference coordinate system. The calibration parameters include the sensor's intrinsic parameter matrix and extrinsic parameter matrix. The intrinsic parameter matrix includes the focal length f and the principal point coordinates (c x ,c y ), the external parameter matrix includes the rotation matrix R and the translation vector T;
[0023] S23, synchronously capture the color image, infrared image and depth image of the structured light pattern to generate an initial image data set D initial , where the color image data is represented by I RGB (x, y), infrared image data is represented by I IR (x, y), the depth image data is represented as I Depth (x,y):
[0024] D initial = {I RGB (x,y),I IR (x,y),I Depth (x,y)}for(x,y)∈Ω;
[0025] Where Ω is the spatial domain of the image.
[0026] S24, using multi-sensor data fusion technology, fuses the data from the RGB camera, infrared camera and depth sensor to generate high-quality initial image data I fused (x, y, z), where x and y are the spatial coordinates of the image and z is the depth information:
[0027] I fused (x,y,z)=α1I RGB (x,y)+α2I IR (x,y)+α3I Depth (x,y);
[0028] Among them, α1, α2, and α3 are fusion weight coefficients, which are determined by minimizing the following error function:
[0029] E(α1,α2,α3)=∑ (x,y)∈Ω (I fused (x,y,z)-(α1I RGB (x,y)+α2I IR(x,y)+
[0030] α3I Depth (x,y))) 2 ;
[0031] S25. Perform geometric correction and radiation correction on the captured initial image data. The geometric correction includes distortion correction and projection transformation. The radiation correction includes brightness correction and contrast correction to eliminate the influence of the sensor's system error and ambient light:
[0032]
[0033] Among them, (x', y') is the corrected coordinate, k1, k2, k3, ... are distortion correction coefficients;
[0034] Radiation correction formula:
[0035]
[0036] Among them, μ is the average brightness of the image, σ is the standard deviation of the brightness;
[0037] S26, converting the corrected image data into a unified format to generate a standardized initial image data set D standard , used for subsequent image processing and 3D point cloud generation:
[0038]
[0039] Optionally, S3 includes the following steps:
[0040] S31, for the initial image data set D standard To remove noise, use Gaussian filter to smooth the color image, infrared image and depth image respectively;
[0041] S32, performing edge detection on the denoised image data, and extracting edge features of the structured light pattern using a Canny edge detection algorithm;
[0042] S33, extracting structured light coding information, performing fringe decoding on the image after edge detection, and using a phase unwrapping algorithm to calculate the phase distribution of the structured light pattern;
[0043] S34, generating preliminary three-dimensional point cloud data, and converting the fringe phase into depth information using projection geometry;
[0044] S35. Represent the three-dimensional point cloud data as a point set:
[0045] P = {(x,y,z)|(x,y)∈Ω};
[0046] Among them, x and y are image coordinates, and z is depth information.
[0047] Optionally, S4 includes the following steps:
[0048] S41. Constructing an improved adaptive convolutional neural network model, wherein the improved adaptive convolutional neural network model includes an input layer, an adaptive convolutional layer, a multi-layer convolutional layer, a pooling layer, a fully connected layer and an output layer;
[0049] S42, input the preliminary three-dimensional point cloud data P = {(x, y, z) | (x, y) ∈ Ω} to the input layer of the ACNN model, which converts the point cloud data into a feature tensor T input ;
[0050] S43, extract local features from the input feature tensor through an adaptive convolution layer. The convolution kernel weight of the adaptive convolution layer is dynamically adjusted according to the input data. The convolution operation formula is:
[0051]
[0052] Among them, T conv is the feature tensor after convolution, w m,n is the convolution kernel weight, i, j, k are the coordinates of the feature tensor;
[0053] S44. In the multi-layer convolution layer, higher-level features are extracted layer by layer, and the convolution result is nonlinearly transformed using the ReLU activation function σ(x)=max(0,x) to generate the intermediate feature tensor T middle ;
[0054] S45. Downsample the intermediate feature tensor through the pooling layer to reduce the size of the feature tensor and improve the translation invariance of the feature. The pooling operation formula is:
[0055] T pool (i,j,k)=max m,n (T middle (i+m,j+n,k));
[0056] Among them, T pool is the feature tensor after pooling;
[0057] S46: Input the pooled feature tensor into the fully connected layer to learn the global features and generate the final feature representation T fc ;
[0058] S47, classify the final feature representation through the output layer, identify and mark invalid points, and output the invalid point marking matrix M invalid , marking each element of the matrix M invalid(i,j) indicates the validity of the (i,j)th point in the point cloud data. A value of 0 indicates an invalid point, and a value of 1 indicates a valid point.
[0059] Optionally, S5 includes the following steps:
[0060] S51, constructing a multi-level attention mechanism network, wherein the network includes a global attention module and a local attention module;
[0061] S52, the three-dimensional point cloud data P after identifying and marking the invalid points marked ={(x,y,z,m)|(x,y)∈Ω,m∈{0,1}} is input into the multi-level attention mechanism network;
[0062] S53. In the global attention module, the global attention weight of each point is calculated, and the global attention weight α global (i) Calculated by the following formula:
[0063]
[0064] Among them, e global (i) = W global ·P marked (i)+b global , W global and b global is the weight and bias of global attention, N is the number of points in the point cloud;
[0065] S54. In the local attention module, the local attention weight of each point is calculated, and the local attention weight α local (i) Calculated by the following formula:
[0066]
[0067] Among them, e local (i) = W local ·P marked (i)+b local , W local and b local is the weight and bias of local attention, N(i) is the neighborhood point set of point i;
[0068] S55, combining the global attention weight and the local attention weight, performing weighted summation on the three-dimensional point cloud data to generate weighted point cloud data P weighted (i), the calculation formula is:
[0069] P weighted (i) = α global (i)·P marked (i)+α local (i)·Pmarked (i);
[0070] S56, perform secondary filtering on the weighted point cloud data, use a denoising algorithm based on weighted features to remove the remaining invalid points, and generate optimized three-dimensional point cloud data P optimized , where invalid points with weights lower than a preset threshold will be removed.
[0071] Optionally, the S6 includes the following steps:
[0072] S61, the optimized three-dimensional point cloud data P optimized Input into the 3D reconstruction algorithm;
[0073] S62, reconstruct the surface of the point cloud data, and use the Delaunay triangulation algorithm to generate a preliminary mesh structure of the three-dimensional face model. The vertex set of the triangulation is V = {(x i ,y i ,z i )|i=1,2,…,N}, the edge set is E={(v i ,v j )|v i ,v j ∈V}, the face set is F={(v i ,v j ,v k )|v i ,v j ,v k ∈V};
[0074] S63, optimizing the generated preliminary mesh structure, using the Laplacian smoothing algorithm to smooth the mesh vertices, and the smoothed vertex coordinates (x′ i ,y′ i ,z′ i ) is calculated by the following formula:
[0075]
[0076] Among them, λ is the smoothing coefficient, N(i) is the set of neighboring vertices of vertex i;
[0077] S64, perform detail enhancement on the smoothed mesh, use a detail enhancement algorithm based on shape histogram, calculate the shape feature histogram of each vertex, and adjust the vertex position according to the histogram information. The adjusted vertex coordinates (x″ i ,y″ i ,z″ i ) is calculated by the following formula:
[0078] (x″ i ,y″ i ,z″i )=(x′ i ,y′ i ,z′ i )+β·h i ·n i ;
[0079] Among them, β is the detail enhancement coefficient, h i is the height of the shape histogram, n i is the normal vector of the vertex;
[0080] S65, texture mapping is performed on the optimized mesh model. A texture mapping algorithm based on spherical projection is used to map the preprocessed texture image T(x, y) to the surface of the three-dimensional mesh model. The mapping function is:
[0081] T(x,y)=T sphere (u,v);
[0082] Where (u, v) is the coordinate of the mesh vertex in the spherical projection coordinate system, T sphere is the texture image of spherical projection;
[0083] S66. Globally optimize the three-dimensional face model after texture mapping, using an optimization algorithm based on energy minimization to minimize the following energy function:
[0084] E=E smooth +αE data +βE texture ,
[0085] Among them, E smooth is the mesh smoothing energy, E data is the data consistency energy, E texture is the texture consistency energy, α and β are weight coefficients;
[0086] S67: Output a final high-precision three-dimensional face model, wherein the model has high-precision geometric details and texture information.
[0087] The beneficial effects of the present invention are:
[0088] (1) The present invention uses coded structured light technology to project an adaptive structured light pattern onto the surface of the face, and can generate and adjust the light pattern in real time according to the shape, posture and surface features of the face, so that the captured three-dimensional information is more accurate, avoiding the errors and limitations caused by fixed grating patterns, and using multi-sensor fusion technology to comprehensively process the data of the RGB camera, infrared camera and depth sensor to generate high-quality initial image data. This multi-source data fusion method can maintain the consistency and reliability of data under various lighting conditions, effectively reduce the influence of noise and occlusion areas, analyze the preliminary three-dimensional point cloud data through an improved adaptive convolutional neural network based on deep learning, adaptively learn the features of invalid points in the image, identify and mark invalid points, ensure accurate identification of invalid points, use an improved multi-level attention mechanism to perform secondary filtering on the marked invalid points, combine global and local information, improve the accuracy of invalid point removal, generate optimized three-dimensional point cloud data, and make the final three-dimensional face model have higher accuracy and detail expression.
[0089] (2) The present invention uses an intelligent projection control system and a machine learning algorithm to analyze ambient lighting, facial posture changes and reflection intensity in real time, dynamically adjust projection parameters, and ensure the stability and clarity of structured light patterns in different environments. This adaptive projection technology can achieve fast and reliable three-dimensional reconstruction in a variety of application scenarios. In the preprocessing step, the initial image data is efficiently subjected to noise removal, edge detection and phase decoding to ensure that the generated three-dimensional point cloud data has high purity and accuracy. The optimized three-dimensional point cloud data is reconstructed and optimized through computational geometry algorithms such as Delaunay triangulation, Laplacian smoothing and detail enhancement based on shape histograms, so that the generated three-dimensional face model not only has high-precision geometric details, but also has good smoothness and detail performance. In the texture mapping and global optimization steps, a texture mapping algorithm based on spherical projection and an optimization algorithm based on energy minimization are used to ensure that the texture information of the three-dimensional face model is highly consistent with the geometric structure, and the final output three-dimensional model has a high sense of reality and consistency. BRIEF DESCRIPTION OF THE DRAWINGS
[0090] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:
[0091] Figure 1 This is a flow chart of a three-dimensional face reconstruction method based on coded structured light to remove invalid points proposed by the present invention. DETAILED DESCRIPTION
[0092] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.
[0093] refer to Figure 1 , a 3D face reconstruction method based on coded structured light to remove invalid points, comprising the following steps:
[0094] S1. Use coded structured light technology to project a structured light pattern containing time coding and space coding onto the surface of the face, and the modulation pattern is dynamically adjusted according to the lighting conditions and facial features;
[0095] S2, capturing the reflected structured light pattern using an image sensor and generating initial image data;
[0096] S3, modulating and decoding the initial image data, extracting the structured light coding information, and generating preliminary three-dimensional point cloud data;
[0097] S4. Analyze the preliminary 3D point cloud data by improving the adaptive convolutional neural network based on deep learning, adaptively learn the features of invalid points in the image, and identify and mark the invalid points;
[0098] S5. Use an improved multi-level attention mechanism to perform secondary filtering on the marked invalid points, combine global and local information, improve the accuracy of invalid point removal, and generate optimized 3D point cloud data;
[0099] S6. Based on the optimized three-dimensional point cloud data, a computational geometry algorithm is applied to reconstruct and optimize the optimized three-dimensional point cloud data to generate a high-precision three-dimensional face model;
[0100] S7. Post-process the high-precision 3D face model to meet application requirements, including smoothing, texture mapping, and model calibration.
[0101] In this implementation, S1 includes the following steps:
[0102] S11, providing a coded structured light projection device including a projector and a coded pattern generator, wherein the projector is used to project an adaptive coded structured light pattern onto a face surface, and the coded pattern generator is used to generate and adjust the structured light pattern in real time according to a shape, posture and surface features of the face;
[0103] S12, calibrating the projection angle and distance of the projector so that the structured light pattern can evenly cover the surface of the face, and the projection angle θ and the projection distance D are adjusted according to the size and position of the face;
[0104] S13, through the intelligent projection control system, using machine learning algorithms to analyze the ambient light, facial posture changes and reflection intensity in real time, and dynamically adjust the projection parameters, including brightness adjustment L(t), contrast adjustment C(t) and color temperature adjustment T c ;
[0105] As a technical means for dynamically adjusting projection parameters, the intelligent projection control system only needs to explain its function and purpose. The intelligent projection control system in S13 has already explained its function and purpose (using machine learning algorithms to analyze ambient lighting, facial posture changes and reflection intensity in real time, and dynamically adjust projection parameters), so there is no need to introduce its internal implementation details in detail. As long as the description is clear and can enable technicians in this field to understand its function and role, it will be fine.
[0106] S14: Select a structured light pattern, where the structured light pattern includes a stripe pattern, a chessboard pattern, or other regular patterns.
[0107] In this implementation, S2 includes the following steps:
[0108] S21. Setting a multi-sensor capture device including an RGB camera, an infrared camera, and a depth sensor, wherein the RGB camera is used to capture a color image of the structured light pattern, the infrared camera is used to capture the structured light pattern under low light conditions, and the depth sensor is used to obtain depth information;
[0109] S21. Set up a three-dimensional reconstruction device including a projector and at least one image sensor, wherein the projector is used to project a coded structured light pattern onto the surface of a face, and the image sensor (depth camera) is used to capture the structured light pattern reflected by the surface of the face. This configuration allows the system to calculate phase information by analyzing changes in the captured pattern, and further obtain depth information by phase decoding, thereby constructing three-dimensional point cloud data.
[0110] S22. Calibrate the multi-sensor to capture the viewing angle and position of the RGB camera, infrared camera, and depth sensor, so that the RGB camera, infrared camera, and depth sensor work in the same reference coordinate system. The calibration parameters include the sensor's intrinsic parameter matrix and extrinsic parameter matrix. The intrinsic parameter matrix includes the focal length f and the principal point coordinates (c x ,c y ), the external parameter matrix includes the rotation matrix R and the translation vector T;
[0111] S23, synchronously capture the color image, infrared image and depth image of the structured light pattern to generate an initial image data set D initial , where the color image data is represented by I RGB (x, y), infrared image data is represented as I IR (x, y), the depth image data is represented as I Deptd (x,y):
[0112] D initial = {I RGB (x,y),I IR (x,y),I Depth (x,y)}for(x,y)∈Ω;
[0113] Where Ω is the spatial domain of the image.
[0114] S24, using multi-sensor data fusion technology, fuses the data from the RGB camera, infrared camera and depth sensor to generate high-quality initial image data I fused (x, y, z), where x and y are the spatial coordinates of the image and z is the depth information:
[0115] I fused (x,y,z)=α1I RGB (x,y)+α2I IR (x,y)+α3I Depth (x,y);
[0116] Among them, α1, α2, and α3 are fusion weight coefficients, which are determined by minimizing the following error function:
[0117] E(α1,α2,α3)=∑ (x,y)∈Ω (I fused (x,y,z)-(α1I RGB (x,y)+α2I IR (x,y)+
[0118] α3I Depth (x,y))) 2 ;
[0119] S25. Perform geometric correction and radiation correction on the captured initial image data. The geometric correction includes distortion correction and projection transformation. The radiation correction includes brightness correction and contrast correction to eliminate the influence of the sensor's system error and ambient light:
[0120]
[0121] Among them, (x', y') is the corrected coordinate, k1, k2, k3, ... are distortion correction coefficients;
[0122] Radiation correction formula:
[0123]
[0124] Among them, μ is the average brightness of the image, σ is the standard deviation of the brightness;
[0125] S26, converting the corrected image data into a unified format to generate a standardized initial image data set D standard , used for subsequent image processing and 3D point cloud generation:
[0126]
[0127] In this implementation, S3 includes the following steps:
[0128] S31, for the initial image data set D standard To remove noise, use Gaussian filter to smooth the color image, infrared image and depth image respectively;
[0129] S32, performing edge detection on the denoised image data, and extracting edge features of the structured light pattern using a Canny edge detection algorithm;
[0130] S33, extracting structured light coding information, performing fringe decoding on the image after edge detection, and using a phase unwrapping algorithm to calculate the phase distribution of the structured light pattern;
[0131] S34, generating preliminary three-dimensional point cloud data, and converting the fringe phase into depth information using projection geometry;
[0132] S35. Represent the three-dimensional point cloud data as a point set:
[0133] P = {(x,y,z)|(x,y)∈Ω};
[0134] Among them, x and y are image coordinates, and z is depth information.
[0135] In this implementation, S4 includes the following steps:
[0136] S41. Construct an improved adaptive convolutional neural network model, wherein the improved adaptive convolutional neural network model includes an input layer, an adaptive convolutional layer, a multi-layer convolutional layer, a pooling layer, a fully connected layer and an output layer;
[0137] The neural network datasets are collected as follows:
[0138] Utilization of public datasets: Use existing public face datasets as the basis for training. For example, commonly used face datasets include LFW, CelebA, and AFEW. These datasets contain face images with a variety of lighting, expressions, and postures, which can help the network learn the general features of the face.
[0139] Synthetic data generation: If the existing dataset is not sufficient to cover all required scenarios (such as structured light reflection patterns under specific lighting conditions), consider using graphics rendering technology to generate synthetic data. By simulating face images under different lighting and structured light conditions, a large number of training samples can be artificially created.
[0140] Data enhancement: Enhance existing data, including rotation, scaling, shearing, and illumination changes, to increase the generalization ability of the model. In addition, structured light patterns can also be generated by algorithms and applied to face data to simulate actual application situations.
[0141] S42. Input the preliminary 3D point cloud data P = {(x, y, z) | (x, y) ∈ Ω} into the input layer of the ACNN model, which converts the point cloud data into a feature tensor T input ;
[0142] S43, extract local features from the input feature tensor through an adaptive convolution layer. The convolution kernel weight of the adaptive convolution layer is dynamically adjusted according to the input data. The convolution operation formula is:
[0143]
[0144] Among them, T conv is the feature tensor after convolution, w m,n is the convolution kernel weight, i, j, k are the coordinates of the feature tensor;
[0145] S44. In the multi-layer convolution layer, higher-level features are extracted layer by layer, and the convolution result is nonlinearly transformed using the ReLU activation function σ(x)=max(0,x) to generate the intermediate feature tensor T middle ;
[0146] S45. Downsample the intermediate feature tensor through the pooling layer to reduce the size of the feature tensor and improve the translation invariance of the feature. The pooling operation formula is:
[0147] T pool (i,j,k)=max m,n (T middle (i+m,j+n,k));
[0148] Among them, T pool is the feature tensor after pooling;
[0149] S46: Input the pooled feature tensor into the fully connected layer to learn the global features and generate the final feature representation T fc ;
[0150] S47, classify the final feature representation through the output layer, identify and mark invalid points, and output the invalid point marking matrix M invalid , marking each element of the matrix M invalid (i,j) indicates the validity of the (i,j)th point in the point cloud data. A value of 0 indicates an invalid point, and a value of 1 indicates a valid point.
[0151] In this implementation, S5 includes the following steps:
[0152] S51. Construct a multi-level attention mechanism network, which includes a global attention module and a local attention module;
[0153] S52, the three-dimensional point cloud data P after identifying and marking the invalid points marked ={(x,y,z,m)|(x,y)∈Ω,m∈{0,1}} is input into the multi-level attention mechanism network;
[0154] S53. In the global attention module, the global attention weight of each point is calculated. The global attention weight α global (i) Calculated by the following formula:
[0155]
[0156] Among them, e global (i) = W global ·P marked (i)+b glbal , W global and b global is the weight and bias of global attention, N is the number of points in the point cloud;
[0157] S54. In the local attention module, the local attention weight of each point is calculated, and the local attention weight α local (i) Calculated by the following formula:
[0158]
[0159] Among them, e local (i) = W local ·P marked (i)+b local , W local and b local is the weight and bias of local attention, N(i) is the neighborhood point set of point i;
[0160] S55, combining the global attention weight and the local attention weight, performing weighted summation on the three-dimensional point cloud data to generate weighted point cloud data P weighted (i), the calculation formula is:
[0161] Pweighted (i) = α global (i)·P marked (i)+α global (i)·P marked (i);
[0162] S56, perform secondary filtering on the weighted point cloud data, use a denoising algorithm based on weighted features to remove the remaining invalid points, and generate optimized three-dimensional point cloud data P optimized , where invalid points with weights lower than a preset threshold will be removed.
[0163] In this implementation, S6 includes the following steps:
[0164] S61, the optimized three-dimensional point cloud data P optimized Input into the 3D reconstruction algorithm;
[0165] S62, reconstruct the surface of the point cloud data, and use the Delaunay triangulation algorithm to generate a preliminary mesh structure of the three-dimensional face model. The vertex set of the triangulation is V = {(x i ,y i ,z i )|i=1,2,…,N}, the edge set is E={(v i ,v j )|v i ,v j ∈V}, the face set is F={(v i ,v j ,v k )|v i ,v j ,v k ∈V};
[0166] S63, optimize the generated preliminary mesh structure, use the Laplacian smoothing algorithm to smooth the mesh vertices, and the smoothed vertex coordinates (x i ,y i ,z i ) is calculated by the following formula:
[0167]
[0168] Among them, λ is the smoothing coefficient, N(i) is the set of neighboring vertices of vertex i;
[0169] S64, perform detail enhancement on the smoothed mesh, use a detail enhancement algorithm based on shape histogram, calculate the shape feature histogram of each vertex, and adjust the vertex position according to the histogram information. The adjusted vertex coordinates (x″ i ,y″ i ,z″ i) is calculated by the following formula:
[0170] (x″ i ,y″ i ,z″ i )=(x′ i ,y′ i ,z′ i )+β·h i ·n i ;
[0171] Among them, β is the detail enhancement coefficient, h i is the height of the shape histogram, n i is the normal vector of the vertex;
[0172] S65, texture mapping is performed on the optimized mesh model. A texture mapping algorithm based on spherical projection is used to map the preprocessed texture image T(x, y) to the surface of the three-dimensional mesh model. The mapping function is:
[0173] T(x,y)=T sphere (u,v);
[0174] Where (u, v) is the coordinate of the mesh vertex in the spherical projection coordinate system, T sphere is the texture image of spherical projection;
[0175] S66. Globally optimize the three-dimensional face model after texture mapping, using an optimization algorithm based on energy minimization to minimize the following energy function:
[0176] E=E smooth +αE data +βE texture ,
[0177] Among them, E smooth is the mesh smoothing energy, B data is the data consistency energy, E texture is the texture consistency energy, α and β are weight coefficients;
[0178] S67. Output the final high-precision three-dimensional face model, which has high-precision geometric details and texture information.
[0179] Embodiment 1:
[0180] This embodiment selects a computer vision laboratory of a university as the test scene. The laboratory is equipped with high-performance computers, RGB cameras, infrared cameras and depth sensors. The test subjects are 50 volunteers of different ages and genders, who are tested for 3D face reconstruction under different lighting conditions and postures. The experiment time is from 9 am to 5 pm, and the tests are carried out under three lighting conditions: natural light, artificial light and mixed light to verify the accuracy and robustness of the method of the present invention.
[0181] In the laboratory, the RGB camera, infrared camera and depth sensor are fixed on a tripod and adjusted to a suitable height and angle so that they can cover the entire facial area of the volunteer. First, a coded structured light projection device including a projector and a coded pattern generator is set up to project an adaptive coded structured light pattern onto the facial surface of the volunteer. The projection parameters of the projector such as brightness, contrast and color temperature are adjusted in real time according to the ambient lighting conditions and the changes in the volunteer's posture to ensure the clarity and resolution of the structured light pattern.
[0182] The multi-sensor capture device starts working, the RGB camera captures color images, the infrared camera captures structured light patterns under low light conditions, and the depth sensor obtains depth information. Multi-sensor data fusion technology fuses these data to generate high-quality initial image data.
[0183] The initial image data is preprocessed, and the color image, infrared image and depth image are smoothed using Gaussian filters to reduce the impact of noise. Then, the Canny edge detection algorithm is used to extract the edge features of the structured light pattern, the image after edge detection is stripe decoded, the phase distribution of the structured light pattern is calculated, and preliminary three-dimensional point cloud data is generated.
[0184] The improved deep learning-based adaptive convolutional neural network is used to analyze the preliminary 3D point cloud data, identify and mark invalid points. Then, the improved multi-level attention mechanism is used to perform secondary filtering on the marked invalid points, combining global and local information to improve the accuracy of invalid point removal and generate optimized 3D point cloud data.
[0185] Based on the optimized 3D point cloud data, the Delaunay triangulation algorithm is used to generate the preliminary mesh structure of the 3D face model. The mesh vertices are smoothed by the Laplacian smoothing algorithm to ensure the smoothness and consistency of the mesh. Then, the shape histogram-based detail enhancement algorithm is used to calculate the shape feature histogram of each vertex, and the vertex position is adjusted according to the histogram information to enhance the detail performance of the model.
[0186] Finally, the optimized mesh model is texture mapped, and the preprocessed texture image is mapped to the surface of the 3D mesh model using a texture mapping algorithm based on spherical projection. The energy minimization-based optimization algorithm ensures that the texture information of the 3D face model is highly consistent with the geometric structure, generating the final high-precision 3D face model.
[0187] The experimental data are shown in Table 1 below:
[0188] Table 1 Comparison of 3D face reconstruction experimental data
[0189]
[0190]
[0191] The data in Table 1 show that under different lighting conditions and postures, the reconstruction time of the method of the present invention is significantly shorter than that of the traditional method, and the reconstruction accuracy is also significantly improved. Specifically, under natural light frontal posture, the reconstruction time of the traditional method is 30 seconds, the number of invalid points is 150, and the reconstruction accuracy is 2.5 mm; while the reconstruction time of the method of the present invention is only 20 seconds, the number of invalid points is reduced to 30, and the reconstruction accuracy is improved to 1.0 mm. Similarly, under other lighting conditions and postures, the method of the present invention shows faster reconstruction speed, fewer invalid points and higher reconstruction accuracy.
[0192] In summary, the present invention significantly improves the efficiency and accuracy of three-dimensional face reconstruction through innovative coded structured light technology, multi-sensor data fusion and advanced image processing algorithms, and solves the problems of traditional methods such as many invalid points and low reconstruction accuracy in complex environments and under different lighting conditions.
[0193] The present invention uses coded structured light technology to project an adaptive structured light pattern onto the surface of the face, and can generate and adjust the light pattern in real time according to the shape, posture and surface features of the face, so that the captured three-dimensional information is more accurate, avoiding the errors and limitations caused by the fixed grating pattern, and using multi-sensor fusion technology to comprehensively process the data of the RGB camera, infrared camera and depth sensor to generate high-quality initial image data. This multi-source data fusion method can maintain the consistency and reliability of data under various lighting conditions, effectively reduce the influence of noise and occlusion areas, analyze the preliminary three-dimensional point cloud data through an improved adaptive convolutional neural network based on deep learning, adaptively learn the features of invalid points in the image, identify and mark invalid points, ensure accurate identification of invalid points, use an improved multi-level attention mechanism to perform secondary filtering on the marked invalid points, combine global and local information, improve the accuracy of invalid point removal, generate optimized three-dimensional point cloud data, and make the final three-dimensional face model have higher accuracy and detail performance.
[0194] The present invention uses an intelligent projection control system and a machine learning algorithm to analyze ambient lighting, facial posture changes and reflection intensity in real time, dynamically adjust projection parameters, and ensure the stability and clarity of structured light patterns in different environments. This adaptive projection technology can achieve fast and reliable three-dimensional reconstruction in a variety of application scenarios. In the preprocessing step, the initial image data is efficiently subjected to noise removal, edge detection and phase decoding to ensure that the generated three-dimensional point cloud data has high purity and accuracy. Through computational geometry algorithms such as Delaunay triangulation, Laplacian smoothing and detail enhancement based on shape histograms, the optimized three-dimensional point cloud data is reconstructed and optimized, so that the generated three-dimensional face model not only has high-precision geometric details, but also has good smoothness and detail performance. In the texture mapping and global optimization steps, a texture mapping algorithm based on spherical projection and an optimization algorithm based on energy minimization are used to ensure that the texture information of the three-dimensional face model is highly consistent with the geometric structure, and the final output three-dimensional model has a high sense of reality and consistency.
[0195] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.
Claims
1. A 3D face reconstruction method based on coded structured light to remove invalid points, characterized in that: The steps include: S1. Use coded structured light technology to project a specific structured light pattern onto the surface of the face; S2, capturing the reflected structured light pattern using an image sensor and generating initial image data; S3, modulating and decoding the initial image data, extracting the structured light coding information, and generating preliminary three-dimensional point cloud data; S4. Analyze the preliminary 3D point cloud data by improving the adaptive convolutional neural network based on deep learning, adaptively learn the features of invalid points in the image, and identify and mark the invalid points; S5. Use an improved multi-level attention mechanism to perform secondary filtering on the marked invalid points, combine global and local information, and generate optimized 3D point cloud data; S6. Based on the optimized three-dimensional point cloud data, a computational geometry algorithm is applied to reconstruct and optimize the optimized three-dimensional point cloud data to generate a high-precision three-dimensional face model; S7. Post-process the high-precision 3D face model to meet application requirements, including smoothing, texture mapping, and model calibration.
2. The method for 3D face reconstruction based on coded structured light to remove invalid points according to claim 1, characterized in that: The S1 comprises the following steps: S11, providing a coded structured light projection device including a projector and a coded pattern generator, wherein the projector is used to project an adaptive coded structured light pattern onto a face surface, and the coded pattern generator is used to generate and adjust the structured light pattern in real time according to a shape, posture and surface features of the face; S12, calibrating the projection angle and distance of the projector so that the structured light pattern can evenly cover the surface of the face, and the projection angle θ and the projection distance D are adjusted according to the size and position of the face; S13, through the intelligent projection control system, using machine learning algorithms to analyze the ambient light, facial posture changes and reflection intensity in real time, and dynamically adjust the projection parameters, including brightness adjustment L(t), contrast adjustment C(t) and color temperature adjustment T c ; S14. Select a structured light pattern, where the structured light pattern includes a stripe pattern, a chessboard pattern, or other regular patterns.
3. The 3D face reconstruction method based on coded structured light to remove invalid points according to claim 2, characterized in that: The S2 comprises the following steps: S21, setting a multi-sensor capture device including an RGB camera, an infrared camera and a depth sensor, wherein the RGB camera is used to capture a color image of the structured light pattern, the infrared camera is used to capture the structured light pattern under low light conditions, and the depth sensor is used to obtain depth information; S22. Calibrate the multi-sensor to capture the viewing angle and position of the RGB camera, infrared camera, and depth sensor, so that the RGB camera, infrared camera, and depth sensor work in the same reference coordinate system. The calibration parameters include the sensor's intrinsic parameter matrix and extrinsic parameter matrix. The intrinsic parameter matrix includes the focal length f and the principal point coordinates (c x ,c y ), the external parameter matrix includes the rotation matrix R and the translation vector T; S23, synchronously capture the color image, infrared image and depth image of the structured light pattern to generate an initial image data set D initial , where the color image data is represented by I RGB (x, y), infrared image data is represented as I IR (x, y), the depth image data is represented as I Depth (x,y): D initial ={I RGB (x,y),I IR (x,y),I Depth (x,y)}for(x,y)∈Ω; Where Ω is the spatial domain of the image. S24, using multi-sensor data fusion technology, fuses the data from the RGB camera, infrared camera and depth sensor to generate high-quality initial image data I fused (x, y, z), where x and y are the spatial coordinates of the image and z is the depth information: I fused (x,y,z)=α1I RGB (x,y)+α2I IR (x,y)+α3I Depth (x,y); Among them, α1, α2, and α3 are fusion weight coefficients, which are determined by minimizing the following error function: E(α1,α2,α3)=∑ (x,y)∈Ω (I fused (x,y,z)-(α1I RGB (x,y)+α2I IR (x,y)+α3I Depth (x,y))) 2 ; S25. Perform geometric correction and radiation correction on the captured initial image data. The geometric correction includes distortion correction and projection transformation. The radiation correction includes brightness correction and contrast correction to eliminate the influence of the sensor's system error and ambient light: Among them, (x', y') is the corrected coordinate, k1, k2, k3, ... are distortion correction coefficients; Radiation correction formula: Among them, μ is the average brightness of the image, σ is the standard deviation of the brightness; S26, converting the corrected image data into a unified format to generate a standardized initial image data set D standard , used for subsequent image processing and 3D point cloud generation:
4. The method for 3D face reconstruction based on coded structured light to remove invalid points according to claim 3, characterized in that: The S3 comprises the following steps: S31, for the initial image data set D standard To remove noise, use Gaussian filter to smooth the color image, infrared image and depth image respectively; S32, performing edge detection on the denoised image data, and extracting edge features of the structured light pattern using a Canny edge detection algorithm; S33, extracting structured light coding information, performing fringe decoding on the image after edge detection, and using a phase unwrapping algorithm to calculate the phase distribution of the structured light pattern; S34, generating preliminary three-dimensional point cloud data, and converting the fringe phase into depth information using projection geometry; S35. Represent the three-dimensional point cloud data as a point set: P = {(x,y,z)|(x,y)∈Ω}; Among them, x and y are image coordinates, and z is depth information.
5. The 3D face reconstruction method based on coded structured light to remove invalid points according to claim 4, characterized in that: The S4 comprises the following steps: S41. Constructing an improved adaptive convolutional neural network model, wherein the improved adaptive convolutional neural network model includes an input layer, an adaptive convolutional layer, a multi-layer convolutional layer, a pooling layer, a fully connected layer and an output layer; S42, input the preliminary three-dimensional point cloud data P = {(x, y, z) | (x, y) ∈ Ω} to the input layer of the ACNN model, which converts the point cloud data into a feature tensor T input ; S43, extract local features from the input feature tensor through an adaptive convolution layer. The convolution kernel weight of the adaptive convolution layer is dynamically adjusted according to the input data. The convolution operation formula is: Among them, T conv is the feature tensor after convolution, w m,n is the convolution kernel weight, i, j, k are the coordinates of the feature tensor; S44. In the multi-layer convolution layer, higher-level features are extracted layer by layer, and the convolution result is nonlinearly transformed using the ReLU activation function σ(x)=max(0,x) to generate the intermediate feature tensor T middle ; S45. Downsample the intermediate feature tensor through the pooling layer to reduce the size of the feature tensor and improve the translation invariance of the feature. The pooling operation formula is: T pool (i,j,k)=max m,n (T middle (i+m,j+n,k)); Among them, T pool is the feature tensor after pooling; S46: Input the pooled feature tensor into the fully connected layer to learn the global features and generate the final feature representation T fc ; S47, classify the final feature representation through the output layer, identify and mark invalid points, and output the invalid point marking matrix M invalid , marking each element of the matrix M invalid (i,j) indicates the validity of the (i,j)th point in the point cloud data. A value of 0 indicates an invalid point, and a value of 1 indicates a valid point.
6. The method for 3D face reconstruction based on coded structured light to remove invalid points according to claim 5, characterized in that: The S5 comprises the following steps: S51, constructing a multi-level attention mechanism network, wherein the network includes a global attention module and a local attention module; S52, the three-dimensional point cloud data P after identifying and marking the invalid points marked ={(x,y,z,m)|(x,y)∈Ω,m∈{0,1}} is input into the multi-level attention mechanism network; S53. In the global attention module, the global attention weight of each point is calculated, and the global attention weight α global (i) Calculated by the following formula: Among them, e global (i) = W global ·P marked (i)+b global , W global and b global is the weight and bias of global attention, N is the number of points in the point cloud; S54. In the local attention module, the local attention weight of each point is calculated, and the local attention weight α local (i) Calculated by the following formula: Among them, e local (i) = W local ·P marked (i)+b local , W local and b local is the weight and bias of local attention, N(i) is the neighborhood point set of point i; S55, combining the global attention weight and the local attention weight, performing weighted summation on the three-dimensional point cloud data to generate weighted point cloud data P weighted (i), the calculation formula is: P weighted (i)=α global (i)·P marked (i)+α global (i)·P marked (i); S56, perform secondary filtering on the weighted point cloud data, use a denoising algorithm based on weighted features to remove the remaining invalid points, and generate optimized three-dimensional point cloud data P optimized , where invalid points with weights lower than a preset threshold will be removed.
7. The method for 3D face reconstruction based on coded structured light to remove invalid points according to claim 6, characterized in that: The S6 comprises the following steps: S61, the optimized three-dimensional point cloud data P optimized Input into the 3D reconstruction algorithm; S62, reconstruct the surface of the point cloud data, and use the Delaunay triangulation algorithm to generate a preliminary mesh structure of the three-dimensional face model. The vertex set of the triangulation is V = {(x i ,y i ,z i )|i=1,2,…,N}, the edge set is E={(v i ,v j )|v i ,v j ∈V}, the face set is F={(v i ,v j ,v k )|v i ,v j ,v k ∈V}; S63, optimizing the generated preliminary mesh structure, using the Laplacian smoothing algorithm to smooth the mesh vertices, and the smoothed vertex coordinates (x′ i ,y′ i ,z′ i ) is calculated by the following formula: Among them, λ is the smoothing coefficient, N(i) is the set of neighboring vertices of vertex i; S64, perform detail enhancement on the smoothed mesh, use a detail enhancement algorithm based on shape histogram, calculate the shape feature histogram of each vertex, and adjust the vertex position according to the histogram information. The adjusted vertex coordinates (x″ i ,y″ i ,z″ i ) is calculated by the following formula: (x″ i ,y′′ i ,z′′ i )=(x′ i ,y′ i ,z′ i )+β·h i ·n i 4 Among them, β is the detail enhancement coefficient, h i is the height of the shape histogram, n i is the normal vector of the vertex; S65, texture mapping is performed on the optimized mesh model. A texture mapping algorithm based on spherical projection is used to map the preprocessed texture image T(x, y) to the surface of the three-dimensional mesh model. The mapping function is: T(x,y)=T sphere (u,v); Where (u, v) is the coordinate of the mesh vertex in the spherical projection coordinate system, T sphere is the texture image of spherical projection; S66. Globally optimize the three-dimensional face model after texture mapping, using an optimization algorithm based on energy minimization to minimize the following energy function: E=E smooth +αE data +βE texture , Among them, E smooth is the mesh smoothing energy, E data is the data consistency energy, E texture is the texture consistency energy, α and β are weight coefficients; S67: Output a final high-precision three-dimensional face model, wherein the model has high-precision geometric details and texture information.
Citation Information
Cited By
High-precision three-dimensional terrain reconstruction method based on deep learning
CN120655849A
A high-precision three-dimensional terrain reconstruction method based on deep learning
CN120655849B
Security monitoring camera detection method and system based on face recognition
CN121121822A
A security monitoring camera detection method and system based on face recognition
CN121121822B
