A high-fidelity facial expression capture method based on binocular video

Abstract

The application discloses a kind of face expression high-fidelity capture methods based on binocular video, this method is mainly used to generate the three-dimensional deformable face model coefficient corresponding to the face expression of character in video, these coefficients are rendered through traditional rendering pipeline, and high-fidelity video equivalent to original video is rendered, to realize accurate reproduction and expression of face expression.The present technology is given neutral expression model and appearance map, and a binocular video, innovatively combines face feature point detection, binocular matching and differentiable rendering technology, and the three-dimensional deformable face model coefficient is optimized by the method of gradient descent, and the geometric representation of face expression is optimized, and high-quality face capture of face in video is realized.Experiments show that the method of the application can not only realize high-precision face motion capture, but also produce high-fidelity video output in various rendering scenes through the three-dimensional deformable face model coefficient after optimization.

Description

Technical Field

[0001] This invention relates to the field of computer graphics, and more particularly to a high-fidelity facial expression capture method based on binocular video. Background Technology

[0002] Facial expression capture without relying on explicit markers has been a major research direction in computer graphics and computer vision for the past three decades. Many applications, such as creating animations for characters in movies or enhancing interactive experiences in augmented reality and virtual reality, require driving digital avatars from video and demanding photorealistic results. A major challenge in these scenarios is the ability to transfer subtle facial expressions (e.g., eye contact, lip chewing, tongue movements, blinking) in real time under various lighting conditions.

[0003] There are two main technical challenges: First, it's necessary to accurately separate rigid body motion (i.e., 3D head pose) from non-rigid body motion (i.e., facial expressions). Second, we need to render the appearance of the person's digital avatar. A common approach to separating rigid and non-rigid body motion is to use facial tracking or alignment algorithms. However, in natural scenes, obtaining accurate and dense facial feature points is difficult due to the noise and varied lighting in photographs. Lighting variations always dominate facial detail information, and attempting to separate lighting from an image often destroys the information necessary to capture subtle facial movements. Early methods for rendering the user's appearance in a digital avatar used geometry-free or geometry-only techniques to generate facial animation, but these methods were significantly inadequate in recovering important facial details such as blinks or subtle wrinkles. Meanwhile, some implicit representation-based works, while capable of reproducing facial animation relatively well, struggled to generate the geometry and textures required for industrial art workflows, limiting their application in areas such as changing lighting, slimming faces, and smoothing skin imperfections. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by providing a high-fidelity facial expression capture method based on binocular video. Given a model and texture map of a person's neutral facial expression, along with a binocular video clip, the geometric representation of that person in each frame of the video can be quickly obtained. This invention innovatively combines facial feature point detection, binocular matching, and differentiable rendering techniques. By using gradient descent to optimize the coefficients of a 3D deformable face model, it further optimizes the geometric representation of facial expressions, achieving high-quality facial capture from videos.

[0005] The objective of this invention is achieved through the following technical solution:

[0006] A high-fidelity facial expression capture method based on binocular video includes the following steps:

[0007] (1) Record a binocular video starting with a neutral expression, and reconstruct the first frame using 3D reconstruction and texture reconstruction methods to obtain the geometry and diffuse texture of the first frame of the neutral expression.

[0008] (2) Use Google's MediaPipe library to detect facial feature points in the video, calculate the mean square error loss of the projection of the feature points on the eyes, nose and mouth and the geometrically corresponding points under the viewpoint, and optimize the initial rigid body transformation matrix and the coefficients of the three-dimensional deformable face model 3DMM by the Adam gradient descent method.

[0009] (3) The positions of some geometric points in two viewpoints are obtained by dense matching, and the depth is calculated by parallax. The accurate three-dimensional coordinates of the geometric points of the face are reconstructed. The mean square error loss obtained in step (2) is used to optimize the rigid body transformation matrix and 3DMM coefficients by Adam gradient descent method.

[0010] (4) Render the geometry of the first frame using ray-tracing-based differentiable rendering. Calculate the mean square error loss between the rendered result of the first frame geometry and the image of the neutral expression in the first frame of the video. Optimize the spherical harmonic function coefficients representing the lighting of the scene using the Adam gradient descent method. Then calculate the mean square error loss between the image under this viewpoint and the rendering result of the current geometry under the above lighting in the subsequent frames of the video. Optimize the 3DMM coefficients and the low-resolution light map under this viewpoint using the Adam gradient descent method to obtain the final geometry.

[0011] Furthermore, step (2) is implemented through the following sub-steps:

[0012] (2.1) Mark the geometric points of 40 feature points on the eyes, nose and mouth of the template character. The 40 feature points are as follows: 12 feature points on the upper and lower eyelids of the left and right eyes, 6 feature points on the left and right nostrils, 3 feature points on both sides of the nostrils, 1 feature point on the tip of the nose, 2 feature points on the corners of the mouth, 10 feature points on the outer edge of the upper lip and 6 feature points on the inner edge of the lower lip.

[0013] (2.2) Using Google's MediaPipe library, face feature point detection was performed on each frame of video from two perspectives to obtain the pixel coordinates of 40 feature points on the image for each frame of video from two perspectives.

[0014] (2.3) The intrinsic and extrinsic parameters of the two cameras are calculated using the dual-target calibration function of OpenCV. The current geometry is subjected to rigid body transformation and non-rigid body transformation represented by 3DMM coefficients. The 40 geometric points in the current geometry are projected under the two cameras to obtain the pixel coordinates of the 40 geometric points under the two cameras. The mean square error loss is calculated with the pixel coordinates obtained in step (2.2). The rigid body transformation matrix and 3DMM coefficients are optimized by the Adam gradient descent method. The rigid body transformation matrix is ​​obtained by optimizing quaternions and translations under the three coordinate axes, a total of 7 parameters.

[0015] Specifically, in step (2.3), the current geometry undergoes rigid body transformation and non-rigid body transformation represented by 3DMM coefficients, the mathematical expressions of which are as follows:

[0016] M=s*R*3DMM(coefficient,identity)+t;

[0017] Where s is the scaling factor, R is the rotation matrix of the rigid body transformation, t is the translation vector of the rigid body transformation, 3DMM is the 3DMM non-rigid body transformation, coefficient is the 3DMM coefficient, and identity represents the person's 3DMM representation.

[0018] Furthermore, the 40 geometric points mentioned in step (2.3) are projected under the two cameras to obtain the pixel coordinates of the 40 geometric points under the two cameras, and their mathematical expressions are as follows:

[0019]

[0020]

[0021] Where fx is the x-axis focusing distance, fy is the y-axis focusing distance, cx is the x-axis optical center coordinate, and cy is the y-axis optical center coordinate. These four values ​​together constitute the camera's intrinsic parameters. R is the camera's rotation matrix relative to the world coordinate system, t is the camera's translation vector relative to the world coordinate system, and R and t together constitute the camera's extrinsic parameters. M is the geometry's three-dimensional coordinates in the world coordinate system, and the final generated Mv is the pixel coordinates of the geometry in the camera's coordinate system.

[0022] Furthermore, the rigid body transformation matrix in step (2.3) is calculated by optimizing quaternions and seven parameters of translation across the three coordinate axes using the following expression:

[0023] q = xi + yj + zk + w;

[0024]

[0025]

[0026]

[0027] Where x, y, z and w are quaternions, tx, ty and tz are translations along the x, y and z axes respectively, and H is the rigid body transformation matrix.

[0028] Furthermore, step (3) is implemented through the following sub-steps:

[0029] (3.1) Fine-tuning the pre-trained model on the face dataset using a dense matching neural network to adapt it to the task of dense face matching;

[0030] (3.2) Project the neutral expression in the first viewpoint. Then, perform dense matching between the neutral expression image in the first viewpoint and each frame of video in the first viewpoint. Filter out the matches that correspond to the previously projected pixel coordinates to obtain the pixel position of the geometric point in the first viewpoint. Then, perform dense matching between each frame of video in the first viewpoint and each frame of video in the second viewpoint to filter out the matching pairs of pixels where the geometric point is located in the first viewpoint. Combine the camera intrinsic and extrinsic parameters and use OpenCV's triangulation reconstruction function to calculate the three-dimensional coordinates of the geometric point in each frame.

[0031] (3.3) The three-dimensional coordinates of the geometric points of each frame obtained in sub-step (3.2) are used to calculate the mean square error loss of the geometric points corresponding to the current geometry of each frame. The rigid body transformation matrix and 3DMM coefficients of each frame are optimized by the Adam gradient descent method. The rigid body transformation matrix is ​​further calculated by optimizing quaternions and translations under the three coordinate axes, a total of 7 parameters.

[0032] Furthermore, step (4) is implemented through the following sub-steps:

[0033] (4.1) Use a face region segmentation neural network to obtain the face region mask of each frame of video from two perspectives, remove hair, mouth, neck and below and background, and keep only the face and eyes as the real image for subsequent differentiable rendering optimization process;

[0034] (4.2) Use Mitsuba3 to apply the diffuse texture of the person’s appearance to the geometry obtained in the previous steps of the first frame, render it in two viewpoints, calculate the mean square error loss of each pixel by comparing the rendering results in the two viewpoints with the real images in the two viewpoints, and optimize the spherical harmonic function coefficients representing the lighting of the scene by the Adam gradient descent method.

[0035] (4.3) The illumination simulated by the spherical harmonic function coefficients obtained in step (4.2) is used as the ambient light of the entire scene. The geometry of each frame obtained in the previous step is applied with the appearance diffuse map and rendered in two viewpoints. The mean square error loss of each pixel is calculated by comparing the rendering results of the two viewpoints with the real image. The 3DMM coefficients of each frame and the low-resolution light map of that viewpoint are optimized by the Adam gradient descent method. The low-resolution light map is one-eighth the resolution of the diffuse map. It is upsampled to the resolution of the diffuse map and the values ​​are added to the diffuse map to serve as the final map for rendering.

[0036] (4.4) Apply the 3DMM coefficients of each frame obtained in step (4.3) and the rigid body transformation matrix of each frame obtained in step (3.3) to the neutral expression to obtain the geometric representation of each frame of the video.

[0037] Further, step (4.2) specifically involves: applying a non-rigid transformation to the obtained 3DMM coefficients and rigid body transformation matrix to calculate the current complete geometry; applying a diffuse texture of the person's appearance to the geometry of the first frame using Mitsuba3; and rendering the geometry from both perspectives using path tracing, as shown in the following expression:

[0038] L(x,ω0)=∫ Ω L i (x,ω i )f r (x,ω i ,ω0)(ω i ·n)dωi;

[0039] Where L(x,ω0) is the radiance of point x in the image along the direction ω0, i.e., the color exhibited by that pixel, and ω0 is the current viewing direction. i (x,ω i ) is the direction ω i The intensity of the incoming light, f r This is the bidirectional reflection distribution function, which describes how light rays entering from ω0 are reflected to ω. i ,(ω i ·n) is the dot product of the incident light direction and the geometric normal n at that point, ∫ Ω The integral is a hemispherical surface integral, which is then approximated using random sampling and Monte Carlo integration. 64 incident light rays are randomly sampled at each point. i ;

[0040] The mean squared error loss of each pixel is calculated between the rendered images from the two perspectives and the real image. The losses are then added together to optimize the coefficients of a third-order spherical harmonic function. The expression for the spherical harmonic function to generate ambient light is as follows:

[0041]

[0042] Where N is the maximum order, which is 3 in this method; L(θ, φ) represents the intensity of ambient light in a given direction; and c IM Y represents the coefficients of the spherical harmonic function. IM It is a spherical harmonic function, which can be obtained through pre-calculation;

[0043] The coefficients of the third-order spherical harmonic function representing the lighting in this scene were optimized using the Adam gradient descent method.

[0044] Furthermore, in step (4.3), the low-resolution lightmap is one-eighth the resolution of the diffuse map, and is divided into gain and bias maps. The final map is calculated using the following expression and used for rendering:

[0045] T = F(gain) * T0 + F(bias);

[0046] Where T0 is the diffuse face map, and F operation is bicubic interpolation, upsampled to the diffuse map resolution.

[0047] The beneficial effects of this invention are as follows:

[0048] The differentiable rendering framework of this invention satisfies the rendering equation, and the generated geometry supports industrial art workflow applications such as changing lighting, slimming faces, and beautifying skin surface blemishes. Experimental verification shows that each reconstructed frame can closely match the original image, and it supports the capture of various exaggerated expressions, including speaking, laughing, frowning, and grimacing. Attached Figure Description

[0049] Figure 1 This is a framework diagram of the binocular video high-fidelity facial expression capture method of the present invention;

[0050] Figure 2 This is a schematic diagram of the layout of facial feature points according to the present invention;

[0051] Figure 3 This is a comparison image between the rendering result of this invention and the original image;

[0052] Figure 4 This is a comparison diagram of the geometry generated by this invention and the geometry reconstructed in 3D. Detailed Implementation

[0053] The present invention will now be described in detail with reference to the accompanying drawings.

[0054] like Figure 1 As shown, the present invention provides a high-fidelity facial expression capture method based on binocular video, comprising the following steps:

[0055] Step 1: Record a binocular video starting with a neutral expression. Reconstruct the first frame using 3D reconstruction and texture reconstruction methods to obtain the geometry and diffuse texture of the first frame with the neutral expression. Use Google's MediaPipe library to detect facial feature points in the video, obtaining a total of 40 feature points on the eyes, nose, and mouth. Calculate the mean squared error loss by projecting the geometrically corresponding points onto the viewpoint. Optimize the initial rigid body transformation matrix and 3D deformable face model (3DMM) coefficients using the Adam gradient descent method.

[0056] The initial values ​​of the 3DMM coefficients and rigid body transformation matrix obtained in this step provide the basis for subsequent optimization, and are divided into the following sub-steps.

[0057] (1.1) Mark the geometric points of 40 feature points on the template figure, including the eyes, nose, and mouth. The positions of the 40 feature points are shown in the attached figure. Figure 2 The area shown in red (Note:) Figure 2 (The images are with the inventor's consent and do not involve any infringement of portrait rights.) The features are as follows: 12 feature points on the upper and lower eyelids of the left and right eyes, 6 feature points on the left and right nostrils, 3 feature points on both sides of the nostrils, 1 feature point on the tip of the nose, 2 feature points on the corners of the mouth, 10 feature points on the outer edge of the upper and lower lips, and 6 feature points on the inner edge.

[0058] (1.2) Using Google's MediaPipe library, facial feature point detection is performed on each frame of video from two perspectives, and the pixel coordinates corresponding to these 40 feature points are selected from the obtained pixel coordinates.

[0059] (1.3) The intrinsic and extrinsic parameters of the two cameras are calculated using the pre-shot calibration board and the two-camera calibration function of OpenCV. Specifically, 15 to 20 photos of the hand holding the chessboard calibration board need to be taken in the area that can be seen by both cameras. First, the pixel coordinates of the chessboard are obtained from the two perspectives using the findChessboardCorners function of OpenCV. Then, the intrinsic parameters of the two cameras are obtained using the calibrateCamera function of OpenCV. Finally, the extrinsic parameters of the two cameras are obtained using the stereoCalibrate function of OpenCV.

[0060] Next, the current geometry will undergo rigid body transformation and non-rigid body transformation represented by 3DMM coefficients, as shown in formula (1).

[0061] M=s*R*3DMM(coefficient,identity)+t (1)

[0062] Where s is the scaling factor, R is the rotation matrix of the rigid body transformation, t is the translation vector of the rigid body transformation, 3DMM is the 3DMM non-rigid body transformation, coefficient is the 3DMM coefficient, and identity represents the person's 3DMM representation.

[0063] The 40 geometric points described in step (1.1) of this geometry are projected onto two cameras as shown in formula (2):

[0064]

[0065]

[0066] Where fx is the x-axis focusing distance, fy is the y-axis focusing distance, cx is the x-axis optical center coordinate, and cy is the y-axis optical center coordinate; these four values ​​together constitute the camera's intrinsic parameters. R is the camera's rotation matrix relative to the world coordinate system, t is the camera's translation vector relative to the world coordinate system, and R and t together constitute the camera's extrinsic parameters. M is the geometric three-dimensional coordinates in the world coordinate system, and the final generated M... v Let be the pixel coordinates of this geometry under this camera.

[0067] The pixel coordinates of the 40 geometric points obtained are compared with the pixel coordinates of the frame image obtained in step (1.2) to calculate the mean square error loss. The rigid body transformation matrix and 3DMM coefficients are optimized by the Adam gradient descent method. The rigid body transformation matrix is ​​obtained by optimizing quaternions and translations under the three coordinate axes, a total of 7 parameters, through formula (3).

[0068] q = xi + yj + zk + w;

[0069]

[0070]

[0071]

[0072] Where x, y, z and w are quaternions, tx, ty and tz are translations along the x, y and z axes respectively, and H is the rigid body transformation matrix.

[0073] Step 2: Obtain the positions of some geometric points in two viewpoints through dense matching, calculate the depth using parallax, reconstruct the precise 3D coordinates of the geometric points in the face, calculate the mean square error loss with the corresponding points of the geometry obtained in Step 1, and optimize the rigid body transformation matrix and 3DMM coefficients using the Adam gradient descent method.

[0074] This step fully utilizes the depth information provided by the binocular video, providing crucial data for accurately reconstructing the 3D facial model, and is divided into the following sub-steps:

[0075] (2.1) The dense matching neural network is fine-tuned on a face dataset to better adapt the pre-trained model to the task of dense face matching. Specifically, this method uses LoFTR: Detector-Free LocalFeature Matching with Transformers to perform dense matching. This work uses a coarse-to-fine design to achieve dense matching. In the coarse stage, the image is downsampled to 1 / 8 resolution for matching. In the fine stage, the center pixel of each 8*8 patch in the coarse stage is selected for matching with the corresponding patch in another image. In this method, the pixel selection rule in the patch is modified. Instead of selecting the center pixel, a specific pixel in the patch can be specified for subsequent matching. Meanwhile, since the pre-trained model of the dense matching neural network did not perform well in matching faces, this method used geometry from 200 different people with different expressions. The Mitsuba3 renderer was used to calculate the depth map of each geometry in two viewpoints, and the same region was randomly cropped in both viewpoints. Pixel pairs with the same depth in the region in both viewpoints were considered as correct matches. A total of 5000 pairs of image patches were cropped, and together with the original image taken during acquisition, they were made into labels. The learning rate was adjusted to 1 / 10 of the original learning rate, and the pre-trained model was used as the initial value to relearn, resulting in a dense matching neural network for the face dataset that can match each pixel.

[0076] (2.2) Project the neutral expression using formula (2) from the first-person perspective to obtain M. v1 Next, the neutral expression images from the first-person perspective are densely matched with each frame of the video from the first-person perspective using the neural network obtained in step 1), specifying M. v1 Matching the pixel coordinates of each pixel in frame i yields the pixel coordinate matching pairs of the geometric point distribution in frame i. <P i Q i Next, perform dense matching between each frame of video from the first perspective and each frame of video from the second perspective, specifying pixel coordinates Qi for matching, to obtain the pixel coordinate matching pair of the geometric point binocular matching for frame i. i ,R i ​The intrinsic and extrinsic parameters of the two cameras, Qi and Ri, are combined. The 3D coordinates of the geometric points in each frame are calculated using the OpenCV triangulation reconstruction function triangulatePoints. The average depth is calculated based on the depth of all the obtained geometric points. The depth of each geometric point is subtracted from the average depth, and geometric points that exceed the threshold are discarded.

[0077] (2.3) The three-dimensional coordinates of the geometric points of each frame obtained in step (2.2) are used to calculate the mean square error loss of the geometric points corresponding to the current geometry of each frame. The rigid body transformation matrix and 3DMM coefficients of each frame are optimized by the Adam gradient descent method. The rigid body transformation matrix is ​​obtained by optimizing quaternions and translations under the three coordinate axes, a total of 7 parameters, through formula (3).

[0078] Step 3: Using ray tracing-based differentiable rendering, firstly, calculate the mean squared error loss for the first frame of the video (image of a neutral expression) and the rendering result of the geometry obtained in Step 2. Optimize the spherical harmonic function coefficients that can represent the lighting of the scene using the Adam gradient descent method. Next, calculate the mean squared error loss for the image and the rendering result of the current geometry under the same lighting and viewpoint for subsequent frames of the video. Optimize the 3DMM coefficients and low-resolution lightmap using the Adam gradient descent method to obtain the final geometry.

[0079] This step is the core technology of this invention, and it consists of the following sub-steps:

[0080] (3.1) A face region segmentation neural network is used to obtain the face region mask of each frame of video from two perspectives. The present invention uses a non-open source neural network method, which can obtain 14 masks: background, skin, nose, eyeball, left eye, right eye, left eyebrow, right eyebrow, left ear, right ear, upper lip, lower lip, hair, and neck. The mask from the skin to the lower lip is selected to make a black and white mask image, which is used to remove the image parts that do not participate in differentiable rendering and participate in the subsequent differentiable rendering optimization process.

[0081] (3.2) Apply formula (1) to the 3DMM coefficients and rigid body transformation matrix obtained in the previous steps to obtain the current complete geometry. Apply the diffuse texture of the person's appearance to the geometry of the first frame using Mitsuba3. Render the geometry in the two viewpoints using the path tracing method, as shown in formula (4).

[0082] L(x,ω0)=∫ Ω L i (x,ω i )f r (x,ω i ,ω0)(ω i ·n)dωi (4)

[0083] Where L(x,ω0) is the radiance of point x in the image along the direction ω0, i.e., the color exhibited by that pixel, and ω0 is the current viewing direction. i (x,ω i ) is the direction ω i The intensity of the incoming light, f r This is the bidirectional reflection distribution function, which describes how light rays entering from ω0 are reflected to ω. i ,(ω i ·n) is the dot product of the incident light direction and the geometric normal n at that point, ∫ Ω The integral is a hemispherical surface integral. In this method, it is approximated by random sampling and Monte Carlo integration. 64 incident light rays are randomly sampled at each point. i .

[0084] The mean square error loss of each pixel is calculated between the rendered images from the two perspectives and the real image. The losses are then added together to optimize a third-order spherical harmonic function coefficient, where the formula for generating ambient light using the spherical harmonic function is shown in formula (5).

[0085]

[0086] Where N is the maximum order, which is 3 in this method; L(θ, φ) represents the intensity of ambient light in a given direction; and c IM Y represents the coefficients of the spherical harmonic function. IM It is a spherical harmonic function, which can be obtained through pre-calculation.

[0087] The coefficients of the third-order spherical harmonic function that can represent the lighting of the scene were optimized using the Adam gradient descent method.

[0088] (3.3) The ambient light of the entire scene is simulated by the spherical harmonic function coefficients obtained in step (3.2). The geometry of each frame obtained in the previous step is applied with the diffuse reflection map of the person's appearance. The images are rendered in two different viewpoints. The mean square error loss of each pixel is calculated by comparing the rendering results of the two viewpoints with the real image. The 3DMM coefficients of each frame and the low-resolution light map of the viewpoint are optimized by the Adam gradient descent method. The low-resolution light map is one-eighth the resolution of the diffuse reflection map and is divided into gain and bias maps. The final map is calculated by formula (6) and participates in the rendering.

[0089] T = F(gain) * T0 + F(bias) (6)

[0090] Where T0 is the diffuse face map, and F operation is bicubic interpolation, upsampled to the diffuse map resolution.

[0091] (3.4) Apply the 3DMM coefficients of each frame obtained in step (3.3) and the rigid body transformation matrix of each frame obtained in step (2.3) to the neutral expression using formula (1) to obtain the geometric representation of each frame of the video.

[0092] Experimental results show that each frame reconstructed using this method closely matches the original image, supporting the capture of various exaggerated expressions including speaking, laughing, frowning, and grimacing. (See attached image.) Figure 3 This is a comparison image between the rendered result and the original image. Figure 4 This is a comparison image of the geometry generated by this method and the geometry reconstructed in 3D. The differentiable rendering framework of this invention satisfies the rendering equation, and the generated geometry supports industrial art workflow applications such as changing lighting, slimming faces, and beautifying skin surface blemishes.

[0093] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0094] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A method for high-fidelity facial expression capture based on binocular video, characterized in that, Includes the following steps: (1) Record a binocular video starting with a neutral expression, and reconstruct the first frame using 3D reconstruction and texture reconstruction methods to obtain the geometry and diffuse texture of the first frame with a neutral expression; (2) Use Google's MediaPipe library to detect facial feature points in the video, calculate the mean square error loss of the projection of the feature points on the eyes, nose and mouth and the geometrically corresponding points under each viewpoint, and optimize the initial rigid body transformation matrix and the coefficients of the three-dimensional deformable face model 3DMM by the Adam gradient descent method. (3) The positions of some geometric points in two perspectives are obtained by dense matching, and the depth is calculated by parallax. The accurate three-dimensional coordinates of the geometric points of the face are reconstructed. The mean square error loss between the reconstructed three-dimensional coordinates and the corresponding points of the current geometry is calculated. The rigid body transformation matrix and 3DMM coefficients are optimized by Adam gradient descent method. (4) Render the geometry of the first frame using ray tracing-based differentiable rendering. Calculate the mean square error loss between the rendered result of the first frame geometry and the image of the neutral expression in the first frame of the video. Optimize the spherical harmonic function coefficients representing scene lighting using the Adam gradient descent method. Then calculate the mean square error loss between the image under this viewpoint and the current geometry under the above lighting in the subsequent frames of the video. Optimize the 3DMM coefficients and the low-resolution light map under this viewpoint using the Adam gradient descent method to obtain the final geometry.

2. The method for high-fidelity facial expression capture based on binocular video according to claim 1, characterized in that, Step (2) is achieved through the following sub-steps: (2.1) Mark the geometric points of 40 feature points on the eyes, nose and mouth on the geometry of the template character. The 40 feature points are as follows: 12 feature points on the upper and lower eyelids of the left and right eyes, 6 feature points on the left and right nostrils, 3 feature points on both sides of the nostrils, 1 feature point on the tip of the nose, 2 feature points on the corners of the mouth, 10 feature points on the outer edge of the upper lip and 6 feature points on the inner edge of the lower lip. (2.2) Using Google's MediaPipe library, face feature point detection was performed on each frame of video from two perspectives to obtain the pixel coordinates of 40 feature points on the image for each frame of video from two perspectives. (2.3) The intrinsic and extrinsic parameters of the two cameras are calculated using the dual-target calibration function of OpenCV. The current geometry is subjected to rigid body transformation and non-rigid body transformation represented by 3DMM coefficients. The 40 geometric points in the current geometry are projected under the two cameras to obtain the pixel coordinates of the 40 geometric points under the two cameras. The mean square error loss is calculated with the pixel coordinates obtained in step (2.2). The rigid body transformation matrix and 3DMM coefficients are optimized by the Adam gradient descent method. The rigid body transformation matrix is ​​obtained by optimizing quaternions and translations under the three coordinate axes, a total of 7 parameters.

3. The method for high-fidelity facial expression capture based on binocular video according to claim 2, characterized in that, In step (2.3), the current geometry undergoes rigid body transformation and non-rigid body transformation represented by 3DMM coefficients. The mathematical expressions for these transformations are as follows: ； Where s is the scaling factor, R is the rotation matrix of the rigid body transformation, t is the translation vector of the rigid body transformation, 3DMM is the 3DMM non-rigid body transformation, coefficient is the 3DMM coefficient, and identity represents the person's 3DMM representation.

4. The method for high-fidelity facial expression capture based on binocular video according to claim 2, characterized in that, The 40 geometric points mentioned in step (2.3) are projected under the two cameras to obtain the pixel coordinates of the 40 geometric points under the two cameras. The mathematical expression is as follows: ； ； in, for Axis focusing distance, for Axis focusing distance, for Axial optical center coordinates, for The optical axis coordinates, these four values ​​together constitute the camera's intrinsic parameters. Let be the rotation matrix of the camera relative to the world coordinate system. Let be the translation vector of the camera relative to the world coordinate system. and Together they form the external parameters of the camera. The final generated three-dimensional coordinates of the geometry in the world coordinate system. Let be the pixel coordinates of this geometry under this camera.

5. The method for high-fidelity facial expression capture based on binocular video according to claim 2, characterized in that, The rigid body transformation matrix in step (2.3) is calculated by optimizing quaternions and seven parameters of translation across the three coordinate axes using the following expression: ； ； ； ； Where x, y, z and w are quaternions, tx, ty and tz are translations along the x, y and z axes respectively, and H is the rigid body transformation matrix.

6. The method for high-fidelity facial expression capture based on binocular video according to claim 1, characterized in that, Step (3) is achieved through the following sub-steps: (3.1) Fine-tuning the dense matching neural network on the face dataset to adapt the pre-trained model to the task of dense face matching; (3.2) Project the neutral expression in the first viewpoint. Then, perform dense matching between the neutral expression image in the first viewpoint and each frame of video in the first viewpoint. Filter out the matches that correspond to the previously projected pixel coordinates to obtain the pixel position of the geometric point in the first viewpoint. Then, perform dense matching between each frame of video in the first viewpoint and each frame of video in the second viewpoint to filter out the matching pairs of pixels where the geometric point is located in the first viewpoint. Combine the camera intrinsic and extrinsic parameters and use OpenCV's triangulation reconstruction function to calculate the three-dimensional coordinates of the geometric point in each frame. (3.3) The three-dimensional coordinates of the geometric points of each frame obtained in sub-step (3.2) are used to calculate the mean square error loss of the geometric points corresponding to the current geometry of each frame. The rigid body transformation matrix and 3DMM coefficients of each frame are optimized by the Adam gradient descent method. The rigid body transformation matrix is ​​further calculated by optimizing quaternions and translations under the three coordinate axes, a total of 7 parameters.

7. The method for high-fidelity facial expression capture based on binocular video according to claim 1, characterized in that, Step (4) is achieved through the following sub-steps: (4.1) Use a face region segmentation neural network to obtain the face region mask of each frame of video from two perspectives, remove hair, mouth, neck and below and background, and keep only the face and eyes as the real image for subsequent differentiable rendering optimization process; (4.2) Use Mitsuba3 to apply the diffuse texture of the person’s appearance to the geometry obtained in the previous steps of the first frame, render it in two viewpoints, calculate the mean square error loss of each pixel by comparing the rendering results in the two viewpoints with the real images in the two viewpoints, and optimize the spherical harmonic function coefficients representing the lighting of the scene by the Adam gradient descent method. (4.3) The illumination simulated by the spherical harmonic function coefficients optimized in step (4.2) is used as the ambient light of the entire scene. The geometry optimized in the previous steps of each frame is applied with the appearance diffuse map and rendered in two viewpoints. The mean square error loss of each pixel is calculated by comparing the rendering results in the two viewpoints with the real image. The 3DMM coefficients of each frame and the low-resolution light map in that viewpoint are optimized by the Adam gradient descent method. The low-resolution light map is one-eighth the resolution of the diffuse map. It is upsampled to the resolution of the diffuse map and the values ​​are added to the diffuse map to serve as the final map for rendering. (4.4) Apply the 3DMM coefficients of each frame obtained in step (4.3) and the rigid body transformation matrix of each frame obtained in step (3.3) to the neutral expression to obtain the geometric representation of each frame of the video.

8. The method for high-fidelity facial expression capture based on binocular video according to claim 7, characterized in that, Step (4.2) specifically involves: applying a non-rigid transformation to the obtained 3DMM coefficients and rigid body transformation matrix to calculate the current complete geometry; applying a diffuse texture of the person's appearance to the geometry of the first frame using Mitsuba3; and rendering the geometry from both perspectives using path tracing, as shown in the following expression: ； Where L(x, ω0) is the radiance of point x in the image along the direction ω0, i.e., the color exhibited by that pixel, and ω0 is the current viewing direction. i (x, ω i ) is the direction ω i The intensity of the incoming light, f r This is the bidirectional reflection distribution function, which describes how light rays entering from ω0 are reflected to ω. i ,(ω i ·n) is the dot product of the incident light direction and the geometric normal n at that point, ∫ Ω The integral is a hemispherical surface integral, which is then approximated using random sampling and Monte Carlo integration. 64 incident light rays are randomly sampled at each point. i ; The mean squared error loss of each pixel is calculated between the rendered images from the two perspectives and the real image. The losses are then added together to optimize the coefficients of a third-order spherical harmonic function. The expression for the spherical harmonic function to generate ambient light is as follows: ； Where N is the maximum order, which is 3, L(θ, ϕ) represents the intensity of ambient light in a given direction, and c IM Y represents the coefficients of the spherical harmonic function. IM It is a spherical harmonic function, which can be obtained through pre-calculation; The coefficients of the third-order spherical harmonic function representing the lighting in this scene were optimized using the Adam gradient descent method.

9. A method for high-fidelity facial expression capture based on binocular video according to claim 7, characterized in that, In step (4.3), the low-resolution lightmap is one-eighth the resolution of the diffuse map, and is divided into gain and bias maps. The final map is calculated using the following expression and used for rendering: ； Where T0 is the diffuse face map, and F operation is bicubic interpolation, upsampled to the diffuse map resolution.

Images