Lip sound synchronization method and system for Chinese based on audio driving

Abstract

The invention discloses a lip sound synchronization method and system for Chinese based on audio driving. The method comprises the following steps: judging whether video data input by a user meets requirements or not; extracting a face region from the video data meeting the requirements; extracting audio features from the video data meeting the requirements; and calculating internal and external parameters of the camera based on the extracted face region, inputting audio features, the internal and external parameters of the camera and 3D into a model for model training, generating a head region rendering picture based on an audio file provided by a user by using the trained model, and synchronizing the head region rendering picture to an original video to generate lip audio synchronous video data based on audio driving. The method has the advantages that voice and lip actions can be more naturally synchronized, and rigid or unnatural lip actions possibly occurring in a traditional method are reduced.

Description

Technical Field

[0001] The present invention relates to the field of computer graphics and speech-driven animation technology, and in particular to an audio-driven lip-sync method and system for Chinese. Background Art

[0002] Traditional 3D modeling software requires manual creation of 3D models and animations, which usually requires a lot of manual work and expertise. Traditional facial capture technology requires the use of professional equipment. Although it can achieve highly realistic facial animation, the equipment is expensive and complicated to operate. Existing rule-based lip synchronization systems can convert speech into lip movements according to predefined rules. It is relatively simple to implement, but may not be natural and accurate enough. Some existing deep learning-driven lip synchronization algorithms can automatically learn from data, but may lack the ability to handle complex dynamics and subtle movements. Summary of the invention

[0003] The object of the present invention is to provide a lip synchronization method and system for Chinese based on audio drive, so as to solve the above-mentioned problems existing in the prior art.

[0004] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0005] A lip synchronization method for Chinese based on audio drive, comprising the following steps:

[0006] S1. Face region extraction: Perform face detection and body posture estimation on the video data that meets the requirements input by the user, integrate the face detection results and body posture estimation results to determine the exact position of the head region boundary, and optimize the head region boundary to obtain accurate head region video;

[0007] S2, audio feature extraction: extract audio features from the video data that meets the requirements and input by the user, and convert the audio features related to the mouth shape changes into phoneme sequences;

[0008] S3, camera internal and external parameter acquisition: perform facial key point detection, facial expression detection and facial area segmentation on the video of the head area, reconstruct the face model based on the results of facial key point detection, facial expression detection and facial area segmentation, and calculate the camera internal and external parameters by tracking the changes of the reconstructed face model in consecutive frames using the PnP algorithm;

[0009] S4, model training: input the phoneme sequence, camera internal and external parameters and 3D coordinates into the NeRF network, and train the lip synchronization ability of the NeRF network by introducing the attention mechanism and setting the corresponding loss function. Use the gradient descent method to optimize the network parameters until the loss function converges to obtain the trained NeRF network;

[0010] S5, model reasoning: Input the audio file submitted by the user into the trained NeRF network, obtain the head area rendering image matching the voice, and synchronize it with the original video data, and finally generate audio-driven lip synchronization video data.

[0011] Preferably, before step S1, the process further includes checking the video data input by the user to determine whether the input video data meets the requirements, and if so, proceeding to the subsequent operation, otherwise, new video data needs to be re-input;

[0012] The judgment of video data includes the following aspects:

[0013] (1) Ensure that there is at least one human body or face in the video data;

[0014] (2) Ensure that the video data contains the complete facial area, which should be clearly visible without severe blur or loss of focus;

[0015] (3) Assess lighting conditions. The facial area should be evenly illuminated to avoid strong shadows or reflections.

[0016] (4) The face should not be blocked by objects, especially the lip area;

[0017] (5) The video data should have sufficient resolution to clearly see the subtle movements of the lips;

[0018] (6) The audio in the video data should be in Chinese and mainly consist of Chinese characters.

[0019] Preferably, step S2 specifically includes the following contents:

[0020] S11, using a face detection algorithm to slide windows of different sizes and positions on the image of the video data, analyzing the area within each window, determining whether it is a face and classifying it, performing deduplication detection and merging adjacent areas on the detected face area, obtaining an accurate and complete face detection result, and outputting the position and bounding box information of the face;

[0021] S12, using a pre-trained human posture estimation model to output the positions of human joints in the image of the video data, and correcting unreasonable joint positions through geometric constraints and prior knowledge of human structure, thereby inferring the human posture and approximate body contour;

[0022] S13, integrating the output of the face detection algorithm and the output of the human posture estimation model to obtain the relative position relationship between the head and the body and the coordinates of important key points of the head, so as to comprehensively determine the accurate position of the boundary of the head area, and continuously fine-tune the determined boundary of the head area to obtain an accurate head area video;

[0023] S14, post-processing the precise head region video to optimize the boundary of the head region.

[0024] Preferably, in step S2, Mel-frequency cepstral coefficients or other spectral features are used to capture the audio signal in the video data, and the features related to the mouth shape changes in the audio signal are converted into a phoneme sequence.

[0025] Preferably, step S3 specifically includes the following contents:

[0026] S31, using the trained facial key point algorithm to locate key feature points on the facial image;

[0027] S32, using the expression detection model based on CNN that has been trained on a large number of face image data sets to classify the expression state of the face;

[0028] S33, using semantic segmentation algorithm to divide the face into different regions;

[0029] S34, reconstructing a face model using three-dimensional modeling technology according to the acquired key feature points, expression classification and area division information;

[0030] S35. By tracking and reconstructing the position and posture changes of the face model in consecutive frames, the PnP algorithm is used to calculate the internal and external parameters of the camera according to the coordinates of the known face key points in the image and the corresponding three-dimensional space coordinates.

[0031] Preferably, the reconstruction of the face model is specifically as follows: firstly, the general shape and outline of the face are determined according to the key points; then, the deformation of the facial muscles and skin is adjusted in combination with the expression classification result to simulate the morphological changes under different expressions;

[0032] Among them, the shape parameters of the model include the length, width and height of each part of the face; the texture parameters describe the color and roughness characteristics of the skin; the posture parameters include the rotation, tilt and pitch angles of the face.

[0033] Preferably, step S4 specifically includes the following contents:

[0034] S41, passing the phoneme sequence, camera internal and external parameters and 3D coordinates as input data into the NeRF network;

[0035] S42, position encoding of 3D coordinates to enhance the network's ability to model spatial information;

[0036] S43, learning the mapping relationship between 3D coordinates and audio features through the network;

[0037] S44, output color information of each point;

[0038] S45. Introduce an attention mechanism in the edge area of ​​the portrait to improve the reconstruction quality of edge details; add a separate attention mechanism in the lip area to focus on the key information of lip synchronization; for blink control, the attention mechanism will focus on the action generation in the eye area;

[0039] S46. Design a loss function to calculate the difference between the reconstructed image and the real image, as well as the lip key point distance loss and the lip action classification loss;

[0040] S47. While maintaining the head reconstruction capability, the lip synchronization capability of the NeRF network is trained, and the network parameters are optimized using the gradient descent method until the loss function converges.

[0041] Preferably, step S5 specifically includes the following contents:

[0042] S51, load the trained NeRF network and its parameter weight file;

[0043] S52, performing feature extraction operation on the input audio data;

[0044] S53, inputting the extracted speech features into the trained NeRF network, the model processes the speech features through the encoder, and generates corresponding 3D head movement and lip action sequences through the decoder;

[0045] S54, using the density and color information output by the NeRF network to perform volume rendering and generate a 3D rendered image of the head;

[0046] S55, post-processing the 3D rendered image of the head and the lip movements to obtain a final rendered image;

[0047] S56: Map the final rendered image back to the original video data to achieve lip synchronization.

[0048] The present invention also aims to provide a lip synchronization system for Chinese based on audio drive, which can implement the above-mentioned method. The system comprises:

[0049] Facial region extraction module: performs face detection and human posture estimation on the video data that meets the requirements input by the user, integrates the face detection results and human posture estimation results to determine the exact position of the head region boundary, and optimizes the head region boundary to obtain accurate head region video;

[0050] Audio feature extraction module: extracts audio features from the video data that meets the requirements and converts the audio features related to the mouth shape changes into phoneme sequences;

[0051] Camera internal and external parameter acquisition module: perform facial key point detection, facial expression detection and facial area segmentation on the video of the head area, reconstruct the face model based on the results of facial key point detection, facial expression detection and facial area segmentation, and calculate the camera internal and external parameters using the PnP algorithm by tracking the changes of the reconstructed face model in consecutive frames;

[0052] Model training module: input the phoneme sequence, camera internal and external parameters and 3D coordinates into the NeRF network, and train the lip synchronization ability of the NeRF network by introducing the attention mechanism and setting the corresponding loss function. The network parameters are optimized by the gradient descent method until the loss function converges to obtain the trained NeRF network.

[0053] Model inference module: Input the audio file submitted by the user into the trained NeRF network, obtain the head area rendering image that matches the voice, and synchronize it with the original video data, and finally generate audio-driven lip synchronization video data.

[0054] The beneficial effects of the present invention are: 1. It can synchronize speech and lip movements more naturally, reducing the stiff or unnatural lip movements that may occur in traditional methods. 2. By applying an attention mechanism to specific areas, the algorithm can generate more detailed and realistic facial expressions, improving the overall realism of the video. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 is a flow chart of a lip synchronization method according to an embodiment of the present invention;

[0056] Figure 2 is a flowchart of facial region extraction in an embodiment of the present invention;

[0057] Figure 3 is a flow chart of obtaining camera internal and external parameters in an embodiment of the present invention;

[0058] Figure 4 is a flow chart of model training in an embodiment of the present invention;

[0059] Figure 5 4 is a flow chart of model reasoning in an embodiment of the present invention. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation methods described herein are only used to explain the present invention and are not used to limit the present invention.

[0061] like Figure 1 As shown, in this embodiment, a lip synchronization method for Chinese based on audio drive is provided, which can synchronize speech and lip movements more naturally, reducing the stiff or unnatural lip movements that may occur in traditional methods. By applying an attention mechanism to specific areas, the algorithm can generate more detailed and realistic facial expressions, improving the overall realism of the video. The method includes the following parts:

[0062] 1. Input video

[0063] The video data input by the user is checked, including human body detection, face detection and voice recognition, to determine whether the input video data meets the requirements. If it meets the requirements, the subsequent operation will be carried out, otherwise new video data needs to be re-entered.

[0064] The judgment of video data includes the following aspects:

[0065] (1) Ensure that there is at least one human body or face in the video data;

[0066] (2) Ensure that the video data contains the complete facial area, which should be clearly visible without severe blur or loss of focus;

[0067] (3) Assess lighting conditions. The facial area should be evenly illuminated to avoid strong shadows or reflections.

[0068] (4) The face should not be blocked by objects, especially the lip area;

[0069] (5) The video data should have sufficient resolution to clearly see the subtle movements of the lips;

[0070] (6) The audio in the video data should be in Chinese and mainly consist of Chinese characters.

[0071] 2. Facial Region Extraction

[0072] Perform face detection and body posture estimation on the video data that meets the requirements input by the user, integrate the face detection results and body posture estimation results to determine the exact position of the head area boundary, and optimize the head area boundary to obtain accurate head area video. Figure 2 As shown, it specifically includes the following contents:

[0073] 2.1. Use the face detection algorithm to slide windows of different sizes and positions on the image of the video data, analyze the area within each window, determine whether it is a face and classify it, perform deduplication detection and merge adjacent areas on the detected face area, obtain accurate and complete face detection results, and output the face location and bounding box information.

[0074] 2.2. Use the pre-trained human posture estimation model to output the positions of human joints in the image of video data, and correct unreasonable joint positions through some geometric constraints and prior knowledge of human structure, so as to infer the human posture and approximate body contour.

[0075] 2.3. The face detection algorithm outputs the position, shape and size of the bounding box of the face, which can determine the approximate area of ​​the head; the human posture estimation algorithm obtains the position of the joint points and the human posture, which can analyze the body information, such as judging the body direction, tilt angle, etc.; by integrating the two parts of data, the relative position relationship between the head and the body and the coordinates of the important key points of the head can be obtained, so as to comprehensively judge the accurate range of the boundary of the head area, and continuously fine-tune the initially determined boundary of the head area to obtain a more accurate head area video.

[0076] 2.4. Post-process the precise head area video, such as removing some noise or abnormal detection results, and using some smoothing or filtering operations to optimize the boundary of the head area.

[0077] In this embodiment, if the background needs to be replaced, a portrait cutout operation may be performed on the video data.

[0078] 3. Audio Feature Extraction

[0079] Audio features are extracted from the video data that meets the requirements and the audio features related to the mouth shape changes are converted into phoneme sequences.

[0080] Specifically, the audio signal in the video data is captured using Mel frequency cepstral coefficients or other spectral features, and the features related to mouth shape changes in the audio signal are converted into phoneme sequences. These phonemes are related to specific shapes such as mouth shapes.

[0081] 4. Obtaining the internal and external parameters of the camera

[0082] For the head area video obtained in the facial area extraction, the facial key point detection, facial expression detection and facial area division processes are performed. Based on the results of facial key point detection, facial expression detection and facial area division, the facial model is reconstructed. By tracking the changes of the reconstructed facial model in consecutive frames, the PnP algorithm is used to calculate the internal and external parameters of the camera. Figure 3As shown, it specifically includes the following contents:

[0083] 4.1. Use the trained facial key point algorithm to locate key feature points on the face image, such as eyes, nose, mouth and other parts. The algorithm will detect them according to their unique shape, contour and grayscale distribution, and mark the corresponding contour points. The location of these key points provides an accurate position reference for subsequent analysis.

[0084] 4.2. Use the CNN-based expression detection model that has been trained on a large number of face image datasets to classify facial expressions. For example, a happy expression may be manifested by an upturned mouth corner and narrowed eyes; a sad expression may be a drooping mouth corner and wrinkled eyebrows; an angry expression may be a frowning brow and wide eyes, etc. The model classifies based on these learned feature patterns to provide more information for subsequent reconstruction and parameter estimation.

[0085] 4.3. Use semantic segmentation algorithms to divide the face into different regions. For example, the skin region has relatively uniform color and texture; the hair region has darker color and unique texture; the eye region has obvious eyeball and eye white features; and the mouth region has the shape and color features of the lips.

[0086] 4.4. Reconstruct the face model using 3D modeling technology based on the key feature points, expression classification and region division information obtained. First, determine the approximate shape and outline of the face based on the key points. Then, adjust the deformation of facial muscles and skin based on the expression classification results to simulate the morphological changes under different expressions. Shape parameters may include the length, width and height of each part of the face; texture parameters describe the color, roughness and other characteristics of the skin; posture parameters represent the rotation, tilt and pitch angles of the face.

[0087] 4.5. In the video, the position and posture of the face will change over time. First, the position and posture changes of the face model in consecutive frames are reconstructed by tracking. Then, the PnP algorithm is used to calculate the camera's intrinsic parameters (such as focal length, principal point, etc.) and extrinsic parameters (such as rotation matrix, translation vector, etc.) based on the known coordinates of the face key points in the image and the corresponding three-dimensional space coordinates.

[0088] 5. Model Training

[0089] The phoneme sequence, camera internal and external parameters and 3D coordinates are input into the NeRF network for model training. In special areas, such as the edge of the portrait, the control of blinking and the control of lip sync, separate attention mechanisms are added to obtain better rendering results and control capabilities. The training process includes two steps: one is to train the reconstruction ability of the head, and the other is to train the lip sync ability. Figure 4As shown, it specifically includes the following contents:

[0090] 5.1. Input data: The phoneme sequence, camera internal and external parameters and 3D coordinates are passed into the NeRF network as input data.

[0091] 5.2. Position encoding: Encode the position of 3D coordinates to enhance the network's ability to model spatial information.

[0092] 5.3. Fully connected layer: The network learns the mapping relationship between 3D coordinates and audio features.

[0093] 5.4. Color coding layer: output the color information of each point.

[0094] 5.5. Add attention mechanism: Introduce attention mechanism in the edge area of ​​the portrait to improve the reconstruction quality of edge details; add a separate attention mechanism in the lip area to focus on the key information of lip synchronization; for blink control, the attention mechanism will focus on action generation in the eye area.

[0095] 5.6. Loss function: Calculate the difference between the reconstructed image and the real image, such as L1 loss, L2 loss, etc.; design lip synchronization loss function, such as lip key point distance loss, lip movement classification loss, etc.

[0096] 5.7. Iterative training: While maintaining the head reconstruction capability, the lip synchronization capability of the NeRF network is trained, and the network parameters are optimized using the gradient descent method until the loss function converges.

[0097] 6. Model Reasoning

[0098] The audio file submitted by the user is input into the trained NeRF network to obtain the head area rendering image matching the speech, and synchronize it with the original video data, and finally generate audio-driven lip synchronization video data. Figure 5 As shown, it specifically includes the following contents:

[0099] 6.1. Model loading: Load the trained NeRF network and its parameter weight file.

[0100] 6.2. Feature extraction: Perform feature extraction on the input audio data. The feature extraction process is the same as the extraction process in the third part.

[0101] 6.3. Forward propagation: The extracted speech features are input into the trained NeRF network. The model processes the speech features through the encoder and generates the corresponding 3D head movement and lip action sequences through the decoder.

[0102] 6.4. Volume rendering: Use the density and color information output by the NeRF network for volume rendering to generate a 3D rendered image of the head.

[0103] 6.5. Post-processing: Post-process the 3D rendered image of the head and the lip movements, including smoothing or interpolation, to improve the naturalness of the movements and obtain the final rendered image.

[0104] 6.6. Synchronized output: The final rendered image contains head movements and lip movements synchronized with the input audio, mapping the movement sequence back to the original video data to achieve lip synchronization.

[0105] In this embodiment, if the background or transparent channel needs to be replaced, the portrait cutout result obtained in the second part can be used to perform corresponding replacement processing, and finally generate an audio-driven lip-sync video.

[0106] In this embodiment, a lip synchronization system for Chinese based on audio drive is provided, and the system can implement the method described above. The system includes:

[0107] (1) Facial region extraction module: Perform face detection and body posture estimation on the video data that meets the requirements input by the user, integrate the face detection results and body posture estimation results to determine the exact position of the head region boundary, and optimize the head region boundary to obtain accurate head region video.

[0108] (2) Audio feature extraction module: extracts audio features from the video data that meets the requirements and converts the audio features related to mouth shape changes into phoneme sequences.

[0109] (3) Camera internal and external parameter acquisition module: Face key point detection, face expression detection and face area segmentation are performed on the video of the head area. Based on the results of face key point detection, face expression detection and face area segmentation, the face model is reconstructed. By tracking the changes of the reconstructed face model in consecutive frames, the camera internal and external parameters are calculated using the PnP algorithm.

[0110] (4) Model training module: The phoneme sequence, camera internal and external parameters, and 3D coordinates are input into the NeRF network. The lip synchronization ability of the NeRF network is trained by introducing the attention mechanism and setting the corresponding loss function. The network parameters are optimized using the gradient descent method until the loss function converges to obtain a trained NeRF network.

[0111] (5) Model inference module: The audio file submitted by the user is input into the trained NeRF network, the rendered image of the head area matching the speech is obtained, and it is synchronized with the original video data, and finally the audio-driven lip-sync video data is generated.

[0112] By adopting the above technical solution disclosed in the present invention, the following beneficial effects are obtained:

[0113] The present invention provides an audio-driven lip synchronization method and system for Chinese, which can synchronize speech and lip movements more naturally, reducing the stiff or unnatural lip movements that may occur in traditional methods. By applying an attention mechanism to specific areas, the algorithm can generate more detailed and realistic facial expressions, improving the overall realism of the video.

[0114] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be considered as the scope of protection of the present invention.

Claims

1. A lip synchronization method for Chinese based on audio drive, characterized by: The following steps are included: S1. Face region extraction: Perform face detection and body posture estimation on the video data that meets the requirements input by the user, integrate the face detection results and body posture estimation results to determine the exact position of the head region boundary, and optimize the head region boundary to obtain accurate head region video; S2, audio feature extraction: extract audio features from the video data that meets the requirements and input by the user, and convert the audio features related to the mouth shape changes into phoneme sequences; S3, camera internal and external parameter acquisition: perform facial key point detection, facial expression detection and facial area segmentation on the video of the head area, reconstruct the face model based on the results of facial key point detection, facial expression detection and facial area segmentation, and calculate the camera internal and external parameters by tracking the changes of the reconstructed face model in consecutive frames using the PnP algorithm; S4, model training: input the phoneme sequence, camera internal and external parameters and 3D coordinates into the NeRF network, and train the lip synchronization ability of the NeRF network by introducing the attention mechanism and setting the corresponding loss function. Use the gradient descent method to optimize the network parameters until the loss function converges to obtain the trained NeRF network; S5, model reasoning: Input the audio file submitted by the user into the trained NeRF network, obtain the head area rendering image matching the voice, and synchronize it with the original video data, and finally generate audio-driven lip synchronization video data.

2. The lip synchronization method for Chinese based on audio drive according to claim 1, characterized in that: Before step S1, the process also includes checking the video data input by the user to determine whether the input video data meets the requirements. If so, the subsequent operation is performed; otherwise, new video data needs to be re-input; The judgment of video data includes the following aspects: (1) Ensure that there is at least one human body or face in the video data; (2) Ensure that the video data contains the complete facial area, which should be clearly visible without severe blur or loss of focus; (3) Assess lighting conditions. The facial area should be evenly illuminated to avoid strong shadows or reflections. (4) The face should not be blocked by objects, especially the lip area; (5) The video data should have sufficient resolution to clearly see the subtle movements of the lips; (6) The audio in the video data should be in Chinese and mainly consist of Chinese characters.

3. The lip synchronization method for Chinese based on audio drive according to claim 1, characterized in that: Step S2 specifically includes the following contents: S11, using a face detection algorithm to slide windows of different sizes and positions on the image of the video data, analyzing the area within each window, determining whether it is a face and classifying it, performing deduplication detection and merging adjacent areas on the detected face area, obtaining an accurate and complete face detection result, and outputting the position and bounding box information of the face; S12, using a pre-trained human posture estimation model to output the positions of human joints in the image of the video data, and correcting unreasonable joint positions through geometric constraints and prior knowledge of human structure, thereby inferring the human posture and approximate body contour; S13, integrating the output of the face detection algorithm and the output of the human posture estimation model to obtain the relative position relationship between the head and the body and the coordinates of important key points of the head, so as to comprehensively determine the accurate position of the boundary of the head area, and continuously fine-tune the determined boundary of the head area to obtain an accurate head area video; S14, post-processing the precise head region video to optimize the boundary of the head region.

4. The lip synchronization method for Chinese based on audio drive according to claim 1, characterized in that: In step S2, the audio signal in the video data is captured using Mel-frequency cepstral coefficients or other spectral features, and the features related to the mouth shape changes in the audio signal are converted into a phoneme sequence.

5. The lip synchronization method for Chinese based on audio drive according to claim 1, characterized in that: Step S3 specifically includes the following contents: S31, using the trained facial key point algorithm to locate key feature points on the facial image; S32, using the expression detection model based on CNN that has been trained on a large number of face image data sets to classify the expression state of the face; S33, using semantic segmentation algorithm to divide the face into different regions; S34, reconstructing a face model using three-dimensional modeling technology according to the acquired key feature points, expression classification and area division information; S35. By tracking and reconstructing the position and posture changes of the face model in consecutive frames, the PnP algorithm is used to calculate the internal and external parameters of the camera according to the coordinates of the known face key points in the image and the corresponding three-dimensional space coordinates.

6. The lip synchronization method for Chinese based on audio drive according to claim 5, characterized in that: The reconstruction of the face model is as follows: first, the general shape and outline of the face are determined based on the key points; then, the deformation of the facial muscles and skin is adjusted based on the expression classification results to simulate the morphological changes under different expressions; Among them, the shape parameters of the model include the length, width and height of each part of the face; the texture parameters describe the color and roughness characteristics of the skin; the posture parameters include the rotation, tilt and pitch angles of the face.

7. The lip synchronization method for Chinese based on audio drive according to claim 1, characterized in that: Step S4 specifically includes the following contents: S41, passing the phoneme sequence, camera internal and external parameters and 3D coordinates as input data into the NeRF network; S42, position encoding of 3D coordinates to enhance the network's ability to model spatial information; S43, learning the mapping relationship between 3D coordinates and audio features through the network; S44, output color information of each point; S45. Introduce an attention mechanism in the edge area of ​​the portrait to improve the reconstruction quality of edge details; add a separate attention mechanism in the lip area to focus on the key information of lip synchronization; for blink control, the attention mechanism will focus on the action generation in the eye area; S46. Design a loss function to calculate the difference between the reconstructed image and the real image, as well as the lip key point distance loss and the lip action classification loss; S47. While maintaining the head reconstruction capability, the lip synchronization capability of the NeRF network is trained, and the network parameters are optimized using the gradient descent method until the loss function converges.

8. The lip synchronization method for Chinese based on audio drive according to claim 1, characterized in that: Step S5 specifically includes the following contents: S51, load the trained NeRF network and its parameter weight file; S52, performing feature extraction operation on the input audio data; S53, inputting the extracted speech features into the trained NeRF network, the model processes the speech features through the encoder, and generates corresponding 3D head movement and lip action sequences through the decoder; S54, using the density and color information output by the NeRF network to perform volume rendering and generate a 3D rendered image of the head; S55, post-processing the 3D rendered image of the head and the lip movements to obtain a final rendered image; S56: Map the final rendered image back to the original video data to achieve lip synchronization.

9. An audio-driven lip synchronization system for Chinese, characterized in that: The system can implement the method described in any one of claims 1 to 8 above, and the system includes: Facial region extraction module: performs face detection and human posture estimation on the video data that meets the requirements input by the user, integrates the face detection results and human posture estimation results to determine the exact position of the head region boundary, and optimizes the head region boundary to obtain accurate head region video; Audio feature extraction module: extracts audio features from the video data that meets the requirements and converts the audio features related to the mouth shape changes into phoneme sequences; Camera internal and external parameter acquisition module: perform facial key point detection, facial expression detection and facial area segmentation on the video of the head area, reconstruct the face model based on the results of facial key point detection, facial expression detection and facial area segmentation, and calculate the camera internal and external parameters using the PnP algorithm by tracking the changes of the reconstructed face model in consecutive frames; Model training module: input the phoneme sequence, camera internal and external parameters and 3D coordinates into the NeRF network, and train the lip synchronization ability of the NeRF network by introducing the attention mechanism and setting the corresponding loss function. The network parameters are optimized by the gradient descent method until the loss function converges to obtain the trained NeRF network. Model inference module: Input the audio file submitted by the user into the trained NeRF network, obtain the head area rendering image that matches the voice, and synchronize it with the original video data, and finally generate audio-driven lip synchronization video data.

Images