Virtual anchor real-time driving system based on facial motion capture

Abstract

The application discloses a virtual anchor real-time driving system based on facial motion capture, aims to solve the problems of insufficient precision, lack of naturalness and weak scene adaptation of virtual anchor facial motion in the prior art, and through multi-thread parallel collection of audio and video streams, optimizes data quality through adaptive preprocessing; adopts multi-dimensional facial feature collaborative extraction, double-branch phoneme recognition, face function area semantic segmentation and confidence dynamic weight adjustment technology, realizes accurate representation and collaborative fusion of expression and lip feature; finally, through the lip-expression collaborative driving mechanism, the real-time driving characteristics of the adaptive virtual image are output; thereby effectively improving the precision and naturalness of virtual anchor facial motion, enhancing the complex scene adaptability, giving consideration to real-time response and low-cost deployment, and being applicable to virtual live broadcast, online education and other scenes, and having good application value.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and more particularly to a real-time driving system for virtual anchors based on facial motion capture. Background Technology

[0002] With the rapid development of the digital economy, virtual anchors, as a core carrier of human-computer interaction, directly determine user experience and commercial success based on their technological maturity. Currently, real-time driven technology for virtual anchors based on facial motion capture has gradually replaced traditional optical marker motion capture technology, becoming the mainstream technology. Its core principle is to extract facial features and pronunciation-related information by collecting real-person facial video and audio streams, thereby driving the virtual avatar to generate synchronized facial expressions and lip movements. Currently, most mainstream solutions in the industry are based on data collection using monocular cameras or depth cameras, using deep learning algorithms to achieve integrated processing of feature extraction, phoneme recognition, and virtual avatar driving. For example, the real-time expression-driven solutions using monocular cameras launched by companies such as Tencent and Baidu have been initially implemented in scenarios such as live-streaming e-commerce and virtual customer service.

[0003] However, research has revealed that existing real-time driving technologies for virtual anchors based on facial motion capture still suffer from numerous technical bottlenecks and shortcomings. Specifically, current virtual avatar driving technologies rely on static rules to coordinate conflicts between facial expressions and lip movements in multimodal fusion, resulting in unsatisfactory performance. Current computer vision technology lacks dynamic adaptation capabilities in facial region function recognition, making it difficult to distinguish between emotional expression attributes and vocalization attributes of different facial regions during speech in real time. This leads to functional confusion when fusion of visual and audio features, affecting the naturalness of the driving process. Summary of the Invention

[0004] This invention overcomes the shortcomings of the prior art and provides a real-time driving system for virtual anchors based on facial motion capture.

[0005] To achieve the above objectives, the technical solution adopted by this invention is as follows: a real-time driving system for virtual anchors based on facial motion capture, comprising: a processor and a memory; the memory stores a computer program, which, when executed by the processor, performs the following steps:

[0006] S1. Acquire the user's audio stream and facial video stream in parallel, and perform computer vision preprocessing on the facial video stream;

[0007] S2. Visual feature extraction is performed on the facial video stream to obtain dynamic visual features of key facial regions, overall facial expression visual features, and visual semantic features of the mouth shape region.

[0008] S3. Perform phoneme identification on the audio stream to obtain a phoneme sequence;

[0009] S4. Input the dynamic visual features of the key facial regions, the overall facial expression visual features, the visual semantic features of the mouth shape region, and the phoneme sequence into the facial region visual semantic segmentation and functional discrimination network, and output the facial functional region semantic segmentation map and the visual discrimination confidence map.

[0010] S5. Based on the semantic segmentation map of the facial functional region and the visual discrimination confidence map, the weights of the overall facial expression visual features are adjusted using a visual smoothing algorithm to generate driving features;

[0011] S6. Based on the driving features, drive the virtual avatar model to generate and output a virtual anchor video stream with coordinated lip movements and facial expressions.

[0012] In a preferred embodiment of the present invention, the computer vision preprocessing includes:

[0013] S101. Gaussian bilateral filtering algorithm is used to suppress intra-frame noise in facial video stream, and illumination compensation is achieved through adaptive histogram equalization.

[0014] S102. Based on the MTCNN algorithm, locate and crop the facial ROI region, and extract the RGB three-channel pixel data of the ROI region;

[0015] S103. The RGB three-channel pixel data is standardized into video frames of a fixed size, and the video frames are used as input data for the visual feature extraction step.

[0016] In a preferred embodiment of the present invention, a neural network model is trained using training data with regional semantic annotations; the training data includes speaking videos containing various phonemes and expressions, as well as manually annotated labels in the video frames that indicate whether the movement of a specific facial region mainly serves pronunciation or emotional expression; the neural network model learns and establishes a facial region visual semantic segmentation and functional discrimination network between phoneme sequences, facial expression features and facial region movement semantics through training;

[0017] The facial region visual semantic segmentation and function discrimination network includes: a visual feature alignment layer, a U-Net improved segmentation layer, and a confidence calculation layer;

[0018] The visual feature alignment layer maps the dynamic visual features of the key facial regions, the overall facial expression visual features, the visual semantic features of the mouth shape region, and the phoneme sequence of different dimensions to the same feature space.

[0019] The U-Net improved segmentation layer outputs functional attribute labels for each region through an encoder-decoder structure;

[0020] The confidence calculation layer calculates the visual confidence of the functional attribute labels of each region based on the IoU loss function, and generates the semantic segmentation map and visual discrimination confidence map of the facial functional region.

[0021] The facial functional region semantic segmentation map represents the probability distribution of each small region of the face being identified as an expression-dominant region or a lip-dominant region at the current moment; the visual discrimination confidence map represents the degree of confidence of the conflict detection and facial region visual semantic segmentation and functional discrimination network in determining each region in the facial functional region semantic segmentation map.

[0022] In a preferred embodiment of the present invention, the process of generating the driving feature includes:

[0023] S501, the probability value of each region in the facial functional region semantic segmentation map belonging to the expression-dominant region is combined with the confidence value of the corresponding region in the visual discrimination confidence map to calculate the fusion weight coefficient of each region.

[0024] S502, the overall facial expression visual features are spatially aligned and feature sliced ​​according to the division of the face region to obtain the local expression feature vector corresponding to each region;

[0025] S503, multiply the local expression feature vector of each region by the fusion weight coefficient corresponding to each region to obtain the weighted local expression features;

[0026] S504, aggregate the local facial expression features of all regions to generate the driving feature.

[0027] In a preferred embodiment of the present invention, the process of obtaining the phoneme sequence includes:

[0028] S301. The audio stream is segmented into frames and features are extracted to obtain an audio feature sequence;

[0029] S302. Input the audio feature sequence into the phoneme recognition model and output the phoneme sequence, which represents the phoneme categories and their time boundary information arranged in chronological order.

[0030] In a preferred embodiment of the present invention, the phoneme recognition model is a dual-branch encoder structure, comprising:

[0031] The first encoding branch is used to identify and output the phoneme sequence;

[0032] The second coding branch shares some low-level features with the first coding branch and is used to extract generalized articulation motion features related to facial muscle movements.

[0033] The generalized articulation action features and the phoneme sequence together serve as the input to the facial region visual semantic segmentation and functional discrimination network in S4.

[0034] In a preferred embodiment of the present invention, the visual feature extraction process includes:

[0035] S201. Process each frame of the facial video stream using a face detection and alignment model to obtain a standardized face image;

[0036] S202. Input the face image into the facial expression encoder to extract the overall facial expression features;

[0037] S203. Based on the coordinates of the key facial points, the dynamic visual features of the key facial regions, the overall facial expression visual features, and the visual semantic features of the mouth shape region are cropped from the face image.

[0038] In a preferred embodiment of the present invention, the visual semantic features of the lip shape region and the phoneme sequence are input into a lip shape driving generation network to generate lip shape driving parameters synchronized with pronunciation; then the driving features and the lip shape driving parameters are combined to drive the virtual avatar model.

[0039] In a preferred embodiment of the present invention, the facial functional region semantic segmentation map is represented in matrix form. Each element value in the matrix corresponds to a predefined small region in the face. The magnitude of each element value represents the probability value of a predefined small region belonging to the expression-dominant region. The probability value close to 1 indicates that it is judged as the expression-dominant region, and the probability value close to 0 indicates that it is judged as the lip-shape-dominant region.

[0040] In a preferred embodiment of the present invention, the processing procedure of the visual smoothing algorithm includes:

[0041] S511. Take the probability value of a certain region in the semantic segmentation map of the facial functional region at the current time and the probability value of the same region at the previous time, and perform a weighted average, wherein the weight of the weighted average is determined by the confidence value of the corresponding region in the visual discrimination confidence map at the current time.

[0042] S512. When the confidence value is high, the weighted average depends on the determination at the current time.

[0043] S513. When the confidence value is low, the weighted average depends on the determination of historical time.

[0044] This invention addresses the shortcomings of the prior art and has the following beneficial effects:

[0045] (1) By using multi-dimensional visual feature collaborative extraction technology, dynamic features of key facial regions, overall facial expression features, and lip shape semantic features are captured simultaneously, breaking through the limitations of single feature extraction in existing technologies and laying a data foundation for driving accuracy. At the same time, by using an integrated network of facial region visual semantic segmentation and functional discrimination, the dynamics of the face are divided into expression-dominant and lip shape-dominant regions, and feature weights are dynamically adjusted based on confidence, fundamentally solving the problem of conflict between expression and lip shape driving. Combined with phoneme recognition technology with a dual-branch coding structure, it can not only accurately output phoneme sequences to match lip shapes, but also extract generalized pronunciation action features and establish a connection between pronunciation and facial muscle movement, making the linkage between lip shape movement and facial muscle more accurate. Ultimately, it achieves accurate matching of virtual anchor's expression, lip shape, and facial muscle movement, avoiding problems such as lip shape misalignment and stiff expression.

[0046] (2) The lip-syncing control parameters are accurately generated and synchronized with the pronunciation through a lip-syncing-driven generative network. Then, the facial features are optimized for temporal smoothing by combining confidence-based visual smoothing and dynamic weight adjustment techniques, so that the transition between facial expressions and lip movements is natural and conforms to the facial expression habits of real humans. Among them, the visual smoothing algorithm uses a weighted average of current and historical confidence to avoid fluctuations in feature judgment while ensuring real-time performance, making the facial movements of the virtual anchor (such as emotion switching and continuous pronunciation) more coherent and closer to real people. The adaptive fusion coefficient dynamically adjusts the driving weight according to the amplitude of lip movements, further improving the coordination and realism of facial movements and significantly optimizing the user viewing experience.

[0047] (3) Through adaptive preprocessing optimization technology, a combination of Gaussian bilateral filtering for noise reduction, CLAHE illumination compensation, and MTCNN precise cropping is adopted to effectively cope with complex scene interferences such as changes in light intensity, intra-frame noise, and facial angle shifts, ensuring high quality and standardization of input data. At the same time, based on the confidence-based dynamic weight adjustment mechanism, in extreme cases such as sudden changes in light or facial occlusion, the system prioritizes historical stable judgment results to avoid driving feature fluctuations and ensure the stability of the virtual anchor's facial movements. In addition, the standardized feature extraction and driving process can adapt to audio and video inputs of different resolutions and different types of virtual avatar models, greatly expanding the application scenarios of the system.

[0048] (4) By using dual threads to collect video streams and audio streams respectively, and combining the timestamp synchronization algorithm, the synchronization error is controlled within ±10ms, which greatly reduces the data processing delay. At the same time, the computational efficiency is optimized while ensuring accuracy, and redundant computation is avoided, so that the system can respond to audio and video inputs in real time, quickly generate driving features and drive virtual images, fully meeting the needs of latency-sensitive scenarios such as live streaming and real-time interaction.

[0049] (5) This solution uses computer vision technology to achieve accurate and real-time recognition of facial functional areas, providing a visually driven fusion basis for virtual anchors and ensuring that the virtual anchors' lip movements and expressions are coordinated and natural. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is a system flowchart of a preferred embodiment of the present invention; Detailed Implementation

[0052] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0053] like Figure 1 As shown, the real-time driving system for a virtual anchor based on facial motion capture includes: a processor and a memory; the memory stores a computer program, which, when executed by the processor, performs the following steps:

[0054] S1. Acquire the user's audio stream and facial video stream in parallel, and perform computer vision preprocessing on the facial video stream.

[0055] Furthermore, computer vision preprocessing includes:

[0056] S101. Gaussian bilateral filtering algorithm is used to suppress intra-frame noise in facial video stream, and illumination compensation is achieved through adaptive histogram equalization.

[0057] S102. Based on the MTCNN algorithm, locate and crop the facial ROI region, and extract the RGB three-channel pixel data of the ROI region;

[0058] S103. Standardize the RGB three-channel pixel data into fixed-size video frames, and use the video frames as input data for the visual feature extraction step.

[0059] It should be noted that obtaining the user's audio stream and facial video stream requires the user's consent. In existing virtual anchor driving technologies, most solutions extract only single-dimensional visual features (such as extracting only overall facial expression features or focusing only on lip-shape features), resulting in poor coordination between the virtual avatar's facial expressions and lip movements, and insufficient accuracy in action driving. This technology achieves efficient collaborative extraction of multi-dimensional features through three steps: First, it uses a face detection and alignment model to standardize each frame of the facial video stream, ensuring consistency in subsequent feature extraction; second, it extracts overall facial expression features through an expression feature encoder, capturing macroscopic changes in facial emotions; third, based on precise facial key point coordinates, it selectively crops dynamic features of key facial regions (such as the eyes, eyebrows, and other core areas affecting emotional expression) and semantic features of the lip-shape region from the standardized facial image, achieving comprehensive feature coverage of macroscopic expression + local dynamics + lip-shape semantics. This technology solves the pain point of existing technologies' single-dimensional feature structure, allowing the subsequent driving process to match overall emotions while accurately reproducing local action details, laying a data foundation for natural virtual avatar driving.

[0060] In one embodiment, the 68-point facial landmark detection model from the Dlib library is used to obtain the set of facial landmark coordinates for each frame of the image. ,in Indicates the first The pixel coordinates of key points are used. The face image is standardized to normalize the key point coordinates to... The normalization formula for the interval is: , ,in , Each of the key points Minimum and maximum values ​​of coordinates , Similarly, based on the preset region division rules, extract key facial regions (eyes: key points). Eyebrows: Key Points ), mouth shape area (key point) The bounding boxes of each region are determined by coordinate calculation, and the coordinates of the bounding boxes are: , , , The corresponding region image is cropped based on the bounding box. The cropped region image is then input into a lightweight CNN encoder, which outputs dynamic visual features of key facial regions. Visual semantic features of the mouth shape region Simultaneously, the standardized face image is input into the ResNet-18 expression feature encoder, which outputs the overall facial expression features. All three dimensions are unified at 64×128.

[0061] S2. Visual features are extracted from the facial video stream to obtain dynamic visual features of key facial regions, overall facial expression visual features, and visual semantic features of the mouth shape region.

[0062] Furthermore, the visual feature extraction process includes:

[0063] S201. Process each frame of facial video stream using a face detection and alignment model to obtain a standardized face image;

[0064] S202. Input the face image into the facial expression encoder to extract the overall facial expression features;

[0065] S203. Based on the coordinates of facial key points, extract the dynamic visual features of key facial regions, the overall facial expression visual features, and the visual semantic features of the mouth shape region from the facial image.

[0066] Furthermore, the facial functional region semantic segmentation map is represented in matrix form. Each element value in the matrix corresponds to a predefined small region in the face. The magnitude of each element value represents the probability value of a predefined small region belonging to the expression-dominant region. A probability value close to 1 indicates that it is judged as the expression-dominant region, and a probability value close to 0 indicates that it is judged as the lip-shape-dominant region.

[0067] S3. Perform phoneme identification on the audio stream to obtain a phoneme sequence.

[0068] Furthermore, the process of obtaining the phoneme sequence includes:

[0069] S301. Perform frame segmentation and feature extraction on the audio stream to obtain an audio feature sequence;

[0070] S302. Input the audio feature sequence into the phoneme recognition model and output the phoneme sequence. The phoneme sequence represents the phoneme categories arranged in chronological order and their temporal boundary information.

[0071] Furthermore, the phoneme recognition model is a dual-branch encoder structure, including:

[0072] The first encoding branch is used to identify and output the phoneme sequence;

[0073] The second coding branch shares some low-level features with the first coding branch and is used to extract generalized articulation features related to facial muscle movements.

[0074] In this context, generalized articulation action features and phoneme sequences serve together as inputs to the facial region visual semantic segmentation and functional discrimination network in S4.

[0075] It should be noted that existing phoneme recognition models are mostly single-branch structures, which can only output phoneme sequences for matching lip movements and cannot establish a direct relationship between phonemes and facial muscle movements, resulting in a disconnect between the virtual anchor's lip movements and facial expressions (such as no corresponding movement of facial muscles when speaking).

[0076] This technology employs a dual-branch coding structure: the first branch performs the traditional phoneme recognition function, accurately outputting phoneme categories and time boundary information arranged in chronological order to ensure the synchronization of lip movements and pronunciation; the second branch shares the underlying audio features with the first branch, eliminating the need for additional computational costs, and is specifically designed to extract generalized pronunciation action features.

[0077] This feature reflects the movement patterns of facial muscles (such as the chin, cheeks, and muscles around the lips) during the pronunciation of different phonemes, establishing a mapping relationship between pronunciation and muscle movement. The dual-branch feature is collaboratively input into the subsequent network, enabling the system to not only accurately drive lip movements but also to naturally match facial muscle movements with pronunciation actions. This solves the problem of stiffness in existing technologies where lip movements are dynamic while the face remains static, enhancing the realism of virtual anchor driving.

[0078] In one embodiment, the audio stream is subjected to frame-by-frame processing with 16kHz sampling, 20ms frame length, and 10ms frame shift. The Mel-frequency cepstral coefficients (MFCCs) of each frame are extracted as audio features to obtain an audio feature sequence. (in 13 represents the number of audio frames and 13 represents the feature dimension of Mel-frequency cepstral coefficients (MFCC).

[0079] A dual-branch encoder is constructed, with a 3-layer 1D-CNN shared feature extraction module at the bottom layer. The shared features are calculated as follows: ,in This indicates a shared convolutional layer, with the input being... Output shared features The dimension is .

[0080] The first branch (phoneme recognition branch) shares features. Then connect to a 2-layer bidirectional LSTM ( ) and fully connected layers ( The output phoneme probability distribution is generated by the following formula: ,in (41 is the preset number of phoneme categories), obtained through the Viterbi algorithm from The final phoneme sequence is obtained by decoding. .

[0081] The second branch (the generalized articulation action feature branch) is also based on shared features. Access Layer 1 unidirectional and feature mapping fully connected layer ( Extracting generalized articulation action features, the calculation formula is as follows: ,in As the activation function, it outputs generalized articulation action features. This feature is related to the phoneme sequence. Collaborative input subsequent semantic segmentation and function discrimination network.

[0082] S4. Input the dynamic visual features of key facial regions, the overall facial expression visual features, the visual semantic features of the mouth shape region, and the phoneme sequence into the facial region visual semantic segmentation and functional discrimination network, and output the facial functional region semantic segmentation map and the visual discrimination confidence map.

[0083] Furthermore, the neural network model is trained using training data with region semantic annotations. The training data includes speaking videos containing various phonemes and expressions, as well as manually annotated labels in the video frames that indicate whether the movement of a specific facial region mainly serves pronunciation or emotional expression. The neural network model learns and establishes a facial region visual semantic segmentation and functional discrimination network that integrates phoneme sequences, facial expression features, and facial region movement semantics.

[0084] The facial region visual semantic segmentation and function discrimination network includes: a visual feature alignment layer, a U-Net improved segmentation layer, and a confidence calculation layer;

[0085] The visual feature alignment layer maps dynamic visual features of key facial regions, overall facial expression visual features, visual semantic features of mouth shape regions, and phoneme sequences from different dimensions to the same feature space.

[0086] The U-Net improved segmentation layer outputs functional attribute labels for each region through an encoder-decoder structure;

[0087] The confidence calculation layer calculates the visual confidence of the functional attribute labels of each region based on the IoU loss function, and generates a semantic segmentation map of facial functional regions and a visual discrimination confidence map.

[0088] The facial functional region semantic segmentation map represents the probability distribution of each small region of the face being classified as an expression-dominant region or a lip-dominant region at the current moment; the visual discrimination confidence map represents the degree of confidence of the conflict detection and facial region visual semantic segmentation and functional discrimination network in classifying each region in the facial functional region semantic segmentation map.

[0089] It should be noted that existing virtual anchor driving technologies do not functionally divide the facial region, and all use uniform weights to process facial expression and lip-shape features. This leads to problems such as the system failing to prioritize the core functional areas (lip shape is prioritized during pronunciation, and facial expression is prioritized during emotional expression) when pronunciation and facial expression change simultaneously (e.g., speaking happily). This results in conflicts between facial expression and lip-shape and driving confusion. This technology achieves end-to-end processing of feature alignment, region segmentation, and confidence determination through an integrated network.

[0090] Specifically, a visual feature alignment layer maps features of different dimensions (visual and audio) and scales to the same feature space, addressing the heterogeneity problem of multi-source data. The improved U-Net segmentation layer, using an encoder-decoder structure and training data with semantically labeled regions (labeling each region's motion to serve pronunciation or emotional expression), divides the face into multiple small regions, outputting the probability distribution (semantic segmentation map) of each region belonging to either the expression-dominant or lip-shape-dominant region. A confidence calculation layer calculates the confidence level of each region's determination based on the IoU loss function (confidence map), representing the system's grasp of region function discrimination. This technology achieves dynamic intelligent segmentation of facial functional regions for the first time, allowing the system to adaptively adjust the driving priority of expression and lip shape according to real-time scenarios, fundamentally solving the technical pain point of conflict between expression and lip shape driving.

[0091] In one embodiment, the visual feature alignment layer performs dimensionality unification and spatial alignment on the input multi-source features, including overall facial expression visual features. Dynamic visual features of key facial regions Visual semantic features of the mouth shape region Generalized features of articulation .

[0092] The four types of features are mapped to the same feature space through a linear transformation, and the alignment formula is as follows: (in }, which correspond to the four types of input features mentioned above; To adapt the weight matrix to the input and output dimensions of each feature, (For the corresponding bias vector), after alignment, all feature dimensions are unified to . .

[0093] Then, the four types of alignment features , , , By performing fusion, fusion characteristics are obtained. Frame-level alignment of audio and video features is achieved through a time synchronization algorithm. An improved U-Net segmentation layer is constructed to fuse features. The input network employs a 4-layer convolutional + pooling operation at the encoder to extract deep features, and a 4-layer deconvolution + feature concatenation operation at the decoder to reconstruct region features. The output is a face region segmentation map (dividing the face into 32×32 small regions), with each region... The formula for calculating the dominant expression probability (where m and n are region coordinates, ranging from 1 to 32) is as follows: ,in For the output of the segmentation layer The original feature values ​​of the region, The function maps probability values ​​to The intervals are ultimately used to form a semantic segmentation map of facial functional regions. .

[0094] The confidence calculation layer calculates the confidence level for each region based on the IoU loss function. The IoU loss formula is as follows: ,in For the first Predicted labels for the region (If it is determined to be the expression-dominant area, then it is the lip-shape-dominant area.) The area is manually labeled with real data. This indicates the number of pixels in the calculation area; the confidence score for each area is calculated using the following formula: Ultimately, a visual discrimination confidence map is formed. .

[0095] S5. Based on the semantic segmentation map of facial functional regions and the visual discrimination confidence map, the weights of the overall facial expression visual features are adjusted through a visual smoothing algorithm to generate driving features.

[0096] Furthermore, the process of generating driving features includes:

[0097] S501, combine the probability value of each region in the facial functional region semantic segmentation map belonging to the expression-dominant region with the confidence value of the corresponding region in the visual discrimination confidence map to calculate the fusion weight coefficient of each region;

[0098] S502, spatially align and slice the overall facial expression visual features according to the division of the face region to obtain the local expression feature vector corresponding to each region;

[0099] S503, multiply the local expression feature vector of each region by the fusion weight coefficient corresponding to each region to obtain the weighted local expression features;

[0100] S504 aggregates the local facial expression features of all regions to generate driving features.

[0101] Furthermore, the processing steps of the visual smoothing algorithm include:

[0102] S511. Take the probability value of a certain region in the semantic segmentation map of the facial functional region at the current time and the probability value of the same region at the previous time, and take a weighted average. The weight of the weighted average is determined by the confidence value of the corresponding region in the visual discrimination confidence map at the current time.

[0103] S512. When the confidence level is high, the weighted average depends on the judgment at the current moment.

[0104] S513. When the confidence level is low, the weighted average depends on the judgment of historical time.

[0105] It should be noted that in existing technologies, the generation of virtual anchor driving features often adopts fixed weights or simple smoothing algorithms, without considering the reliability and temporal continuity of feature judgment. This leads to fluctuations in feature judgment in complex scenarios (such as sudden changes in lighting or facial occlusion), resulting in stiff transitions in virtual avatar expressions / lip movements and unstable driving.

[0106] This technology achieves dynamic weight adjustment and smoothing optimization in two steps: First, it combines the expression dominance probability of each region in the semantic segmentation map with the confidence of the corresponding region in the confidence map to calculate the fusion weight coefficient of each region. The higher the confidence of the region judgment, the more the weight coefficient is inclined to the feature at the current moment.

[0107] The second step employs a confidence-based visual smoothing algorithm, which weights the probability value of the region at the current moment with the probability value at the previous moment. When the confidence level is high, the current judgment is prioritized (ensuring real-time performance); when the confidence level is low, historical judgments are prioritized (ensuring stability). Simultaneously, the overall facial expression features are sliced ​​and weighted according to facial regions, and then aggregated to generate driving features. This technology not only solves the instability problem of feature judgment in complex scenes but also achieves dynamic adaptation of facial expression and lip-sync driving weights, making the virtual anchor's movement transitions more natural and the driving force more precise.

[0108] In one embodiment, the fusion weight coefficient for each region is calculated using the following formula: ,in for time The region's fusion weighting coefficient This is the balance coefficient (value 0.5). for The probability that the expression in that region is unsmoothed at any given moment. for The confidence level of the judgment in this region at any given time.

[0109] The visual smoothing algorithm is executed to perform temporal smoothing (eliminating fluctuations) on the dominant facial expression probabilities that are not smoothed at the current time. The smoothing formula is as follows: ,in for time The dynamic weighting factor of the region is calculated using the following formula: , for The dominant expression probability in this smoothed region at any given time.

[0110] Will Overall facial expression features at any moment Feature slices are generated according to the 32×32 region division rule of the face to obtain each Local expression feature vector corresponding to the region Then, the local feature vectors are weighted using the fusion weight coefficients and the smoothed probabilities. The weighting formula is as follows: The weighted local feature dimension is still 1. .

[0111] The weighted local features of the 32×32 regions are aggregated to generate the final driving features.

[0112] The aggregation formula is:

[0113] ,in For mean pooling operation, the output drives the features. This is used to drive the subsequent virtual avatar model.

[0114] S6. Based on the driving features, drive the virtual avatar model to generate and output a virtual anchor video stream with coordinated lip movements and facial expressions.

[0115] Furthermore, the visual semantic features of the lip shape region and the phoneme sequence are input into the lip shape-driven generation network to generate lip shape-driven parameters that are synchronized with the pronunciation; then the driving features and lip shape-driven parameters are combined to drive the virtual avatar model.

[0116] It should be noted that in existing technologies, the lip-syncing and facial expression-driven processes of virtual anchors are mostly processed independently and in parallel, lacking an effective coordination mechanism. This leads to problems such as asynchronous lip-syncing and facial expression (e.g., the lip-syncing remains calm when smiling) and uncoordinated movements (e.g., stiff facial expression when speaking).

[0117] This technology constructs a lip-shape-expression collaborative fusion mechanism: on the one hand, a specially designed lip-shape-driven generation network is used to input visual semantic features of the lip-shape region (capturing dynamic details of the lip shape) and phoneme sequences (ensuring pronunciation synchronization) to accurately generate lip-shape-driven parameters that perfectly match the pronunciation rhythm; on the other hand, the technology described above is used to generate comprehensive driving features that integrate multi-dimensional visual features and audio features (dominating the overall facial expression and muscle movement).

[0118] By synergistically integrating lip-shape driving parameters with comprehensive driving features, lip-shape movements are embedded into the overall facial expression framework, achieving precise linkage between pronunciation and lip shape, and between emotion and facial expression. This solves the problem of insufficient synchronization and coordination between the two, making the facial movements of virtual anchors more in line with the expression habits of real humans and improving the user viewing experience.

[0119] In one embodiment, a lip-shape-driven generation network is constructed, and visual semantic features of the lip-shape region are input. (Output of the visual feature extraction steps above, dimensions) ) and phoneme sequence (The output of the previous dual-branch encoder, after being converted into a one-hot encoded vector, has different dimensions.) The network uses a 2-layer architecture. (LSTM layer for lip shape feature modeling) +1 layer (A dedicated fully connected layer for mouth shape parameter output) structure; the formula for generating mouth shape driving parameters is:

[0120] ;

[0121] in For feature splicing operations (to and (assembled according to time dimension) The activation function outputs the lip-sync driving parameters. (Corresponds to 40 lip-sync control parameters, used to precisely control the lip movements of the virtual character).

[0122] Calculate the lip-to-facial expression co-fusion coefficient This coefficient is adaptively adjusted based on the amplitude of the lip movements at the current moment (the more intense the lip movements, the higher the amplitude). The smaller the value, the higher the lip-sync weight. The formula for calculating the fusion coefficient is: ,in Preset static lip shape reference parameters (dimensions) ), Calculated for the mean. It is an absolute value (to avoid interference from negative amplitude). The range of values ​​is .

[0123] The previously generated expression-driven features (dimension) ) and lip-sync parameters To achieve collaborative fusion, since the two dimensions are inconsistent, a projection operation is first performed. Will Mapped to (and (Dimension matching), the fusion formula is: ,in For the final driving feature vector (dimension) This allows for the input of the data into a virtual avatar model (such as a BlendShape model) to achieve coordinated driving of facial expressions and lip movements.

[0124] Furthermore, some virtual anchor driving systems use a serial audio and video processing method, resulting in high driving latency. At the same time, the video preprocessing scheme is too simple (such as using only simple filtering or fixed-size cropping), which cannot adapt to complex scenes (such as changes in light intensity and facial angle shifts), affecting the accuracy of subsequent feature extraction.

[0125] This technology is optimized from two dimensions: First, it adopts a parallel audio and video acquisition and processing mechanism to avoid the latency caused by serial processing and ensure the real-time performance of the virtual anchor driver (meeting the needs of real-time scenarios such as live streaming); second, it designs a multi-step adaptive preprocessing scheme: it uses Gaussian bilateral filtering to suppress intra-frame noise (preserving facial details while removing interference), adaptive histogram equalization to achieve illumination compensation (solving the problem of facial feature blurring caused by changes in light intensity), the MTCNN algorithm to accurately locate and crop the facial ROI region, and then normalizes the RGB three-channel data into fixed-size video frames (ensuring consistency in subsequent feature extraction). This technology solves the problems of insufficient real-time performance and poor preprocessing adaptability in existing technologies, providing high-quality, standardized input data for all subsequent feature extraction and driving steps, which is the foundation for the stable and efficient operation of the entire system.

[0126] In one embodiment, a multi-threaded parallel mechanism is used to acquire audio and video streams. Thread 1 acquires facial video streams (30fps, 1920×1080 resolution) through a USB camera, and thread 2 acquires audio streams (16kHz sampling rate, 16-bit depth) through a microphone. Audio and video frame-level synchronization is achieved through a timestamp synchronization algorithm, and the synchronization error is controlled within ±10ms (to ensure the accuracy of subsequent audio and video feature alignment).

[0127] Perform adaptive preprocessing on the synchronized video frames:

[0128] 1) Gaussian bilateral filtering for noise reduction (preserving facial details and suppressing intra-frame noise), the filtering formula is: ;

[0129] in, As the normalization factor, ; Gaussian function with range , Gaussian function in spatial domain , These are the single-channel pixel values ​​of the original video frame. These are the filtered single-channel pixel values. Indicates the pixel Neighborhood Sum all pixels within the range.

[0130] 2) Adaptive Histogram Equalization (CLAHE) for illumination compensation (solving facial feature blurring caused by changes in light intensity), the grayscale mapping formula is: ,in The original grayscale level (0~255). grayscale The cumulative distribution function, for The minimum value, This represents the total number of pixels in a single-channel image. This is a rounding operation, outputting the image after illumination compensation.

[0131] 3) Locate the facial Region of Interest (ROI), output the facial bounding box coordinates, crop the facial region based on the bounding box, and scale it to a fixed size of 256×256 to obtain the cropped RGB three-channel image. Standardize the cropped RGB three-channel image (to ensure consistency in subsequent feature extraction), using the following standardization formula: ,in For RGB channels (c=1 corresponds to the R channel, c=2 corresponds to the G channel, and c=3 corresponds to the B channel). For the cut channel pixel values, For channel Pre-calculated mean ( ), For channel Pre-calculated standard deviation ( Output standardized video frames This serves as input data for subsequent visual feature extraction steps.

[0132] Based on the preferred embodiments of the present invention described above, those skilled in the art can make various changes and modifications without departing from the inventive concept. The technical scope of this invention is not limited to the contents of the specification, but must be determined according to the scope of the claims.

Claims

1. A real-time driven system for virtual anchors based on facial motion capture, characterized in that, include: A processor and a memory; the memory stores a computer program that, when executed by the processor, performs the following steps: S1. Acquire the user's audio stream and facial video stream in parallel, and perform computer vision preprocessing on the facial video stream; S2. Visual feature extraction is performed on the facial video stream to obtain dynamic visual features of key facial regions, overall facial expression visual features, and visual semantic features of the mouth shape region. S3. Perform phoneme identification on the audio stream to obtain a phoneme sequence; S4. Input the dynamic visual features of the key facial regions, the overall facial expression visual features, the visual semantic features of the mouth shape region, and the phoneme sequence into the facial region visual semantic segmentation and functional discrimination network, and output the facial functional region semantic segmentation map and the visual discrimination confidence map. S5. Based on the semantic segmentation map of the facial functional region and the visual discrimination confidence map, the weights of the overall facial expression visual features are adjusted using a visual smoothing algorithm to generate driving features; S6. Based on the driving features, drive the virtual avatar model to generate and output a virtual anchor video stream with coordinated lip movements and facial expressions; The neural network model is trained using training data with region semantic annotations. The training data includes speaking videos containing various phonemes and expressions, as well as manually annotated labels in the video frames that indicate whether the movement of a specific facial region mainly serves pronunciation or emotional expression. The neural network model learns and establishes a facial region visual semantic segmentation and functional discrimination network that integrates phoneme sequences, facial expression features, and facial region movement semantics. The facial region visual semantic segmentation and function discrimination network includes: a visual feature alignment layer, a U-Net improved segmentation layer, and a confidence calculation layer; The visual feature alignment layer maps the dynamic visual features of the key facial regions, the overall facial expression visual features, the visual semantic features of the mouth shape region, and the phoneme sequence of different dimensions to the same feature space. The U-Net improved segmentation layer outputs functional attribute labels for each region through an encoder-decoder structure; The confidence calculation layer calculates the visual confidence of the functional attribute labels of each region based on the IoU loss function, and generates the semantic segmentation map and visual discrimination confidence map of the facial functional region. The facial functional region semantic segmentation map represents the probability distribution of each small region of the face being classified as an expression-dominant region or a mouth-dominant region at the current moment; the visual discrimination confidence map represents the degree of confidence of the conflict detection and facial region visual semantic segmentation and functional discrimination network in determining each region in the facial functional region semantic segmentation map. The facial functional region semantic segmentation map is represented in matrix form. Each element value in the matrix corresponds to a predefined small region in the face, and the magnitude of each element value represents the probability value that a predefined small region belongs to the expression-dominant region.

2. The real-time driving system for virtual anchors based on facial motion capture according to claim 1, characterized in that: The computer vision preprocessing includes: S101. Gaussian bilateral filtering algorithm is used to suppress intra-frame noise in facial video stream, and illumination compensation is achieved through adaptive histogram equalization. S102. Based on the MTCNN algorithm, locate and crop the facial ROI region, and extract the RGB three-channel pixel data of the ROI region; S103. The RGB three-channel pixel data is standardized into video frames of a fixed size, and the video frames are used as input data for the visual feature extraction step.

3. The real-time driving system for virtual anchors based on facial motion capture according to claim 1, characterized in that: The process of generating the driving features includes: S501, the probability value of each region in the facial functional region semantic segmentation map belonging to the expression-dominant region is combined with the confidence value of the corresponding region in the visual discrimination confidence map to calculate the fusion weight coefficient of each region. S502, the overall facial expression visual features are spatially aligned and feature sliced ​​according to the division of the face region to obtain the local expression feature vector corresponding to each region; S503, multiply the local expression feature vector of each region by the fusion weight coefficient corresponding to each region to obtain the weighted local expression features; S504, aggregate the local facial expression features of all regions to generate the driving feature.

4. The real-time driving system for virtual anchors based on facial motion capture according to claim 1, characterized in that: The process of obtaining the phoneme sequence includes: S301. The audio stream is segmented into frames and features are extracted to obtain an audio feature sequence; S302. Input the audio feature sequence into the phoneme recognition model and output the phoneme sequence, which represents the phoneme categories and their time boundary information arranged in chronological order.

5. The real-time driving system for virtual anchors based on facial motion capture according to claim 4, characterized in that: The phoneme recognition model is a dual-branch encoder structure, including: The first encoding branch is used to identify and output the phoneme sequence; The second coding branch shares some low-level features with the first coding branch and is used to extract generalized articulation motion features related to facial muscle movements. The generalized articulation action features and the phoneme sequence together serve as the input to the facial region visual semantic segmentation and functional discrimination network in S4.

6. The real-time driving system for virtual anchors based on facial motion capture according to claim 1, characterized in that: The visual feature extraction process includes: S201. Process each frame of the facial video stream using a face detection and alignment model to obtain a standardized face image; S202. Input the face image into the facial expression encoder to extract the overall facial expression features; S203. Based on the coordinates of the key facial points, the dynamic visual features of the key facial regions, the overall facial expression visual features, and the visual semantic features of the mouth shape region are cropped from the face image.

7. The real-time driving system for virtual anchors based on facial motion capture according to claim 1, characterized in that: The visual semantic features of the mouth shape region and the phoneme sequence are input into the mouth shape driving generation network to generate mouth shape driving parameters synchronized with pronunciation; then the driving features and the mouth shape driving parameters are combined to drive the virtual image model.

Images