Consistent identity recognition-based fine-grained facial expression digital human avatar generation method and system

Abstract

The application provides a fine-grained facial expression digital human portrait generation method and system based on consistent identity recognition, and relates to the technical field of computer vision and digital human, wherein the method comprises the following steps: non-contact collection of multi-angle expression images and infrared light reflection signals in a virtual reality environment, conversion of the multi-angle expression images and the infrared light reflection signals into texture and depth electrical signals, and preprocessing of the texture and depth electrical signals to obtain purified expression data; then separation of identity features and noisy expression features through an automatic encoder, reconstruction of the identity features and the noisy expression features through decoding to obtain pure fine-grained expression features; then generation of enhanced local expression features based on a consistent attention mechanism according to eye and mouth region weight distribution, fusion of the enhanced local expression features and the identity features into expression-driven features with consistent identity, and finally driving of a three-dimensional face model to generate a digital human portrait with consistent identity and expression details. The application improves the realism and identity stability of expression interaction in virtual reality.

Description

Technical Field

[0001] This application relates to the fields of computer vision and digital human technology, and in particular to a method and system for generating fine-grained digital human avatars with facial expressions based on consistent identity recognition. Background Technology

[0002] With the popularization of applications such as virtual reality and online social networking, users have placed higher demands on the realism and expressiveness of digital avatars. They need to generate rich and delicate facial expressions while maintaining the stability and consistency of the user's own identity characteristics to support a natural and credible real-time interactive experience.

[0003] Existing solutions employ text-guided image generation models, directly generating avatars with corresponding expressions by inputting a single user facial image and a text description of the target expression. This method leverages a pre-trained image generation network, combining facial identity feature encoding and expression semantic embedding to achieve end-to-end avatar generation.

[0004] However, this approach presents a contradiction between rendering facial expression details and preserving identity. The generated expressions are prone to blurring or exaggerating key features, leading to decreased expression recognizability. Furthermore, the stability of identity features is easily compromised during cross-expression generation, resulting in visible differences in facial structure across different expressions. In addition, this method suffers from generation efficiency bottlenecks when handling complex, continuous expressions, making it difficult to meet the demands for coherent multi-expression output in real-time interactive scenarios. Summary of the Invention

[0005] This application provides a fine-grained method and system for generating digital human avatars with facial expressions based on consistent identity recognition, in order to solve the problems of poor realism and poor identity stability in virtual reality facial expression interaction in the prior art.

[0006] To address the aforementioned technical problems, in a first aspect, this application provides a method for generating fine-grained digital avatars with facial expressions based on consistent identity recognition, comprising:

[0007] Acquire multi-angle facial expression images and infrared light reflection signals of real people in a virtual reality environment using non-contact methods;

[0008] Based on the multi-angle facial expression image data and the infrared light reflection signal, purified facial expression data is generated;

[0009] The purified facial expression data is input into an autoencoder network. Through the encoding stage of the autoencoder network, the identity feature vector representing static identity and the initial facial expression feature vector containing environmental noise are separated. Through the decoding stage of the autoencoder network, the initial facial expression feature vector is reconstructed to remove the environmental noise and obtain a pure, fine-grained facial expression feature vector.

[0010] Using a feature fusion network based on a consistent attention mechanism, enhanced local expression features are generated based on the weight distribution of the pure fine-grained expression feature vector in the eye and mouth regions. The enhanced local expression features are then fused with the identity feature vector to generate identity-consistent expression-driven features.

[0011] Based on the identity-consistent expression-driven features, the 3D face model is driven to generate fine-grained digital avatars with consistent expressions.

[0012] Optionally, the feature fusion network based on the consistency attention mechanism generates enhanced local expression features based on the weight distribution of the pure fine-grained expression feature vector in the eye and mouth regions, and fuses the enhanced local expression features with the identity feature vector to generate identity-consistent expression-driven features, including:

[0013] The purified fine-grained facial expression feature vector is input into a feature fusion network based on a consistent attention mechanism. The region segmentation module of the feature fusion network divides the purified fine-grained facial expression feature vector into eye region feature blocks and mouth region feature blocks.

[0014] The consistency detection module of the feature fusion network performs identity consistency analysis on the feature blocks of the eye region and the feature blocks of the mouth region respectively to obtain the identity consistency parameters of each region feature block.

[0015] The weight calculation module of the feature fusion network generates an eye region weight map and a mouth region weight map based on the identity consistency parameters.

[0016] The feature enhancement module of the feature fusion network performs weighted processing on the corresponding regional feature blocks according to the eye region weight map and the mouth region weight map to obtain enhanced eye region features and enhanced mouth region features.

[0017] The enhanced eye region features and the enhanced mouth region features are spatially spliced ​​together by the feature splicing module of the feature fusion network to form enhanced local expression features. The enhanced local expression features are then spliced ​​together with the identity feature vector along the channel dimension to obtain the spliced ​​fusion features.

[0018] The spliced ​​fused features are compressed and integrated through the multi-layer fully connected network module of the feature fusion network to generate identity-consistent expression-driven features.

[0019] Secondly, this application provides a fine-grained digital avatar generation system for facial expressions based on consistent identity recognition, comprising:

[0020] The acquisition module is used to acquire multi-angle facial expression image data and infrared light reflection signals of real people in a virtual reality environment through non-contact methods.

[0021] The conversion module is used to generate purified facial expression data based on the multi-angle facial expression image data and the infrared light reflection signal;

[0022] The input module is used to input the purified facial expression data into the autoencoder network. Through the encoding stage of the autoencoder network, the identity feature vector representing static identity and the initial facial expression feature vector containing environmental noise are separated. Through the decoding stage of the autoencoder network, the initial facial expression feature vector is reconstructed to remove the environmental noise and obtain a pure fine-grained facial expression feature vector.

[0023] The fusion module is used to utilize a feature fusion network based on a consistent attention mechanism to generate enhanced local expression features based on the weight distribution of the pure fine-grained expression feature vector in the eye and mouth regions, and to fuse the enhanced local expression features with the identity feature vector to generate identity-consistent expression-driven features.

[0024] The generation module is used to drive the 3D face model to generate a fine-grained digital avatar with consistent facial expressions based on the identity-consistent expression-driven features.

[0025] Thirdly, this application provides an electronic device, comprising:

[0026] Memory, used to store computer programs;

[0027] A processor, configured to execute the computer program to implement the steps of the fine-grained facial expression digital human avatar generation method based on consistent identity recognition as described in the first aspect above.

[0028] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps of the fine-grained facial expression digital human avatar generation method based on consistent identity recognition as described in the first aspect above.

[0029] The technical solution provided in this application has the following beneficial effects:

[0030] This application achieves the acquisition of complementary raw data containing facial texture and 3D depth information under virtual reality non-contact interaction conditions, providing a comprehensive data foundation for subsequent processing. By converting optical signals into electrical signals and performing time-series alignment and redundancy suppression, data distortion caused by asynchronous acquisition devices, head movements, and environmental interference is eliminated, ensuring the synchronization and purity of the input data. Stable identity attributes and dynamic expression changes in facial data are effectively separated, and environmental noise mixed in with expression features is removed through the reconstruction process, resulting in clear and detailed fine-grained expression features. By dynamically weighting and enhancing key expression regions such as the eyes and mouth, and controlling their fusion with identity features, the generated expressions are ensured to be both vivid and delicate, while strictly conforming to the individual's identity characteristics. Finally, the optimized driving features are applied to a 3D model to generate a digital avatar that maintains high identity consistency and realistic expression details throughout continuous expression changes.

[0031] Furthermore, this application utilizes a consistent attention mechanism to dynamically focus on and enhance key facial expression region features that are consistent with identity during the feature fusion stage. This effectively constrains the interference of facial expression changes on identity features while generating rich facial expression details, ensuring the high stability and uniformity of the final digital human avatar's identity attributes.

[0032] These or other aspects of this application will become more apparent from the description of the following embodiments. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 A flowchart illustrating a fine-grained facial expression digital avatar generation method based on consistent identity recognition, provided for embodiments of this application;

[0035] Figure 2 A schematic diagram illustrating a specific implementation of a fine-grained facial expression digital avatar generation method based on consistent identity recognition, provided in this application embodiment;

[0036] Figure 3 This is a schematic diagram of the structure of a fine-grained facial expression digital avatar generation system based on consistent identity recognition, provided in an embodiment of this application. Detailed Implementation

[0037] In applications such as virtual reality interaction and digital social networking, existing methods for generating digital emoji avatars based on text or single images generally face the challenge of insufficient detail in facial expressions and difficulty in maintaining the stability of identity features. Specifically, the generated expressions often lack subtle variations in key areas such as the eyes and lips, resulting in stiff or distorted expressions. Furthermore, when different expressions are generated for the same identity, facial contours and features tend to shift, disrupting the user's consistent perception of the identity. These problems stem from the insufficient handling of the coupling relationship between facial expressions and identity features in existing methods, as well as the lack of effective suppression of data interference in complex environments, thus limiting the realism and usability of digital avatars.

[0038] To address the aforementioned issues, this application proposes a method and system for generating fine-grained digital avatars with facial expressions based on consistent identity recognition. The method first acquires complementary data containing texture and depth information through multi-angle optical and infrared signal collaborative acquisition, and obtains high-quality expression data after signal preprocessing. Then, an autoencoder is used to separate stable identity features and pure fine-grained expression features. Based on this, a consistent attention mechanism is introduced to dynamically enhance and fuse key expression areas such as the eyes and mouth with identity consistency, generating expression-driven signals that retain rich expression details while strictly conforming to identity features. Finally, the signal drives a 3D face model to output a digital avatar with consistent identity and natural expression. This method, through multi-source data collaborative processing, feature decoupling reconstruction, and attention-guided fusion, effectively unifies expression delicacy and identity stability, solving the core problems of missing expression details and identity feature drift in existing technologies, and improving the realism and user identification of digital avatars in interactive applications.

[0039] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0040] The core of this application is to provide a fine-grained method for generating digital avatars with facial expressions based on consistent identity recognition. A flowchart of one specific implementation is shown below. Figure 1 As shown, the method includes:

[0041] Step 101: Acquire multi-angle facial expression image data and infrared light reflection signals of real people in a virtual reality environment through non-contact methods.

[0042] In step 101, the non-contact method refers to the data acquisition method of acquiring facial images and light signals through optical sensors without physical contact with the human body; the multi-angle expression image data refers to the continuous image sequence of the user's face at multiple angles, which is acquired simultaneously by image acquisition devices placed at different positions in the virtual reality scene. This image data includes facial texture and contour information.

[0043] Infrared light reflection signal refers to the optical signal obtained by actively emitting an invisible infrared beam and receiving it after reflection from the face. The intensity of this signal is related to the depth variation of the three-dimensional surface of the face and is used to compensate for data loss in visible light images under occlusion or low light conditions.

[0044] In this embodiment of the application, in a virtual reality interactive environment, the system synchronously activates multiple visible light image sensors and infrared light projection receiving devices deployed in space. The visible light sensors capture continuous images of the user's facial expressions from different directions, forming multi-angle facial expression image data.

[0045] At the same time, the infrared device emits infrared light of a specific wavelength toward the facial area and receives the reflected signal. The two types of data are synchronized by a hardware mechanism to ensure that the acquisition timestamps are consistent, forming the original multimodal signal required for subsequent processing.

[0046] Step 102: Generate purified facial expression data based on the multi-angle facial expression image data and the infrared light reflection signal.

[0047] In this embodiment of the application, step 102 can be implemented in the following manner: converting the multi-angle expression image data and the infrared light reflection signal into facial texture electrical signals and facial depth electrical signals respectively, and preprocessing the facial texture electrical signals and the facial depth electrical signals respectively to obtain purified expression data. The preprocessing includes temporal alignment and redundancy suppression processing.

[0048] Among them, facial texture electrical signal refers to the conversion of pixel brightness information in visible light images into a continuous electrical signal sequence through photoelectric conversion, reflecting the surface attributes of facial skin texture, color, etc.; facial depth electrical signal refers to the conversion of infrared reflected light intensity into an electrical signal through a photodiode, and its amplitude change corresponds to the relative depth distance of each point on the facial surface.

[0049] The purified facial expression data refers to the standardized multimodal signal set obtained after time alignment and redundancy suppression. This data set is composed of facial texture electrical signals and facial depth electrical signals precisely registered according to time frames, eliminating signal distortion and inconsistency caused by asynchronous acquisition devices, head movements, or environmental interference, forming a complete, clean, and standardized facial expression data source that can be used for subsequent feature extraction.

[0050] In this embodiment, multi-angle facial expression image data generates facial texture electrical signals through a photoelectric conversion circuit within the image sensor, and infrared light reflection signals are converted into facial depth electrical signals through a photoelectric detection module.

[0051] The two types of signals are then time-aligned, and frame rate conversion is used to make the two types of signals have the same time resolution. Next, redundancy analysis is performed on the aligned signals to detect and remove abnormal pulses caused by instantaneous occlusion or sudden changes in lighting. Finally, the processed texture and depth electrical signals are combined frame by frame to form multidimensional purified facial expression data.

[0052] For example, 450 frames of data are obtained. The image data is converted into a facial texture electrical signal with a resolution of 640×480 by a CMOS sensor. The infrared signal is output as a facial depth electrical signal with a resolution of 320×240 by a photoelectric conversion module. The frame rate of the texture electrical signal is increased from 30 frames to 60 frames, which is consistent with the depth signal, through linear interpolation, so that each frame of depth signal corresponds to two frames of texture signal.

[0053] The signal difference between consecutive frames was then calculated. When the signal difference of a certain segment was less than the threshold of 5, it was determined to be redundant and removed. A total of 12 frames of abnormal data were removed. Finally, the remaining 438 frames of texture electrical signals and 219 frames of depth electrical signals were combined in chronological order to form the purified facial expression data.

[0054] Step 103: Input the purified facial expression data into the autoencoder network. Through the encoding stage of the autoencoder network, separate the identity feature vector representing static identity and the initial facial expression feature vector containing environmental noise. Through the decoding stage of the autoencoder network, reconstruct the initial facial expression feature vector to remove the environmental noise and obtain a pure, fine-grained facial expression feature vector.

[0055] In step 103, the identity feature vector refers to the low-dimensional dense vector extracted by the neural network that represents the inherent static features of the user's face. This vector may contain invariant attributes such as face shape and facial feature distribution.

[0056] The initial facial expression feature vector refers to the intermediate feature representation extracted from the input data, which includes real facial expression changes and environmental noise; the pure fine-grained facial expression feature vector refers to the refined facial expression features obtained after noise removal through reconstruction, which retains only subtle muscle movements.

[0057] In this embodiment, the purified facial expression data is input into the encoder part of the autoencoder, and high-level features are extracted step by step through multi-layer convolution operations. At the bottleneck layer, the feature flow is separated into two branches. The identity branch generates an identity feature vector through global pooling and fully connected layers, and the facial expression branch generates an initial facial expression feature vector through convolutional layers that preserve the spatiotemporal structure.

[0058] The initial facial expression feature vector is input into the decoder, which gradually reconstructs the feature map through deconvolution layers and performs skip connections with the multi-scale detail features of the corresponding layers in the encoder to supplement the details. Finally, environmental noise is removed through convolution operations in the output layer, and a clean, fine-grained facial expression feature vector is output.

[0059] For example, the 438 frames of cleaned facial expression data obtained in the above embodiment are input into a pre-trained autoencoder. The encoder compresses each frame of data into a 160×120×64 feature map through 5 convolutional layers, and separates a 512-dimensional identity feature vector and a 256-dimensional initial facial expression feature vector at the bottleneck layer.

[0060] The initial facial expression feature vector is reconstructed into a 320×240-dimensional feature vector through three layers of deconvolution. The feature vector is then added element-wise with the feature output from the second layer of the encoder to restore details. Finally, the feature vector is filtered through an output layer with a convolution kernel size of 3×3 to generate a clean, fine-grained facial expression feature vector of 320×240×32, in which the noise energy is reduced by 70% compared to the original signal.

[0061] Step 104: Using a feature fusion network based on a consistent attention mechanism, based on the weight distribution of the pure fine-grained expression feature vector in the eye and mouth regions, enhanced local expression features are generated, and the enhanced local expression features are fused with the identity feature vector to generate identity-consistent expression-driven features.

[0062] In step 104, the enhanced local expression feature refers to the enhanced expression formed by weighting and amplifying the features of key areas such as the eyes and mouth through the consistent attention mechanism. This feature highlights the expression change areas that are consistent with the identity while retaining the original details, so that it can drive expression generation more effectively in the subsequent fusion process, while avoiding the dilution or distortion of identity features.

[0063] Identity-consistent expression-driven features refer to driving signals generated by fusing enhanced local expression features with identity features, which both preserve details and conform to identity constraints.

[0064] In this embodiment of the application, after the pure fine-grained facial expression feature vector is input into the feature fusion network, it is first divided into eye region feature blocks and mouth region feature blocks by the region segmentation module. The consistency detection module calculates the cosine similarity between the two region blocks and the identity feature vector as identity consistency parameters.

[0065] The weight calculation module generates weight maps for the eye region and mouth region based on consistency parameters. The feature enhancement module amplifies the regional features based on the weight maps. The enhanced regional features are spatially stitched to form enhanced local expression features, which are then stitched with the identity feature vector in the channel dimension. Finally, the feature is reduced and integrated through a multi-layer fully connected network to output expression-driven features that are consistent with the identity.

[0066] For example, by inputting a clean, fine-grained facial expression feature vector into a feature fusion network, the region segmentation module extracts an 80×60 eye feature block and a 100×80 mouth feature block, and the consistency detection module calculates a similarity of 0.92 for the eye region and 0.88 for the mouth region.

[0067] The weight calculation module generates a weight map, where the weight coefficient for the eyes is 0.92 divided by the sum of 0.92 and 0.88, which equals 0.511, and the weight coefficient for the mouth is 0.489. The feature enhancement module multiplies the features of the two regions by their corresponding weights and then concatenates them into an enhanced local expression feature of 180×80.

[0068] This feature is concatenated with a 512-dimensional identity feature vector and then compressed into a 256-dimensional identity-consistent expression-driven feature through a 3-layer fully connected network.

[0069] Step 105: Based on the identity-consistent expression-driven features, drive the 3D face model to generate a fine-grained digital avatar with consistent expressions.

[0070] In step 105, the three-dimensional face model collects a large amount of real face scanning data under multiple poses and lighting conditions, and uses three-dimensional deformation model technology to learn the statistical change space of face shape and expression, and constructs a parameterized model jointly controlled by identity parameters and expression parameters.

[0071] During training, high-precision 3D mesh and texture data are used, and the model parameters are fitted through a nonlinear optimization algorithm, enabling it to generate 3D faces with realistic geometric and texture details in real time based on the input feature vectors.

[0072] Fine-grained facial expression digital avatars refer to digital virtual avatars generated based on this method that can accurately present subtle muscle movements such as slight wrinkles in the eyes and slight upturns in the corners of the mouth while maintaining a high degree of consistency in user identity characteristics. Their facial expression changes are not only anatomically reasonable, but also have rich visual details and emotional expressiveness, making them suitable for highly realistic virtual interaction scenarios.

[0073] In this embodiment, the identity-consistent expression-driven features are decomposed into region deformation parameters and texture change parameters by the parsing module. The deformation parameters are mapped to the corresponding mesh vertices of the 3D face model. The displacement vector of each vertex is calculated and the coordinates are updated to realize model deformation.

[0074] Texture variation parameters are input into the surface shader to adjust local colors and normal maps; the deformed model is enhanced with details by adding skin pores and fine wrinkles through high-frequency maps; finally, a digital human head is rendered that maintains identity consistency while including fine-grained facial expression changes.

[0075] For example, the 256-dimensional identity-consistent expression-driven features obtained in the above embodiments are parsed into 192-dimensional region deformation parameters and 64-dimensional texture change parameters; the deformation parameters are mapped to a 3D model containing 15,230 vertices;

[0076] The formula for calculating the vertex displacement vector can be: ,in, Let be the vertex displacement vector. For the first Weights of each control point The deformation parameter components are used to calculate the new coordinates of all vertices; the texture parameter control shader increases the red channel of the corner of the mouth area by 15% and the normal map intensity of the eye area by 8%, and finally overlays the pore detail map to generate a fine-grained digital human head with a resolution of 1024×1024.

[0077] This method systematically solves the problems of missing facial details and identity feature drift in complex environments, improving the realism and identity stability of digital avatars.

[0078] To address the issue of inconsistent identity in existing digital facial expression generation, some embodiments include step 104: utilizing a feature fusion network based on a consistency attention mechanism, enhanced local facial expression features are generated based on the weight distribution of the pure, fine-grained facial expression feature vector in the eye and mouth regions. These enhanced local facial expression features are then fused with the identity feature vector to generate identity-consistent expression-driven features, such as... Figure 2 As shown, it includes:

[0079] Step 201: Input the purified fine-grained facial expression feature vector into a feature fusion network based on a consistent attention mechanism. Through the region segmentation module of the feature fusion network, the purified fine-grained facial expression feature vector is divided into eye region feature blocks and mouth region feature blocks.

[0080] In step 201, the eye region feature block is a local feature set containing eye-related feature data such as eye socket and eyelid, and the mouth region feature block is a local feature set containing mouth-related feature data such as lips and corners of mouth.

[0081] In this embodiment, the feature fusion network first extracts rectangular feature regions corresponding to the eye position from the pure fine-grained expression feature vector through a preset spatial coordinate range to form an eye region feature block, and at the same time extracts rectangular feature regions corresponding to the mouth position to form a mouth region feature block. These two feature blocks retain the channel dimension and local spatial structure of the original feature vector.

[0082] Step 202: Through the consistency detection module of the feature fusion network, perform identity consistency analysis on the feature blocks of the eye region and the feature blocks of the mouth region respectively, and obtain the identity consistency parameters of each region feature block.

[0083] In step 202, the identity consistency parameter is a numerical indicator that characterizes the degree of matching between the facial expression features of each region and the identity feature vector.

[0084] In this embodiment, the consistency detection module calculates the similarity between the feature block of the eye region and the identity feature vector, and the similarity between the feature block of the mouth region and the identity feature vector. The identity consistency parameter of the two regions is obtained by vector inner product operation. The larger the value of the parameter, the higher the compatibility between the facial expression feature and the identity feature of the region.

[0085] Step 203: Based on the identity consistency parameters, generate an eye region weight map and a mouth region weight map using the weight calculation module of the feature fusion network.

[0086] In step 203, the eye region weight map is a two-dimensional distribution map that identifies the importance of each location in the eye region, and the mouth region weight map is a two-dimensional distribution map that identifies the importance of each location in the mouth region.

[0087] In this embodiment, the weight calculation module converts the identity consistency parameter of the eye region into a uniform weight value for the entire region, generating an eye region weight map with constant weight. At the same time, it converts the identity consistency parameter of the mouth region into a uniform weight value for the entire region, generating a mouth region weight map. The weight value is proportional to the identity consistency parameter of the corresponding region, and this embodiment does not specifically limit the expression of this relationship.

[0088] Step 204: Through the feature enhancement module of the feature fusion network, the corresponding regional feature blocks are weighted according to the eye region weight map and the mouth region weight map to obtain enhanced eye region features and enhanced mouth region features.

[0089] In step 204, the enhanced eye region features are eye feature data after weight enhancement, and the enhanced mouth region features are mouth feature data after weight enhancement.

[0090] In this embodiment, the feature enhancement module multiplies each feature value in the eye region feature block by the weight coefficient of the corresponding position in the eye region weight map to obtain enhanced eye region features. At the same time, it multiplies each feature value in the mouth region feature block by the weight coefficient of the corresponding position in the mouth region weight map to obtain enhanced mouth region features. This process amplifies the feature region that is compatible with the identity.

[0091] Step 205: Using the feature splicing module of the feature fusion network, the enhanced eye region features and the enhanced mouth region features are spatially spliced ​​to form enhanced local expression features. The enhanced local expression features are then spliced ​​with the identity feature vector along the channel dimension to obtain the spliced ​​fusion features.

[0092] In step 205, the spliced ​​fusion feature is a composite feature formed by connecting the enhanced local expression features and the identity feature vector in the channel dimension.

[0093] In this embodiment, the feature splicing module first splices the enhanced eye region features and the enhanced mouth region features according to their spatial positions to form a complete enhanced local expression feature. Then, it connects this feature with the identity feature vector in the channel direction to form a spliced ​​fusion feature with an increased number of channels.

[0094] Step 206: Through the multi-layer fully connected network module of the feature fusion network, the spliced ​​fused features are compressed and integrated to generate identity-consistent expression-driven features.

[0095] In this embodiment, the multi-layer fully connected network module processes the spliced ​​fused features through three fully connected layers: the first layer performs feature dimensionality reduction, the second layer performs nonlinear transformation, and the third layer further compresses the feature dimension, ultimately outputting identity-consistent expression-driven features.

[0096] It should be noted that the structure of the feature fusion network and the specific structure of each module described above are exemplary, and the embodiments of this application do not impose specific limitations on them. They can be set according to the actual situation.

[0097] Here is a specific example:

[0098] Based on a pure, fine-grained facial expression feature vector of 320×240×32 dimensions, when generating a smiling expression-driven feature for user B in Company A's virtual reality social platform, this feature vector is input into a feature fusion network based on a consistent attention mechanism. First, the region segmentation module extracts the eye region feature block with upper left corner coordinates of [120, 80] and size of 80×60, and the mouth region feature block with upper left corner coordinates of [110, 160] and size of 100×80 from the feature vector according to the preset facial key point coordinate range.

[0099] Next, the consistency detection module calculates the cosine similarity between the two region feature blocks and the 512-dimensional identity feature vector. The dot product of the eye region feature block and the identity feature vector is 286. The modulus of the eye region feature block is 310, and the modulus of the identity feature vector is 312. Therefore, the calculation process for the eye region identity consistency parameter is as follows: Similarly, the dot product of the mouth region feature block and the identity feature vector is 275, the magnitude of the mouth region feature block is 312, the magnitude of the identity feature vector is also 312, and the mouth region identity consistency parameter is equal to... ;

[0100] The weight calculation module then generates a weight map based on the identity consistency parameters, where the weight coefficient for the eye region is equal to... The mouth area weighting coefficient is equal to This generates a weighted map of the eye region with a weight of 0.511 for all locations and a weighted map of the mouth region with a weight of 0.489 for all locations.

[0101] The feature enhancement module multiplies each element in the eye region feature block by 0.511 to obtain enhanced eye region features, and multiplies each element in the mouth region feature block by 0.489 to obtain enhanced mouth region features. The feature stitching module stitches the two enhanced region features together in spatial dimension to obtain an enhanced local expression feature of 180×80, and then stitches it together with the 512-dimensional identity feature vector in channel dimension to obtain a 1292-dimensional stitched fusion feature.

[0102] Finally, the multi-layer fully connected network module compresses the feature through three fully connected layers. The first layer maps the 1292-dimensional feature to 512-dimensional, the second layer maintains the 512-dimensional feature and performs non-linear activation, and the third layer outputs 256-dimensional identity-consistent expression-driven features, completing the conversion from multimodal features to driving signals.

[0103] In this embodiment, the key areas of facial expressions are enhanced in a targeted manner through a consistent attention mechanism and fused with identity features. This ensures that the generated expression-driven features retain subtle details of facial expression changes while maintaining high compatibility with identity features. This effectively solves the identity drift problem in the expression generation process and improves the identity stability and expression realism of digital avatars.

[0104] To further improve the compatibility between expression-driven features and identity features, in some embodiments, step 206 involves: compressing and integrating the spliced ​​fused features through a multi-layer fully connected network module of the feature fusion network to generate expression-driven features with consistent identity, including:

[0105] Step 301: The first fully connected layer of the multi-layer fully connected network module is used to perform the first feature dimensionality reduction on the spliced ​​fused features to obtain the first dimensionality-reduced features.

[0106] In step 301, the first dimensionality reduction feature is the feature representation after channel dimension reduction.

[0107] In this embodiment, the first fully connected layer maps the high-dimensional spliced ​​fused features to a low-dimensional space through linear transformation, retaining the main feature information while reducing the data dimension, thus forming the first dimensionality-reduced feature.

[0108] Step 302: The first dimensionality reduction feature is nonlinearly transformed through the second fully connected layer of the multilayer fully connected network module to obtain the nonlinear transformed feature.

[0109] In step 302, the nonlinear transformation feature is the feature representation after being processed by the activation function.

[0110] In this embodiment, the second fully connected layer performs a linear transformation on the first dimensionality-reduced feature and then processes it through an activation function to introduce nonlinear characteristics, thereby enhancing the expressive power of the feature and obtaining a nonlinear transformation feature.

[0111] Step 303: Perform a second feature dimensionality reduction on the nonlinear transformation features through the third fully connected layer of the multilayer fully connected network module to obtain the second dimensionality-reduced features.

[0112] In step 303, the second dimensionality-reduced feature is a further compressed, compact feature representation. The first fully connected layer is constructed as a linear layer with 512 neurons, receiving 1024-dimensional spliced ​​and fused features and outputting a 512-dimensional first dimensionality-reduced feature;

[0113] The second fully connected layer is constructed as a nonlinear layer with 512 neurons connected to the ReLU activation function, which performs a nonlinear transformation on the 512-dimensional first dimensionality reduction features;

[0114] The third fully connected layer is constructed as a linear layer with 256 neurons, compressing the 512-dimensional nonlinear transformation features into 256-dimensional second-dimensional features.

[0115] The output layer is constructed as a composite layer containing an identity consistency comparison unit and a linear transformation unit. The comparison unit calculates the identity consistency constraint parameters through cosine similarity, and the linear transformation unit maps the 256-dimensional calibrated features into 128-dimensional expression-driven features.

[0116] In this embodiment, the third fully connected layer performs dimensionality reduction on the nonlinear transformation features again, compressing the features to the target dimension through linear transformation to obtain the second dimensionality-reduced features.

[0117] Step 304: Through the output layer of the multilayer fully connected network module, the second dimensionality reduction feature is compared with the identity feature vector to obtain the identity consistency constraint parameter. Based on the identity consistency constraint parameter, the second dimensionality reduction feature is calibrated for identity consistency.

[0118] In step 304, the identity consistency constraint parameter is a quantitative indicator that characterizes the degree of matching between features.

[0119] In this embodiment, the output layer calculates the similarity between the second dimensionality reduction feature and the identity feature vector as an identity consistency constraint parameter, and scales and adjusts the second dimensionality reduction feature according to the parameter to make it more consistent with the identity feature distribution.

[0120] Step 305: Process the calibrated features through a linear transformation of the output layer to generate identity-consistent expression-driven features.

[0121] In this embodiment, the output layer performs a final linear transformation on the calibrated features, adjusts the feature dimensions and optimizes the feature distribution, generating expression-driven features that can be directly used to drive the identity consistency of the 3D model.

[0122] In this embodiment, the progressive feature compression and identity consistency calibration of the multi-layer fully connected network ensures that the expression-driven features strictly adhere to identity feature constraints while retaining necessary details, effectively improving the identity stability and expression realism of the generated digital avatar.

[0123] Optionally, step 103: The purified facial expression data is input into an autoencoder network, and through the encoding stage of the autoencoder network, the identity feature vector representing static identity and the initial facial expression feature vector containing environmental noise are separated, including:

[0124] Step 401: Input the purified facial expression data into the encoder part of the autoencoder network, and extract basic facial features from the purified facial expression data through the first convolutional layer of the encoder part.

[0125] In step 401, the "basic" in basic facial features refers to the initial and core facial composition information extracted from the original input data through the first layer of convolution. This feature contains primary but indispensable structured information such as the basic outline of the face, the location and orientation of major organs, and is the basis for all subsequent feature processing.

[0126] In this embodiment, the first convolutional layer of the encoder uses multiple convolutional kernels to scan and calculate the purified facial expression data, and captures local facial patterns through local connectivity and weight sharing mechanisms to output a basic facial feature map that preserves spatial structure.

[0127] Step 402: Enhance the basic facial features through the second convolutional layer of the encoder part to obtain enhanced facial features.

[0128] In step 402, for example, the first convolutional layer structure is designed to use 32 3×3 convolutional kernels with a stride of 2 to reduce the dimensionality of the 640×480 input to a 320×240×32 feature map. The second convolutional layer structure is designed to use 64 3×3 convolutional kernels with a stride of 2 to further reduce the dimensionality of the 320×240×32 input to an enhanced feature map of 160×120×64. The enhanced facial features are high-level feature representations that contain more detailed information.

[0129] In this embodiment, the second convolutional layer of the encoder performs deeper feature extraction based on the basic facial features. By increasing the number of convolutional kernels and layers, the receptive field is expanded, multi-scale information is fused, and enhanced facial features with stronger expressive power are obtained.

[0130] Step 403: Through the bottleneck layer of the encoder section, the enhanced facial features are respectively sent to the identity feature branch and expression feature branch of the bottleneck layer.

[0131] In step 403, the identity feature branch is a processing path specifically for extracting identity-related features, and the expression feature branch is a processing path specifically for extracting expression-related features.

[0132] In this embodiment, at the bottleneck layer of the encoder, enhanced facial features are copied and input into two independent processing branches, where the identity feature branch focuses on extracting stable features related to individual identity, and the expression feature branch focuses on extracting dynamic features related to expression changes.

[0133] Step 404: In the identity feature branch, the enhanced facial features are spatially compressed through a global pooling layer, and an identity feature vector is generated based on the compressed enhanced facial features through a fully connected layer.

[0134] In this embodiment, the identity feature branch first uses a global pooling layer to compress the spatial dimension of the enhanced facial features into a single point, retaining the feature information of all channels, and then maps these features into a fixed-dimensional identity feature vector through a fully connected layer.

[0135] Step 405: In the expression feature branch, the enhanced facial features are processed by a spatiotemporal convolutional layer to generate an initial expression feature vector that includes environmental noise.

[0136] In this embodiment, the facial expression feature branch uses a convolution kernel that combines temporal and spatial domains to perform sliding calculations on enhanced facial features, while capturing the spatial distribution features and temporal change patterns of facial expressions, and outputting an initial facial expression feature vector that retains the spatiotemporal dimensions.

[0137] In this embodiment, the hierarchical feature extraction and dual-branch separation mechanism of the encoder effectively decouples identity features from facial expression features, laying a solid foundation for subsequent noise removal and feature fusion, and improving the purity and discriminability of feature representation.

[0138] To further improve the purity and detail of facial expression features, in some embodiments, step 103: reconstructing the initial facial expression feature vector through the decoding stage of the autoencoder network to remove environmental noise and obtain a pure, fine-grained facial expression feature vector, includes:

[0139] Step 501: Input the initial facial expression feature vector into the decoder part of the autoencoder network, and perform feature reconstruction on the initial facial expression feature vector through the first deconvolution layer of the decoder part to obtain preliminary reconstructed features.

[0140] In step 501, the core difference between the preliminary reconstructed features and the initial expression feature vector lies in the different resolution and structuring of the features: the initial expression feature vector is a compact but abstract feature representation after being compressed by the encoder, which mainly carries the semantic information of the expression but loses a lot of spatial details;

[0141] The initial reconstructed features are those processed by the first deconvolution layer. They have recovered some spatial dimensions and richer texture details, laying the foundation for fine-grained reconstruction of expressions. However, they still contain environmental noise and lack sufficient detail.

[0142] In this embodiment, the first deconvolution layer of the decoder uses a deconvolution kernel to upsample and transform the initial facial expression feature vector. Through inverse convolution operations, the spatial size of the feature map is gradually restored to form preliminary reconstructed features containing the basic facial expression contour.

[0143] Step 502: Perform a skip connection between the preliminary reconstructed features and the multi-scale detail features of the corresponding layer in the decoder part to obtain the enhanced features with supplemented details.

[0144] In step 502, the corresponding layer refers to the layer in the decoder that corresponds to the layer in the encoder with the same feature map size. Specifically, the output of the first deconvolutional layer corresponds to the input of the second convolutional layer in the encoder. In this architecture, there is a main corresponding layer designed for this skip connection.

[0145] Multi-scale detail features are feature sets containing rich detail information extracted from different levels of the encoder.

[0146] In this embodiment, a skip connection mechanism is used to concatenate or add elements of the multi-scale detail features extracted from specific levels of the encoder with the preliminary reconstructed features, so that the reconstruction process can reuse the detail information captured in the encoding stage and obtain more detailed enhanced features.

[0147] Step 503: The enhanced features after supplementing details are upsampled through the second deconvolution layer of the decoder part to obtain upsampled features.

[0148] In step 503, the upsampled features are feature representations with further expanded spatial dimensions and containing more refined spatial information.

[0149] In this embodiment, the second deconvolution layer of the decoder uses a larger deconvolution kernel to resample the enhanced features after supplementing details, further expanding the spatial size of the feature map while maintaining important facial expression details, thus obtaining upsampled features with higher resolution.

[0150] Step 504: Through the output layer of the decoder part, environmental noise is removed from the upsampled features to generate a clean, fine-grained facial expression feature vector.

[0151] In this embodiment, the output layer of the decoder performs upsampling feature filtering through convolution operations, uses trained weight parameters to identify and suppress environmental noise components, and retains the true facial expression change features, ultimately outputting a clean, fine-grained facial expression feature vector.

[0152] In this embodiment, the layered reconstruction, detail supplementation and noise removal mechanism of the decoder effectively eliminates the influence of environmental interference on facial expression features, while preserving rich facial expression details, providing high-quality facial expression feature input for subsequent feature fusion.

[0153] To further improve the quality and reliability of the input data, in some embodiments, step 102 involves preprocessing the facial texture electrical signal and the facial depth electrical signal to obtain purified expression data, including:

[0154] Step 601: Establish a multi-signal synchronization channel, and perform clock domain synchronization processing on the facial texture electrical signal and the facial depth electrical signal in the multi-signal synchronization channel to obtain a group of synchronized electrical signals with a unified timing reference.

[0155] In step 601, the multi-signal synchronization channel performs clock domain synchronization processing on the facial texture electrical signal and the facial depth electrical signal to obtain a synchronization electrical signal group, which is a collection of multiple electrical signals with a unified time reference.

[0156] In this embodiment, a dedicated signal synchronization channel is established to perform timestamp alignment and frame rate matching processing on the facial texture electrical signals and facial depth electrical signals generated by different acquisition devices, so that all signals have a consistent time reference and form a synchronized electrical signal group.

[0157] Step 602: Perform inter-frame correlation analysis on the facial texture electrical signals in the synchronization electrical signal group to remove timing jitter and obtain stable facial texture electrical signals.

[0158] In step 602, the stable facial texture electrical signal is the texture signal after eliminating temporal instability factors.

[0159] In this embodiment of the application, by calculating the feature similarity between adjacent frames, the timing inconsistency problem caused by the fluctuation of the acquisition device or the transmission delay is identified, and the abnormal frames are corrected or replaced to obtain a continuous and stable facial texture electrical signal in the time dimension.

[0160] Step 603: Perform pulse interference detection on the facial depth electrical signal in the synchronized electrical signal group to identify abnormal pulse signal segments.

[0161] In step 603, the abnormal pulse signal segment is a segment whose signal is distorted due to environmental interference.

[0162] In the embodiments of this application, sudden signal mutations that exceed the normal fluctuation range can be detected by analyzing the amplitude variation law of the depth electrical signal. These mutations are usually caused by environmental infrared interference or equipment noise and are marked as abnormal pulse signal segments.

[0163] Step 604: Using the waveform characteristics of adjacent normal pulse signal segments, the abnormal pulse signal segments are reconstructed and repaired to obtain complete facial depth electrical signals.

[0164] In step 604, the complete facial depth electrical signal is a continuous depth signal after repairing all abnormal segments.

[0165] In this embodiment, the waveform characteristics of the normal signals before and after the abnormal segment are used to reconstruct the signal morphology of the damaged segment through interpolation algorithms or waveform matching techniques, thereby ensuring the continuity and integrity of the depth signal.

[0166] Step 605: Combine the stable facial texture electrical signal and the complete facial depth electrical signal in frame order to form purified expression data.

[0167] In this embodiment of the application, the processed texture electrical signal and depth electrical signal are combined and packaged according to the correspondence of time frames to form a standardized multimodal data set that can be used for subsequent feature extraction.

[0168] In this embodiment, the timing inconsistencies and signal interference problems in the acquisition process are effectively eliminated through processing steps such as multi-signal synchronization, timing stabilization, and interference detection and repair, providing high-quality and highly reliable standardized input data for subsequent feature extraction.

[0169] To further improve the quality and realism of digital avatar generation, in some embodiments, step 105: based on the identity-consistent expression-driven features, driving the 3D face model to generate a fine-grained expression-based digital avatar that maintains identity consistency, including:

[0170] Step 701: Parse the identity-consistent expression-driven features into a set of expression control parameters that include region deformation parameters and texture change parameters.

[0171] In step 701, the region deformation parameters refer to the control data used to control the geometric shape changes of specific facial regions on the 3D face model. These parameters determine the displacement direction and magnitude of the model mesh vertices, thereby forming different facial expression contours.

[0172] Texture variation parameters refer to attribute data used to adjust the visual appearance of a 3D human face model's surface, including changes in texture features such as skin color, gloss, and wrinkle details. These parameters enhance the realism of facial expressions by affecting the rendering effect of the surface shader.

[0173] In this embodiment of the application, the expression-driven features that are consistent with identity are decomposed into two independent parts by the parameter parsing module. The region deformation parameter is responsible for controlling the geometric deformation caused by facial muscle movement, and the texture change parameter is responsible for controlling the changes in surface properties such as skin color and gloss.

[0174] Step 702: Map the region deformation parameters to the corresponding mesh vertices of the 3D face model and calculate the vertex displacement vector.

[0175] In step 702, the vertex displacement vector is a mathematical vector that describes the direction and magnitude of the position change of the vertices in the 3D model mesh.

[0176] In this embodiment of the application, the region deformation parameters are assigned to each mesh vertex of the three-dimensional face model according to the pre-established parameter-vertex correspondence table, and the displacement vector that each vertex needs to move is calculated by weighting coefficients.

[0177] Step 703: Based on the vertex displacement vector, perform real-time deformation processing on the 3D face model to obtain the deformed 3D face model.

[0178] In step 703, the deformed 3D face model is a 3D model that conforms to the target expression state after geometric adjustment.

[0179] In this embodiment, the calculated vertex displacement vector is applied to each corresponding vertex of the 3D face model. By updating the vertex coordinates, the model is deformed in real time, so that the model presents the geometric features of the target expression.

[0180] Step 704: Apply the texture change parameters to the surface shader of the deformed 3D face model to adjust the local texture features of the deformed 3D face model.

[0181] In step 704, the surface shader is a program module that controls the rendering of the visual effects of the model's surface.

[0182] In this embodiment, texture variation parameters are input into the surface shader. By adjusting the shader's color calculation, normal mapping, and specular parameters, the texture representation of specific areas of the model is changed, enhancing the realism of facial expressions.

[0183] Step 705: Perform detail enhancement processing on the adjusted 3D face model, and generate a fine-grained digital avatar with fine expressions based on the enhanced 3D face model.

[0184] In this embodiment of the application, after completing geometric deformation and texture adjustment, high-frequency detail mapping is used to add skin pores, fine wrinkles and other detailed features to the model, and finally, a fine-grained digital human head with rich details is rendered.

[0185] In this embodiment, a complete process of parameter parsing, geometric deformation, texture adjustment and detail enhancement is used to realize the transformation from driving features to high-quality digital avatars, which not only maintains the consistency of identity features, but also presents rich facial expression details, thereby improving the realism and expressiveness of digital avatars.

[0186] Here is a specific example:

[0187] In the embodiments of this application, this application can generate high-precision fine-grained expressions. Specifically, it supports 135 types of expressions, such as melancholy, disgust, and joy, with an expression error as low as 11.09 and a CLIP score of 0.678, surpassing existing methods, such as PhotoMaker, which has an expression error of 13.39 and a CLIP score of 0.661.

[0188] This application embodiment can also maintain strong identity consistency. Specifically, a single identity image can generate multi-expression avatars with an ID similarity of 0.361, approaching the cross-expression consistency of real human images. This application embodiment can also achieve efficient batch generation. Specifically, the consistency attention mechanism enables a single inference to generate a full set of 135 expressions, improving inference speed by 40%, and ensuring consistent styles, such as LEGO and clay styles.

[0189] This application embodiment also features multi-style compatibility, specifically: controlling the art style through text prompts to adapt to the needs of scenarios such as games and education; the specific implementation process is as follows:

[0190] Phase 1: Model Training. During training, data preparation begins with the CelebA-HQ dataset, containing 30,000 high-resolution face images, divided into training and testing sets at a 99:1 ratio. Reference images are retrieved from the Emo135 dataset for each expression category, constructing expression-image pairs. Feature extraction and fusion are then performed. During fusion, a multi-objective joint training approach can be used, such as the loss function formula... ,in, Represents the loss function. This indicates a correction for flow loss to ensure generation quality; This represents the identity loss, used to maximize the cosine similarity of ArcFace features between the generated graph and the reference graph; The expression loss is used to minimize the POSTER feature MSE between the generated image and the expression reference image. Then, the parameters are trained through the optimizer. For example, the parameters of AdamW include learning rate lr=1e-4, weight decay=1e-2, batch size of 4, resolution of 512×512, and number of steps of 515k.

[0191] Phase Two: Inference Generation. During this phase, the user inputs: a single identity image + a list of target expression categories, for example: [“Homesickness”, “Anxiety”] + optional style prompts. This embodiment then automatically retrieves reference images from Emo135 for each expression category. A consistency generation process is then executed. After this process, post-processing optimization is performed, specifically: super-resolution processing is applied to the generated images to improve detail clarity. Abnormal samples with ID similarity <0.3 are filtered using DINOv2 feature verification. Finally, Phase Three is entered to deploy the method corresponding to this embodiment and provide a real-time interactive interface.

[0192] Figure 3 A schematic diagram of the structure of a fine-grained facial expression digital avatar generation system based on consistent identity recognition provided in this application embodiment is shown below. Figure 3 As shown, the detailed implementation section describes:

[0193] The acquisition module 31 is used to acquire multi-angle facial expression image data and infrared light reflection signals of real people in a virtual reality environment through non-contact methods.

[0194] The conversion module 32 is used to generate purified facial expression data based on the multi-angle facial expression image data and the infrared light reflection signal.

[0195] The input module 33 is used to input the purified facial expression data into the autoencoder network. Through the encoding stage of the autoencoder network, the identity feature vector representing static identity and the initial facial expression feature vector containing environmental noise are separated. Through the decoding stage of the autoencoder network, the initial facial expression feature vector is reconstructed to remove the environmental noise and obtain a pure fine-grained facial expression feature vector.

[0196] The fusion module 34 is used to utilize a feature fusion network based on a consistent attention mechanism to generate enhanced local expression features based on the weight distribution of the pure fine-grained expression feature vector in the eye and mouth regions, and to fuse the enhanced local expression features with the identity feature vector to generate identity-consistent expression-driven features.

[0197] The generation module 35 is used to drive the 3D face model to generate a fine-grained digital avatar with consistent facial expressions based on the identity-consistent expression-driven features.

[0198] The fine-grained facial expression digital avatar generation system based on consistent identity recognition in this application embodiment is used to implement the aforementioned fine-grained facial expression digital avatar generation method based on consistent identity recognition. Therefore, the specific implementation of the fine-grained facial expression digital avatar generation system based on consistent identity recognition can be found in the embodiment section of the fine-grained facial expression digital avatar generation method based on consistent identity recognition above. The specific implementation can be referred to the description of the corresponding embodiments, and will not be repeated here.

[0199] This application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of the fine-grained facial expression digital human avatar generation method based on consistency identity recognition described above.

[0200] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the above-described methods for generating fine-grained facial expression digital avatars based on consistent identity recognition.

[0201] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memory, random access memory, portable hard drives, magnetic disks, or optical disks.

[0202] The embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the embodiments of the fine-grained facial expression digital human avatar generation method based on consistent identity recognition.

[0203] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0204] The foregoing has provided a detailed description of a fine-grained digital avatar generation method and system for facial expressions based on consistent identity recognition, as provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A method for generating fine-grained digital avatars with facial expressions based on consistent identity recognition, characterized in that, include: Acquire multi-angle facial expression images and infrared light reflection signals of real people in a virtual reality environment using non-contact methods; Based on the multi-angle facial expression image data and the infrared light reflection signal, purified facial expression data is generated; The purified facial expression data is input into an autoencoder network. Through the encoding stage of the autoencoder network, the identity feature vector representing static identity and the initial facial expression feature vector containing environmental noise are separated. Through the decoding stage of the autoencoder network, the initial facial expression feature vector is reconstructed to remove the environmental noise and obtain a pure, fine-grained facial expression feature vector. Using a feature fusion network based on a consistent attention mechanism, enhanced local expression features are generated based on the weight distribution of the pure fine-grained expression feature vector in the eye and mouth regions. The enhanced local expression features are then fused with the identity feature vector to generate identity-consistent expression-driven features. Based on the identity-consistent expression-driven features, the 3D face model is driven to generate fine-grained digital avatars with consistent expressions. The feature fusion network based on a consistent attention mechanism generates enhanced local expression features based on the weight distribution of the pure fine-grained expression feature vector in the eye and mouth regions. These enhanced local expression features are then fused with the identity feature vector to generate identity-consistent expression-driven features, including: The purified fine-grained facial expression feature vector is input into a feature fusion network based on a consistent attention mechanism. The region segmentation module of the feature fusion network divides the purified fine-grained facial expression feature vector into eye region feature blocks and mouth region feature blocks. The consistency detection module of the feature fusion network performs identity consistency analysis on the feature blocks of the eye region and the feature blocks of the mouth region respectively to obtain the identity consistency parameters of each region feature block. The weight calculation module of the feature fusion network generates an eye region weight map and a mouth region weight map based on the identity consistency parameters. The feature enhancement module of the feature fusion network performs weighted processing on the corresponding regional feature blocks according to the eye region weight map and the mouth region weight map to obtain enhanced eye region features and enhanced mouth region features. The enhanced eye region features and the enhanced mouth region features are spatially spliced ​​together by the feature splicing module of the feature fusion network to form enhanced local expression features. The enhanced local expression features are then spliced ​​together with the identity feature vector along the channel dimension to obtain the spliced ​​fusion features. The spliced ​​fused features are compressed and integrated through the multi-layer fully connected network module of the feature fusion network to generate identity-consistent expression-driven features; The step of driving a 3D face model to generate a fine-grained digital avatar with consistent facial expressions based on the identity-consistent expression-driven features includes: The identity-consistent expression-driven features are parsed into a set of expression control parameters, including region deformation parameters and texture change parameters; The region deformation parameters are mapped to the corresponding mesh vertices of the 3D face model, and the vertex displacement vectors are calculated. Based on the vertex displacement vector, the 3D face model is subjected to real-time deformation processing to obtain the deformed 3D face model; The texture change parameters are applied to the surface shader of the deformed 3D face model to adjust the local texture features of the deformed 3D face model. The adjusted 3D face model is enhanced with detail, and a fine-grained digital avatar with facial expressions is generated based on the enhanced 3D face model.

2. The method according to claim 1, characterized in that, The multi-layer fully connected network module of the feature fusion network compresses and integrates the spliced ​​fused features to generate identity-consistent expression-driven features, including: The first fully connected layer of the multi-layer fully connected network module is used to perform a first feature dimensionality reduction on the spliced ​​fused features to obtain the first dimensionality reduction features. The first dimensionality reduction feature is nonlinearly transformed through the second fully connected layer of the multi-layer fully connected network module to obtain the nonlinear transformed feature. The nonlinear transformation features are subjected to a second feature dimensionality reduction through the third fully connected layer of the multi-layer fully connected network module to obtain the second dimensionality-reduced features. The second dimensionality reduction feature is compared with the identity feature vector through the output layer of the multi-layer fully connected network module to obtain the identity consistency constraint parameter. Based on the identity consistency constraint parameter, the second dimensionality reduction feature is calibrated for identity consistency. The calibrated features are processed by a linear transformation of the output layer to generate identity-consistent expression-driven features.

3. The method according to claim 1, characterized in that, The step of inputting the purified facial expression data into an autoencoder network, and separating the identity feature vector representing static identity and the initial facial expression feature vector containing environmental noise through the encoding stage of the autoencoder network, includes: The purified facial expression data is input into the encoder part of the autoencoder network, and basic facial features are extracted from the purified facial expression data through the first convolutional layer of the encoder part. The basic facial features are enhanced by using the second convolutional layer of the encoder section to obtain enhanced facial features; The enhanced facial features are fed to the identity feature branch and expression feature branch of the bottleneck layer through the bottleneck layer of the encoder section, respectively. In the identity feature branch, the enhanced facial features are spatially compressed through a global pooling layer, and an identity feature vector is generated based on the compressed enhanced facial features through a fully connected layer. In the facial expression feature branch, the enhanced facial features are processed by a spatiotemporal convolutional layer to generate an initial facial expression feature vector that includes environmental noise.

4. The method according to claim 1, characterized in that, The step of reconstructing the initial facial expression feature vector through the decoding stage of the autoencoder network to remove environmental noise and obtain a clean, fine-grained facial expression feature vector includes: The initial facial expression feature vector is input into the decoder part of the autoencoder network. The initial facial expression feature vector is reconstructed through the first deconvolution layer of the decoder part to obtain preliminary reconstructed features. The preliminary reconstructed features are skip-connected with the multi-scale detail features of the corresponding layer in the decoder part to obtain the enhanced features with supplemented details. The enhanced features after supplementing details are upsampled through the second deconvolution layer of the decoder section to obtain upsampled features; The output layer of the decoder section removes environmental noise from the upsampled features to generate a clean, fine-grained facial expression feature vector.

5. The method according to claim 1, characterized in that, The process of generating purified facial expression data based on the multi-angle facial expression image data and the infrared light reflection signal includes: The multi-angle facial expression image data and the infrared light reflection signal are respectively converted into facial texture electrical signals and facial depth electrical signals; The facial texture electrical signal and the facial depth electrical signal are preprocessed to obtain purified facial expression data.

6. A fine-grained digital avatar generation system for facial expressions based on consistent identity recognition, characterized in that, include: The acquisition module is used to acquire multi-angle facial expression image data and infrared light reflection signals of real people in a virtual reality environment through non-contact methods. The conversion module is used to generate purified facial expression data based on the multi-angle facial expression image data and the infrared light reflection signal; The input module is used to input the purified facial expression data into the autoencoder network. Through the encoding stage of the autoencoder network, the identity feature vector representing static identity and the initial facial expression feature vector containing environmental noise are separated. Through the decoding stage of the autoencoder network, the initial facial expression feature vector is reconstructed to remove the environmental noise and obtain a pure fine-grained facial expression feature vector. The fusion module is used to utilize a feature fusion network based on a consistent attention mechanism to generate enhanced local expression features based on the weight distribution of the pure fine-grained expression feature vector in the eye and mouth regions, and to fuse the enhanced local expression features with the identity feature vector to generate identity-consistent expression-driven features. The generation module is used to drive the 3D face model to generate a fine-grained digital avatar with consistent facial expressions based on the identity-consistent expression-driven features. The feature fusion network based on a consistent attention mechanism generates enhanced local expression features based on the weight distribution of the pure fine-grained expression feature vector in the eye and mouth regions. These enhanced local expression features are then fused with the identity feature vector to generate identity-consistent expression-driven features, including: The purified fine-grained facial expression feature vector is input into a feature fusion network based on a consistent attention mechanism. The region segmentation module of the feature fusion network divides the purified fine-grained facial expression feature vector into eye region feature blocks and mouth region feature blocks. The consistency detection module of the feature fusion network performs identity consistency analysis on the feature blocks of the eye region and the feature blocks of the mouth region respectively to obtain the identity consistency parameters of each region feature block. The weight calculation module of the feature fusion network generates an eye region weight map and a mouth region weight map based on the identity consistency parameters. The feature enhancement module of the feature fusion network performs weighted processing on the corresponding regional feature blocks according to the eye region weight map and the mouth region weight map to obtain enhanced eye region features and enhanced mouth region features. The enhanced eye region features and the enhanced mouth region features are spatially spliced ​​together by the feature splicing module of the feature fusion network to form enhanced local expression features. The enhanced local expression features are then spliced ​​together with the identity feature vector along the channel dimension to obtain the spliced ​​fusion features. The spliced ​​fused features are compressed and integrated through the multi-layer fully connected network module of the feature fusion network to generate identity-consistent expression-driven features; The step of driving a 3D face model to generate a fine-grained digital avatar with consistent facial expressions based on the identity-consistent expression-driven features includes: The identity-consistent expression-driven features are parsed into a set of expression control parameters, including region deformation parameters and texture change parameters; The region deformation parameters are mapped to the corresponding mesh vertices of the 3D face model, and the vertex displacement vectors are calculated. Based on the vertex displacement vector, the 3D face model is subjected to real-time deformation processing to obtain the deformed 3D face model; The texture change parameters are applied to the surface shader of the deformed 3D face model to adjust the local texture features of the deformed 3D face model. The adjusted 3D face model is enhanced with detail, and a fine-grained digital avatar with facial expressions is generated based on the enhanced 3D face model.

7. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the steps of the fine-grained facial expression digital human avatar generation method based on consistent identity recognition as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, enables the fine-grained facial expression digital avatar generation method based on consistent identity recognition as described in any one of claims 1 to 5.

Images