Digital human expression interaction generation method and device based on organizational dynamics law

Abstract

The embodiment of the application discloses a digital human expression interaction generation method and device based on the law of organizational dynamics, comprising: extracting the rhythm features and semantic features of the to-be-processed voice content; combining the personalized expression base and the weighting coefficient through the linear skinning formula to generate linear expression features, wherein the personalized expression base is generated by combining the initial expression base with the individual feature vector, and the weighting coefficient is generated by encoding the rhythm features through a voice time sequence feature encoder; the semantic features and the linear expression features are extracted through a conditional prior encoder, and are jointly input into a preset diffusion model as a condition to generate nonlinear expression features and virtual skeleton control parameters; the nonlinear expression features and the virtual skeleton control parameters are input into a preset reinforcement learning model, and through left-right symmetry constraint, the final generated expression result is generated, and the final generated expression result is the digital human expression corresponding to the to-be-processed voice content.

Description

Technical Field

[0001] This specification relates to the fields of computer technology and digital human technology, and in particular to a method and device for generating digital human facial expressions based on the laws of organizational dynamics. Background Technology

[0002] 3D digital human facial expression interaction generation is a core technology in the field of virtual digital humans. It is widely used in immersive interaction in virtual reality (VR) and augmented reality (AR) scenarios, emotional services in intelligent customer service systems, real-time performances of virtual anchors and virtual idols, virtual teachers in online education, character creation in film and animation, dynamic design of intelligent characters in games, as well as in fields such as medical rehabilitation, cultural heritage protection, and smart government.

[0003] In existing technologies, the Emotalk facial expression interaction algorithm, as a method for generating digital human facial expressions, can achieve interactive generation of digital human facial expressions. However, this method has obvious limitations in emotional expression. The generated expressions often appear stiff and rigid, lacking accurate capture and dynamic presentation of subtle emotional changes, and are difficult to show true human emotions.

[0004] Based on this, this specification provides a method and apparatus for generating digital human facial expressions based on the laws of tissue dynamics. Summary of the Invention

[0005] This specification provides a digital human facial expression interaction generation method and apparatus based on the laws of tissue dynamics to solve the following technical problems: In the prior art, the Emotalk facial expression interaction algorithm, as a digital human facial expression interaction generation method, can realize the interactive generation of digital human facial expressions, but this facial expression generation method has obvious limitations in emotional expression. The generated facial expressions often appear stiff and rigid, lacking accurate capture and dynamic presentation of subtle emotional changes, and are difficult to show real human emotions.

[0006] To solve the above-mentioned technical problems, the embodiments in this specification are implemented as follows:

[0007] This specification provides an embodiment of a digital human facial expression interaction generation method based on the laws of tissue dynamics. The digital human facial expression interaction generation method includes:

[0008] Extract the phonetic and semantic features of the speech content to be processed;

[0009] A personalized expression base and weighted coefficients are combined to generate linear expression features through a linear skinning formula. The personalized expression base is generated by combining an initial expression base with individual feature vectors, and the weighted coefficients are generated by encoding the phonetic features through a speech temporal feature encoder.

[0010] The semantic features and the linear facial expression features extracted by the conditional prior encoder are used together as conditional inputs into a preset diffusion model to generate nonlinear facial expression features and virtual skeleton control parameters.

[0011] The nonlinear facial expression features and the virtual skeleton control parameters are input into a preset reinforcement learning model. Through left-right symmetry constraints, the final facial expression result is generated, which serves as the digital human expression corresponding to the speech content to be processed.

[0012] This specification also provides an embodiment of a digital human facial expression interaction generation device based on tissue dynamics, the digital human facial expression interaction generation device comprising:

[0013] The extraction module extracts the phonetic and semantic features of the speech content to be processed;

[0014] The linear table case-varying module combines personalized expression bases and weighted coefficients to generate linear expression features through a linear skinning formula. The personalized expression base is generated by combining an initial expression base with individual feature vectors, and the weighted coefficients are generated by encoding the phonological features through a speech temporal feature encoder.

[0015] The nonlinear expression change module uses the semantic features and the linear expression features extracted by the conditional prior encoder as conditional inputs into a preset diffusion model to generate nonlinear expression features and virtual skeleton control parameters.

[0016] The facial expression fine modeling module inputs the nonlinear facial expression features and the virtual skeleton control parameters into a preset reinforcement learning model. Through left-right symmetry constraints, it generates the final facial expression result, which serves as the digital human expression corresponding to the speech content to be processed.

[0017] The digital human facial expression interaction generation method based on organizational dynamics provided in this specification extracts the phonetic and semantic features of the speech content to be processed. A personalized expression base and weighted coefficients are combined using a linear skinning formula to generate linear facial expression features. The personalized expression base is generated by combining an initial expression base with individual feature vectors, and the weighted coefficients are generated by encoding the phonetic features using a speech temporal feature encoder. The semantic features and the linear facial expression features extracted by a conditional prior encoder are input as conditions into a preset diffusion model to generate nonlinear facial expression features and virtual skeleton control parameters. The nonlinear facial expression features and the virtual skeleton control parameters are input into a preset reinforcement learning model, and through left-right symmetry constraints, the final generated facial expression result is generated. This final generated facial expression result serves as the digital human expression corresponding to the speech content to be processed, enabling the generated digital human expression to possess high realism, high naturalness, and physical consistency, improving the naturalness and expressiveness of the generated expression, and ultimately achieving coordinated generation of digital human expressions and movements. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A schematic diagram of the system architecture of a digital human facial expression interaction generation method based on the laws of organizational dynamics provided in the embodiments of this specification;

[0020] Figure 2 A flowchart illustrating a digital human facial expression interaction generation method based on organizational dynamics principles, provided as an embodiment of this specification;

[0021] Figure 3 A framework diagram of a digital human facial expression interaction generation method based on the laws of tissue dynamics provided in the embodiments of this specification;

[0022] Figure 4 This is a schematic diagram of a digital human facial expression interaction generation device based on the laws of tissue dynamics, provided as an embodiment of this specification. Detailed Implementation

[0023] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0024] In existing technologies, linear deformation is typically used to describe simple geometric changes in facial expression generation, while nonlinear deformation involves complex dynamic features. Currently, linear and nonlinear deformations are often modeled separately, leading to a lack of subtlety in expression generation. There is an urgent need to study the regular integration of these two methods and construct a unified deformation space modeling method to achieve coordinated generation of expressions and movements. To address the problem of unified modeling of linear and nonlinear deformation spaces, this application uses pseudo-muscle model theory as the basic mathematical theory to explore a modeling method for soft tissue dynamics in voice-driven digital human facial expression generation. By combining the linear deformation modeling method of Blendshape technology with a skeletal-driven model, the composite modeling mechanism of linear and nonlinear deformations in facial expression generation is studied. Simultaneously, based on the laws of soft tissue dynamics, the subtle changes in facial expressions are modeled to simulate the complex movement characteristics of facial tissues, thereby improving the naturalness and expressiveness of generated expressions.

[0025] Figure 1 This is a schematic diagram of the system architecture for a digital human facial expression interaction generation method based on organizational dynamics, provided as an embodiment of this specification. Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0026] Terminal devices 101, 102, and 103 interact with server 105 via network 104 to receive or send messages, etc. Various client applications can be installed on terminal devices 101, 102, and 103. For example, dedicated programs may be used for generating digital human facial expressions based on organizational dynamics.

[0027] Terminal devices 101, 102, and 103 can be either hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various dedicated or general-purpose electronic devices, including but not limited to smartphones, tablets, laptops, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software programs or software modules (e.g., multiple software programs or software modules used to provide distributed services) or as a single software program or software module.

[0028] Server 105 can be a server that provides various services, such as a backend server that provides services to client applications installed on terminal devices 101, 102, and 103. For example, the server can generate digital human facial expressions based on organizational dynamics principles, so that the results of generating digital human facial expressions based on organizational dynamics principles can be displayed on terminal device servers 101, 102, and 103.

[0029] Server 105 can be either hardware or software. When server 105 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When server 105 is software, it can be implemented as multiple software programs or software modules (e.g., multiple software programs or software modules used to provide distributed services), or as a single software program or software module.

[0030] Figure 2 This is a flowchart illustrating a digital human facial expression interaction generation method based on organizational dynamics, provided as an embodiment of this specification. From a programming perspective, the execution entity of the process can be a program hosted on an application server or application terminal. It is understood that this method can be executed by any device, equipment, platform, or cluster of devices with computing and processing capabilities. Figure 2 As shown, the digital human facial expression interaction generation method includes:

[0031] Step S201: Extract the phonetic and semantic features of the speech content to be processed.

[0032] In the embodiments of this specification, the speech content to be processed is the acquired speech content, which is the speech content of a real person acquired through the acquisition device.

[0033] In the embodiments of this specification, the extraction of phonological and semantic features of the speech content to be processed specifically includes:

[0034] The speech content to be processed is passed through a speech feature encoder to extract phonetic features;

[0035] The speech content to be processed is subjected to a large visual language model to extract semantic features.

[0036] In the embodiments of this specification, the speech feature encoder adopts existing technology. Specifically, it can perform frame segmentation, windowing, Fourier transform and other processing on the original speech signal through an audio encoder (such as CNN, Transformer or SoundStream) to extract phonological features. Specifically, the phonological features include: MFCC (Mel frequency cepstral coefficients), spectrogram, fundamental frequency (F0) and energy.

[0037] The Visual Language Model (VLM) is a multimodal, generative AI model capable of understanding and processing video, images, and text. In the embodiments described in this specification, the Visual Language Model employs existing technologies, which will not be elaborated upon further.

[0038] Step S203: The personalized expression base and weighted coefficients are combined using a linear skinning formula to generate linear expression features, wherein the personalized expression base is generated by combining the initial expression base with individual feature vectors, and the weighted coefficients are generated by encoding the phonological features through a speech temporal feature encoder.

[0039] In the embodiments of this specification, the initial expression base is a preset standardized general basic facial deformation data (standard general Blendshape), which is the basic unit for all facial expressions, mouth shapes, and eyebrow and eye movements of a three-dimensional digital human. It has no personalized facial attributes and is a general standard template for all types of digital humans.

[0040] In the embodiments of this specification, the individual feature vector includes: facial contour feature vector, facial feature structure feature vector, facial physiological deformation feature vector, skeletal movement constraint feature vector, and style differentiation feature vector;

[0041] The personalized expression base is generated by combining the initial expression base with individual feature vectors, specifically including:

[0042] Using the initial expression base as the deformation basis, the individual feature vector is linearly transformed and feature fused to generate the personalized expression base;

[0043] The weighting coefficients are generated by encoding the phonetic features using a speech temporal feature encoder, specifically including:

[0044] The phonological features are input into the speech temporal feature encoder, which performs dimensional alignment, temporal context modeling, and deep feature extraction to obtain dynamic emotional association information, which includes speech rhythm, energy fluctuations, and fundamental frequency changes.

[0045] The dynamic emotional association information is linearly mapped and weighted adaptively changed to generate a high-dimensional real-time weight vector, which includes: mouth shape deformation weight, eyebrow and eye movement weight, facial contour traction weight, cheek muscle deformation weight, jaw movement weight, and global expression intensity adjustment weight.

[0046] The high-dimensional real-time weight vector is normalized to generate the weighting coefficients;

[0047] The personalized expression base and weighting coefficients are combined using a linear skinning formula to generate linear expression features, specifically including:

[0048] ;

[0049] in,

[0050] This represents the linear facial expression feature;

[0051] Indicates the number of personalized emoji bases;

[0052] Indicates the total number of personalized emoji bases;

[0053] Indicates the weighting coefficients;

[0054] This refers to the personalized emoticon base.

[0055] In the embodiments of this specification, the facial contour feature vector includes: recording global morphological parameters such as the length-to-width ratio of the face, the curvature of the jaw, the prominence of the cheekbone, and the offset of the outer contour of the face, and correcting the overall contour adaptation deviation of the initial expression base.

[0056] The facial feature vector includes local facial topological offset features such as eye shape, eye distance, nose bridge shape, lip thickness, brow bone shape, and relative position of facial features, to achieve accurate matching between facial expressions and facial features.

[0057] Facial physiological deformation feature vectors characterize the range of facial muscle traction, skin elasticity coefficient, and soft tissue compression and stretching response characteristics, and are used to regulate the deformation amplitude and intensity of facial expression bases in different regions.

[0058] The skeletal motion constraint feature vector covers the activity threshold and motion boundary parameters of the maxillofacial bones, orbital bones, and temporomandibular joint, constraining the limit deformation range of the eyebrows, eyes, jaw, and mouth to avoid clipping and excessive stretching.

[0059] Style differentiation feature vectors carry out the appearance attributes of a person, such as age, gender, style and temperament, and make stylistic modifications to the exaggeration, softness and muscle movement habits of the basic expression base to ensure the consistency of the visual style of the expression.

[0060] In the embodiments of this specification, linear expression features are features of global facial morphological changes described by coarse-grained parameters, such as basic facial expressions like upturned corners of the mouth, raised eyebrows, and widened eyes. These features can generate basic expressions through linear combination. The coarse-grained parameters can be 3DMM coefficients or Blendshape weights. In this embodiment, Blendshape weights are used as the coarse-grained parameters, and each Blendshape represents a vertex displacement pattern of a basic expression.

[0061] In this embodiment of the specification, the step of using the initial expression base as the deformation basis to perform linear transformation and feature fusion on the individual feature vector to generate the personalized expression base specifically includes:

[0062] A phase transition mapping matrix is ​​constructed based on the initial expression base and the individual feature vectors to obtain adaptive correction weights;

[0063] The initial expression base is modified using a linear weighting method based on the adaptive correction weights to generate modified features;

[0064] The modified features are then subjected to feature concatenation and linear fusion operations to generate the personalized expression base.

[0065] In this embodiment of the specification, a phase transition mapping matrix is ​​constructed based on the initial expression base and the individual feature vectors to obtain adaptive correction weights, specifically including:

[0066] The individual feature vector is decomposed into individual feature components. The initial expression base is divided into regions according to the facial region, which are then divided into expression base facial regions. The individual feature components are then bound one-to-one with the expression base facial regions.

[0067] Based on individual feature vectors, a personalized deformation mapping weight matrix, i.e. a phase transformation mapping matrix, is generated through matrix transformation operations;

[0068] Based on the phase transition mapping matrix, a nonlinear mapping relationship between the initial expression basis and individual feature vectors is established;

[0069] The initial expression base is deformed and corrected using the nonlinear mapping relationship. The individual feature components are then embedded into the initial expression base through feature splicing, linear weighted fusion and spatial dimension mapping operations to obtain the fused expression.

[0070] The fused facial expressions, after undergoing smoothing constraint processing, are then processed through a phase transition mapping matrix to generate adaptive correction weights.

[0071] Individual feature components include: contour feature components, facial topology components, soft tissue elasticity components, skeletal constraint components, and style correction components. Among them, the contour feature component is used to characterize the overall facial shape differentiation attributes of the digital human, including global shape parameters such as face length-to-width ratio, facial contour curvature, cheekbone width, jawline curvature, and profile contour shape; the facial feature topology component is used to characterize the inherent structure and spatial distribution differences of facial features, covering detailed parameters such as eyebrow-eye distance, eyebrow-eye shape, eyelid opening and closing limit, brow bone undulation, eye corner shape, nasal alar structure, and the topological relationship of the relative positions of facial features; the soft tissue elasticity component is used to characterize the physical properties of facial subcutaneous soft tissue, including mechanical characteristics such as skin tightness, cheek and perioral soft tissue stretching limit, muscle flexibility, skin-muscle adhesion characteristics, and soft tissue accumulation and contraction coefficient; the skeletal constraint component is used to characterize the facial underlying skeletal movement restriction parameters, including physiological limit conditions such as mandibular movement range, maxillofacial rotation angle, skeletal linkage constraint, occlusal movement range, and craniofacial skeletal support boundary; and the style correction component is used to characterize the digital human's global personalized expression style attributes, including global control parameters such as expression softness, deformation subtlety, overall exaggeration coefficient, expression transition smoothness, and visual style characteristics.

[0072] The facial regions include: eyebrows, eye area, mouth area, jawline, and facial contour; the expression-based facial regions include: facial contour area, eyebrows area, eye area, mouth area, jawline area, and cheek area.

[0073] The one-to-one correspondence between individual feature components and facial expression base regions is as follows: contour feature components correspond to facial contour regions; facial feature topology components correspond to eyebrow regions and perioral regions; soft tissue elasticity components correspond to perioral regions; skeletal constraint components correspond to mandibular regions; and style correction components correspond to the entire face fusion region. The entire face fusion region refers to the entire facial region including facial contour regions, eyebrow regions, perioral regions, perioral regions, mandibular regions, and cheek regions.

[0074] In the embodiments described in this specification, the phase transition mapping matrix is ​​used to assign independent differential correction coefficients to each mesh vertex of the face and each type of basic initial expression basis.

[0075] The deformation correction of the initial facial expression base using the aforementioned nonlinear mapping relationship specifically refers to: using contour feature components and facial feature topology components to correct the global and local vertex offset amplitude of the initial facial expression base, adapting it to the facial proportions, facial feature shapes, and facial feature spacing of the target digital human; using soft tissue elasticity components to dynamically adjust the deformation amplitude coefficients of the local traction, compression, and stretching of the facial expression base, adapting to the physical deformation differences in skin tightness and soft tissue flexibility of different individuals; and using skeletal constraint components to limit and constrain the extreme deformation range of the initial facial expression base, regulating the movement range of key areas such as jaw opening and closing and eye rotation, preventing deformation beyond physiological limits, thereby completing the deformation correction of the initial facial expression base.

[0076] The individual feature components are embedded into the deformation data corresponding to the initial expression base through feature splicing, linear weighted fusion and spatial dimension mapping operations. The purpose is to obtain the fused expression by completing the scaling of expression deformation, structural morphology correction and local detail adaptation while preserving the original expression base's motion logic, expression semantics and action linkage relationship.

[0077] After the fused facial expressions undergo smoothing constraint processing, they are then processed by a phase transition mapping matrix to generate adaptive correction weights. This smoothing constraint process includes global smoothing constraints and local deformation equalization and normalization, thereby suppressing abnormal deformations such as vertex abrupt changes, excessive local stretching, and topological distortion. The phase transition mapping matrix is ​​used for convergence calculation to obtain the adaptive correction weights, which include adaptive weights for physiological structure and adaptive weights for motion characteristics. The adaptive weights for physiological structure consider the differences in physiological structure between the initial feature base and individual feature components, while the adaptive weights for motion characteristics consider the differences in motion between the initial feature base and individual feature components.

[0078] Step S205: The semantic features and the linear expression features extracted by the conditional prior encoder are used together as conditional inputs into the preset diffusion model to generate nonlinear expression features and virtual skeleton control parameters.

[0079] In this embodiment of the specification, the semantic features and the linear expression features extracted by the conditional prior encoder are jointly used as conditional inputs into a preset diffusion model to generate nonlinear expression features and virtual skeleton control parameters, specifically including:

[0080] After preprocessing, the linear facial expression features are encoded by a conditional prior encoder and compressed into a latent vector.

[0081] After fusing the temporal feature sequence generated based on the semantic features with the latent vector, a time step embedding is introduced and input into a preset diffusion model to generate nonlinear facial expression features and virtual skeleton control parameters.

[0082] In the embodiments described in this specification, the nonlinear facial expression features are generated by fusing virtual skeleton displacement, skeleton rotation parameters, and linear facial expression features;

[0083] The calculation formula for the nonlinear facial expression features is as follows:

[0084] ;

[0085] in,

[0086] Represents non-linear facial expression features;

[0087] Indicates the number of personalized emoji bases;

[0088] Indicates the total number of personalized emoji bases;

[0089] Indicates the weight of the skeletal points;

[0090] This represents the i-th personalized emoji base;

[0091] Represents the bone rotation matrix;

[0092] Represents the bone displacement vector. Indicates the increment symbol. Represents the basic position vector of the skeleton;

[0093] This represents the translation parameters of the skeleton.

[0094] In the embodiments of this specification, the nonlinear facial expression features include subtle muscle deformation expressions, local micro-expressions, facial soft tissue compression and stretching expressions, muscle detail deformation expressions, and skeletal micro-movement related expressions, which are used to characterize the refined facial dynamic changes that linear expressions cannot cover.

[0095] Among them, subtle muscle deformation expressions refer to small, low-amplitude contraction and relaxation changes in the superficial facial muscles, including subtle tremors of the brow muscles, slight tightening of the muscles around the eyes, and weak pulling of the orbicularis oris muscles, which are not easily noticeable local muscle movements. This type of expression does not change the overall facial posture, but only reflects subtle emotional fluctuations through slight changes in muscle tension. It belongs to the refined micro-dynamic expressions that linear expressions cannot simulate.

[0096] Local micro-expressions refer to instantaneous, subtle, and short-term transitional facial expressions limited to local areas such as the eyes, eyebrows, corners of the mouth, and nostrils. These include subtle movements such as slightly lowering the corners of the eyes, slightly raising or lowering the eyebrows, slightly shifting the corners of the mouth, and slightly opening or closing the nostrils. They are used to reproduce the subconscious emotional expressions of real humans and enhance the naturalness and realism of digital human emotions.

[0097] Facial soft tissue compression and stretching expressions refer to the natural compression, stretching, relaxation, and adhesion deformation of the soft tissues of the face, such as the cheeks, around the mouth, and around the eyes, as they move with facial expressions. This includes changes in the adhesion of facial skin and flesh, local soft tissue accumulation and stretching, and the soft physical deformation effects such as slight bulging and sagging of the cheeks, simulating the mechanical movement characteristics of real human facial skin and soft tissue.

[0098] Texture detail deformation expression refers to the subtle changes in skin texture, such as slight undulations, minor changes in local wrinkles, and subtle fluctuations in facial texture that occur synchronously with facial expressions. It does not change the overall geometric shape of the face, but only enhances the realistic texture of the expression, solving the problem of the stiffness of linear expressions that are smooth and lack skin details.

[0099] Skeletal micro-movement related expressions refer to refined, linked expressions driven by minute displacements, slight rotations, and small-angle coordinated movements of the facial virtual skeleton. These include subtle skeletal movements such as slight jaw shifts, slight jaw and facial rotations, and minor movements of the orbital bones. Through these skeletal micro-movements, the surface soft tissues undergo coordinated deformation, resulting in a richly layered and physiologically consistent natural non-linear expression effect.

[0100] The virtual skeleton control parameters include: three-dimensional translation of the skeleton, three-axis small-angle rotation, partitioned skeleton linkage constraint coefficient, and skeleton motion limit threshold. Specifically, the three-dimensional translation of the skeleton represents the tiny linear offset distance of a single facial virtual bone along the three orthogonal axes X, Y, and Z in the three-dimensional spatial coordinate system. It is used to precisely control the spatial position of the bone points in front-back, left-right, and up-down directions, adapting to the subtle displacement deformation of the facial contour, perioral region, and jaw area. The three-axis small-angle rotation represents the arc-level micro-rotation angle of the virtual bone around its own three-axis coordinate system, adapting to the pitch, yaw, and roll weak-angle linkage posture adjustment of the eyebrows, perioral region, and maxillofacial bones, conforming to the physiological rotation range of micro-movements of real facial features. The partitioned skeleton linkage... The constraint coefficient is the weight of the collaborative coupling between multiple adjacent bones in the same region. It is used to constrain the synchronous movement rhythm and linkage amplitude matching relationship of the same group of bones, so as to avoid the problem of local facial tearing and linkage disconnection caused by the independent movement of a single bone. The bone movement limit threshold is the safety boundary of the maximum translational stroke and maximum rotation angle of the bone preset according to the human maxillofacial anatomy and physiology. It is used to limit the bone movement beyond the limit in real time, and prevent the deformation beyond the physiological limit from causing mesh pulling, clipping, and skin and flesh misalignment distortion. In specific embodiments, the bone movement limit threshold is determined according to the business scenario, and is not specifically limited here.

[0101] In the embodiments of this specification, the preset diffusion model adopts DDPM (Denoising Diffusion Probability Model), which uses existing technology and will not be described in detail here.

[0102] The conditional prior encoder is the core component of the conditional variational autoencoder. The conditional prior encoder uses existing technology, which will not be elaborated here.

[0103] Step S207: The nonlinear facial expression features and the virtual skeleton control parameters are input into a preset reinforcement learning model. Through left-right symmetry constraints, the final generated facial expression result is generated, and the final generated facial expression result serves as the digital human facial expression corresponding to the speech content to be processed.

[0104] In this embodiment of the specification, the nonlinear facial expression features and the virtual skeleton control parameters are input into a preset reinforcement learning model. Through left-right symmetry constraints, a final facial expression result is generated. This final facial expression result serves as the digital human expression corresponding to the speech content to be processed, specifically including:

[0105] The nonlinear facial features and the virtual skeleton parameters are input into the soft tissue stress distribution function to obtain candidate expressions under physical constraints.

[0106] Using the candidate expressions under the physical constraints as the initial state, the nonlinear expression features and virtual skeleton control parameters are adjusted by the proximal policy optimization algorithm to update the candidate expressions under the physical constraints. The final generated expression result is determined by the reward function of the left-right symmetry constraint.

[0107] In the implementation of this specification, the preset reinforcement learning model adopts the Markov framework and uses existing technology, which will not be elaborated here.

[0108] In the embodiments described in this specification, the expression for the soft tissue stress distribution function is:

[0109] ;

[0110] in,

[0111] This represents the stress distribution function of the soft tissue;

[0112] This represents the strain of soft tissue at position x and time t;

[0113] Indicates the initial strain;

[0114] Indicates the elastic modulus;

[0115] Indicates the viscosity coefficient;

[0116] This indicates that the partial differential equation is true;

[0117] The expression for the reward function of the left-right symmetry constraint is:

[0118] ;

[0119] in,

[0120] R represents the reward function of the left-right symmetry constraint;

[0121] This indicates the final generated facial expression result;

[0122] This represents the result of mapping facial expressions to be symmetrical from left to right.

[0123] Indicates stress;

[0124] A symmetric mapping representing stress;

[0125] and Indicates the weighting factor;

[0126] Indicates the number of personalized emoji bases;

[0127] This represents the total number of personalized emoji bases.

[0128] To further understand the digital human facial expression interaction generation method based on tissue dynamics provided in the embodiments of this specification, Figure 3 This is a framework diagram of a digital human facial expression interaction generation method based on organizational dynamics, provided for embodiments of this specification. Figure 3 As shown, the process first employs a linear expression-based dynamic modeling module weighted by facial expression bases. This module uses voice-driven methods to establish a precise mapping between semantics and facial expression bases, enabling the generated linear expressions to possess preliminary semantic consistency and emotional expression capabilities, providing conditional input for subsequent nonlinear modeling. Subsequently, a nonlinear expression-based dynamic modeling module driven by virtual skeletons, building upon pseudo-muscles, achieves complex and nuanced expression changes, making the generated expressions richer and more natural at the detail level. This accurately expresses complex emotional information and provides higher-quality basic expression data for refined modeling. Finally, an embedded soft tissue dynamics-based refined expression modeling module, through the fusion of soft tissue dynamics laws and symmetry constraints, aims to achieve refined expression modeling, ensuring the final generated results possess high realism, high naturalness, and physical consistency.

[0129] The digital human facial expression interaction generation method based on organizational dynamics provided in this specification extracts the phonetic and semantic features of the speech content to be processed. A personalized expression base and weighted coefficients are combined using a linear skinning formula to generate linear facial expression features. The personalized expression base is generated by combining an initial expression base with individual feature vectors, and the weighted coefficients are generated by encoding the phonetic features using a speech temporal feature encoder. The semantic features and the linear facial expression features extracted by a conditional prior encoder are input as conditions into a preset diffusion model to generate nonlinear facial expression features and virtual skeleton control parameters. The nonlinear facial expression features and the virtual skeleton control parameters are input into a preset reinforcement learning model, and through left-right symmetry constraints, the final generated facial expression result is generated. This final generated facial expression result serves as the digital human expression corresponding to the speech content to be processed, enabling the generated digital human expression to possess high realism, high naturalness, and physical consistency, improving the naturalness and expressiveness of the generated expression, and ultimately achieving coordinated generation of digital human expressions and movements.

[0130] The above describes in detail a method for generating digital human facial expressions based on organizational dynamics. Correspondingly, this specification also provides a device for generating digital human facial expressions based on organizational dynamics, such as... Figure 4 As shown. Figure 4 A digital human facial expression interaction generation device based on tissue dynamics is provided as an embodiment of this specification. The digital human facial expression interaction generation device includes:

[0131] Extraction module 401 extracts the phonetic and semantic features of the speech content to be processed;

[0132] The linear table case-changing module 403 generates linear expression features by combining personalized expression base and weighting coefficients through a linear skinning formula. The personalized expression base is generated by combining an initial expression base with an individual feature vector, and the weighting coefficients are generated by encoding the phonological features through a speech temporal feature encoder.

[0133] The nonlinear expression change module 405 uses the semantic features and the linear expression features extracted by the conditional prior encoder as conditional inputs into a preset diffusion model to generate nonlinear expression features and virtual skeleton control parameters.

[0134] The facial expression fine modeling module 407 inputs the nonlinear facial expression features and the virtual skeleton control parameters into a preset reinforcement learning model. Through left-right symmetry constraints, it generates the final facial expression result, which serves as the digital human expression corresponding to the speech content to be processed.

[0135] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0136] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments for apparatus, electronic devices, and non-volatile computer storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0137] The apparatus, electronic device, and non-volatile computer storage medium and method provided in the embodiments of this specification are corresponding. Therefore, the apparatus, electronic device, and non-volatile computer storage medium also have similar beneficial technical effects as the corresponding method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the corresponding apparatus, electronic device, and non-volatile computer storage medium will not be repeated here.

[0138] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0139] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside on local and remote computer storage media, including storage devices.

[0140] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0141] The above description is merely an embodiment of this specification and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of this application should be included within the scope of the claims of this application.

Claims

1. A method for generating digital human facial expression interaction based on organizational dynamics, characterized in that, The digital human facial expression interaction generation method includes: Extract the phonetic and semantic features of the speech content to be processed; A personalized expression base and weighted coefficients are combined to generate linear expression features through a linear skinning formula. The personalized expression base is generated by combining an initial expression base with individual feature vectors, and the weighted coefficients are generated by encoding the phonetic features through a speech temporal feature encoder. The semantic features and the linear facial expression features extracted by the conditional prior encoder are used together as conditional inputs into a preset diffusion model to generate nonlinear facial expression features and virtual skeleton control parameters. The nonlinear facial expression features and the virtual skeleton control parameters are input into a preset reinforcement learning model. Through left-right symmetry constraints, the final facial expression result is generated, and the final facial expression result serves as the digital human expression corresponding to the speech content to be processed.

2. The digital human facial expression interaction generation method as described in claim 1, characterized in that, The extraction of phonetic and semantic features of the speech content to be processed specifically includes: The speech content to be processed is passed through a speech feature encoder to extract phonetic features; The speech content to be processed is subjected to a large visual language model to extract semantic features.

3. The digital human facial expression interaction generation method as described in claim 1, characterized in that, The individual feature vectors include: facial contour feature vector, facial feature structure feature vector, facial physiological deformation feature vector, skeletal movement constraint feature vector, and style differentiation feature vector; The personalized expression base is generated by combining the initial expression base with individual feature vectors, specifically including: Using the initial expression base as the deformation base, the individual feature vector is linearly transformed and feature fused to generate the personalized expression base.

4. The digital human facial expression interaction generation method as described in claim 1, characterized in that, The weighting coefficients are generated by encoding the phonetic features using a speech temporal feature encoder, specifically including: The phonological features are input into the speech temporal feature encoder, which performs dimensional alignment, temporal context modeling, and deep feature extraction to obtain dynamic emotional association information, which includes speech rhythm, energy fluctuations, and fundamental frequency changes. The dynamic emotional association information is linearly mapped and weighted adaptively changed to generate a high-dimensional real-time weight vector, which includes: mouth shape deformation weight, eyebrow and eye movement weight, facial contour traction weight, cheek muscle deformation weight, jaw movement weight, and global expression intensity adjustment weight. The high-dimensional real-time weight vector is normalized to generate the weighting coefficients; The personalized expression base and weighting coefficients are combined using a linear skinning formula to generate linear expression features, specifically including: ; in, This represents the linear facial expression feature; Indicates the number of personalized emoji bases; Indicates the total number of personalized emoji bases; Indicates the weighting coefficients; This refers to the personalized emoticon base.

5. The digital human facial expression interaction generation method as described in claim 3, characterized in that, The process of using the initial expression base as the deformation basis to perform linear transformation and feature fusion on the individual feature vector to generate the personalized expression base specifically includes: A phase transition mapping matrix is ​​constructed based on the initial expression base and the individual feature vectors to obtain adaptive correction weights; The initial expression base is modified using a linear weighting method based on the adaptive correction weights to generate modified features; The modified features are then subjected to feature concatenation and linear fusion operations to generate the personalized expression base.

6. The digital human facial expression interaction generation method as described in claim 1, characterized in that, The semantic features and the linear facial expression features extracted by the conditional prior encoder are jointly used as conditional inputs into a preset diffusion model to generate nonlinear facial expression features and virtual skeleton control parameters, specifically including: After preprocessing, the linear facial expression features are encoded by a conditional prior encoder and compressed into a latent vector. After fusing the temporal feature sequence generated based on the semantic features with the latent vector, a time step embedding is introduced and input into a preset diffusion model to generate nonlinear facial expression features and virtual skeleton control parameters.

7. The digital human facial expression interaction generation method as described in claim 1, characterized in that, The nonlinear facial expression features are generated by fusing virtual skeleton displacement, skeleton rotation parameters, and linear facial expression features. The calculation formula for the nonlinear facial expression features is as follows: ； in, Represents non-linear facial expression features; Indicates the number of personalized emoji bases; Indicates the total number of personalized emoji bases; Indicates the weight of the skeletal points; This represents the i-th personalized emoji base; Represents the bone rotation matrix; Represents the bone displacement vector. Indicates the increment symbol. Represents the basic position vector of the skeleton; This represents the translation parameters of the skeleton.

8. The digital human facial expression interaction generation method as described in claim 1, characterized in that, The nonlinear facial expression features and the virtual skeleton control parameters are input into a preset reinforcement learning model. Through left-right symmetry constraints, a final facial expression result is generated. This final generated facial expression result serves as the digital human expression corresponding to the speech content to be processed, specifically including: The nonlinear facial features and the virtual skeleton parameters are input into the soft tissue stress distribution function to obtain candidate expressions under physical constraints. The candidate expressions under the physical constraints are used as the initial state. The nonlinear expression features and virtual skeleton control parameters are adjusted by the proximal policy optimization algorithm to update the candidate expressions under the physical constraints. The final expression result is determined by the reward function with left-right symmetry constraints.

9. The digital human facial expression interaction generation method as described in claim 8, characterized in that, The expression for the soft tissue stress distribution function is: ; in, This represents the stress distribution function of the soft tissue; This represents the strain of soft tissue at position x and time t; Indicates the initial strain; Indicates the elastic modulus; Indicates the viscosity coefficient; This indicates that the partial differential equation is true; The expression for the reward function of the left-right symmetry constraint is: ； in, R represents the reward function of the left-right symmetry constraint; This indicates the final generated facial expression result; This represents the result of mapping facial expressions to be symmetrical from left to right. Indicates stress; A symmetric mapping representing stress; and Indicates the weighting factor; Indicates the number of personalized emoji bases; This represents the total number of personalized emoji bases.

10. A digital human facial expression interaction generation device based on the laws of organizational dynamics, characterized in that, The digital human facial expression interaction generation device includes: The extraction module extracts the phonetic and semantic features of the speech content to be processed; The linear table case-varying module combines personalized expression bases and weighted coefficients to generate linear expression features through a linear skinning formula. The personalized expression base is generated by combining an initial expression base with individual feature vectors, and the weighted coefficients are generated by encoding the phonological features through a speech temporal feature encoder. The nonlinear expression change module uses the semantic features and the linear expression features extracted by the conditional prior encoder as conditional inputs into a preset diffusion model to generate nonlinear expression features and virtual skeleton control parameters. The facial expression fine modeling module inputs the nonlinear facial expression features and the virtual skeleton control parameters into a preset reinforcement learning model. Through left-right symmetry constraints, it generates the final facial expression result, which serves as the digital human expression corresponding to the speech content to be processed.

Images