Interactive digital human generation method and system based on picture and audio synthesis

By using image and audio synthesis technology, personalized 3D digital humans with natural interactive capabilities are generated, solving the problems of expensive equipment and high professional skill requirements in existing technologies, and realizing the rapid and low-cost generation and application of digital humans.

CN121330134APending Publication Date: 2026-01-13NEW ONE (BEIJING) TECH CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202511527273.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-10-24
Publication Date
2026-01-13

AI Technical Summary

Technical Problem

Existing technologies require high-precision equipment and specialized skills to generate digital humans, resulting in long production cycles and difficulty in rapid customization, thus failing to achieve natural interaction and personalization.

Method used

Using an image and an audio clip, a convolutional neural network is used to extract key points on the face and body. Combined with speech recognition and emotion analysis, a 3D digital human with multimodal interactive capabilities, including facial expressions and body movements, is generated.

Benefits of technology

It enables the low-cost and high-efficiency generation of personalized digital humans with natural interaction capabilities, suitable for scenarios such as virtual anchors, online education, and intelligent customer service, thus improving user experience and application efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN121330134A_ABST
    Figure CN121330134A_ABST
Patent Text Reader

Abstract

The invention discloses an interactive digital human generation method based on picture and audio synthesis, and belongs to the crossing field of artificial intelligence and computer graphics. According to the method, a full-link process of'feature extraction-model construction-emotion driving-real-time interaction-video output 'can be automatically completed only by uploading a character picture and a section of audio by a user: key point and semantic feature extraction is performed on the picture to obtain face / posture information; performing voice recognition, semantic analysis and emotion recognition on the audio to obtain a semantic tag and an emotion parameter; generating a personalized three-dimensional digital human based on the information, and driving the personalized three-dimensional digital human to generate facial expressions and limb actions which are synchronous with emotions; the user intention is analyzed in real time through natural language understanding and computer vision, and multi-modal interaction is achieved. According to the method, digital human creation can be completed without professional modeling and motion capture equipment, the generation cost is reduced, and the method can be widely applied to virtual anchors, online education, intelligent customer service and movie and television entertainment scenes.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of digital human technology, which is an intersection of artificial intelligence and computer graphics technology, and in particular to an interactive digital human generation method and system based on image and audio synthesis. Background Technology

[0002] With the rapid development of the metaverse and the live-streaming economy, the demand for digital humans is exploding. Existing solutions mainly rely on high-precision 3D scanning, multi-camera arrays, professional motion capture suits, and manual modeling. The equipment is expensive, the process is lengthy, the production cycle for a single character can be as long as several weeks, and the cost can be tens of thousands of yuan. Moreover, it is difficult to quickly customize based on user input.

[0003] On the other hand, while GAN-based "voice-driven portrait" technology can generate 2D speaking faces with synchronized lip movements, it cannot provide full-body movements and interactive capabilities; and traditional skeletal animation requires animators to adjust it frame by frame, which lacks real-time performance and personalization.

[0004] Therefore, how to generate 3D digital humans with natural interactive capabilities at low cost and high efficiency using only an image and an audio clip under the condition of "zero professional equipment and zero professional skills" has become a technical problem that the industry urgently needs to solve. Summary of the Invention

[0005] The purpose of this invention is to provide an interactive digital human generation method and system based on image and audio synthesis, thereby solving the aforementioned problems existing in the prior art.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] An interactive digital human generation method based on image and audio synthesis includes the following steps:

[0008] S1. Based on the input image of the person, extract facial key points, body posture key points and appearance semantic features to obtain the basic 3D model of the digital human.

[0009] S2. Based on the input audio, semantic labels and sentiment parameters are obtained through speech recognition, semantic analysis, and sentiment recognition.

[0010] S3. Based on semantic tags and emotional parameters, drive the 3D basic model to generate corresponding facial expression sequences and body movement sequences;

[0011] S4. Based on the real-time received user interaction information, analyze the user's intent and generate feedback content, and synchronously drive the facial expression sequence and body movement sequence to obtain a dynamic digital human with multimodal interaction capabilities.

[0012] S5. Based on the dynamic digital human, output a video file with synchronized voice, facial expressions and actions.

[0013] In some specific embodiments, step 1, "extracting facial key points," employs a convolutional neural network to perform feature mapping on the input image I, resulting in a 68×2-dimensional facial key point coordinate matrix.

[0014] P=CNN(I)∈ℝ^{68×2}.

[0015] In some specific embodiments, in step 2, "emotion recognition" obtains the probability distribution of emotion categories by combining the acoustic feature vector x with an emotion classifier:

[0016] p = Softmax(W·x + b),

[0017] Where x is the output of a 40-dimensional log-Mel filter bank, W∈ℝ^{N×40}, and N is the number of sentiment categories.

[0018] In some specific embodiments, the "facial expression sequence" in step 3 is generated by a linear combination of the emotion parameter e∈ℝ^{N} and the facial muscle motion basis vector B∈ℝ^{68×2×K}:

[0019] F(t) = P + B·(Λ·e(t)),

[0020] Where Λ=diag(λ1,…,λ K ) is the diagonal matrix of emotional intensity, and t is the time frame index.

[0021] In some specific embodiments, the "limb movement sequence" in step 3 is calculated using the skeletal joint rotation angle θ∈ℝ^{J×3}, where J is the number of joints, and the angle is calculated using the forward kinematics formula:

[0022] X=FK(θ; T),

[0023] T represents the skeletal hierarchy tree structure, and X∈ℝ^{J×3} represents the world coordinates of the joints.

[0024] In some specific embodiments, step 4, "user intent parsing," employs a Transformer encoder to generate an intent latent vector from the input text sequence S:

[0025] h = Transformer(S) ∈ ℝ^{d},

[0026] The probability of the intent category is p = Softmax(Wh + b).

[0027] In some specific embodiments, the "output video" in step 5 uses H.264 encoding, and the bitrate R is adaptively determined by the target quality factor Q.

[0028] R = α·Q + β·σ

[0029] Where σ is the mean square error of optical flow between consecutive frames, and α and β are preset coefficients.

[0030] In some specific embodiments, the "appearance semantic features" in step 1 are obtained through the output of the last layer of global average pooling in ResNet, denoted as:

[0031] f=GAP(ResNet(I))∈ℝ^{2048},

[0032] Then, f is concatenated with the facial key point coordinates P and input into a fully connected layer to obtain a joint feature vector for 3D modeling;

[0033] v = W[f;vec(P)] + b,

[0034] Where W∈ℝ^{512×2252}, b∈ℝ^{512}, D=512.

[0035] In some specific embodiments, the smoothness of the "limb movement sequence" in step 3 is controlled by Bézier curve interpolation: given the angle θ between adjacent keyframes k θ k+1 The interpolation angle in time t∈[0,1]

[0036] θ(t)=(1−t) 3 θ k +3(1−t) 2 tB1+3(1−t)t 2 B2+t 3 θ k+1 Where, control point B1 = θ k +⅓Δ,

[0037] B2=θ k+1 −⅓Δ,Δ=θ K+1 −θ K This is to ensure continuous action and minimal acceleration.

[0038] An interactive digital human generation system based on the same concept and using image and audio synthesis includes:

[0039] The multimodal input module is used to receive image materials and audio information input by the user;

[0040] The image processing and feature extraction module is used to extract facial features, body posture features, and appearance features of people in images.

[0041] The audio semantic analysis and sentiment recognition module is used to convert audio into text and analyze its semantics and sentiment.

[0042] The digital human modeling and rendering module is used to construct and render digital human models based on extracted feature information.

[0043] The motion and expression generation module is used to generate the motion and expression of the digital human based on the audio analysis results;

[0044] The interactive response module is used to enable natural language interaction between users and the digital human;

[0045] The output and management module is used to output the generated digital human and provide management functions.

[0046] The beneficial effects of this invention are:

[0047] 1. High efficiency and low cost generation: Users only need to provide simple images and audio materials, and the system can automatically complete the complex processes of digital human modeling, motion generation, and interactive logic construction. No professional modeling and animation skills are required, which greatly shortens the digital human generation time, reduces production costs, and enables ordinary users to easily create personalized digital humans.

[0048] 2. High degree of personalization: Based on user-input images and personalized parameter settings, the system can generate digital human images with unique appearances; combined with audio semantics and sentiment analysis, it generates actions and expressions that fit the content and emotions of the digital human, meeting the diverse creative needs of users and giving each generated digital human distinct personality characteristics.

[0049] 3. Natural and smooth interactive experience: Through multimodal interaction technology, the digital human can understand the user's verbal and non-verbal information and make natural and intelligent interactive responses. The interaction process is smooth and natural, which enhances the sense of interaction and immersion between the user and the digital human. It is suitable for scenarios such as virtual anchors and intelligent customer service where the interactive experience is highly demanding.

[0050] 4. Wide Application Adaptability: The interactive digital humans generated by the system can be flexibly applied to multiple fields such as virtual anchors, online education, intelligent customer service, film and television production, and gaming. In virtual anchor scenarios, digital humans can achieve real-time live interaction; in the field of online education, digital humans can act as virtual teachers to explain knowledge; in intelligent customer service scenarios, digital humans can answer users' questions in a more approachable way, expanding the application boundaries of digital humans.

[0051] 5. Convenient Management and Integration: The system provides comprehensive digital human management functions, facilitating user management and maintenance of digital human projects. Simultaneously, it supports rapid integration with third-party applications, reducing the difficulty of deploying digital humans on different platforms and improving the application efficiency and promotional value of digital humans. Attached Figure Description

[0052] Figure 1 This is a flowchart of the method of the present invention;

[0053] Figure 2 This is a system workflow diagram of the present invention. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0055] Reference Figure 1 and Figure 2 The interactive digital human generation method based on image and audio synthesis, as shown, includes the following steps:

[0056] S1. Based on the input image of the person, extract facial key points, body posture key points and appearance semantic features to obtain the basic 3D model of the digital human.

[0057] S2. Based on the input audio, semantic labels and sentiment parameters are obtained through speech recognition, semantic analysis, and sentiment recognition.

[0058] S3. Based on semantic tags and emotional parameters, drive the 3D basic model to generate corresponding facial expression sequences and body movement sequences;

[0059] S4. Based on the real-time received user interaction information, analyze the user's intent and generate feedback content, and synchronously drive the facial expression sequence and body movement sequence to obtain a dynamic digital human with multimodal interaction capabilities.

[0060] S5. Based on the dynamic digital human, output a video file with synchronized voice, facial expressions and actions.

[0061] The implementation details of S1 from "a single image" to "a three-dimensional basic model" are as follows:

[0062] Image preprocessing

[0063] First, the uploaded image is automatically cropped, the person area is centered and expanded to a uniform aspect ratio, and then light noise reduction and color equalization are performed to keep the shadows and highlights within the standard range, so as to avoid deviations in subsequent feature extraction due to exposure differences.

[0064] Facial key point extraction

[0065] A cascaded deep network is used: the first layer quickly locks the face bounding box; the second layer accurately locates 68 semantic points within the bounding box, including the corners of the eyes, the tip of the nose, the corners of the lips, and the chin; the third layer converts the two-dimensional coordinates into a three-dimensional point cloud with depth estimation, providing "skeleton anchor points" for subsequent modeling.

[0066] Extraction of key points of body posture

[0067] The trunk detection branch runs in parallel, first locating the center of the chest cavity from top to bottom, then extending to the lower limbs, upper limbs, and head, outputting the two-dimensional coordinates and visibility confidence of 25 major joints; for invisible joints, the reasonable positions are automatically completed using human kinematic priors, and an initial depth is assigned.

[0068] Appearance semantic feature extraction

[0069] A 2048-dimensional "appearance fingerprint" is output at the bottleneck layer of the network, which encodes global information such as skin color, hair density, clothing pattern, and accessory category; at the same time, the original image's 512×512 texture map is retained as the color source for subsequent rendering.

[0070] Geometric modeling

[0071] The "facial 3D point cloud + body joint chain" obtained in the previous step is aligned with the standard topology template. Through deformation and transfer algorithm, the vertices of the template mesh are stretched to the same proportion and outline as the photo to form a personalized initial mesh. At the same time, the volume of loose parts such as hair and clothing is supplemented by voxelization and sculpting, so that the overall shape and the silhouette of the photo have an error of less than 1mm.

[0072] Material and texture generation

[0073] Using the original image's texture and depth information, the system automatically distinguishes four major material regions: skin, hair, fabric, and accessories. It enables micro-normal detail mapping for the skin region, preserves the pattern for the fabric region, and generates anisotropic specular parameters for the hair region, ultimately outputting a set of PBR texture packages that can be directly used for physically based rendering.

[0074] Basic model encapsulation

[0075] The mesh, skeleton, textures, and material parameters are packaged into a lightweight format and written into the "driving anchor point index" of 68 points for the face and 25 points for the body, for quick access by the subsequent expression and motion modules. At this point, stage S1 is complete, resulting in a 3D basic model that is highly similar to the photograph and possesses topological consistency.

[0076] Implementation details of S2 from "a piece of audio" to "semantic tags + sentiment parameters"

[0077] Voice front-end processing

[0078] First, the input audio is resampled at 16kHz, silence segments are removed, and gain is normalized. Then, a speech activity detection algorithm is used to segment effective sentences, remove breathing and noise, and retain clean segments.

[0079] Speech recognition

[0080] The text segment is fed into an end-to-end acoustic model, which outputs a character sequence with timestamps. Then, combined with a large-scale language model, homophones are automatically corrected to obtain text with an accuracy of over 95%.

[0081] Semantic analysis

[0082] The entire text is segmented, part-of-speech tagging is performed, and entity recognition is performed to extract the subject-verb-object triples. Then, an intent classifier is used to map the sentence to major categories such as "greeting, question and answer, instruction, and small talk" and to give the confidence level. The parts containing numerical values ​​and names are stored separately to provide material for the subsequent "reply" of the digital human.

[0083] Emotion recognition

[0084] While performing semantic analysis, the acoustic channels are processed in parallel: Mel-frequency cepstral coefficients, energy, and fundamental frequency curves of each frame are extracted to form 40-dimensional dynamic features; these are input into a temporal sentiment network to output five-dimensional probabilities of "happy, sad, angry, surprised, and neutral"; if the probability of a certain emotion is higher than 0.6, it is locked as the final emotion category, and an intensity value of 0–1 is given simultaneously.

[0085] Tag and parameter encapsulation

[0086] The intent category, keyword list, emotion category, and emotion intensity are packaged into a single structured data set, with timestamps corresponding one-to-one with voice frames. Subsequent driving modules can directly read the data by timestamps to achieve precise triggering of "which word is spoken and which emotion frame it corresponds to".

[0087] The implementation details of S3, from "semantics + emotion" to "expression and action sequence", are as follows:

[0088] Facial expression generation

[0089] A "feeling-muscle" lookup table is pre-built: each emotion category corresponds to a set of facial muscle contraction intensities; the emotion intensity obtained from S2 is used as a coefficient to linearly scale the muscle vector to obtain the 52-dimensional muscle activation value of the current frame; then, the muscle-skin mapping algorithm is used to convert the activation value into the displacement of 68 facial key points, driving the mesh vertices to deform in real time, forming subtle expressions such as smiling, frowning, and surprise.

[0090] Body movement generation

[0091] First, candidate templates such as "standing explanation, sitting dialogue, waving to emphasize" are pre-selected from the action library according to the intention category; then, the emotional intensity is mapped to the action amplitude curve: when the intensity is high, the arm swing amplitude increases by 30% and the speed increases by 15%; when the intensity is low, the action tends to be gentle.

[0092] rhythm alignment

[0093] Based on the phoneme boundaries of the speech recognition output, a "stress-unstress" rhythm pulse is generated; key gestures are inserted at the stress position and transition interpolation is performed in the unstress interval to align the gesture rise and fall with the speech beat and avoid "sound-image misalignment".

[0094] Smooth transition

[0095] Cubic Bézier curves are used to interpolate adjacent keyframes to ensure the continuity of angular velocity and angular acceleration; output frame rates of 30fps and 60fps are set for facial expressions and movements respectively, and finally merged into a unified 60fps sequence to eliminate the feeling of frame skipping.

[0096] Sequence output

[0097] Write the 68-point facial displacement and 25-joint rotation angle of each frame into the timeline to form a ".seq" driver file; the rendering engine can then read the file by frame number and render it instantly, completing the S3 stage.

[0098] Implementation details of S4 from "user interaction information" to "multimodal dynamic digital human"

[0099] Interactive information capture

[0100] If the user speaks through the microphone, the system repeats the S2 process, outputting new intentions and emotions in real time; if the user turns on the camera, the visual branch runs in parallel: every 100ms, the system detects the user's head posture and gestures, recognizes four types of actions: "nodding, shaking head, waving hand, and pointing", and provides the start and end times.

[0101] Intended fusion

[0102] Combine voice intent and visual intent using an "OR" logic: for example, if a user says "yes" and nods, the confidence level is calculated by the weighted average of the two; if there is a conflict between voice and gesture, the one with higher confidence level is used first to prevent misjudgment.

[0103] Feedback content generation

[0104] After confirming the intent, the system retrieves the best response text from the knowledge graph or casual conversation corpus and generates 16kHz audio using a speech synthesis engine. At the same time, based on the emotional polarity of the response text, the system automatically assigns the digital human "emotion when responding" to achieve "content-emotion" consistency.

[0105] Synchronous drive

[0106] The rhythm pulses and emotional parameters of the new audio are sent to the existing expression-action pipeline in S3 in real time, and the keyframe sequence of the next 2 seconds is refreshed immediately; for the animation that is playing, a 50% mixed weight is used to gradually transition to the new sequence to avoid abrupt jumps.

[0107] loop iteration

[0108] The system continuously executes the "hear-see-think-move" closed loop with a cycle of 200ms, allowing the digital human to maintain eye contact, voice dialogue, and gesture responses, forming a natural and smooth multimodal interactive experience.

[0109] The implementation details of S5's transition from "dynamic digital humans" to "distributable video files" are as follows:

[0110] Audio and video recording

[0111] On the rendering side, enable the 1080p, 60fps off-screen buffer and simultaneously enable the 48kHz audio track; before rendering each frame, first check the audio playback pointer to ensure that the lip movements and gestures on the screen are strictly aligned with the audio samples, with an error of no more than 1 frame.

[0112] Real-time encoding

[0113] The system uses an H.264 hardware encoder to convert RGBA frames to YUV420 and enables variable bitrate: 3Mbps for static images and automatically increases to 8Mbps for fast gesture segments to maintain sharp edges; the audio uses AAC-LC at a fixed 128 kbps in stereo.

[0114] Post-packaging

[0115] After recording, the H.264 video stream and AAC audio stream are written into an MP4 container, and editable metadata is inserted, including the start and end times of each subtitle, emotion tags, and keyframe indexes, to facilitate subsequent editing or direct uploading to a live streaming platform.

[0116] Quality self-inspection

[0117] The system automatically scans the output files. If it finds audio-visual misalignment, abnormal bitrate, or a black screen, it immediately triggers a re-recording. After passing the inspection, it generates a 30-second preview animation for the user to confirm the effect.

[0118] Multi-terminal distribution

[0119] The final video supports three resolutions: 1920×1080, 1280×720, and 854×480. Users can choose to download directly or push the video to third-party platforms such as Douyin, Bilibili, and Zoom with one click, achieving "instant playback after shooting." With this, the S5 stage is complete, and the digital human content can be released to the public.

[0120] In some specific embodiments, step 1, "extracting facial key points," employs a convolutional neural network to perform feature mapping on the input image I, resulting in a 68×2-dimensional facial key point coordinate matrix.

[0121] P=CNN(I)∈ℝ^{68×2}.

[0122] Input normalization

[0123] First, the user-uploaded images of any size are scaled down to 256×256 pixels, and then color-centered according to the ImageNet statistical mean to keep the network input distribution consistent and reduce the offset caused by differences in lighting and skin color.

[0124] Multi-level convolutional downsampling

[0125] The algorithm employs a repeated stacking of triples: convolution → batch normalization → ReLU. After every two convolutions, a 2×2 max pooling operation is performed, halving the spatial size and doubling the number of channels step by step. After four such "downsampling blocks," a high-dimensional feature map of 16×16×128 is obtained. At this point, each pixel contains approximately 16×16 pixels of original receptive field information, enabling the simultaneous capture of facial contours and local textures.

[0126] Multiscale receptive field fusion

[0127] Three types of dilated convolutions (3×3, 5×5, and 7×7) are introduced in parallel on the 16×16 feature map, with dilation rates set to 1, 2, and 3 respectively, to expand the receptive field without sacrificing resolution. The outputs of the three convolutions are concatenated by channel and then reduced to 128 channels by a 1×1 convolution, forming "multi-scale contextual features". This operation enables the network to locate fine details at the corners of the eyes as well as perceive the overall geometry from the chin to the forehead.

[0128] Enhanced attention

[0129] Global average pooling is calculated along the channel dimension to obtain a 128-dimensional vector. This vector is then compressed by a fully connected layer and expanded back to 128 dimensions. Sigmoid is used to generate channel weights of 0 to 1, which are then multiplied channel by channel with the original feature map. This step automatically amplifies channels that contribute significantly to keypoint localization (such as edges and corners) while suppressing background or illumination channels, thus improving robustness.

[0130] Upsampling and pixel-level prediction

[0131] A two-stage upsampling process, combining transposed convolution and regular convolution, is used to progressively restore the 16×16 feature map to 64×64. After each upsampling step, a 3×3 convolution is applied to refine the edges. The final result is a 64×64×68 heatmap stack, with each stack piece corresponding to the probability distribution of a facial key point.

[0132] Coordinate Decoding

[0133] Each 64×64 heatmap is Softmax normalized to become a probabilistic map. The centroid of the probabilistic map is calculated using the "mathematical expectation" method, which involves weighting the coordinates of each pixel according to probability to obtain the sub-pixel-level X and Y coordinates of the key point. The 68 heatmaps are decoded sequentially to finally form a 68×2 facial key point matrix P.

[0134] Geometric constraint post-processing

[0135] After mapping the prediction results back to the original image size, a "shape constraint" is performed using the face contour prior: a 68-point average face template is constructed, and the template is aligned to the prediction points through similarity transformation. Then, the template is used as a reference to slightly pull back single points that deviate too much, which maintains personalization and avoids abnormal jitter.

[0136] Through the above seven steps, the network only needs one forward calculation to output the coordinates of 68 points with a sub-pixel error of less than 1.5 pixels from the real face within 10ms, providing accurate and stable "anchor" input for subsequent 3D modeling.

[0137] In some specific embodiments, in step 2, "emotion recognition" obtains the probability distribution of emotion categories by combining the acoustic feature vector x with an emotion classifier:

[0138] p = Softmax(W·x + b),

[0139] Where x is the output of a 40-dimensional log-Mel filter bank, W∈ℝ^{N×40}, and N is the number of sentiment categories.

[0140] Speech segment extraction and windowing

[0141] The input audio is first subjected to speech activity detection to extract valid sentence segments; each segment is then divided into continuous frames with a frame length of 25ms and a frame shift of 10ms, and a Hamming window is added to each frame to reduce spectral leakage and prepare for subsequent frequency domain analysis.

[0142] Frequency domain transformation and Mel filtering

[0143] A Fast Fourier Transform (FFT) is performed on each frame of the windowed signal to obtain the power spectrum. Then, 40 triangular bandpass filters covering the 0–8kHz range are placed on the Mel scale. The power spectrum energy is integrated into each filter, and the logarithm is taken to form a 40-dimensional log-Mel vector. This vector preserves the main energy distribution of the human voice while also exhibiting robustness to noise and volume variations.

[0144] Dynamic feature supplementation

[0145] Based on the 40-dimensional static log-Mel, its first-order difference (Delta) and second-order difference (Delta-Delta) are calculated to describe the dynamic changes and acceleration changes of the spectrum, respectively. The static, first-order, and second-order differences are concatenated to form a 120-dimensional hypervector. To reduce the computational load, principal component analysis is used to compress the 120-dimensional vector back to 40-dimensional vectors, ultimately obtaining the frame-level acoustic feature vector x.

[0146] Sentiment classifier mapping

[0147] Each row of the pre-trained weight matrix W corresponds to an emotion (such as happiness, sadness, anger, surprise, neutral, etc., for a total of N rows), and each column corresponds one-to-one with a 40-dimensional acoustic feature; the bias vector b provides a baseline threshold for each emotion. During calculation, the feature vector x is multiplied by W, and then b is added to obtain the original N-dimensional score vector.

[0148] Softmax probability normalization

[0149] For each element of the N-dimensional score vector, take the exponent to make large values ​​larger and small values ​​smaller. Then sum all the exponent values ​​and divide each exponent value by the sum to obtain the normalized probability distribution p. At this point, the sum of all elements in the vector is 1, and each element represents the posterior probability of the corresponding sentiment category, which can be directly used for subsequent decision-making.

[0150] Category decision and intensity output

[0151] The dimension with the highest probability is selected as the final emotion category; at the same time, the maximum probability value is used as the emotion intensity, ranging from 0 to 1. The system continuously outputs the category and intensity every 200ms frame, forming an emotion curve that changes over time, providing real-time driving signals for the digital human's facial expressions and body movements.

[0152] Through the above six steps, the network can convert a piece of speech into a reliable emotion category probability in less than 3ms per frame, and synchronize the timestamp with the speech recognition result to ensure strict alignment of the subsequent "sound-emotion-image".

[0153] In some specific embodiments, the "facial expression sequence" in step 3 is generated by a linear combination of the emotion parameter e∈ℝ^{N} and the facial muscle motion basis vector B∈ℝ^{68×2×K}:

[0154] F(t) = P + B·(Λ·e(t)),

[0155] Where Λ=diag(λ1,…,λ K ) is the diagonal matrix of emotional intensity, and t is the time frame index.

[0156] Facial Expression Sequence Generation Process

[0157] Neutral reference positioning

[0158] First, take the coordinates of the 68 three-dimensional facial key points obtained in step 1 as the "neutral face" P. This frame corresponds to a completely relaxed and expressionless state of the face, and serves as the benchmark for all subsequent expression changes.

[0159] Muscle motion basis vector construction

[0160] A "muscle-shape" correspondence was pre-trained using a large amount of facial expression scan data: for six basic emotions—happiness, sadness, surprise, anger, fear, and disgust—as well as a neutral state, the average displacement of each key point from neutral to peak was calculated, forming six sets of independent displacement patterns, called six "muscle movement basis vectors." The length of each vector is consistent with the coordinate dimension of the facial key points, and they are stored in matrix B, corresponding to six columns. Each column represents the direction of shape change caused by a single emotion.

[0161] Emotional Intensity Diagonal Matrix

[0162] In real time, six types of emotion values ​​are extracted from the emotion probability vector obtained in step 2 and placed sequentially into the diagonal positions of the diagonal matrix Λ. The function of this matrix is ​​to transform "whether an emotion occurs" into "how strong the emotion is": when the probability is close to 1, the corresponding emotion is fully activated; when the probability is close to 0, the emotion contributes almost no deformation.

[0163] Linear combined deformation

[0164] Multiplying the sentiment probability vector e by the diagonal matrix Λ on the left yields a six-dimensional scaled sentiment intensity vector; then multiplying this intensity vector by the basis vector matrix B on the right is equivalent to summing the six basic displacement patterns by intensity weights to obtain the total displacement ΔF of the current frame.

[0165] Overlay on neutral face

[0166] The total displacement ΔF is added point-by-point to the neutral reference P to obtain the facial keypoint coordinates F at the current moment. This operation guarantees:

[0167] When there is no emotion, e≈0, F≈P, and the face remains natural;

[0168] When a single emotion is activated, F moves along the corresponding base direction, resulting in a pure expression;

[0169] When multiple emotions are mixed, the basis vectors are superimposed proportionally to generate delicate and complex composite expressions.

[0170] Timing smoothing

[0171] A Gaussian time-domain filter is applied to the emotional intensity vectors of multiple consecutive frames to eliminate sudden jumps; the filtered intensity is then fed into the linear combination mentioned above to ensure smooth facial expression transitions without jagged edges.

[0172] Subpixel refinement

[0173] After obtaining integer-level key points, curve fitting is used to make sub-pixel corrections to contour points such as the corners of the mouth and eyes, making the smile curve or frown depth more realistic. Finally, a 68×2 facial expression sequence F_t that changes continuously over time is output, which is used by the rendering engine to drive the facial mesh frame by frame.

[0174] In some specific embodiments, the "limb movement sequence" in step 3 is calculated using the skeletal joint rotation angle θ∈ℝ^{J×3}, where J is the number of joints, and the angle is calculated using the forward kinematics formula:

[0175] X=FK(θ; T),

[0176] T represents the skeletal hierarchy tree structure, and X∈ℝ^{J×3} represents the world coordinates of the joints.

[0177] "Body Movement Sequence" Generation Process (Plain Text Version)

[0178] Skeletal hierarchy tree construction

[0179] First, a "parent-child" tree-like skeleton is built for the digital human: the root node is placed in the pelvis, branching downwards into the thigh-lower leg-foot, and upwards into the spine-chest-shoulders-arms-hands, with the head hanging at the top of the chest cavity. Each node records the translation, rotation, and length relative to its parent node, forming a standard human topology T.

[0180] Joint rotation angle definition

[0181] For each frame of emotion and semantic tags, the system initially selects a "candidate action" from the action template library. This action is stored as a keyframe every 30ms. Each frame only provides the three-dimensional Euler angles (or quaternions) of each joint relative to its parent node, with a total dimension of J×3, denoted as θ. The value of J typically includes 22 to 25 major joints in the limbs, spine, and neck.

[0182] Emotional intensity mapping

[0183] Using the emotional intensity obtained from S2 as a scaling factor, the large-angle channels in θ (such as arm swing and head nodding) are proportionally magnified or reduced: when happy or angry, the elbow swing amplitude can be magnified by 20%; when sad, the shoulder drooping angle increases and the arm retracts as a whole. The scaling maintains the continuity of angles and avoids sudden jumps.

[0184] Forward kinematics calculation

[0185] Starting from the root node, proceed recursively level by level in depth-first order:

[0186] First, calculate the position of the child node in the parent space based on the rotation angle and length of the current node;

[0187] Then multiply the position by the world transformation matrix of the parent node to obtain the world coordinates of the child node;

[0188] Repeat until the leaf joint.

[0189] The entire process is encapsulated within the FK function: input the entire frame θ and the skeleton tree T, and output the world coordinates X∈ℝ^(J×3) for all joints.

[0190] Semantic rhythm alignment

[0191] For speech stress or punctuation pauses, the system inserts "emphasis keyframes" in real time: at the stress point, the wrist or head reaches maximum amplitude 50ms in advance to create a natural beat; during the unstressed intervals, the speed is reduced to maintain a sense of breathiness. Cubic Bezier interpolation is used between keyframes to ensure continuous angular velocity and angular acceleration, eliminating a mechanical feel.

[0192] Collision and Constraint Correction

[0193] After obtaining the world coordinates X, quickly detect whether there is penetration between the arm and torso, or between the left and right hands; if a collision occurs, the relevant joints are slightly moved along the reverse gradient until the gap is greater than 2cm. At the same time, the torsional angle of single-degree-of-freedom joints such as the elbow and knee is restricted to ensure physiological feasibility.

[0194] Timing output

[0195] The θ sequence is sampled at a frequency of 60fps, and the corresponding X is obtained by calling FK for each frame. t Write it to the timeline; the rendering engine uses X... t By moving the skeleton in real time, you can see smooth and natural arm swings, body tilts, or head nods that are synchronized with the voice and match emotions, thus generating a sequence of limb movements.

[0196] In some specific embodiments, step 4, "user intent parsing," employs a Transformer encoder to generate an intent latent vector from the input text sequence S:

[0197] h = Transformer(S) ∈ ℝ^{d},

[0198] The probability of the intent category is p = Softmax(Wh + b).

[0199] Step 4: Detailed Implementation Process of "User Intent Analysis"

[0200] Text preprocessing

[0201] First, the original sentence input by the user is segmented by character or word, and a special start marker is added at the beginning of the sentence and an end marker is added at the end. At the same time, corresponding position numbers are generated so that the model knows the order of each character in the sentence. Then, these symbols are mapped into a fixed-length vector to obtain a sequence representation with the same length as the sentence, which is used as the input of the Transformer.

[0202] Self-attention coding

[0203] The core of Transformer is the "multi-head self-attention" mechanism:

[0204] For each character vector, generate three temporary vectors: query, key, and value.

[0205] The query is compared with all keys to determine the relevance weight of the current character to other characters in the sentence.

[0206] These weights are used to sum all values ​​in a weighted manner, so that the new representation of each word incorporates global contextual information;

[0207] Multiple "heads" are opened in parallel, allowing different heads to focus on different aspects such as syntax, sentiment, and entities, and then the results of each head are spliced ​​together to restore the original dimension.

[0208] This process is repeated several times, with each layer followed by a residual connection and layer normalization to ensure gradient stability.

[0209] Intent Feature Convergence

[0210] After multiple layers of self-attention, the model generates a new context-rich vector for each character. The output corresponding to the special marker at the beginning of the sentence is taken as the "intent latent vector" h of the entire sentence. It has aggregated all local information, has a fixed dimension, and no longer changes with the sentence length.

[0211] Classification mapping

[0212] Each row of the pre-trained weight matrix W corresponds to a user intent (such as query, affirmation, negation, interruption, small talk, etc., totaling N types), and the number of columns is consistent with the dimension of the latent vector. Multiplying h and W by a matrix, and adding the specific bias b for each intent type, yields the N-dimensional raw score.

[0213] Probability normalization

[0214] The system takes the exponent of each element of the N-dimensional score vector, divides it by the sum, and forms a normalized probability distribution p. The element with the highest probability is the user intent identified by the system. All probability values ​​are retained for the downstream dialogue strategy module to determine the risk threshold: if the highest probability is still below the preset threshold, a safe response or clarification inquiry can be triggered.

[0215] Time sequence update

[0216] Intent parsing is performed once a complete sentence is received using a sliding window approach, outputting real-time labels. For multi-turn dialogues, the intent labels from the previous turn are also appended as additional features to the current input, enabling the model to track context and achieve "reference resolution" and "topic inheritance".

[0217] Through the above six steps, the Transformer encoder can compress text of arbitrary length into a fixed-dimensional latent intent vector in milliseconds and provide a reliable probability distribution, providing a basis for digital humans to choose the correct response strategy and emotional tone.

[0218] In some specific embodiments, the "output video" in step 5 uses H.264 encoding, and the bitrate R is adaptively determined by the target quality factor Q.

[0219] R = α·Q + β·σ

[0220] Where σ is the mean square error of optical flow between consecutive frames, and α and β are preset coefficients.

[0221] Step 5: Detailed Implementation Process of "Output Video" Bitrate Adaptive Control

[0222] Target quality factor setting

[0223] Before recording begins, users can choose one of three settings: "Smooth," "Standard," or "High Definition." The system maps this setting to a target quality factor Q in the range of 0–1: the Smooth setting has a low Q value, resulting in a tighter overall bitrate budget; the High Definition setting has a high Q value, retaining more details.

[0224] Calculation of optical flow mean square error

[0225] After converting two consecutive color images to grayscale, a dense optical flow algorithm is used to obtain the displacement vector field of each pixel. The sum of the squares of the X and Y components of this vector field is then averaged across the entire image to obtain the inter-frame motion intensity σ. The larger the range of motion of the digital human and the more frequent the camera switching, the higher the value of σ; in static dialogue scenes, σ is close to zero.

[0226] Linear combination to calculate code rate

[0227] In the preliminary experiment phase, coefficients α and β were derived through regression analysis using a large number of samples: α mainly reflects the "basic image quality requirements," and β reflects the "motion compensation increment." The formula R = α·Q + β·σ incorporates both static quality requirements and dynamic complexity, and the result R is the suggested bitrate for the current segment, in Mbps.

[0228] Upper and lower limit protection

[0229] To prevent bitrate overflow or excessively low bitrate in extreme scenarios, the system sets hard thresholds for R: an upper limit to avoid network bandwidth overload, and a lower limit to ensure that macroblocks are not over-compressed and produce pixelation. If the calculated value exceeds the range, it is truncated to the nearest boundary.

[0230] Code execution

[0231] After obtaining the frame-level bitrate, it is written into the RC (Rate Control) module of the H.264 encoder. The encoder uses this bitrate as a target and automatically assigns different quantization parameters to I-frames, P-frames, and B-frames to achieve instantaneous adjustment of "increasing the bitrate when there is a lot of motion and decreasing the bitrate when there is no movement" to maintain constant visual quality.

[0232] Fragment-level smoothing

[0233] For longer videos, σ is recalculated and R is updated every 1 second; when the bitrate of adjacent segments changes by more than 20%, a moving average method is used for transition to prevent viewers from perceiving a sudden change in image quality.

[0234] Through the above six steps, the system can save bitrate in "static explanation" segments and automatically provide additional bitrate compensation for "large gestures + turning" segments, while keeping the overall video bandwidth-friendly and avoiding motion blur or block effects.

[0235] In some specific embodiments, the "appearance semantic features" in step 1 are obtained through the output of the last layer of global average pooling in ResNet, denoted as:

[0236] f=GAP(ResNet(I))∈ℝ^{2048},

[0237] Then, f is concatenated with the facial key point coordinates P and input into a fully connected layer to obtain a joint feature vector for 3D modeling;

[0238] v = W[f;vec(P)] + b,

[0239] Where W∈ℝ^{512×2252}, b∈ℝ^{512}, D=512.

[0240] Step 1: Detailed Implementation Process of "Appearance Semantic Feature Extraction and Joint Feature Fusion"

[0241] Image fed into ResNet

[0242] The normalized 256×256 image of a person is input into a pre-trained ResNet network. The network first reduces the spatial size and increases the channel depth through multiple layers of convolution and residual blocks, and finally obtains a feature map with a small spatial size but a high number of channels of 2048 in the last layer.

[0243] Global average pooling

[0244] Global average pooling is performed on each channel of the feature map individually: the pixel values ​​in each channel are summed and then divided by the total number of pixels to obtain a single numerical value that represents the overall appearance information of the image. This process is repeated for all 2048 channels, outputting a 2048-dimensional vector f, called the "appearance semantic feature." It encapsulates global visual cues such as skin tone, hair texture, clothing texture, and accessory style.

[0245] Facial key point flattening

[0246] The coordinates P of the 68 three-dimensional facial key points detected in step 1 (a total of 68×3=204 dimensions) are concatenated into a one-dimensional long vector vec(P) in a fixed order to ensure that the order of geometric information corresponds to the subsequent network weights.

[0247] Feature splicing

[0248] The 2048-dimensional appearance vector f is concatenated with the 204-dimensional geometric vector vec(P) to form a 2252-dimensional composite vector. This vector contains both "what it looks like" and "where it is located," providing a complete basis for subsequent personalized modeling.

[0249] Fully Connected Dimensionality Reduction

[0250] The pre-trained fully connected weight matrix W has 2252 columns and 512 rows; the bias vector b has a dimension of 512. The concatenated long vector is fed into this layer, where a linear mapping is performed: multiplication and addition followed by bias, outputting a 512-dimensional joint feature vector v. This dimension preserves sufficient detail while avoiding parameter explosion in subsequent networks.

[0251] Normalization and activation

[0252] Batch normalization is performed on the 512-dimensional results to stabilize the mean and variance of each dimension. Then, the ReLU activation function is used to remove negative values, enhancing the non-linear expressive power, resulting in the final joint feature v used for 3D modeling. Subsequent deformation, texturing, and stylization modules all use this vector as the "facial fingerprint" input to achieve personalized reconstruction from "one image to one person."

[0253] In some specific embodiments, the smoothness of the "limb movement sequence" in step 3 is controlled by Bézier curve interpolation: given the angle θ between adjacent keyframes K θ k+1 The interpolation angle in time t∈[0,1]

[0254] θ(t)=(1−t) 3 θ k +3(1−t) 2 tB1+3(1−t)t 2 B2+t 3 θ k+1 ,

[0255] Where, control point B1=θ k +⅓Δ,B2=θ k+1 −⅓Δ,Δ=θ k+1 −θ K This is to ensure continuous action and minimal acceleration.

[0256] Step 3: Detailed Implementation Process of Bezier Curve Smoothing Interpolation for "Body Movement Sequence"

[0257] Keyframe selection

[0258] Extract two adjacent keyframes from the motion library or real-time generated joint rotation sequence, and denote them as the starting angle θ. K With the ending angle θK+1 The corresponding time point t K With t k+1 Several intermediate frames need to be created between two frames to ensure a smooth visual transition for the robotic arm or limb.

[0259] Parameterized time variables

[0260] Normalize the actual timeline: Define t = (tt) k ) / (t k+1 -t k The value t changes continuously from 0 to 1, representing the "current transition progress". All subsequent calculations are performed on this 0-1 interval to ensure that the interpolation is independent of the actual time length.

[0261] Control point construction

[0262] To obtain a trajectory with zero initial velocity, zero final velocity, and minimal change in acceleration, two internal control points are introduced:

[0263] The first control point B1 is set outside the starting point, at a distance of 1 / 3 of the angle difference between the two frames, i.e., B1 = θ. k +⅓Δ;

[0264] The second control point B2 is set outside the endpoint, at the same distance of 1 / 3 of the difference, but in the opposite direction.

[0265] That is, B2 = θ k+1 - ⅓Δ,

[0266] Where Δ=θ k+1 -θ k This arrangement ensures that the tangent direction of the curve at the start and end times is consistent with the keyframe connection, avoiding sudden acceleration or abrupt stops.

[0267] cubic Bézier interpolation

[0268] Using four points (θ) k ,B1,B2,θ K+1 ) The cubic Bézier curve is defined to calculate the interpolation angle at any point t:

[0269] The curve passes exactly through θ at t=0. k Furthermore, the tangential direction points to B1, and the initial velocity is moderate.

[0270] At t=1, it passes strictly through θ k+1 The tangential direction comes from B2, and the ending speed is also gentle;

[0271] The intermediate process consists of a fourth-order polynomial weight (1-t). 3 3(1-t) 2 t、3(1-t)t 2 t3 The control is continuous and differentiable in terms of angle change, and the acceleration curve is smooth without abrupt changes.

[0272] frame-by-frame sampling

[0273] Substituting τ at equal intervals of 0, 1 / F, 2 / F, ..., 1 according to the frame rate into the Bessel formula, we obtain the corresponding interpolation angle sequence θ(t). Since the curve shape is determined solely by θ... k θ k+1 The calculation is small and can be completed in milliseconds, depending on the two control points.

[0274] Angular velocity and angular acceleration verification

[0275] The first-order difference of the interpolation result is used to obtain the angular velocity, and the second-order difference is used to obtain the angular acceleration. If the peak acceleration is found to exceed the physiological comfort threshold, the control point offset is automatically reduced (for example, the 1 / 3 coefficient is reduced to 1 / 4), and the curve is regenerated to ensure that the joint movement of the robot or digital human conforms to natural biomechanics.

[0276] Loop splicing

[0277] The above process is repeated for all adjacent keyframes of the entire action sequence. Each curve maintains position and tangent continuity at the connection point, and the overall trajectory has no speed jumps. The final output is a smooth, natural, and low-energy limb action sequence for the rendering engine to drive in real time.

[0278] An interactive digital human generation system based on the same concept and using image and audio synthesis includes:

[0279] The multimodal input module is used to receive image materials and audio information input by the user;

[0280] The image processing and feature extraction module is used to extract facial features, body posture features, and appearance features of people in images.

[0281] The audio semantic analysis and sentiment recognition module is used to convert audio into text and analyze its semantics and sentiment.

[0282] The digital human modeling and rendering module is used to construct and render digital human models based on extracted feature information.

[0283] The motion and expression generation module is used to generate the motion and expression of the digital human based on the audio analysis results;

[0284] The interactive response module is used to enable natural language interaction between users and the digital human;

[0285] The output and management module is used to output the generated digital human and provide management functions.

[0286] 1. Multimodal input module:

[0287] The front-end interface features intuitive image upload and audio recording components. The image upload component supports batch and single image uploads, providing basic editing functions such as image preview, cropping, and rotation. The audio recording function supports one-click recording, pausing, and playback, facilitating audio recording for users, and also supports uploading and replacing local audio files. A simple parameter setting form is included, allowing users to adjust parameters such as gender, age, and style of the digital avatar using dropdown menus, sliders, and other interactive elements.

[0288] 2. Image processing and feature extraction module:

[0289] In the image preprocessing stage, image processing libraries such as OpenCV are used to perform noise reduction (e.g., median filtering), color correction (e.g., histogram equalization), and size normalization on uploaded images. Deep learning-based facial landmark detection models (e.g., MTCNN) and body pose estimation models (e.g., OpenPose) are employed to extract the coordinates of facial and body feature points in the images. Feature encoding algorithms are used to convert feature point information into feature vectors, which are then combined with deep learning models (e.g., ResNet, VGG) to extract semantic features and identify detailed appearance information within the images.

[0290] 3. Audio semantic analysis and sentiment recognition module:

[0291] Open-source speech recognition libraries (such as SpeechRecognition) are used to convert audio into text. Natural language processing toolkits (such as NLTK and spaCy) are then used to perform semantic analysis operations on the text, including word segmentation, part-of-speech tagging, and named entity recognition, to extract key semantic information. Deep learning-based sentiment classification models (such as LSTM and Transformer) are employed, combined with the acoustic features of the audio (such as Mel-frequency cepstral coefficients, MFCC), to determine the category and intensity of the emotion expressed in the audio.

[0292] 4. Digital Human Modeling and Rendering Module:

[0293] Based on the API interfaces of 3D modeling software (such as Blender and Maya), and combined with extracted image feature information, a basic 3D model of the digital human is generated. Mesh optimization algorithms are used to simplify and smooth the model, improving rendering efficiency. Texture mapping technology is employed to map the appearance textures from the image onto the surface of the digital human model. In the stylization stage, neural style transfer algorithms (such as Fast Neural Style and AdaIN) are used to transfer the user-selected style image features onto the digital human model. During rendering, physically based rendering (PBR) techniques, such as the Disney BRDF, are used to simulate the interaction between light and materials, combined with ray tracing algorithms to calculate realistic lighting and shadow effects, achieving high-quality digital human rendering.

[0294] 5. Action and Expression Generation Module:

[0295] For facial expression generation, a facial muscle motion parameter model is established, and muscle parameters are adjusted based on emotion recognition results to drive changes in the digital human's facial expressions. Skeletal animation technology from computer graphics is employed to construct a skeletal system for the digital human body. Based on semantic analysis results and kinematic models, corresponding rotation, translation, and other motion parameters are generated for the skeleton. Recurrent Neural Networks (RNNs) or Long Short-Term Memory Networks (LSTMs) are used to learn and predict action sequences, generating continuous and natural movements. For keyframe interpolation, algorithms such as Bézier curve interpolation and spline curve interpolation are used to generate smooth transition frames between key action frames, ensuring fluid and natural movements.

[0296] 6. Interactive Response Module:

[0297] A dialogue management system based on natural language processing is constructed. An intent recognition model (such as the BERT model based on Transformer) is used to parse the user's input speech or text. Combined with knowledge graphs and dialogue strategies, appropriate response content is generated. Computer vision technology is used to capture the user's body movements and facial expressions through a camera. Pose recognition and facial expression recognition algorithms are employed to analyze the user's nonverbal behavior, enabling the digital human to provide corresponding interactive feedback based on the user's nonverbal information, thus achieving multimodal interaction.

[0298] 7. Output and Management Module:

[0299] For different output formats, corresponding encoding and compression technologies are employed. For MP4 format, H.264 or H.265 encoding standards are used for video encoding, and AAC encoding is used for audio processing; for AVI format, DivX or XviD encoding algorithms are used; and for WebM format, the VP9 encoding standard is used. Video quality adjustment options are provided, allowing users to set parameters such as resolution, frame rate, and bitrate according to their needs. Regarding digital human management, a user management interface is designed, allowing users to view, edit, delete, and copy digital human projects. Batch management and tag categorization are supported, facilitating the organization and management of digital human works. Cloud storage utilizes a distributed storage system (such as Ceph or GlusterFS) to ensure data security and reliability, supporting automatic backup and disaster recovery.

[0300] By adopting the above-disclosed technical solution of this invention, the following beneficial effects are obtained:

[0301] 1. High efficiency and low cost generation: Users only need to provide simple images and audio materials, and the system can automatically complete the complex processes of digital human modeling, motion generation, and interactive logic construction. No professional modeling and animation skills are required, which greatly shortens the digital human generation time, reduces production costs, and enables ordinary users to easily create personalized digital humans.

[0302] 2. High degree of personalization: Based on user-input images and personalized parameter settings, the system can generate digital human images with unique appearances; combined with audio semantics and sentiment analysis, it generates actions and expressions that fit the content and emotions of the digital human, meeting the diverse creative needs of users and giving each generated digital human distinct personality characteristics.

[0303] 3. Natural and smooth interactive experience: Through multimodal interaction technology, the digital human can understand the user's verbal and non-verbal information and make natural and intelligent interactive responses. The interaction process is smooth and natural, which enhances the sense of interaction and immersion between the user and the digital human. It is suitable for scenarios such as virtual anchors and intelligent customer service where the interactive experience is highly demanding.

[0304] 4. Wide Application Adaptability: The interactive digital humans generated by the system can be flexibly applied to multiple fields such as virtual anchors, online education, intelligent customer service, film and television production, and gaming. In virtual anchor scenarios, digital humans can achieve real-time live interaction; in the field of online education, digital humans can act as virtual teachers to explain knowledge; in intelligent customer service scenarios, digital humans can answer users' questions in a more approachable way, expanding the application boundaries of digital humans.

[0305] 5. Convenient Management and Integration: The system provides comprehensive digital human management functions, facilitating user management and maintenance of digital human projects. Simultaneously, it supports rapid integration with third-party applications, reducing the difficulty of deploying digital humans on different platforms and improving the application efficiency and promotional value of digital humans.

[0306] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. An interactive digital human generation method based on image and audio synthesis, characterized in that, Includes the following steps: S1. Based on the input image of the person, extract facial key points, body posture key points and appearance semantic features to obtain the basic 3D model of the digital human. S2. Based on the input audio, semantic labels and sentiment parameters are obtained through speech recognition, semantic analysis, and sentiment recognition. S3. Based on the semantic tags and emotion parameters, drive the three-dimensional basic model to generate corresponding facial expression sequences and body movement sequences; S4. Based on the real-time received user interaction information, analyze the user's intent and generate feedback content, and synchronously drive the facial expression sequence and body movement sequence to obtain a dynamic digital human with multimodal interaction capabilities. S5. Based on the dynamic digital human, output a video file with synchronized voice, facial expressions and actions.

2. The method according to claim 1, characterized in that, In step 1, "extracting facial key points" uses a convolutional neural network to perform feature mapping on the input image I, resulting in a 68×2-dimensional facial key point coordinate matrix: P=CNN(I)∈ℝ^{68×2}.

3. The method according to claim 2, characterized in that, In step 2, "emotion recognition" obtains the probability distribution of emotion categories by combining the acoustic feature vector x with an emotion classifier. p = Softmax(W·x + b), Where x is the output of a 40-dimensional log-Mel filter bank, W∈ℝ^{N×40}, and N is the number of sentiment categories.

4. The method according to claim 3, characterized in that, In step 3, the "facial expression sequence" is generated by a linear combination of the emotion parameter e∈ℝ^{N} and the facial muscle movement basis vector B∈ℝ^{68×2×K}: F(t) = P + B·(Λ·e(t)), Where Λ=diag(λ1,…,λ k ) is the diagonal matrix of emotional intensity, and t is the time frame index.

5. The method according to claim 4, characterized in that, In step 3, the "limb movement sequence" is calculated using the skeletal joint rotation angle θ∈ℝ^{J×3}, where J is the number of joints, and the angle is calculated using the forward kinematics formula: X=FK(θ; T), T represents the skeletal hierarchy tree structure, and X∈ℝ^{J×3} represents the world coordinates of the joints.

6. The method according to claim 5, characterized in that, In step 4, "user intent parsing" uses a Transformer encoder to generate an intent latent vector from the input text sequence S: h = Transformer(S) ∈ ℝ^{d}, The probability of the intent category is p = Softmax(Wh + b).

7. The method according to claim 6, characterized in that, In step 5, the "output video" uses H.264 encoding, and the bitrate R is adaptively determined by the target quality factor Q. R = α·Q + β·σ Where σ is the mean square error of optical flow between consecutive frames, and α and β are preset coefficients.

8. The method according to claim 7, characterized in that, In step 1, the "appearance semantic features" are obtained through the output of the last layer of ResNet's global average pooling, denoted as: f=GAP(ResNet(I))∈ℝ^{2048}, Then, f is concatenated with the facial key point coordinates P and input into a fully connected layer to obtain a joint feature vector for 3D modeling; v = W[f;vec(P)] + b, Where W∈ℝ^{512×2252}, b∈ℝ^{512}, D=512.

9. The method according to any one of claims 1-8, characterized in that, The smoothness of the "limb movement sequence" in step 3 is controlled by Bézier curve interpolation: Given the angle θ of adjacent keyframes k θ k+1 The interpolation angle in time t∈[0,1] θ(t)=(1−t) 3 θ k +3(1−t) 2 tB1+3(1−t)t 2 B2+t 3 θ k+1 , Where, control point B1=θ k +⅓Δ,B2=θ k+1 −⅓Δ,Δ=θ k+1 −θ K This is to ensure continuous action and minimal acceleration.

10. An interactive digital human generation system based on image and audio synthesis, characterized in that, include: The multimodal input module is used to receive image materials and audio information input by the user; The image processing and feature extraction module is used to extract facial features, body posture features, and appearance features of people in images. The audio semantic analysis and sentiment recognition module is used to convert audio into text and analyze its semantics and sentiment. The digital human modeling and rendering module is used to construct and render digital human models based on extracted feature information. The motion and expression generation module is used to generate the motion and expression of the digital human based on the audio analysis results; The interactive response module is used to enable natural language interaction between users and the digital human; The output and management module is used to output the generated digital human and provide management functions.

Citation Information

Cited By

  • Video translation method and device

    CN121814981A