Digital human action generation system and method based on diffusion model and emotion recognition

Abstract

The invention relates to the technical field of image recognition, in particular to a digital human action generation system and method based on a diffusion model and emotion recognition. The method comprises the following steps: synchronously collecting a voice stream and a face video stream of a user, and respectively carrying out short-time Fourier transform and face key point detection to obtain a voice Mel spectrogram sequence and a key point displacement vector sequence; performing multi-modal fusion processing on the voice Mel spectrogram sequence and the key point displacement vector sequence to generate an emotion feature vector; and obtaining a digital human skeleton joint rotation state of the current frame, and carrying out emotion adaptive mapping on the emotion feature vector to obtain a conditional latent variable. By fusing a diffusion model, multi-modal emotion recognition and a physiological constraint mechanism and combining efficient inverse solution and parallel rendering technologies, emotion-driven, high-authenticity and low-delay real-time generation and stable interaction of digital human actions are realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition, and in particular to a digital human action generation system and method based on diffusion model and emotion recognition. BACKGROUND

[0002] Digital human limb action generation technology is gradually evolving from static preset to real-time dynamic generation. Traditional action generation methods mainly rely on action library retrieval, key frame interpolation or reconstruction based on motion capture data. Although basic action synthesis can be achieved, there is a lack of effective response to the real-time emotional state of users. The diffusion model is introduced into the action generation field due to its superior performance in image generation and sequence modeling, and can generate complex action sequences with high fidelity through a gradual denoising process. At the same time, emotion recognition technology has made significant progress in multi-modal feature fusion, speech and facial expression recognition, making it possible to perceive user emotions in real time. However, the diffusion model alone in action generation often faces problems such as insufficient biomechanical constraints, high complexity of inverse kinematics (IK) calculation, and limited real-time performance.

[0003] In the prior art, the diffusion model has not been fully integrated into the kinematic constraint mechanism, and the generated results are prone to produce physiological unreasonable actions such as joint out-of-bound and structure insertion. Traditional inverse calculation algorithms have high computational complexity, making it difficult to meet the high frame rate rendering requirements. Some systems do not introduce a confidence mechanism to dynamically balance accuracy and efficiency, lack an efficient multi-level verification framework, and make it difficult to balance real-time performance and accuracy. SUMMARY

[0004] Therefore, it is necessary to provide a digital human action generation system and method based on diffusion model and emotion recognition to solve at least one of the above technical problems.

[0005] To achieve the above-mentioned purpose, a digital human action generation method based on diffusion model and emotion recognition comprises the following steps:

[0006] Step S1: Synchronize the acquisition of user speech stream and facial video stream, and perform short-time Fourier transform and facial key point detection respectively to obtain a speech mel spectrum graph sequence and a key point displacement vector sequence; perform multi-modal fusion processing on the speech mel spectrum graph sequence and the key point displacement vector sequence to generate an emotional feature vector;

[0007] Step S2: Obtain the rotation state of the digital human skeleton joint of the current frame, and perform emotion adaptation mapping on the emotional feature vector to obtain a conditional latent variable; perform splicing processing on the conditional latent variable and a standard Gaussian noise vector to obtain a joint rotation angle prediction matrix;

[0008] Step S3: The joint rotation angle prediction matrix is subjected to a differentiable probability distribution loss constraint processing to obtain a joint rotation angle adjustment matrix; the joint rotation angle adjustment matrix and the preset end effector target position are jointly inferred to obtain a joint angle compensation amount;

[0009] Step S4: The joint angle compensation amount is calculated to obtain a prediction probability distribution information entropy, and an inverse solution agent confidence value is generated; when the confidence value is lower than a preset threshold, the motion posture of the current frame is verified based on the joint angle compensation amount, and the joint angle compensation amount is corrected based on the verification result; the joint angle compensation amount is cached and rendered;

[0010] Step S5: The joint angle compensation amount is subjected to motion posture correction to obtain a corrected posture parameter matrix; the corrected posture parameter matrix is subjected to continuous motion sequence synthesis to obtain digital human motion rendering data frames;

[0011] Step S6: The digital human motion rendering data frames are output to a display terminal to complete the dynamic response and real-time presentation of the digital human limb motion under the user emotion driving.

[0012] The present application combines a diffusion model and a multi-modal emotion recognition technology, and proposes an emotion-driven motion generation method with multi-level dynamic verification and physical constraint capability to solve the problems of lack of naturalness, poor physiological rationality and low real-time performance in the digital human limb motion generation process. In the specific implementation process, the speech mel spectrum graph and the facial key point displacement vector are fused to improve the accurate recognition ability of the current emotional state of the user. After the emotion features are extracted by the bidirectional long short-term memory network, they are mapped to the diffusion model conditional latent variable through the trainable emotion adapter, thereby realizing the deep coupling of the emotion state and the motion generation process. In the denoising stage, the differentiable probability distribution function based on the exponential distribution is introduced as the constraint mechanism, and the human body biomechanics boundary is embedded in each residual block inside the network to realize the dynamic control of the physiological rationality in the early stage of the motion generation process, significantly reducing the occurrence rate of out-of-bound motion and structure insertion. At the same time, the inverse solution agent mechanism constructed by the graph neural network quickly completes the joint angle adjustment reasoning, and the confidence index is constructed by the probability distribution entropy value to judge the credibility of the output. Only when the confidence is insufficient, the traditional FABRIK reverse solving process is triggered, effectively reducing the frequency of traditional inverse solution operation by more than 60%, and significantly improving the real-time response ability of the system as a whole. In addition, in the motion posture correction stage, the pseudo-inverse operation of the Jacobian matrix is combined with the spherical linear interpolation to ensure that the final output motion not only meets the physical constraints but also has the dynamic change characteristics of emotional expression. Further, the CUDA stream parallel processing mechanism and the double buffer structure are used to optimize the frame rendering process, ensuring that the total delay is controlled within 15 milliseconds. The structured data frame is output to the graphics engine for rendering, finally realizing the low-latency, high-accuracy and high-immersion human-computer interaction response, and ensuring that the performance of the digital human in practical application has continuity, stability and emotional consistency.

[0013] Preferably, the present application also provides a digital human action generation system based on diffusion model and emotion recognition, for executing the digital human action generation method based on diffusion model and emotion recognition described above, the digital human action generation system based on diffusion model and emotion recognition comprises:

[0014] A multi-modal emotion feature extraction module is configured to synchronously collect a user voice stream and a facial video stream, respectively perform short-time Fourier transform and facial key point detection, and obtain a speech mel spectrum graph sequence and a key point displacement vector sequence; and perform multi-modal fusion processing on the speech mel spectrum graph sequence and the key point displacement vector sequence to generate an emotion feature vector.

[0015] An emotion-driven action generation module is configured to obtain a digital human skeleton joint rotation state of a current frame, perform emotion adaptation mapping on the emotion feature vector to obtain a conditional latent variable, and perform splicing processing on the conditional latent variable and a standard Gaussian noise vector to obtain a joint rotation angle prediction matrix.

[0016] A kinematics constraint and inverse solution pre-verification module is configured to perform differentiable probability distribution loss constraint processing on the joint rotation angle prediction matrix to obtain a joint rotation angle adjustment matrix, and perform joint inference on the joint rotation angle adjustment matrix and a preset end effector target position to obtain a joint angle compensation amount.

[0017] An inverse solution confidence dynamic verification module is configured to calculate a prediction probability distribution information entropy of the joint angle compensation amount to generate an inverse solution agent confidence value, verify an action posture of the current frame based on the joint angle compensation amount when the confidence value is lower than a preset threshold, and correct the joint angle compensation amount based on a verification result; and perform result caching and rendering on the joint angle compensation amount.

[0018] An action correction and continuous synthesis module is configured to perform action posture correction on the joint angle compensation amount to obtain a corrected posture parameter matrix, and perform continuous action sequence synthesis on the corrected posture parameter matrix to obtain a digital human action rendering data frame.

[0019] A real-time rendering output module is configured to output the digital human action rendering data frame to a display terminal to complete dynamic response and real-time presentation of a digital human limb action under user emotion driving.

[0020] The modules of the application realize high fidelity, high real-time performance and physiological rationality of the emotion-driven digital human action generation process in cooperation. Firstly, the multi-modal emotion feature extraction module combines voice and facial image data, extracts sound spectrum features and facial micro-expression dynamic changes through short-time Fourier transform and key point detection technology, and then constructs a high-dimensional emotion feature vector through a unified time axis alignment and feature normalization fusion strategy, thereby enhancing the sensitivity and accuracy of the system in recognizing emotional intensity and continuity. The emotion-driven action generation module maps the above-mentioned emotion features to the conditional control variables in the action generation process through a trainable emotion adapter, so that the actions generated by the digital human are consistent with the current psychological state of the user, thereby improving the emotional matching degree of action performance. The kinematic constraint and inverse solution pre-verification module introduces a differentiable probability distribution loss function within the diffusion generation framework to limit the rotation angle of each joint within the physiological range allowed by the human body, and embeds a constraint feedback mechanism in the early denoising stage to effectively avoid problems such as joint penetration and boundary crossing in action generation. At the same time, the joint inference strategy greatly reduces the complexity of the traditional inverse solution operation. The inverse solution confidence dynamic verification module constructs a confidence evaluation index based on the entropy information of the prediction distribution, which can evaluate the stability and reliability of the output results in real time. When the confidence is insufficient, the traditional FABRIK structure optimization solving process is automatically called and verified asynchronously, and the result is cached, thereby effectively balancing the real-time response speed and physical accuracy of action generation. The action correction and continuous synthesis module introduces the pseudo-inverse operation of the Jacobian matrix and the spherical linear interpolation processing method in the correction process after physical verification, thereby maintaining the coherence and spatial coordination of the action. Finally, in the continuous action construction process, high-quality action time sequences are output in the form of structured data with consistent frame rate. The real-time rendering output module ensures that the action completes complete rendering output under low delay conditions through the display buffer loading, graphics pipeline parallel scheduling and frame refresh control strategy, realizes the immediate response and natural synchronization between the user's emotional state change and the digital human's limb action, and improves the immersion, stability and credibility of the whole system in the interaction process. BRIEF DESCRIPTION OF DRAWINGS

[0021] Other features, objects and advantages of the application will become more apparent from the following detailed description of non-limiting embodiments made with reference to the accompanying drawings:

[0022] Figure 1 A step flowchart of a digital human action generation method based on a diffusion model and emotion recognition according to the application;

[0023] Figure 2 A detailed step flowchart of step S1 in the above embodiment; Figure 1

[0024] Figure 3 Figure 1 ​​The detailed step flow diagram of step S2 is shown in the following. DETAILED DESCRIPTION

[0025] The technical method of the present application will be described clearly and completely in combination with the drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0026] In addition, the drawings are only schematic illustrations of the present application and are not necessarily drawn to scale. The same reference numerals in the drawings represent the same or similar parts, and thus repeated description thereof will be omitted. Some of the block diagrams shown in the drawings are functional entities, which do not necessarily have to correspond to physically or logically independent entities. The functional entities can be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different network and / or processor methods and / or microcontroller methods.

[0027] It should be understood that although the terms "first", "second" and the like can be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of the exemplary embodiments, a first element can be called a second element, and similarly a second element can be called a first element. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0028] To achieve the above-mentioned purpose, please refer to Figures 1 to 3 The present application provides a digital human action generation method based on diffusion model and emotion recognition, which comprises the following steps:

[0029] Step S1: synchronously collecting user voice stream and facial video stream, respectively performing short-time Fourier transform and facial key point detection to obtain a sequence of speech mel spectrum graph and a sequence of key point displacement vector; performing multi-modal fusion processing on the sequence of speech mel spectrum graph and the sequence of key point displacement vector to generate an emotion feature vector;

[0030] Step S2: obtaining the rotation state of the digital human skeleton joint of the current frame, and performing emotion adaptive mapping on the emotion feature vector to obtain a conditional latent variable; performing splicing processing on the conditional latent variable and a standard Gaussian noise vector to obtain a joint rotation angle prediction matrix;

[0031] Step S3: The joint rotation angle prediction matrix is subjected to a differentiable probability distribution loss constraint processing to obtain a joint rotation angle adjustment matrix; the joint rotation angle adjustment matrix and the preset end effector target position are jointly inferred to obtain a joint angle compensation amount;

[0032] Step S4: The joint angle compensation amount is calculated to obtain prediction probability distribution information entropy, and an inverse solution proxy confidence value is generated; when the confidence value is lower than a preset threshold, the motion posture of the current frame is verified based on the joint angle compensation amount, and the joint angle compensation amount is corrected based on the verification result; the joint angle compensation amount is cached and rendered;

[0033] Step S5: The joint angle compensation amount is subjected to motion posture correction to obtain a corrected posture parameter matrix; the corrected posture parameter matrix is subjected to continuous motion sequence synthesis to obtain digital human motion rendering data frames;

[0034] Step S6: The digital human motion rendering data frames are output to a display terminal to complete the dynamic response and real-time display of the digital human limb motion under the emotion driving of the user.

[0035] In the embodiment of the present application, as shown in the reference Figure 1 The method comprises the following steps:

[0036] Step S1: Synchronously collect a user voice stream and a face video stream, respectively perform short-time Fourier transform and face key point detection to obtain a speech mel spectrum graph sequence and a key point displacement vector sequence; and perform multi-modal fusion processing on the speech mel spectrum graph sequence and the key point displacement vector sequence to generate an emotion feature vector;

[0037] In the embodiment of the present application, first, the synchronous acquisition unit is used to collect the user's speech stream and facial video stream in real time. The sampling rate of the speech stream collection is fixed at 16 kHz, and the sampling bit depth is 16 bits. The frame rate of the facial video stream collection is set to 30 frames per second, and the resolution is fixed at 1280x720 pixels. The speech stream signal collected is processed by using a short-time Fourier transform tool for framing and windowing. The frame length is set to 25 milliseconds, the frame shift is set to 10 milliseconds, and the window function is a Hamming window. Then, the Mel spectrum graph of each frame of spectral data is extracted, the number of Mel filter banks is limited to 64, and a continuous speech Mel spectrum graph sequence is formed. Synchronously, the 68 facial key point detection method based on the Dlib library is applied to the collected facial video stream, and the two-dimensional coordinate information of each key point is extracted frame by frame. The Euclidean distance change between adjacent frames is calculated, and a key point displacement vector sequence is formed. The dimension of the displacement vector is fixed at 68. In order to ensure the time alignment of the speech features and the facial features, the speech Mel spectrum graph sequence is adjusted in time steps by using the linear interpolation method, so as to be synchronized with the facial video frames. The two types of feature sequences after synchronization and alignment are respectively subjected to Z-score standardization processing, the mean and standard deviation of each feature dimension are calculated, and each feature value is processed by using the standardization formula. The standardization formula is: the standardized value is equal to the original value minus the mean divided by the standard deviation. Then, the standardized speech features and facial features are directly spliced in the feature dimension according to the time step, and a joint feature sequence with the same length is formed. The joint feature sequence is input into the pre-constructed bidirectional long short-term memory sequence processor. The processor includes two sequence processing units in opposite directions, each unit includes 128 hidden states, and uses ReLU as the activation function. The output of each time is composed of the forward and reverse hidden states, and then linearly transformed through a fully connected mapping layer to output the emotion intensity value and the emotion persistence index. The emotion intensity value is limited to 0 to 1, and the emotion persistence index is limited to 0 to 5 seconds. The two are finally spliced to form an emotion feature vector, and the length of the emotion feature vector is set to 32.

[0038] Step S2: obtaining the digital human skeleton joint rotation state of the current frame, and performing emotion adaptive mapping on the emotion feature vector to obtain a conditional latent variable; and splicing the conditional latent variable and a standard Gaussian noise vector to obtain a joint rotation angle prediction matrix;

[0039] In the embodiment of the application, first, the real-time rotation state of the skeleton joints of the digital person in the current frame is acquired by the skeleton capture unit, which contains the rotation angle data of 26 joints, and the rotation angle of each joint is represented in the form of a quaternion, and the value range of each component of the quaternion is limited to-1 to 1, and all quaternions are subjected to unitization processing to meet the rotation specification. Subsequently, the emotional feature vector from the emotion perception and multi-modal fusion module is input into the emotion adaptation mapping unit, which is composed of a group of fully connected mapping layers, the input dimension of the first fully connected layer is set to 32, the output dimension is set to 128, the input dimension of the second fully connected layer is 128, the output dimension is 256, the activation function uses the ReLU function, and the weight initialization of each layer uses the Xavier initialization method. The output of the emotion adaptation mapping unit is the conditional latent variable, and the dimension is 256. In parallel, the standard Gaussian noise generation unit generates a noise vector with a length of 256, and each noise component is independently sampled from a standard normal distribution, and the mean of the standard normal distribution is fixed at 0 and the standard deviation is fixed at 1. Subsequently, the conditional latent variable and the standard Gaussian noise vector perform a splicing operation in the feature dimension to form a composite input vector with a length of 512. The composite input vector is input into the hierarchical denoising restoration unit, which is internally processed by 5 denoising restoration levels in turn, each level contains a two-dimensional convolution operation with a convolution kernel size of 3*3, and the convolution channel number is 64, 128, 256, 128 and 64 in turn, and after each layer of convolution, batch normalization and ReLU activation operation are connected, and residual connection is reserved in the forward transmission process to enhance the information transmission efficiency. Finally, the output is mapped to the joint rotation angle prediction matrix through the fully connected mapping layer, and the output dimension is set to 26*3, which represents the Euler angle rotation increment of each joint, the Euler angle sequence is fixed as ZYX sequence, the unit is radian, and the value range of a single angle component is limited to-π to π.

[0040] Step S3: performing differentiable probability distribution loss constraint processing on the joint rotation angle prediction matrix to obtain a joint rotation angle adjustment matrix; and performing joint inference on the joint rotation angle adjustment matrix and a preset end effector target position to obtain a joint angle compensation amount.

[0041] In the embodiment of the application, first, the joint rotation angle prediction matrix generated by the previous module is subjected to biomechanics constraint processing, and each Euler angle component in the joint rotation angle prediction matrix is first subjected to difference calculation with the biomechanics reference mean value of the corresponding joint, and the difference result is substituted into an exponential deviation penalty function, and the expression of the penalty function is L = λexp (α·|θ i - θ i0 |), wherein θ i is the current predicted angle, θ i0To preset physiological mean, lambda is constraint strength, for hip joint, shoulder joint and other large activity degree joints, lambda is limited between 0.1 and 0.5, for knee joint, elbow joint and other single-axis joints, lambda is limited between 1.0 and 1.5, alpha is amplification coefficient, is uniformly set to 3.0, the function output is used as a correction weight coefficient. According to the correction weight coefficient, each joint rotation angle is weighted and adjusted to obtain a joint rotation angle adjustment matrix. Then, the adjusted joint rotation angle adjustment matrix and the preset end effector target position are jointly input into the inverse solution unit. First, a complete human skeleton topology structure is established in the unit, the number of nodes is fixed to 26, the nodes represent joints, and the edges represent bone connections. The node feature vector is composed of the current joint rotation angle three-dimensional value and the end effector three-dimensional space target position information. The overall feature dimension is 6. In the skeleton graph, a fixed number of layers of graph convolution processing sequence is used. The adjacency matrix of each layer of convolution is constructed according to the skeleton topology structure matrix. The number of output channels of each layer of convolution is 64, 128 and 64 in turn. After convolution, batch normalization and ReLU function are used for nonlinear transformation. The convolution output result is mapped to the compensation angle value of 26 joints through full connection transformation, the unit is radian, the value range is fixed to -pi / 6 to pi / 6, and the final joint angle compensation amount is formed.

[0042] Step S4: calculate the prediction probability distribution information entropy of the joint angle compensation amount, generate the inverse solution proxy confidence value; when the confidence value is lower than the preset threshold, verify the action posture of the current frame based on the joint angle compensation amount, and modify the joint angle compensation amount based on the verification result; the joint angle compensation amount is cached and rendered;

[0043] In the embodiment of the application, first, the joint angle compensation amount output by the previous module is evaluated by the probability distribution entropy calculation unit. For each joint angle compensation value, the compensation range is preset to [-pi / 6, pi / 6], the interval is uniformly divided into 30 subintervals, the discrete probability distribution is generated by counting the frequency of the compensation value in each interval, the entropy value of each joint is calculated, and the entropy value calculation formula is H=-∑(pi·log(pi)), wherein pi represents the probability value of the i-th subinterval, to avoid zero value of logarithm, all probability values less than 10 ―6 are uniformly assigned to 10 ―6The sum of the entropy values of all 26 joints is summed to obtain the overall predicted entropy value. Then, the overall predicted entropy value is normalized with the theoretical maximum entropy value Hmax=log(30) to obtain the confidence value C=1-(H / Hmax). The calculated confidence value is compared with the preset threshold value 0.9, and when the confidence value is less than 0.9, the traditional inverse solution verification process is triggered. In the traditional inverse solution verification process, the improved FABRIK solving process is used, first, the joint weight vector is initialized, and the weight value is set as: the weight of the shoulder and the hip is 1.0, the weight of the elbow and the knee is 2.0, and the weight of the remaining joints is 1.5; the initial posture is the posture after the joint angle compensation, and the Euclidean distance between the target position of the end and the predicted end position is calculated as the convergence standard, and the threshold value is set as 0.01 meters; the forward and backward iterations are executed alternately, and the maximum iteration number is set as 5 times, and the joint rotation angle is detected in real time in each iteration whether it exceeds the physiological limit, and the physiological limit range is the preset legal rotation interval of each joint, and the iteration is stopped and the latest valid solution is returned if it exceeds the range. After the verification is completed, the Jacobian matrix pseudo-inverse method is used to difference fusion of the high-precision inverse solution result and the low-precision compensation amount, and the final modified joint angle compensation amount is generated. The final modified joint angle compensation amount is stored in a special cache unit, and the cache capacity is fixed as 5 frames of future data, and is updated in real time.

[0044] Step S5: performing action posture correction on the joint angle compensation amount to obtain a modified posture parameter matrix; and performing continuous action sequence synthesis on the modified posture parameter matrix to obtain digital human action rendering data frames.

[0045] In the embodiment of the application, first, the posture correction unit is used to perform action posture correction processing on the modified joint angle compensation amount output by the previous module. In the correction process, the residual error between the rotation angle of each joint after compensation and the desired position of the end effector is constructed into a Jacobian matrix, and the number of rows of the Jacobian matrix corresponds to the number of three-dimensional space coordinates (3) of the end effector, and the number of columns corresponds to the total number of degrees of freedom (78) of the joint. The pseudo-inverse of the Jacobian matrix is used to solve the optimal joint angle adjustment amount in the least square sense, and the pseudo-inverse calculation adopts the singular value decomposition method, and the singular value is less than 10 ―3The part of the inverse operation is set to zero to suppress ill-posed solution. The calculated joint angle adjustment amount and the original joint rotation angle are fused by spherical linear interpolation method, and the interpolation coefficient is set to 0.5 to obtain the smooth transition of the modified pose parameter matrix. The modified pose parameter matrix contains the three-axis Euler angle rotation data of 26 joints of each frame, and the rotation angle value range is limited to [-π, π] radian interval. Subsequently, the current frame modified pose parameter matrix and the previous frame modified pose parameter matrix are time series spliced by using the continuous action synthesis unit, and the sliding window smoothing method is used to improve the continuity, and the sliding window length is set to 5 frames, and the weight of each frame in the window is distributed according to the Gaussian distribution, and the weight of the center frame is 0.4, and the weight of the edge frame is 0.1. The smoothed sequence forms a complete continuous action sequence. Finally, the continuous action sequence is input into the action rendering preparation unit, and is uniformly converted into a skinning skeleton system compatible format, and the skinning weight is set in advance by using the linear skinning weight generation tool, and the weight sum is kept as 1.

[0046] Step S6: output the digital human action rendering data frame to the display terminal, complete the dynamic response and real-time display of the digital human limb action under the user emotional driving.

[0047] In the embodiment of the present application, first, the digital human action rendering data frame output by the previous module is received, which contains the rotation matrix of 26 joints, the bone position matrix and the skin deformation parameters. The data frame is transmitted to the rendering preparation unit at a rate of 60 frames per second. In the rendering preparation unit, the bone position matrix and the rotation matrix are first input into the skeleton posture reconstruction unit. The forward kinematics transformation method is used to calculate the absolute pose of each joint in three-dimensional space. The forward kinematics is recursively transmitted from the root bone to the child bone, ensuring the consistency of all joint poses. Subsequently, the skin deformation unit combines the joint pose with the skin weight. The linear mixed skin method is used to calculate the new position of each vertex. The skin formula is V' = ∑Wi·(Ri·Vi+Ti), where V' is the new vertex position, Wi is the corresponding skin weight, Ri is the joint rotation matrix, Vi is the original vertex position, and Ti is the joint translation. The number of weights is fixed at a maximum of 4 joints per vertex, and the weight sum is constant at 1. After the skin calculation is completed, the complete human mesh vertex position information is output. Then, the grid data is input into the rasterization rendering engine. The rendering engine is implemented based on the OpenGL 4.5 version standard. The Phong lighting model is used for the lighting model. The number of light sources is fixed at 3, which are the main light source, the environment light source and the backlight source. The main light intensity is 1.0, the environment light intensity is 0.3, and the backlight intensity is 0.5. The specular reflection coefficient in the material parameter is set to 0.8, and the diffuse reflection coefficient is set to 0.7. In the rendering process, the depth buffer is used to eliminate occlusion, and the double buffering mechanism is enabled to ensure smooth frame switching. The frame buffer resolution is fixed at 1920x1080 pixels, the color depth is 24 bits, and the refresh rate is 60Hz. After rendering is completed, the final image is transmitted to the liquid crystal display terminal through the video output control unit. The display terminal size is 27 inches, and the refresh rate is set to 60Hz synchronously, realizing the dynamic response and real-time display of digital human limb action under user emotional driving.

[0048] The application combines a diffusion model and a multi-modal emotion recognition technology, and proposes an emotion-driven action generation method with multi-level dynamic verification and physical constraint capability to solve the problems of lack of naturalness, poor physiological rationality and low real-time performance in the digital human limb action generation process. In the specific implementation process, the speech mel spectrum graph and the facial key point displacement vector are fused to improve the accurate recognition ability of the current emotional state of the user. After the emotion features are extracted by the bidirectional long short-term memory network, they are mapped to the diffusion model conditional latent variable through the trainable emotion adapter, so that the deep coupling of the emotion state and the action generation process is realized. In the denoising stage, the differentiable probability distribution function based on the exponential distribution is introduced as the constraint mechanism, and the human body biomechanics boundary is embedded in each residual block inside the network to realize the dynamic control of the physiological rationality in the early stage of the action generation process, which significantly reduces the occurrence rate of out-of-bound actions and structural interpenetration. At the same time, the inverse solution agent mechanism constructed by the graph neural network is combined to quickly complete the joint angle adjustment reasoning, and the confidence index is constructed by the probability distribution entropy to judge the credibility of the output. Only when the confidence is insufficient, the traditional FABRIK reverse solution process is triggered, which effectively reduces the frequency of traditional inverse solution operation by more than 60%, and significantly improves the real-time response ability of the system as a whole. In addition, in the action posture correction stage, the pseudo-inverse operation of the Jacobian matrix is combined with the spherical linear interpolation to ensure that the final output action not only meets the physical constraints but also has the dynamic change characteristics of emotional expression. Further, the CUDA stream parallel processing mechanism and the double buffer structure are used to optimize the frame rendering process, so that the total delay is controlled within 15 milliseconds, and the structured data frame is output to the graphics engine for rendering. Finally, the low-delay, high-accuracy and high-immersion human-computer interaction response is realized, and the performance of the digital human in practical application has continuity, stability and emotional consistency.

[0049] Preferably, step S1 comprises the following steps:

[0050] Step S11: synchronously collecting a speech stream and a facial video stream from a user terminal;

[0051] Step S12: performing short-time Fourier transform on the speech stream of the user to obtain a speech time-frequency feature map sequence;

[0052] Step S13: performing facial key point detection on the facial video stream of the user to obtain a 68-facial key point two-dimensional coordinate sequence;

[0053] Step S14: calculating the key point positions between continuous frames based on the difference of the 68-facial key point two-dimensional coordinate sequence to obtain a facial key point displacement vector sequence;

[0054] Step S15: performing time alignment and normalization processing on the speech mel spectrum graph sequence and the facial key point displacement vector sequence to obtain a joint feature sequence;

[0055] Step S16: based on the joint feature sequence, forward and reverse inference of the pre-trained bidirectional LSTM network is performed to generate the sentiment intensity level and the sentiment persistence indicator, and the two are spliced into a sentiment feature vector.

[0056] In the embodiment of the present application, first, a synchronous data acquisition unit is used to collect a speech stream and a facial video stream from a user terminal, wherein the sampling rate of the speech stream is set to 16 kHz, the sampling bit depth is 16 bits, the acquisition frame rate of the facial video stream is set to 30 frames per second, and the image resolution is set to 1280x720 pixels. After the collection is completed, the speech stream signal is processed by short-time Fourier transform, the frame segmentation method with a frame length of 25 milliseconds and a frame shift of 10 milliseconds is adopted in the processing process, the Hamming window is used as the window function, the Mel spectrum diagram is extracted after the spectrum decomposition, the number of Mel filter banks is set to 64, and a speech time-frequency feature diagram sequence is formed. Synchronously, the facial key point detection tool in the Dlib library is applied to the facial video stream, and the two-dimensional coordinate information of 68 standard facial key points is extracted frame by frame, and the two-dimensional coordinate range is limited to the integer values [0, 1279] and [0, 719] in the image pixel space. Subsequently, the coordinate change amount of each key point between consecutive frames is calculated based on the inter-frame difference method, and the Euclidean distance difference formula is used to obtain a facial key point displacement vector sequence, and the dimension of the vector sequence is fixed to 68. The speech Mel spectrum diagram sequence and the facial key point displacement vector sequence are linearly interpolated and aligned on the time axis, the interpolation target time step is set to 30 steps per second, and the synchronization consistency with the facial video frame rate is ensured. After alignment, the Z-score standardization processing is performed on the two types of feature sequences respectively, the mean and standard deviation of each dimension of the feature are calculated during standardization, and all feature values are normalized to dimensionless data according to the standardization formula. The speech features and facial features after standardization are spliced in the feature dimension to form a joint feature sequence, and the sequence dimension is (30xN), wherein N is the feature dimension of a single frame after splicing. Subsequently, the joint feature sequence is input into the bidirectional long short-term memory network sequence processing unit which has been trained, the network internally includes two sequence processing channels in opposite directions, each channel includes 128 state units, the state units use the ReLU function for state updating, the hidden states output at both ends of the sequence are spliced according to the time step and then sent to a fully connected transformation layer, and the sentiment intensity level and the sentiment persistence indicator are output, wherein the sentiment intensity level is a real value between 0 and 1, the sentiment persistence indicator is a real value between 0 and 5 seconds, and finally the two are spliced into a sentiment feature vector with a length of 32 dimensions.

[0057] The application constructs a high-dimensional, time-consistent emotion feature vector by synchronously collecting and jointly processing the user voice stream and the facial video stream, effectively improving the real-time perception ability of the digital person to the user's emotional state. First, the time-frequency feature map of the voice is extracted by short-time Fourier transform, which retains the energy distribution and pitch fluctuation information reflecting the emotional changes in the voice. Further combined with the displacement changes of 68 facial key points between consecutive frames, the dynamic features in the facial micro-expression are captured, such as eyebrow lifting, mouth corner shrinking and other expression details. The facial key point displacement vector is constructed by difference calculation, which reflects the spatial trajectory of emotional changes while maintaining the timing accuracy. Then, through time alignment and normalization processing of the voice and facial information, the multi-modal features are unified in the time scale, avoiding recognition bias caused by asynchronous features. After joint feature input into the bidirectional LSTM network, the forward and reverse paths jointly capture the emotional change trend and context association, realize accurate evaluation of emotional intensity and continuity, and finally construct the emotion feature vector, which enhances the recognition ability of the subsequent action generation module to instantaneous emotional fluctuations and stable emotional state, thereby improving the consistency and natural expression level of the digital person's emotional response in the interaction process.

[0058] Preferably, step S2 comprises the following steps:

[0059] Step S21: reading the digital person skeleton state of the current frame to obtain skeleton joint rotation state data;

[0060] Step S22: performing feature mapping of the emotion adapter based on the emotion feature vector and the skeleton joint rotation state data to obtain a conditional latent variable;

[0061] Step S23: performing splicing processing on the conditional latent variable and a standard Gaussian noise vector to obtain a diffusion model input vector;

[0062] Step S24: performing reverse diffusion sampling of the hierarchical denoising UNet network based on the diffusion model input vector to obtain a joint rotation angle prediction matrix.

[0063] In the embodiment of the present application, first, the digital human skeleton state data of the current frame is obtained by using the skeleton state reading unit, and the skeleton state data contains the rotation state information of 26 joints. The rotation state of each joint is represented in the form of a quaternion, and the value range of each component of the quaternion is limited to [-1, 1], and the length of the quaternion is ensured to be equal to 1 through unitization processing. The emotional feature vector output by the emotion perception and multi-modal fusion module and the read skeleton joint rotation state data are input into the feature mapping unit of the emotion adapter, wherein the length of the emotional feature vector is 32 dimensions, the skeleton joint rotation state is flattened to 78 dimensions after the quaternion to Euler angle transformation, and the input feature vector with a length of 110 dimensions is formed after splicing. The input feature vector is sequentially mapped through two fully connected mapping layers, the input dimension of the first layer is 110, the output dimension is 256, the input dimension of the second layer is 256, and the output dimension is 512. ReLU is used as the activation function of each layer, and the weight parameters are initialized using the Xavier method. The 512-dimensional vector output after feature mapping is the conditional latent variable. A 512-dimensional standard Gaussian noise vector is generated by using the Gaussian noise generation unit, the mean of the noise component is set to 0, and the standard deviation is set to 1, so as to ensure that each sampling is independent and unbiased. The conditional latent variable and the standard Gaussian noise vector are spliced in the feature dimension to form a diffusion model input vector with a length of 1024 dimensions. Then, the input vector is input into the hierarchical denoising UNet network to perform reverse diffusion sampling. The UNet network is set to have 5 down-sampling and up-sampling levels. Each down-sampling layer uses a two-dimensional convolution operation with a convolution kernel size of 3x3, and the number of channels is 64, 128, 256, 512, and 512, respectively, with a step size of 2. The activation function uses ReLU, and the convolution is followed by a batch normalization operation. The up-sampling part uses transposed convolution for feature map enlargement, and the convolution kernel size is also 3x3 with a step size of 2. The number of channels is inversely decreased to 512, 256, 128, 64, and 32, respectively. At the same time, residual skip connections are introduced in the symmetric levels to enhance the feature reuse capability. Finally, the network output is converted into a joint rotation angle prediction matrix through a fully connected mapping layer, and the matrix dimension is 26x3, representing the three-axis Euler angle prediction increment of each joint, with a value range limited to [-pi, pi] radian interval.

[0064] The application realizes high fusion of emotion driving and action generation by joint input of emotional features and current skeleton state of digital human, significantly improves the accuracy and continuity of action in expressing emotion, first, the emotion adapter receives the emotion vector and the joint state, completes the targeted feature mapping, generates the control variable with the emotion expression intention and the current posture condition, ensures that the generated action has context consistency and personalized emotional driving features, then the control variable and the standard noise vector are spliced and input into the diffusion model, so that the subsequent action generation is not only controlled by the emotional state, but also has the action diversity brought by random disturbance, improves the fidelity, in the hierarchical denoising UNet structure, the reverse sampling process is taken as a basic unit by residual block, the potential action details are restored by multi-scale feature restoration strategy, the joint rotation angle prediction matrix which meets the emotional semantics and has reasonable spatial structure is formed, the structure can effectively retain the fine-grained posture changes, while avoiding action stiffness or distortion, the whole process improves the naturalness of expression and the motion continuity of digital human in real-time interaction scene.

[0065] Preferably, step S3 comprises the following steps:

[0066] Step S31: performing joint-by-joint value processing on the joint rotation angle prediction matrix to obtain a joint rotation angle prediction value sequence;

[0067] Step S32: calculating a differentiable probability distribution loss function based on the joint rotation angle prediction value sequence and the preset physiological mean value of the joint to obtain a biomechanics deviation loss value;

[0068] Step S33: performing back propagation processing on the biomechanics deviation loss value to obtain a rotation angle gradient correction direction and amplitude;

[0069] Step S34: adjusting the joint rotation angle prediction matrix based on the rotation angle gradient correction direction and amplitude to obtain a joint rotation angle adjustment matrix;

[0070] Step S35: performing feature splicing on the joint rotation angle adjustment matrix and the preset end effector target position to obtain an inverse solution agent input vector;

[0071] Step S36: forward reasoning the graph neural network agent model based on the inverse solution agent input vector to obtain a joint angle compensation amount.

[0072] In the embodiment of the application, first, the joint rotation angle prediction matrix output by the diffusion model is sequentially processed by joint-by-joint value processing according to the joint order, and the rotation angle prediction value sequence of 26 joints is extracted, each joint contains 3 Euler angle components, and the whole constitutes a prediction angle vector with a length of 78. Then, for the rotation angle prediction value of each joint, the corresponding preset physiological mean value θ i0, the physiological mean is set according to the human motion physiological standard, the mean of the shoulder joint and the hip joint is set to 0 radian, and the mean of the knee joint and the elbow joint is set to π / 12 radian. The biomechanical deviation loss value of each joint is calculated using an exponential offset function L = λexp (α·|θ i -θ i0 |). In the offset function, λ is the joint constraint strength, which is set to 0.1 to 0.5 for joints with large single-axis activity, and is set to 1.0 to 1.5 for single-axis joints with limited physiological activity, and α is set to 3.0. The loss values of all joints are added to obtain the total biomechanical deviation loss value. Then, the partial derivative of the loss function with respect to the rotation angle of each joint is calculated using a numerical differentiation method as the rotation angle gradient correction direction and amplitude, and the finite difference formula ΔL / Δθ is used in the partial derivative calculation, where Δθ takes a value of 10 ―4 radians. Based on the obtained correction direction and amplitude, the joint rotation angle prediction matrix is adjusted by a step size η = 0.05, and the adjustment calculation is The updated joint rotation angle adjustment matrix is obtained. The adjusted joint rotation angle adjustment matrix and the preset end effector target position are spliced to form an inverse solution agent input vector with a length of 81, and the end effector target position is a three-dimensional space coordinate. Subsequently, the inverse solution agent input vector is input into the graph neural network agent model solving unit, the graph structure contains 26 nodes, the node features are the spliced joint angles and target position information, and the node edge connection is set according to the digital human skeleton topology relationship. The graph neural network agent model contains 3 layers of graph convolution processing units, the convolution kernel adjacency matrix is fixed, the convolution channel numbers are 64, 128 and 64 respectively, ReLU is used as the activation function for each layer of convolution, and the convolution output is processed by batch normalization to improve numerical stability. The graph convolution output is finally output through a fully connected mapping layer to output the angle compensation of 26 joints, and the compensation angle value range is set to [-π / 6, π / 6] radians. The obtained joint angle compensation is used as the input data of the confidence evaluation and traditional inverse solution coordination module.

[0073] The application realizes physical rationality control of joint rotation angles in the action generation process by introducing a biomechanical constraint mechanism based on a differentiable probability distribution, effectively avoiding joint out-of-bound and action imbalance problems. Firstly, the rotation angle prediction value is extracted joint by joint and the exponential distribution difference degree is calculated with the preset physiological rotation mean value, which can quantify the deviation degree between the action prediction result and the natural activity range of the human body, thereby accurately reflecting the potential physiological violation risk. The deviation loss value is directly fed back to the rotation angle parameter adjustment process through the back propagation mechanism in network training, generating the gradient direction and amplitude for constraint correction, dynamically adjusting the joint prediction angle, making the action generation approach the physiological feasible range in the preliminary stage, and significantly reducing the subsequent posture checking burden. Further, the corrected rotation angle information and the target end effector position information are spliced as the input of the graph neural network agent model, so that the agent model can simultaneously perceive the posture state and target control intention, improving the accuracy and stability of the reasoning output. The agent model uses the joint-skeleton graph structure to retain the human body topology, so that the joint angle compensation output is more consistent with the overall kinematic chain logic, thereby maintaining the generation efficiency while enhancing the consistency and stability of the action physical structure.

[0074] Preferably, the step S4 includes:

[0075] The joint angle compensation vector is modeled with a probability distribution to obtain a prediction probability distribution of each joint angle compensation value.

[0076] Based on the prediction probability distribution, an information entropy of each joint angle compensation value is calculated to obtain an information entropy sequence.

[0077] The information entropy sequence is normalized to obtain a joint angle compensation overall information entropy.

[0078] Based on the joint angle compensation overall information entropy, a confidence function is calculated to obtain an inverse solution agent confidence value.

[0079] In the embodiment of the present application, firstly, the joint angle compensation value vector output by the previous module is subjected to probability distribution modeling processing, the length of the joint angle compensation value vector is 26, the value range of each compensation value is limited in [-pi / 6, pi / 6] radian interval, the interval is equally divided into 60 subintervals, and the width of each subinterval is pi / 180 radian. For each joint angle compensation value, a discrete probability distribution is constructed in the corresponding interval, a sliding window kernel density estimation method is used for smoothing processing, the sliding window bandwidth is set to pi / 360 radian, the weighted frequency of occurrence in each subinterval is counted as the probability value of the interval, and the sum of all interval probability values is strictly normalized to 1. Subsequently, based on the obtained prediction probability distribution, the information entropy of each joint angle compensation value is calculated respectively, the information entropy calculation formula is Hi=-∑j(pij·log(pij)), wherein pij is the probability value of the jth subinterval of the ith joint, and the logarithm zero value is avoided by uniformly setting all probability values less than 10 -6 to 10 ―6 . After the individual information entropy values of all 26 joints are calculated in turn, an information entropy sequence with a length of 26 is formed. Then, the information entropy sequence is subjected to normalization processing, the normalization adopts division by the theoretical maximum entropy Hmax, Hmax=log(60), the normalized entropy value Hi_norm=Hi / Hmax of each joint is obtained after normalization, and the 26 normalized entropy values are averaged to obtain the joint angle compensation overall information entropy H_total. Finally, the inverse solution agent confidence value is calculated by using the confidence function C=1-H_total, and the confidence value is limited in the range of [0, 1], wherein the closer the confidence value is to 1, the lower the prediction result uncertainty is.

[0080] The present application effectively realizes the quantitative evaluation of the confidence of the agent output by modeling the probability distribution of the joint angle compensation value vector and analyzing the information entropy, thereby enhancing the stability and intelligent decision-making ability of the motion generation system under complex input conditions. Firstly, the probability modeling of each joint angle compensation value can reflect the distribution concentration degree and uncertainty difference of each joint adjustment value in the output result, and the information entropy of each joint is calculated to quantify the confusion degree and confidence level of the prediction result, and an information entropy sequence is formed. After normalization processing, the sequence converges into an overall information entropy index, which reflects the confidence degree of the current frame overall posture adjustment scheme, and the lower the value is, the more concentrated and reliable the agent output is. Finally, by setting a confidence function, the information entropy is mapped to a standardized confidence value, which is used as the basis for triggering the traditional high-precision inverse solution process, effectively realizing the dynamic balance control mechanism between precision and efficiency, avoiding motion distortion or physical distortion caused by low-quality agent reasoning results, reducing redundant calculation under high confidence, and improving the overall response speed and resource utilization efficiency of the system.

[0081] Preferably, step S4, when the confidence value is lower than a preset threshold, verifies the action posture of the current frame based on the joint angle compensation amount, and corrects the joint angle compensation amount based on the verification result, including:

[0082] The confidence value of the reverse engineering proxy is compared with the preset confidence threshold;

[0083] When the confidence value is higher than the threshold, the joint angle compensation amount is cached to obtain the proxy result cache item;

[0084] When the confidence value is lower than the threshold, the joint angle compensation amount is input into the traditional inverse kinematics module to obtain the accurate joint angle adjustment value;

[0085] A difference analysis was performed between the precise joint angle adjustment value and the proxy output value to obtain the error vector;

[0086] Spherical linear interpolation fusion based on error vector is used to correct joint angle compensation.

[0087] In this embodiment of the invention, firstly, the confidence value C output by the reverse engineering proxy confidence evaluation unit is received, and this confidence value is compared with a preset confidence threshold, which is set to 0.9. When the confidence value is greater than or equal to 0.9, the joint angle compensation amount output in the current frame is directly stored in the proxy result buffer. The buffer length is fixed at 5 frames. Whenever new data is written, the oldest data is automatically removed to ensure that the content of the buffer is up-to-date. When the confidence level is below 0.9, the joint angle compensation amount of the current frame is input into the traditional inverse kinematics module. The traditional inverse kinematics module executes according to the improved FABRIK method. First, it reads the skeletal structure and end effector target position of the current frame. According to the skeleton hierarchy, it performs position correction sequentially from the root node to the end node. During the correction process, joint length is used to maintain constraints, and Euclidean distance is used for error judgment. The convergence threshold is set to 0.01 meters. In each iteration step, joint weight control is added: the weight of the shoulder and hip joints is 1.0, the weight of the knee and elbow joints is 2.0, and the weight of the remaining joints is 1.5. The maximum number of iterations is limited to 5. If the convergence condition is not met after 5 iterations, the current optimal solution is retained as the accurate joint angle adjustment value. Subsequently, the difference between the accurate joint angle adjustment value and the joint angle compensation amount output by the proxy is calculated in each joint dimension to obtain the error vector. The error value range is limited to [-π / 6, π / 6] radians. After normalization, the error vector is used as the interpolation weight input to the spherical linear interpolation fusion process. The interpolation coefficient is fixed at 0.5. The quaternion spherical linear interpolation method is used to smoothly fuse the proxy compensation amount and the precise compensation amount to generate the corrected joint angle compensation amount. The correction result is then stored in the buffer and simultaneously passed to the motion correction and continuous synthesis module as the output result of the current frame.

[0088] The application realizes dynamic identification and accurate correction of the agent reasoning result by setting a confidence threshold and introducing a double-path verification mechanism, improves the stability and physiological rationality of the action generation system when facing high uncertainty input, firstly, by comparing the current frame inverse solution agent confidence value with the threshold, it can be judged whether the agent output result is reliable in real time, if the confidence is higher than the set value, the system directly caches the current output, avoids repeated calculation, improves the rendering efficiency and response speed, if the confidence is insufficient, the joint angle compensation amount is sent into the traditional inverse solution module to obtain the accurate adjustment value, to ensure that the generated action meets the kinematic chain geometry and human physiological limitations, on this basis, the accurate output and the agent result are compared joint by joint, an error vector is constructed, the error vector reflects the deviation between the prediction and the actual, then a spherical linear interpolation method is used to fuse between the two results, so that the corrected compensation value takes into account the agent efficiency and physical accuracy, effectively avoids action mutation and structure insertion, improves the natural transition and spatial continuity of the output action, and overall enhances the robustness and performance credibility of the system in real-time interaction scene.

[0089] Preferably, the result caching and rendering of the joint angle compensation amount in step S4 comprises:

[0090] The joint angle compensation amount is subjected to physical boundary detection to obtain a joint angle physical legality flag;

[0091] The boundary crossing flags in the joint angle physical legality flag are screened and logically summed to obtain the total number of boundary crossing joints of the current frame;

[0092] When the number of boundary crossings is lower than the threshold, the joint angle compensation amount is subjected to caching processing to obtain a compensation cache result;

[0093] When the number of boundary crossings exceeds the threshold, the current frame is marked as action generation failure to obtain a failure flag item;

[0094] The compensation cache result is written into a thread-shared cache area to obtain a frame buffer record;

[0095] The rendering engine parameters are prepared based on the frame buffer record to obtain a rendering input structure;

[0096] The rendering input structure is subjected to frame identification and timing correction processing to obtain a synchronously executable rendering instruction stream.

[0097] In the embodiment of the present application, first, the modified joint angle compensation amount is subjected to physical boundary detection, and in the detection process, whether the rotation angles of the 26 joints in the three-axis Euler angle dimension exceed the preset physical limit range is checked in turn. The physical limit range is set according to the physiological kinematics standard, and the value range of the shoulder joint and the hip joint is [-π / 2, π / 2] radian, and the value range of the knee joint and the elbow joint is [-π / 4, π / 4] radian. Whenever the rotation angle of any axis of a joint exceeds the corresponding range, the physical legality flag of the joint is generated as 0, and the legality flag is 1. Then, the logical sum of the physical legality flags of all 26 joints is obtained to obtain the total number of boundary joints S in the current frame. S is compared with the preset threshold T, and the threshold T is set to 3. When S≥(26-T), that is, the number of boundary exceeds the threshold, the current frame is marked as motion generation failure, a generation failure flag item is generated, and the flag value is set to -1; when S<(26-T), it is determined that the compensation result is effective, the modified joint angle compensation amount is written into the compensation cache area, and the compensation cache result is formed. The compensation cache area is a double-buffer structure, and the capacity is fixed to 5 future frames of data, which is automatically updated in a ring coverage manner to ensure that the latest data is always maintained in the cache. Then, the compensation cache result is transmitted into the thread shared cache area, the thread shared cache area is implemented in a multi-producer single-consumer architecture, the buffer depth is set to 10 frames, the mutual exclusion lock mechanism is used to avoid multi-thread writing conflict, and the frame buffer record is formed. The latest data in the frame buffer record is read in real time and used to prepare the rendering engine parameters to generate the rendering input structure. The rendering input structure includes the frame number, the 26 joint Euler angle compensation amount, the skeleton index table and the skinning weight table, the data structure adopts a fixed-length array format, and the data alignment is 64-byte memory alignment. Finally, the frame identification and timing correction processing are performed on the rendering input structure, the frame identification adopts a monotonically increasing frame number to identify the current rendering progress of the system, the timing correction uses the system real-time clock to calibrate the inter-frame interval, and the calibration target frame rate is 60 frames per second. After completion, the synchronous executable rendering instruction stream is output, the instruction stream is input to the rendering pipeline module in the form of a streaming memory buffer, and the subsequent digital human motion is driven for real-time rendering.

[0098] The application effectively guarantees the legitimacy of joint angle data through physical boundary crossing detection, avoids the negative influence of abnormal postures on subsequent action generation and rendering quality, filters and counts the number of boundary crossing joints, enables the system to dynamically judge the effectiveness of the current action, realizes the timely identification and shielding of abnormal actions, and improves the robustness and stability of the overall action generation; when the number of boundary crossings is below the threshold, cache processing is performed to ensure the continuity and consistency of the compensation data, effectively reducing the sudden data fluctuations in the rendering process and improving the smoothness of the animation and the visual effect; when the number of boundary crossings exceeds the threshold, the failure state is marked in time to avoid the waste of invalid rendering resources and optimize the system performance and response efficiency; the compensation result is written into the thread shared cache area to realize efficient synchronization and safe sharing of data in a multi-threaded environment and improve the system concurrent processing capability; based on the cache record, the rendering parameters are prepared, and the time sequence consistency and synchronous execution capability of the rendering instruction stream are ensured through frame identification and time sequence correction, thereby ensuring the coherence of the rendering output and the accuracy of the animation performance. Overall, this step effectively integrates data verification, cache management and rendering scheduling, which helps to improve the stability, real-time performance and rendering quality of the action generation system, and enhances the user experience and system reliability.

[0099] Preferably, step S5 comprises the following steps:

[0100] Step S51: inverse kinematics verification is performed on the joint angle compensation amount to obtain an out-of-boundary determination result of the joint rotation angle physiology

[0101] Step S52: based on the out-of-boundary determination result, Jacobian matrix pseudo-inverse calculation is performed on the joint angle compensation amount to obtain a posture correction increment;

[0102] Step S53: spherical linear interpolation processing is performed on the joint angle compensation amount and the posture correction increment to obtain a corrected posture parameter matrix;

[0103] Step S54: the corrected posture parameter matrix is used to update the digital human skinning skeleton posture to obtain an updated skeleton structure frame;

[0104] Step S55: cloth physical simulation processing is performed on the updated skeleton structure frame to obtain a skeleton animation frame with dynamic clothing effects;

[0105] Step S56: based on the skeleton animation frame, a motion synthesis engine time sequence is reconstructed to obtain a continuous action sequence;

[0106] Step S57: frame delay control and timestamp calibration are performed on the continuous action sequence to obtain frame-level rendering control parameters;

[0107] Step S58: the corrected posture parameter matrix and the frame-level rendering control parameters are encapsulated to obtain a digital human action rendering data frame.

[0108] In the embodiment of the present application, by constructing a virtual environment coordinate system, the joint angle compensation quantity to be verified is input into the physical simulation module, the module sets upper limit value and lower limit value according to the physiological activity range of human joints, and each joint rotation angle is compared one by one, when it is detected that a certain joint rotation angle exceeds the preset range, for example, the shoulder joint rotation angle exceeds ±120° in the three-dimensional coordinate axis or the knee joint exceeds ±150°, it is judged that the joint has physiological out-of-range; then, the motion correction method based on pseudo-inverse calculation of Jacobian matrix is called, first, the joint angle compensation quantity is converted into joint position change vector, the expected displacement vector between the actual position and the target position of the current frame end effector in space is constructed, the derivative matrix of the vector in six-dimensional space, that is, the Jacobian matrix, is calculated, and the least square pseudo-inverse operation is performed on the matrix to obtain the joint angle posture correction increment for compensation; the joint angle compensation quantity generated in step S51 and the posture correction increment calculated in step S52 are respectively converted into quaternion representation form, the angle interpolation calculation is performed through the spherical linear interpolation method with a fixed interpolation coefficient α=0.5, and the corrected posture parameter matrix between the two and considering the original motion trend and the physical feasibility is obtained; subsequently, the corrected posture parameter matrix is used as input to drive the skinning system based on bone weight binding, and the rotation matrix of each vertex position is updated in real time according to the bound joint, so that the skeleton structure of the digital human obtains a unified update state in the current frame, forming an updated skeleton structure frame; the updated skeleton structure frame is input into the simulation engine through the cloth physical simulation module, and the mass-spring system is used to process each point of the clothing mesh node, and the displacement and velocity of each cloth node are updated combined with the acceleration under the action of the gravity and wind of the previous frame, to complete the generation of the skeleton animation frame with dynamic effect of the clothing; the skeleton animation frame is arranged in time sequence to construct the time sequence data of the action synthesis engine, and the process adopts a fixed frame rate of 30FPS to ensure the time consistency and spatial continuity of the continuous action sequence; next, the frame control module calculates the delay time Δt of each frame of skeleton animation, sets the upper limit to not more than 16 milliseconds, and labels the time stamp for each frame, the time stamp format is millisecond-level UNIX time, which ensures that the rendering engine can correctly read and schedule the position information and physical state of the current frame in the frame-by-frame rendering process; finally, the corrected posture parameter matrix generated in step S53 and the frame-level rendering control parameter output in step S57 are combined into a unified data structure, and encapsulated into a JSON format digital human action rendering data frame, each data frame contains joint index, rotation quaternion, time stamp, cloth node displacement vector and other fields, and the data structure can be directly input into the rendering engine for generation and output of the final action image.

[0109] The present application verifies and calculates the inverse kinematics and Jacobian matrix pseudo-inverse, effectively ensures the physiological reasonableness of joint angle compensation, avoids unnatural or non-conforming human motion posture, and improves the authenticity and naturalness of digital human motion. Interpolation processing smooths the posture correction process, enhances the coherence and visual fluency of the animation. Based on the corrected posture, the skeleton is updated to realize the precise adjustment of the dynamic skeleton structure of the digital human model, ensuring the consistency of the motion performance and the skeleton state. The introduction of cloth physical simulation enriches the appearance of the character, increases the dynamic effect of the costume, makes the animation more vivid and delicate, and enhances the sense of immersion. Time sequence reconstruction and frame-level delay control ensure the continuity and time synchronization of motion synthesis, effectively avoiding motion breakage or asynchronous phenomenon, and improving the overall animation quality. Finally, by encapsulating the posture parameters and rendering control parameters into a unified data frame, the motion rendering data is standardized and efficiently transmitted, promoting the stability and real-time performance of the rendering process, and comprehensively improving the performance and user experience of the digital human motion rendering system.

[0110] Preferably, step S6 comprises the following steps:

[0111] Step S61: performing frame rate adaptation processing on the digital human motion rendering data frame to obtain a frame control instruction set;

[0112] Step S62: performing video memory buffer loading of the rendering data frame based on the frame control instruction set to obtain a rendering frame buffer block;

[0113] Step S63: performing parallel graphics pipeline scheduling on the rendering frame buffer block to obtain a real-time rendering image stream;

[0114] Step S64: performing frame refresh rendering of the display terminal graphics engine based on the real-time rendering image stream to obtain a digital human visual display image frame;

[0115] Step S65: performing context association recording on the digital human visual display image frame to obtain an emotion-driven response recording log;

[0116] Step S66: calling back the synchronous human-computer interaction component based on the emotion-driven response recording log to obtain a user-perceptible real-time interaction feedback result.

[0117] In the embodiment of the present application, the digital human motion rendering data frame output from step S58 is input into a frame rate adaptation module. The module sets the target output frame rate to 30 frames per second. By calculating the current system rendering delay and the difference between the previous frame timestamp, according to Δt = current timestamp - last frame timestamp, if Δt < 33.33 ms, some non-critical frames are discarded, if Δt > 33.33 ms, intermediate frames are inserted by quaternion interpolation to make up for frame loss, and finally a frame control instruction set conforming to a fixed frame rate is formed, wherein each instruction includes a frame number, a frame timestamp, a rendering priority and a memory location pointer. The memory buffer loading is completed by the GPU resource scheduling module. According to the memory location pointer in the frame control instruction set, the pose parameter matrix and the cloth physical data in the corresponding digital human rendering data frame are directly written into the pre-allocated buffer area in the GPU memory to form a rendering frame buffer block. The buffer block is a two-dimensional structure array, each array unit records all joint quaternions, cloth vertex displacement and texture UV coordinate information in a single frame. The memory writing adopts the CUDA stream asynchronous submission mode to ensure that the CPU thread is not blocked during data transmission. Then, the graphics pipeline scheduling is completed by the graphics engine scheduler. The scheduler calls the vertex shader, geometry processor, rasterizer, fragment processor and blender in the parallel rendering pipeline. In the CUDA parallel thread, vertex transformation, bone weight binding, cloth deformation simulation and lighting texture mapping operations are performed pixel by pixel according to the buffer block data. Each graphics pipeline submodule sets the number of parallel threads ≥ 1024 to ensure that the total processing time of a single frame does not exceed 12 ms, so as to output a frame of image to the frame buffer to build a real-time rendering image stream. Subsequently, the image stream is input into the display terminal graphics engine. The display module refreshes the frame buffer content in FIFO order according to the panel parameters with a refresh rate of 60 Hz, and real-time decodes and draws the skeleton motion and clothing effect to form the visual display image frame of the digital human on the display device. The content of the display image frame is synchronously collected by the context recording module. The recording parameters include display frame timestamp, current rendering emotion label, pose key frame index and cloth disturbance coefficient. All data is written into a local log file or a database in JSON format. The log is named according to the date + timestamp rule, such as "emotion_log_20250611_153000.json", which is used for subsequent calling. Finally, the emotion-driven response recording log is input into the human-computer interaction controller. The controller reads the emotion intensity value and persistence parameter and executes synchronous update commands on the voice playback module, expression rendering unit and haptic output device, etc. For example, when the emotion intensity > 0.8 and the emotion persistence > 2 seconds, the system automatically triggers the digital human voice to increase the tone, the blinking frequency to increase and the haptic motor vibration frequency to increase, so as to realize the real-time interactive feedback result that the user can perceive.

[0118] The application effectively improves the fluency and resource utilization efficiency of digital human motion rendering through frame rate adaptation and video memory buffer optimization, ensuring stable operation and performance balance of the rendering process under different hardware environments. Parallel graphics pipeline scheduling speeds up the rendering speed, realizing efficient real-time image generation and meeting the visual performance requirements in dynamic scenes. The frame refresh mechanism of the display terminal ensures the continuity and picture quality of visual output, enhancing the immersion and viewing experience of users. The introduction of context association records realizes the close combination of rendering content and user interaction state, providing a data foundation for emotion-driven and promoting more natural and intelligent interaction between digital humans and users. Through the callback mechanism of the synchronous human-computer interaction component, the system can respond to user behavior in real time, generating immediate and personalized feedback, greatly improving the immediacy of interaction and user satisfaction. Overall, the process effectively integrates rendering performance optimization and intelligent interaction feedback, significantly enhancing the expressiveness of digital human visual display and the interactivity of interactive experience, providing efficient, stable and expressive technical support for digital human systems in application scenarios.

[0119] Preferably, the application also provides a digital human motion generation system based on a diffusion model and emotion recognition, for executing the above-mentioned digital human motion generation method based on a diffusion model and emotion recognition, the digital human motion generation system based on a diffusion model and emotion recognition comprising:

[0120] A multi-modal emotion feature extraction module is configured to synchronously collect a user voice stream and a facial video stream, respectively perform short-time Fourier transform and facial key point detection, and obtain a voice mel-frequency spectrum graph sequence and a key point displacement vector sequence; the voice mel-frequency spectrum graph sequence and the key point displacement vector sequence are subjected to multi-modal fusion processing to generate an emotion feature vector;

[0121] An emotion-driven motion generation module is configured to obtain a digital human skeleton joint rotation state of a current frame, and perform emotion adaptation mapping on the emotion feature vector to obtain a conditional latent variable; the conditional latent variable and a standard Gaussian noise vector are subjected to splicing processing to obtain a joint rotation angle prediction matrix;

[0122] A kinematics constraint and inverse solution pre-verification module is configured to perform differentiable probability distribution loss constraint processing on the joint rotation angle prediction matrix to obtain a joint rotation angle adjustment matrix; the joint rotation angle adjustment matrix and a preset end effector target position are subjected to joint inference to obtain a joint angle compensation amount;

[0123] An inverse solution confidence dynamic verification module is configured to calculate a prediction probability distribution information entropy of the joint angle compensation amount to generate an inverse solution agent confidence value; when the confidence value is lower than a preset threshold, the motion posture of the current frame is verified based on the joint angle compensation amount, and the joint angle compensation amount is corrected based on the verification result; the joint angle compensation amount is subjected to result caching and rendering;

[0124] The action correction and continuous synthesis module is configured to correct the joint angle compensation amount according to the action posture to obtain a corrected posture parameter matrix, and to synthesize a continuous action sequence according to the corrected posture parameter matrix to obtain a digital human action rendering data frame.

[0125] The real-time rendering output module is configured to output the digital human action rendering data frame to a display terminal to complete dynamic response and real-time display of a digital human limb action under user emotional driving.

[0126] Therefore, from any viewpoint, the embodiments should be considered as exemplary and non-limiting, and the scope of the present application is not limited by the above description, and all changes falling within the meaning and scope of the equivalent elements of the application file are intended to be included in the present application.

[0127] The above description is merely a specific implementation of the present application, enabling those skilled in the art to understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to these embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for generating digital human actions based on diffusion models and emotion recognition, characterized in that, Includes the following steps: Step S1: Simultaneously acquire user voice stream and facial video stream, perform short-time Fourier transform and facial key point detection respectively to obtain voice Mel spectrogram sequence and key point displacement vector sequence; perform multimodal fusion processing on voice Mel spectrogram sequence and key point displacement vector sequence to generate emotion feature vector; Step S2: Obtain the rotation state of the digital human skeleton joints in the current frame, and perform emotion adaptation mapping on the emotion feature vector to obtain conditional latent variables; concatenate the conditional latent variables and the standard Gaussian noise vector to obtain the joint rotation angle prediction matrix. Step S3: Perform differentiable probability distribution loss constraint processing on the joint rotation angle prediction matrix to obtain the joint rotation angle adjustment matrix; perform joint reasoning on the joint rotation angle adjustment matrix and the preset end effector target position to obtain the joint angle compensation amount; Step S4: Calculate the information entropy of the predicted probability distribution of the joint angle compensation amount and generate the inverse solution proxy confidence value; when the confidence value is lower than the preset threshold, verify the action posture of the current frame based on the joint angle compensation amount, and correct the joint angle compensation amount based on the verification result; cache and render the joint angle compensation amount results. Step S5: Correct the joint angle compensation amount for the movement posture to obtain the corrected posture parameter matrix; The corrected posture parameter matrix is ​​used to synthesize a continuous motion sequence to obtain a digital human motion rendering data frame; Step S6: Output the digital human motion rendering data frame to the display terminal to complete the dynamic response and real-time display of the digital human's body movements driven by the user's emotions.

2. The digital human action generation method based on diffusion model and emotion recognition according to claim 1, characterized in that, Step S1 includes the following steps: Step S11: Synchronously acquire voice stream and facial video stream from the user terminal; Step S12: Perform a short-time Fourier transform on the user's speech stream to obtain a speech time-frequency feature map sequence; Step S13: Perform facial key point detection on the user's facial video stream to obtain a two-dimensional coordinate sequence of 68 facial key points; Step S14: Calculate the position of key points between consecutive frames based on the difference of the two-dimensional coordinate sequence of 68 facial key points to obtain the facial key point displacement vector sequence. Step S15: Perform time alignment and normalization on the speech Mel spectrogram sequence and the facial key point displacement vector sequence to obtain the joint feature sequence; Step S16: Perform forward and backward inference of a pre-trained bidirectional LSTM network based on the joint feature sequence to generate the sentiment intensity level and sentiment persistence index, and concatenate them into a sentiment feature vector.

3. The digital human action generation method based on diffusion model and emotion recognition according to claim 1, characterized in that, Step S2 includes the following steps: Step S21: Read the digital human skeleton state of the current frame to obtain the skeleton joint rotation state data; Step S22: Perform feature mapping of the emotion adapter based on the emotion feature vector and skeletal joint rotation state data to obtain conditional latent variables; Step S23: Concatenate the conditional latent variables and the standard Gaussian noise vector to obtain the diffusion model input vector; Step S24: Perform back-diffusion sampling of the hierarchical denoising UNet network based on the diffusion model input vector to obtain the joint rotation angle prediction matrix.

4. The digital human action generation method based on diffusion model and emotion recognition according to claim 1, characterized in that, Step S3 includes the following steps: Step S31: Perform joint-by-joint value taking on the joint rotation angle prediction matrix to obtain the sequence of predicted rotation angle values ​​for each joint; Step S32: Calculate the differentiable probability distribution loss function based on the predicted rotation angle sequence of each joint and the preset physiological mean of the joint to obtain the biomechanical deviation loss value; Step S33: Perform backpropagation processing on the biomechanical deviation loss value to obtain the direction and magnitude of the rotation angle gradient correction; Step S34: Adjust the joint rotation angle prediction matrix based on the direction and magnitude of the rotation angle gradient correction to obtain the joint rotation angle adjustment matrix; Step S35: Perform feature concatenation between the joint rotation angle adjustment matrix and the preset end effector target position to obtain the inverse solution surrogate input vector; Step S36: Based on the forward inference graph neural network surrogate model of the inverse solution surrogate input vector, obtain the joint angle compensation amount.

5. The digital human action generation method based on diffusion model and emotion recognition according to claim 1, characterized in that, Step S4 involves calculating the information entropy of the predicted probability distribution for the joint angle compensation amount and generating the inverse solution surrogate confidence value, including: By modeling the probability distribution of the joint angle compensation vector, the predicted probability distribution of each joint angle compensation value is obtained. The information entropy is calculated for each joint angle compensation value based on the predicted probability distribution, resulting in an information entropy sequence. The information entropy sequence is normalized to obtain the overall information entropy compensated for joint angles; The confidence function is calculated based on the overall information entropy of joint angle compensation to obtain the confidence value of the inverse solution surrogate.

6. The digital human action generation method based on diffusion model and emotion recognition according to claim 1, characterized in that, Step S4: When the confidence value is lower than a preset threshold, verify the action pose of the current frame based on the joint angle compensation amount, and correct the joint angle compensation amount based on the verification result, including: The confidence value of the reverse engineering proxy is compared with the preset confidence threshold; When the confidence value is higher than the threshold, the joint angle compensation amount is cached to obtain the proxy result cache item; When the confidence value is lower than the threshold, the joint angle compensation amount is input into the traditional inverse kinematics module to obtain the accurate joint angle adjustment value; A difference analysis was performed between the precise joint angle adjustment value and the proxy output value to obtain the error vector; Spherical linear interpolation fusion based on error vector is used to correct joint angle compensation.

7. The digital human action generation method based on diffusion model and emotion recognition according to claim 1, characterized in that, Step S4, which involves caching and rendering the joint angle compensation results, includes: Physical over-limit detection is performed on the joint angle compensation amount to obtain the physical legality mark of the joint angle; Filter out the out-of-bounds flags from the joint angle physical validity flags, and perform logical summation to obtain the total number of out-of-bounds joints in the current frame; When the number of out-of-bounds errors is lower than the threshold, the joint angle compensation amount is cached to obtain the compensation cache result; When the number of out-of-bounds actions exceeds the threshold, the current frame is marked as an action generation failure, and a failure flag is obtained. Write the compensation cache result to the thread-shared buffer to obtain the frame buffer record; The rendering engine parameters are prepared based on the frame buffer records, and the rendering input structure is obtained. The rendering input structure is processed with frame identification and timing correction to obtain a rendering instruction stream that can be executed synchronously.

8. The digital human action generation method based on diffusion model and emotion recognition according to claim 1, characterized in that, Step S5 includes the following steps: Step S51: Perform inverse kinematic verification of the joint angle compensation amount to obtain the physiological boundary judgment result of the joint rotation angle. Step S52: Based on the boundary violation determination result, perform Jacobian matrix pseudo-inverse calculation on the joint angle compensation amount to obtain the attitude correction increment; Step S53: Perform spherical linear interpolation on the joint angle compensation amount and attitude correction increment to obtain the corrected attitude parameter matrix; Step S54: Update the pose of the digital human skin skeleton based on the corrected pose parameter matrix to obtain the updated skeleton structure frame; Step S55: Perform cloth physics simulation processing on the updated skeletal structure frame to obtain skeletal animation frames with clothing dynamic effects; Step S56: Reconstruct the motion synthesis engine time sequence based on the skeletal animation frames to obtain a continuous motion sequence; Step S57: Perform frame delay control and timestamp calibration on the continuous action sequence to obtain frame-level rendering control parameters; Step S58: Encapsulate the corrected pose parameter matrix and frame-level rendering control parameters to obtain digital human motion rendering data frames.

9. The digital human action generation method based on diffusion model and emotion recognition according to claim 1, characterized in that, Step S6 includes the following steps: Step S61: Perform frame rate adaptation processing on the digital human motion rendering data frames to obtain a frame control instruction set; Step S62: Load the video memory buffer of the rendering data frame based on the frame control instruction set to obtain the rendering frame buffer block; Step S63: Perform parallel graphics pipeline scheduling on the rendering frame buffer block to obtain a real-time rendered image stream; Step S64: Execute frame refresh rendering of the display terminal graphics engine based on the real-time rendered image stream to obtain digital human visual display image frames; Step S65: Record the context association of the visual display image frames of the digital human to obtain the emotion-driven response recording log; Step S66: Synchronize the human-computer interaction component based on the emotion-driven response recording log callback to obtain real-time interactive feedback results that are perceptible to the user.

10. A digital human motion generation system based on diffusion model and emotion recognition, characterized in that, For executing the digital human motion generation method based on diffusion model and emotion recognition as described in claim 1, the digital human motion generation system based on diffusion model and emotion recognition comprises: The multimodal emotion feature extraction module is used to simultaneously acquire user speech streams and facial video streams, perform short-time Fourier transform and facial key point detection respectively, and obtain speech Mel spectrogram sequence and key point displacement vector sequence; multimodal fusion processing is performed on speech Mel spectrogram sequence and key point displacement vector sequence to generate emotion feature vector; The emotion-driven motion generation module is used to obtain the rotation state of the skeletal joints of the digital human in the current frame, and to perform emotion adaptation mapping on the emotion feature vector to obtain conditional latent variables; the conditional latent variables and the standard Gaussian noise vector are concatenated to obtain the joint rotation angle prediction matrix. The kinematic constraint and inverse kinematics pre-verification module is used to perform differentiable probability distribution loss constraint processing on the joint rotation angle prediction matrix to obtain the joint rotation angle adjustment matrix; and to perform joint reasoning on the joint rotation angle adjustment matrix and the preset end effector target position to obtain the joint angle compensation amount. The inverse kinematics confidence dynamic verification module is used to calculate the information entropy of the predicted probability distribution of the joint angle compensation amount and generate the inverse kinematics proxy confidence value. When the confidence value is lower than the preset threshold, the action posture of the current frame is verified based on the joint angle compensation amount, and the joint angle compensation amount is corrected based on the verification result. The joint angle compensation amount is cached and rendered. The motion correction and continuous synthesis module is used to correct the motion posture by adjusting the joint angle compensation amount to obtain the corrected posture parameter matrix; and to synthesize the corrected posture parameter matrix into a continuous motion sequence to obtain the digital human motion rendering data frame. The real-time rendering output module is used to output the digital human motion rendering data frames to the display terminal, so as to complete the dynamic response and real-time display of the digital human's body movements driven by the user's emotions.

Images