Digital human dialogue and action coordination method and system based on knowledge base enhancement

Abstract

This invention discloses a knowledge-based augmented digital human dialogue and action coordination method and system, relating to the field of computer technology. The method includes receiving natural language input from a user and obtaining a text sequence to be broadcast from a large language model; inputting the text sequence into a causal language model to obtain the probability distribution of lexical units and calculating word-level information entropy; obtaining the difference between the moving average information entropy of the current time window and the historical baseline entropy value and inputting it into a damping mapping model pre-trained based on reinforcement learning to obtain the damping adjustment increment; and accumulating this increment into the current physical mechanical damping coefficient of key joints in the digital human 3D skeletal animation system; finally, inputting the updated damping coefficient and the initial skeletal motion trajectory output by the action generation network into a physics engine to perform dynamic calculations and output the target animation. This invention solves the problem of modal mismatch and visual incongruity caused by mismatched action amplitudes when switching broadcast styles in digital humans.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more specifically, to a knowledge-based enhanced digital human dialogue and action collaboration method and system. Background Technology

[0002] With the deep integration of artificial intelligence and computer graphics, highly realistic virtual digital humans have been widely used in interactive scenarios such as intelligent customer service, education and training, and virtual broadcasting. In particular, the introduction of large language models and knowledge base retrieval technologies has enabled digital humans to break through the limitations of traditional preset scripts and provide highly professional open-domain answers to users' complex questions. This multimodal interactive system based on external knowledge enhancement requires the coordinated processing of multiple stages, including speech recognition, semantic understanding, answer generation, and 3D motion and facial expression rendering.

[0003] In existing technologies, some solutions have been dedicated to addressing the multimodal collaboration and motion generation issues of digital humans. For example, Chinese invention patent application publication number CN120318388A, entitled "An Intelligent Real-Time Interactive Question-Answering System Based on Virtual Digital Humans," discloses receiving user voice or text interaction requests and generating emotionally adaptive response text and voice using a knowledge retrieval module. Subsequently, it combines these features to generate lip-sync animation keyframes, micro-expression parameter sequences, and limb movement trajectories for the virtual digital human. Another example is Chinese invention patent authorization announcement number CN115797606B, entitled "A Method and System for Generating Interactive Motions in 3D Virtual Digital Humans Based on Deep Learning." This patent discloses constructing reference motion sequences for virtual digital humans based on local human body features and generating motion information through an interactive motion generation model combined with motion feature descriptions, thereby allowing the virtual human to provide feedback information during simulation.

[0004] While the aforementioned technologies have achieved, to some extent, the automatic generation of digital human responses, voice broadcasts, and body movements remains a challenge in complex interactive scenarios involving deep integration with external knowledge bases. Existing motion generation solutions typically rely on voice rhythm, emotional tags, or preset motion styles to generate body trajectories, making it difficult to perceive the information density and professional level of the response text itself. Therefore, when the digital human's dialogue switches between casual conversation and highly specialized knowledge such as legal provisions, technical specifications, and financial indicators, the system often still outputs large, lively gestures, resulting in a noticeable incongruity between serious content and physical expression. Directly switching motion templates or motion style parameters can easily cause discontinuities in joint posture, speed, and acceleration, leading to abrupt changes in visuals that contradict physical inertia, such as arm jumps and torso flickers.

[0005] Meanwhile, knowledge-based enhanced digital humans typically need to perform processes such as document retrieval, content rearrangement, and large language model generation before responding, inevitably leading to network and computational latency. Existing digital humans often employ fixed standby animations or static staring states during the waiting period, lacking the eye shifts, gaze avoidance, and subtle eye movements characteristic of human thinking. This makes users perceive the waiting process as system lag or stiff pauses.

[0006] Furthermore, external knowledge bases often contain documents with spatial structures such as comparison tables, hierarchical tree diagrams, and flowcharts. When large language models convert these document contents into plain text for narration, the left-right comparisons, hierarchical relationships, and node subordination in the original documents are easily weakened. Existing digital human motion generation networks typically cannot obtain the document layout or map these spatial relationships into pointing actions in three-dimensional space. This results in digital humans being limited to rhythmic waving without clear objectives when explaining complex knowledge, leading to a disconnect between visual actions and the semantics of the knowledge space. Consequently, this weakens the digital human's sense of professional authority and the user's immersive experience. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to provide a knowledge-based enhanced digital human dialogue and action collaboration method and system to solve the problems mentioned in the background art.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] A knowledge-based enhanced digital human dialogue and action collaboration method includes:

[0010] Receive natural language input from the user, input the natural language input into the large language model, obtain the text sequence to be played output by the large language model, and convert the text sequence to be played into an audio waveform;

[0011] The text sequence to be broadcast is input into a causal language model for word segmentation. The probability distribution of each word after segmentation is obtained and the word-level information entropy is calculated. Based on the word-level information entropy, the moving average information entropy within the current time window is obtained, and the information entropy difference between the moving average information entropy and the preset historical baseline entropy value is calculated.

[0012] The information entropy difference is input into a damping mapping model pre-trained based on reinforcement learning to obtain the damping adjustment increment output by the damping mapping model.

[0013] The damping adjustment increment is accumulated and added to the current physical and mechanical damping coefficient of the key joint preset in the digital human 3D skeletal animation system to obtain the updated physical and mechanical damping coefficient.

[0014] The audio waveform is input into the motion generation network to obtain the initial skeletal motion trajectory output by the motion generation network. The updated physical mechanical damping coefficient and the initial skeletal motion trajectory are then input into the physics engine of the digital human 3D skeletal animation system to perform dynamic calculations and output the target motion animation.

[0015] Preferably, the damping mapping model is a machine learning model trained using a reinforcement learning algorithm, and the training process of the damping mapping model includes:

[0016] Obtain the set of environmental states containing historical entropy differences;

[0017] The historical information entropy difference in the environmental state set is input into the initial machine learning model, and the predicted damping adjustment increment output by the initial machine learning model is obtained as the action output.

[0018] Obtain a preset smoothness target, and calculate the reward function value between the predicted damping adjustment increment and the smoothness target;

[0019] The gradient ascent algorithm is used to maximize the reward function value to update the model parameters of the initial machine learning model, and the damping mapping model is obtained when the iteration termination condition is met.

[0020] Preferably, the key joints include the spinal joint, shoulder joint, and elbow joint in the digital human 3D skeletal animation system;

[0021] The steps to obtain the moving average information entropy within the current time window include:

[0022] The text sequence to be broadcast is divided into a time window by a preset number of word units after word segmentation according to the chronological order.

[0023] Extract the word-level information entropy of all words within the same time window and calculate the arithmetic mean. Use the arithmetic mean as the moving average information entropy within the current time window.

[0024] Preferably, the step of inputting the updated physical-mechanical damping coefficient and the initial skeletal motion trajectory into the physics engine of the digital human 3D skeletal animation system to perform dynamic calculations and output the target motion animation includes:

[0025] The initial skeletal motion trajectory is converted into the expected rotational angular velocity of the key joint;

[0026] The updated physical-mechanical damping coefficient is introduced into the physics engine to generate a physical damping torque in the opposite direction to the expected rotational angular velocity;

[0027] The physical damping torque is superimposed on the kinematic equation of the key joint to obtain the actual rotational angular velocity of the key joint, and the key joint is driven by the actual rotational angular velocity to generate the target motion animation.

[0028] Preferably, before inputting the natural language input into the large language model, the method further includes:

[0029] The natural language input is fed into the predictive language model, and the predicted sentence information entropy of the natural language input is obtained from the output of the predictive language model.

[0030] The predicted statement information entropy is input into a preset judgment model to obtain the Boolean classification result output by the judgment model;

[0031] Under the condition that the Boolean classification result is true, extract the current gaze point coordinates and the current rotation matrix of the eyeball skeleton in the digital human 3D skeletal animation system;

[0032] The current gaze point coordinates are modified to a pre-defined lateral coordinate using a preset nonlinear interpolation function to generate an eye deflection command;

[0033] A white noise signal of a preset frequency is superimposed onto the current rotation matrix to generate a micro-scanning command;

[0034] The eye-turning command and the micro-sagittation command are input together into the digital human 3D skeletal animation system to control the movement of the eye skeleton, and when the text sequence to be broadcast is obtained from the output of the large language model, the eye-turning command and the micro-sagittation command are cancelled.

[0035] Preferably, the determination model is a binary classification model, and the training process of the determination model includes:

[0036] Obtain a judgment sample set containing the information entropy of historical prediction statements and manually labeled true and false classification tags;

[0037] Input the historical prediction statement information entropy from the judgment sample set into the initial classification model to obtain the predicted true / false labels output by the initial classification model;

[0038] Calculate the cross-entropy loss value between the predicted true / false labels and the manually labeled true / false classification labels;

[0039] The network weights of the initial classification model are updated using the cross-entropy loss value, and the decision model is obtained when the cross-entropy loss value converges.

[0040] Preferably, before inputting the updated physical-mechanical damping coefficient and the initial skeletal motion trajectory into the physics engine of the digital human 3D skeletal animation system to perform dynamic calculations, the method further includes:

[0041] Extract the updated physical-mechanical damping coefficient and compare it with a preset damping opening threshold.

[0042] Under the condition that the updated physical and mechanical damping coefficient is greater than the damping opening threshold, the external knowledge base documents referenced by the large language model when generating the text sequence to be broadcast are parsed.

[0043] The external knowledge base document is input into the layout analysis model to obtain the two-dimensional spatial layout metadata output by the layout analysis model;

[0044] The two-dimensional spatial layout metadata is converted into three-dimensional spatial anchor point coordinates in a preset virtual world coordinate system;

[0045] The coordinates of the three-dimensional spatial anchor point are input into the inverse kinematics solver as the target attraction point. The inverse kinematics solver is used to obtain the corrected joint rotation amount that guides the end of the digital human's hand to the coordinates of the three-dimensional spatial anchor point.

[0046] The corrected joint rotation is incorporated into the initial bone motion trajectory to update the initial bone motion trajectory.

[0047] Preferably, the step of obtaining the two-dimensional spatial layout metadata output by the layout analysis model includes:

[0048] Extract the chart row and column boundary coordinates and node hierarchy relationships output by the layout analysis model, and use the chart row and column boundary coordinates and node hierarchy relationships as the two-dimensional spatial layout metadata;

[0049] The steps for converting the two-dimensional spatial layout metadata into three-dimensional spatial anchor point coordinates in a preset virtual world coordinate system include:

[0050] The pelvic bone points in the digital human 3D skeletal animation system are set as the origin of the coordinate system to construct a local 3D coordinate system.

[0051] Replace the row and column boundary coordinates of the chart with the horizontal axis coordinates of the local three-dimensional coordinate system;

[0052] Replace the node hierarchy relationship with the vertical axis coordinate in the local three-dimensional coordinate system;

[0053] Obtain the preset depth axis coordinates, and concatenate the depth axis coordinates with the horizontal axis coordinates and the vertical axis coordinates to generate the three-dimensional space anchor point coordinates.

[0054] Preferably, the layout analysis model is trained using the following steps:

[0055] Obtain a document dataset containing document images and manually annotated boundary coordinates and hierarchical labels;

[0056] The document images in the document dataset are input into the initial graph neural network to obtain the predicted boundary coordinates and predicted level labels output by the initial graph neural network;

[0057] Calculate the coordinate error between the predicted boundary coordinates and the manually labeled boundary coordinates, and calculate the classification error between the predicted hierarchical labels and the manually labeled hierarchical labels;

[0058] The parameters of the initial graph neural network are updated by combining the coordinate error value and the classification error value, and the layout analysis model is obtained after completing a preset number of training cycles.

[0059] This invention also discloses a knowledge-based enhanced digital human dialogue and action collaboration system for implementing the above method, comprising:

[0060] The text sequence acquisition module is configured to receive natural language input from the user, input the natural language input into a large language model, obtain the text sequence to be played output by the large language model, and convert the text sequence to be played into an audio waveform.

[0061] The information entropy processing module is configured to input the text sequence to be broadcast into a causal language model for word segmentation, obtain the probability distribution of each word after segmentation and calculate the word-level information entropy, obtain the moving average information entropy within the current time window based on the word-level information entropy, and calculate the information entropy difference between the moving average information entropy and the preset historical benchmark entropy value.

[0062] The damping increment acquisition module is configured to input the information entropy difference into a damping mapping model pre-trained based on reinforcement learning, and acquire the damping adjustment increment output by the damping mapping model;

[0063] The damping update module is configured to add the damping adjustment increment to the current physical-mechanical damping coefficient of the preset key joint in the digital human 3D skeletal animation system to obtain the updated physical-mechanical damping coefficient.

[0064] The motion calculation output module is configured to input the audio waveform into the motion generation network, obtain the initial skeletal motion trajectory output by the motion generation network, input the updated physical mechanical damping coefficient and the initial skeletal motion trajectory into the physics engine of the digital human 3D skeletal animation system to perform dynamic calculation, and output the target motion animation.

[0065] The advantages of this invention compared to existing technologies lie in its provision of a knowledge-based enhanced digital human dialogue and action collaboration method and system, which achieves the following significant beneficial effects. First, this invention cleverly utilizes word-level information entropy from natural language processing to characterize the obscurity and information density of text, and employs a damping mapping model trained by reinforcement learning to transform the abstract information entropy difference into the physical and mechanical damping coefficients of key joints in a 3D skeletal animation system. This innovative design based on interdisciplinary principles completely solves the modal dissonance problem caused by sudden changes in context in digital humans. When the digital human delivers obscure professional knowledge with high information entropy, the increased physical damping acts like a shock absorber, forcibly absorbing and suppressing the overly lively motion trajectory output by the action generation network in the underlying physics engine, causing the body expression to automatically conform to a restrained and stable expert posture; conversely, when the context returns to low-entropy everyday conversation, the damping adjustment increment automatically decreases, and the movements immediately regain their lightness and agility. This method achieves an extremely smooth and physically consistent style transition without reloading the animation network.

[0066] This invention introduces a pre-judgment mechanism based on the information entropy of predicted statements, extracts the current gaze coordinates and the current rotation matrix of the eyeball skeleton, modifies the current gaze coordinates to pre-set lateral coordinates using a nonlinear interpolation function, and superimposes a high-frequency white noise signal onto the current rotation matrix. This technique applies the biological laws of gaze avoidance and microsalivation that humans instinctively exhibit when facing high cognitive loads and memory retrieval to digital humans, transforming the originally tedious knowledge base retrieval delays into a realistic cognitive preparation process of deep thought in the digital human, effectively eliminating the artificiality caused by delays.

[0067] This invention further extracts two-dimensional spatial layout metadata of external knowledge base documents by combining a layout analysis model, under the condition that the updated physical and mechanical damping coefficient is greater than a pre-set damping activation threshold, and converts it into three-dimensional spatial anchor point coordinates in a preset virtual world coordinate system. Then, using an inverse kinematics solver, the digital human's damped hand end-effectors are precisely guided to these invisible three-dimensional spatial anchor point coordinates. This technique allows the digital human to maintain an expert-like, restrained posture while accurately indicating the left-right contrast or hierarchical relationship of a document in three-dimensional space, just like a real presenter. This completely solves the problem of semantic disconnect between action space and textual content caused by traditional one-dimensional text-driven methods, significantly enhancing the authority of knowledge delivery. Attached Figure Description

[0068] Figure 1 This is an overall schematic diagram of the invention;

[0069] Figure 2 This is a schematic diagram of the reinforcement learning strategy network of the present invention;

[0070] Figure 3 This is a schematic diagram of the damping adjustment of the present invention;

[0071] Figure 4 This is a schematic diagram of the initial classification model of the present invention;

[0072] Figure 5 This is a schematic diagram of the system of the present invention. Detailed Implementation

[0073] The present invention will be described below with reference to embodiments and accompanying drawings. The technical features in each embodiment can be combined with each other without conflict, and the parameter ranges can be calibrated according to the size of the digital human model, animation frame rate, physics engine unit system, and application scenario.

[0074] In one embodiment, such as Figure 1 As shown, the digital human dialogue and action collaboration method is applied to an interactive platform that includes a large language model, a speech synthesis engine, a causal language model, an action generation network, a digital human 3D skeletal animation system, and a physics engine. Users can send natural language input through text input boxes, speech recognition interfaces, or smart terminals. If the input is speech, the system first converts it into text using a speech recognition model; if the input itself is text, it directly proceeds to the subsequent processing flow.

[0075] The fundamental components of this invention, such as the large language model, speech synthesis engine, motion generation network, digital human 3D skeletal animation system, physics engine, and inverse kinematics solver, can all be implemented using existing mature software frameworks or neural network models. The improvement of this invention does not lie in reintroducing the underlying algorithms of these fundamental components, but rather in using the information entropy of the text to be broadcast to adjust the physical and mechanical damping coefficients of the digital human's key joints, and further combining the information entropy of the predicted statement, the document space layout of the knowledge base, and the inverse kinematics results to ensure that the digital human's semantic expression, eye behavior, limb movements, and physical motion state remain coordinated.

[0076] After receiving the user's natural language input, the system provides this input to the large language model. To enhance the knowledge base, the large language model can call an external knowledge base retrieval module before generating an answer. This module extracts keywords, semantic vectors, or entity tags based on the user's question, and retrieves content such as legal provisions, product manuals, teaching materials, technical documents, tables, or charts from the knowledge base. The retrieved content is then concatenated into prompts for the large language model to reference. The large language model outputs a text sequence for the digital human to read aloud, while retaining the document identifiers, page numbers, paragraph numbers, or chart numbers of the external knowledge base referenced during the answer process. This facilitates the subsequent conversion of the spatial structure in the knowledge base into the digital human's directional actions.

[0077] The large language model can employ existing encoder-decoder structures, decoder-only structures, or other models capable of generating natural language text based on context. The knowledge base retrieval module can also employ existing keyword retrieval, vector retrieval, hybrid retrieval, or reordering retrieval methods. For this invention, as long as this part can output the sequence of text to be played and record the external knowledge base documents referenced when generating the sequence, the input requirements for subsequent collaborative action control can be met.

[0078] After the large language model outputs the text sequence to be played, the speech synthesis engine converts the text sequence into an audio waveform. The speech synthesis engine can use an end-to-end neural speech synthesis model or a combination of an acoustic model and a vocoder. To ensure that the subsequent action generation network can obtain stable audio features, the sampling rate of the audio waveform can be set to 16000Hz to 48000Hz, and the audio frame length can be set to 10ms to 40ms. During speech synthesis, phoneme boundaries, stress positions, and pause positions can also be output simultaneously. This auxiliary information can be input into the action generation network, but the subsequent action rhythm control is still based on the audio waveform.

[0079] After obtaining the text sequence to be broadcast, the system inputs it into a causal language model and performs lexicalization using the accompanying word segmenter. Here, the causal language model is not used to regenerate the response, but rather to measure the lexical uncertainty and information density of the generated text. The causal language model processes the text sequence in an autoregressive manner, calculating the probability distribution of the entire vocabulary for each lexical element given the context prior to the current position. For Chinese text, lexical elements can be characters, subwords, or phrases segmented by the word segmenter; for mixed Chinese and English text, English words, numbers, technical abbreviations, and symbols can be processed according to the same lexicalization rules. The system calculates word-level information entropy based on the probability distribution corresponding to each lexical element. The more dispersed the probability distribution, the less predictable the lexical element is in the context, typically corresponding to more obscure, specialized, or information-dense content.

[0080] Causal language models can employ a decoder-only Transformer architecture. This architecture typically includes a lexical embedding layer, a positional encoding layer, multiple self-attention layers with causal masks, a feedforward neural network layer, a normalization layer, and an output mapping layer. The causal mask restricts the model to utilize only contextual information preceding the current position when processing a lexical at that position, thus obtaining the probability distribution corresponding to that position. The output mapping layer maps the hidden states to the vocabulary space and then normalizes the probability of each candidate lexical. In implementation, a pre-trained general causal language model can be directly used, with its parameters frozen, and used only as an information entropy estimator; alternatively, it can be fine-tuned using business domain texts to better suit vertical domain texts such as medical, financial, government, and educational texts. Using a pre-trained model without fine-tuning does not affect the execution of this invention, as this invention only requires the model to output a stable lexical probability distribution.

[0081] In a further embodiment, the causal language model and the large language model can be the same model or two independent models. If the same model is used, the lexical probability distribution generated by the large language model when generating the text sequence to be broadcast can be reused, thereby reducing additional inference time. If an independent model is used, a causal language model with fewer parameters and faster inference speed can be selected specifically to calculate information entropy. To ensure the stability of the information entropy calculation results, the system can fix the causal language model version, word segmenter version, and vocabulary version to avoid the historical baseline entropy value becoming invalid after the model is updated.

[0082] Word-level information entropy can be calculated using entropy operations with a base of 2 or a base of the natural constant. To ensure comparability of outputs across different models, the system can be normalized using a fixed set of validation texts before deployment, ensuring that the word-level information entropy falls within the range of 0 to 12. For common greetings and simple confirmation phrases, word-level information entropy is typically low; for professional regulations, medical terms, financial indicators, and complex technical parameters, word-level information entropy is typically high. This design aims to avoid directly relying on manually set emotion or scene labels, but rather to utilize the language model's own judgment of text predictability to automatically perceive the cognitive difficulty and seriousness of the currently broadcast content.

[0083] In a further embodiment, the system divides a preset number of word units after segmentation of the text sequence to be broadcast into a time window according to the chronological order. The preset number can be 3 to 50 word units, preferably 8 to 20 word units. The time window can move by one word unit at a time, or it can move according to the broadcast progress corresponding to the speech frame. The system extracts the word-level information entropy of each word unit within the same time window and calculates the arithmetic mean, using this arithmetic mean as the moving average information entropy within the current time window. The moving average is used instead of directly using the entropy value of a single word unit because a single uncommon word may only have local fluctuations, and directly driving the skeleton damping can easily cause motion jitter; the averaging of the time window can reflect the overall information density of a small segment of broadcast content, making the change in motion style smoother.

[0084] The system also pre-sets historical baseline entropy values. These values ​​can be derived from historical dialogue data of the digital human in the same business scenario, or from general casual conversation corpora, business Q&A corpora, or manually calibrated corpora. For example, in a customer service scenario, the historical baseline entropy value can be obtained by the average moving average information entropy of the most recent 1,000 to 100,000 rounds of normal dialogue; in an education and training scenario, different baselines can be set for beginner, intermediate, and advanced courses. Historical baseline entropy values ​​are generally set between 1.0 and 8.0. The system calculates the information entropy difference between the moving average information entropy of the current time window and the historical baseline entropy value. A positive information entropy difference indicates that the current broadcast content is more complex or professional than regular content; a negative information entropy difference indicates that the current broadcast content is closer to everyday communication or low-load expression. To avoid extreme text causing abnormal physical parameters, the information entropy difference can be limited to a range of -5 to +5.

[0085] After obtaining the information entropy difference, the system inputs it into a damping mapping model pre-trained based on reinforcement learning. The damping mapping model transforms the complexity changes on the language side into damping adjustment increments on the skeletal physics side. These damping adjustment increments can be a scalar or vectors corresponding to the spinal, shoulder, and elbow joints, respectively. When using a scalar, the system can assign preset weights to different key joints; for example, the weights for the spinal joints are 0.4 to 0.8, the shoulder joints 0.2 to 0.6, and the elbow joints 0.1 to 0.4. When using vectors, the damping mapping model directly outputs the damping adjustment increments corresponding to each key joint. The damping adjustment increments are generally limited to between -2.0 and +5.0, with the actual range determined by the physics engine units.

[0086] This invention maps information entropy difference to a damping coefficient because digital humans should not continue to use large, rapid, and frequent gestures when delivering high-information-density content. Instead of directly switching to a completely new movement style, it adds damping to key joints in the underlying physics engine, naturally suppressing and converging the movement trajectories originally generated by the motion generation network. This preserves the synchronization between the motion generation network and the speech rhythm while using physical constraints to make the body movements more stable, avoiding sudden animation transitions that cause screen flickering.

[0087] In one embodiment, such as Figure 2 As shown, the damping mapping model can employ a multilayer perceptron containing an input layer, hidden layers, and an output layer, or a recurrent neural network or a lightweight Transformer structure. The input layer receives the current entropy difference, or it can receive a sequence of entropy differences from several recent time windows. The number of recent time windows can be 2 to 20, used to help the model determine whether the text complexity fluctuates briefly or increases continuously. The hidden layers can be 2 to 6 layers, with 16 to 256 neurons per layer. The output layer outputs the damping adjustment increment. To prevent sudden jumps in the damping increment, a smoothing constraint unit can be connected after the output layer to limit the change in damping increment between adjacent animation frames to between 0.01 and 0.5.

[0088] The training process of the damping mapping model is as follows. The system first acquires an environmental state set containing historical information entropy differences. This environmental state set can come from real digital human broadcast logs or from simulation-generated text complexity sequences. Each environmental state includes the current information entropy difference, and preferably also includes the information entropy differences from the previous few time windows, the current joint damping coefficient, the current joint angular velocity, and the motion trajectory smoothness index. The historical information entropy differences in the environmental state set are input into the initial machine learning model. The initial machine learning model outputs a predicted damping adjustment increment, which is then used as the action output.

[0089] During training, the system pre-sets a smoothness target. This target can be obtained from statistical analysis of real-person demonstration motion samples; for example, in professional demonstration scenarios, the peak angular velocity of the shoulder joint, the peak angular velocity of the elbow joint, and the amplitude of spinal sway should all be lower than in casual conversation scenarios. Alternatively, it can be specified by animators, such as requiring the hand-eye tremor amplitude in high-information-entropy segments to be less than 2cm to 8cm, and the angular acceleration of key joints to be below a preset upper limit. The system calculates a reward function value based on the degree of closeness between the simulated motion smoothness resulting from the incremental adjustment of predictive damping and the smoothness target. A higher reward function value is achieved when predictive damping reduces unnecessary rapid swaying, maintains motion continuity, and prevents the digitizer from becoming completely stiff; a lower reward function value is achieved when predictive damping is too small, resulting in overly lively motion, or too large, causing limb freezing.

[0090] The system employs a gradient ascent algorithm to maximize the reward function value, thereby updating the model parameters of the initial machine learning model. The gradient ascent algorithm can utilize policy gradient, proximal policy optimization, or other reinforcement learning optimization methods suitable for continuous action output. The learning rate can be set from 0.00001 to 0.01, the batch size from 16 to 1024, and the training epochs from 1000 to 200000. The iteration termination condition can be that the reward function value changes less than a preset range over several consecutive training epochs, or that the smoothness of the actions on the validation set reaches a target range. After training, a damped mapping model is obtained. Through this training method, the model does not simply linearly amplify the information entropy difference into damping, but rather learns to provide more natural damping increments under different action states and contextual changes.

[0091] The system adds the damping adjustment increment to the current physical-mechanical damping coefficient of the key joints in the digital human 3D skeletal animation system to obtain the updated physical-mechanical damping coefficient, such as... Figure 3 As shown. Key joints include the spinal joint, shoulder joint, and elbow joint. The spinal joint primarily affects the forward and backward swaying of the torso and overall postural stability; the shoulder joint affects the range of arm raising and pointing direction; and the elbow joint affects the flexibility of hand gestures. These joints significantly impact the professionalism and naturalness of the digital human, and are therefore prioritized for damping control. The physical mechanical damping coefficient can be set within the range of 0.01 to 30.0, preferably between 0.05 and 15.0. After updates, upper and lower limits can be trimmed to prevent the damping coefficient from being less than 0 or exceeding the stable solution range of the physics engine.

[0092] In the audio-driven motion generation stage, the system inputs the audio waveform into the motion generation network. The motion generation network can include an audio feature encoder, a temporal modeling network, and a skeletal trajectory decoder. The audio feature encoder can extract features such as Mel spectrum, energy, pitch, speech rate, pauses, and phoneme boundaries; the temporal modeling network can employ a Long Short-Term Memory network, a temporal convolutional network, or a Transformer; the skeletal trajectory decoder outputs the initial rotation, displacement, or joint angular velocity of the digital human skeleton in each animation frame. The initial skeletal motion trajectory is synchronized with the speech rhythm, but this trajectory itself does not understand the level of detail in the spoken text, therefore it needs to be fed into the physics engine along with updated physical damping coefficients.

[0093] The action generation network can employ existing speech-driven pose generation models or pre-trained, deployed end-to-end action generation models. Training of this network can be supervised using real-person speech and motion capture data to align the input audio waveform with the output skeletal motion trajectory in time. The action generation network itself is not required to be retrained in this invention; as long as it can output an initial skeletal motion trajectory based on the audio waveform, it can serve as the input module. This invention further utilizes information entropy-driven damping control to correct the physical properties of this initial skeletal motion trajectory, ensuring a match between audio rhythm, semantic complexity, and skeletal motion state.

[0094] In a further embodiment, the system first converts the initial skeletal motion trajectory into the expected rotational angular velocity of the key joints. The expected rotational angular velocity can be calculated from the joint rotation difference between adjacent animation frames, or it can be directly output by the motion generation network. Then, an updated physical damping coefficient is introduced into the physics engine to generate a physical damping torque in the opposite direction to the expected rotational angular velocity. This physical damping torque can be understood as the deceleration resistance encountered when the joint rotates; the faster the expected rotation, the more obvious the damping suppression; the larger the damping coefficient, the stronger the suppression effect.

[0095] Subsequently, the physics engine superimposes the physical damping torque into the dynamics equations of the key joints to solve for the actual rotational angular velocity of the key joints. The actual rotational angular velocity then drives the movement of the spine, shoulder, and elbow, generating the target motion animation. Because the damping torque participates in the dynamics calculation within the physics engine, rather than simply reducing the amplitude in the animation result, the motion transition has inertial continuity. For example, if the digital human is about to make a large-amplitude wave, when the broadcast content enters the high-information-entropy professional interpretation, the increased damping will cause the arm to naturally decelerate and retract, rather than suddenly stopping or instantly switching to another animation.

[0096] The dynamics solution of the physics engine can be achieved using rigid body dynamics, articulated body dynamics, joint constraint solving, constraint dynamics solving, semi-implicit integration, position constraint solving, or other physical simulation methods found in existing 3D animation or game engines. Dynamics solution itself is a problem that can be solved by existing technologies; this invention does not limit the specific type of physics engine, nor does it require redesigning the underlying integrator or collision solver. The key to this invention lies in introducing the updated physical-mechanical damping coefficients obtained from the information entropy difference into the dynamics solution process of key joints, enabling the physics engine to produce motion convergence effects corresponding to semantic complexity based on its existing solution capabilities.

[0097] In another embodiment, the system can perform a pre-cognitive load assessment before the natural language input enters the large language model. The system inputs the user's natural language input into the predictive language model, and the predictive language model outputs the information entropy of the predicted statement based on the natural language input. The predictive language model can be a causal language model, a masked language model, or a lightweight text encoding model. Here, the complexity of the user's question itself is calculated, not the complexity of the digital human's response text. If the user's question contains more technical terms, long-distance dependencies, complex conditions, or rare entities, the information entropy of the predicted statement will increase, which usually means that the large language model or knowledge base retrieval will require a longer processing time.

[0098] The predictive language model can share the same model as the aforementioned causal language model, or it can be set up as a separate text complexity evaluation model with fewer parameters. If a causal language model is used, the system can obtain the probability distribution of each word in the user input in word order and calculate the average information entropy of the entire sentence. If a masking language model is used, the system can estimate the predictability of each word in the user input through word-by-word masking or batch masking, and then obtain the information entropy of the predicted sentence. The predictive language model can directly use a pre-trained model without retraining, or it can be fine-tuned using historical user questions. Fine-tuning is not a necessary condition for the implementation of this invention; it is only used to improve the recognition accuracy of high cognitive load problems in specific business scenarios.

[0099] The system inputs the predicted statement information entropy into a preset judgment model, which outputs a Boolean classification result. The judgment model is a binary classification model; a true value indicates the digital human is about to enter a state of high cognitive load or knowledge retrieval waiting, while a false value indicates the user's input is relatively simple and can maintain normal gaze and standby behavior. The judgment model can be a logistic regression model, a shallow multilayer perceptron, or a small binary classification network. The input can contain only the predicted statement information entropy, or auxiliary quantities such as text length, whether it contains professional entities, and knowledge base recall time estimation can be added without changing the main judgment criteria. The predicted statement information entropy can be normalized to 0 to 1, or retained in the entropy range of 0 to 12. The classification threshold of the judgment model can be set between 0.4 and 0.8.

[0100] The training process of the judgment model includes obtaining a judgment sample set containing the information entropy of historical predicted statements and manually labeled true / false classification tags, such as... Figure 4As shown. Manual labeling of true / false classification tags can be done by labelers based on whether the user's question requires complex reasoning, whether it requires external knowledge base retrieval, and whether it has caused significant delays. The information entropy of historical predicted statements in the judgment sample set is input into the initial classification model, which outputs predicted true / false tags. The system calculates the cross-entropy loss value between the predicted true / false tags and the manually labeled true / false classification tags, and uses this cross-entropy loss value to update the network weights of the initial classification model. When the cross-entropy loss value converges on the validation set, or decreases by less than a preset value for several consecutive training epochs, the judgment model is obtained. The training epochs can be set to 50 to 5000, and the learning rate can be set to 0.0001 to 0.01.

[0101] When the Boolean classification result is true, the system extracts the current gaze coordinates and current rotation matrix of the eye skeleton in the digital human 3D skeletal animation system. The current gaze coordinates can be the user's face position in the direction of the virtual camera, the center point of the screen, or the position of the current interactive object. The current rotation matrix is ​​used to describe the posture of the eye skeleton in 3D space. The system uses a preset nonlinear interpolation function to modify the current gaze coordinates to a preset lateral coordinate, generating an eye deflection command. The nonlinear interpolation function can use a smooth step function, a cubic easing-in / easing-out function, or an S-shaped function, and the interpolation duration can be set from 150ms to 1000ms. The lateral coordinates can be offset 5 degrees to the left or right and 25 degrees relative to the current gaze point, or slightly offset 1 degree to 10 degrees upwards or downwards. Lateral gaze simulates the behavior of a person briefly avoiding a directly viewed object when thinking or retrieving memories, making the waiting process no longer like a system lag.

[0102] Simultaneously, the system generates minute angular perturbations based on white noise signals with a preset sampling or update frequency, and synthesizes these perturbations with the current rotation matrix to generate miniature saccadic commands. The sampling or update frequency of the white noise signal can be set from 10Hz to 120Hz, preferably from 20Hz to 80Hz; the angular amplitude can be set from 0.05 degrees to 2 degrees. To ensure that the eye rotation matrix still satisfies the orthogonality of the rotation matrix, the system can perform normalization or orthogonalization processing after the posture synthesis. Miniature saccadic commands allow the eyeballs to make subtle, irregular, but very small movements, preventing the digital human from having a completely frozen gaze during the retrieval waiting period.

[0103] Eye skeleton control can be implemented based on the existing skeleton driving interface of the digital human 3D skeletal animation system. In other words, the system does not need to redesign the underlying rendering or skinning algorithms of the eye skeleton; it only needs to send gaze deflection instructions and micro-sagittation instructions to the existing animation system. Gaze deflection is used to create gaze avoidance during thinking, while micro-sagittation is used to avoid static and stiff eye movements. Together, they address the problem of digital humans lacking natural feedback during knowledge base retrieval waiting periods.

[0104] The system inputs both eye-shifting and micro-sagittation commands into the digital human 3D skeletal animation system to control eye skeletal movement. Once the large language model outputs the text sequence to be read, the system cancels the eye-shifting and micro-sagittation commands. The cancellation doesn't necessarily involve an instantaneous return to focus; instead, it uses non-linear interpolation of 150ms to 800ms to gradually return to the gaze point facing the user or the object being explained. This creates a natural transition from thinking and retrieval to beginning the response.

[0105] In a further embodiment, before the updated physical-mechanical damping coefficient and the initial skeletal motion trajectory are entered into the physics engine for dynamic calculation, the system also determines whether the knowledge base spatial anchor point action needs to be activated. The system extracts the updated physical-mechanical damping coefficient and compares it with a pre-set damping activation threshold. The damping activation threshold can be set from 0.5 to 10.0, preferably from 1.0 to 6.0. If there are multiple damping coefficients for key joints, the average value, the maximum value, or the comprehensive value weighted by joint weights can be compared. When the updated physical-mechanical damping coefficient is greater than the damping activation threshold, it indicates that the current broadcast content is likely in a high information density or professional explanation state. At this time, the convergence of motion amplitude alone is not enough to express the knowledge structure, and it is also necessary to establish a connection between the digital human's hand movements and the spatial structure in the knowledge base document.

[0106] When the damping activation condition is met, the system parses the external knowledge base documents referenced by the large language model when generating the text sequence to be broadcast. These external knowledge base documents can be PDF pages, webpage screenshots, scanned copies, spreadsheets, tree diagrams, flowcharts, or slideshow pages. The system inputs these external knowledge base documents into the layout analysis model, which outputs two-dimensional spatial layout metadata. This metadata includes the coordinates of chart row and column boundaries and the hierarchical relationships between nodes. The chart row and column boundary coordinates describe the positions of table cells, chart areas, title areas, horizontal comparison items, or vertical category items; the node hierarchical relationships describe the parent-child, sibling, and depth relationships between nodes in tree diagrams, organizational charts, flowcharts, or chapter levels.

[0107] The layout analysis model can employ a graph neural network. Specifically, the system first extracts text blocks, table lines, graphic nodes, title blocks, and image blocks from the page using a document image encoder, treating these elements as graph nodes. Then, it constructs graph edges based on spatial proximity, alignment, containment, and connectivity relationships. The graph neural network performs message passing between the graph nodes and edges, outputting the predicted boundary coordinates and predicted hierarchy label for each element. The predicted boundary coordinates can use page-normalized coordinates, ranging from 0 to 1, or pixel coordinates. The predicted hierarchy label can include categories such as root node, first-level node, second-level node, table header, row header, column header, data cell, legend, and explanatory text.

[0108] The training process of the layout analysis model includes acquiring a document dataset containing document images and manually labeled boundary coordinates and hierarchical labels. The document dataset can include table documents, hierarchical graph documents, flowchart documents, and ordinary paragraph documents from a business knowledge base. The document images from the dataset are input into an initial graph neural network, which outputs predicted boundary coordinates and predicted hierarchical labels. The system calculates the coordinate error between the predicted boundary coordinates and the manually labeled boundary coordinates, and also calculates the classification error between the predicted hierarchical labels and the manually labeled hierarchical labels. The parameters of the initial graph neural network are then updated by combining the coordinate error and classification error values. The training cycle can be set to 20 to 300 rounds, the batch size can be set to 1 to 64, and the learning rate can be set to 0.00001 to 0.001. After completing a preset number of training cycles, the layout analysis model is obtained.

[0109] The system converts 2D spatial layout metadata into 3D spatial anchor point coordinates in a preset virtual world coordinate system. Specifically, the system sets the pelvic bone point in the digital human 3D skeletal animation system as the origin of the coordinate system to construct a local 3D coordinate system. The advantage of using the pelvic bone point as the origin is that arm movements, torso movements, and body orientation can all be stably calculated around the body center, so that even if the digital human moves as a whole in the virtual scene, it will not affect the relative position of the local indicated movements.

[0110] The system maps the horizontal axis coordinates in the chart's row and column boundary coordinates, which represent left-right distribution, column position, or horizontal comparison relationships, to a local 3D coordinate system. For tables, the left column can be mapped to the left front of the digital human's body, and the right column can be mapped to the right front of the body; for horizontal comparison charts, the left and right schemes can also be mapped to different horizontal positions. The horizontal axis coordinates can be set between -0.8m and +0.8m. The system maps the node hierarchy relationship to the vertical axis coordinates in the local 3D coordinate system. Higher-level nodes can be mapped to higher positions, and child nodes can be mapped to lower positions; the vertical coordinates can be set between 0.2m and 1.6m. The system then obtains the preset depth axis coordinates, which can be set between 0.3m and 1.2m in front of the digital human's body. Finally, the depth axis coordinates are concatenated with the horizontal axis coordinates and the vertical axis coordinates to generate 3D spatial anchor point coordinates.

[0111] For example, when a digital human explains a left-right comparison table, the boundary coordinates of the left column are mapped to a 3D spatial anchor point in front of the body's left side, and the boundary coordinates of the right column are mapped to a 3D spatial anchor point in front of the body's right side. When the digital human reads the content of the left column, the end of its hand naturally points to the front left; when reading the content of the right column, the end of its hand turns to the front right. When the digital human explains a tree-like hierarchy, parent nodes are mapped to higher positions, and child nodes are mapped to lower positions. The digital human's gestures can express hierarchical relationships in a vertical direction. In this way, even if the user does not directly see the original document, they can still perceive the spatial logic in the knowledge structure through the digital human's movements.

[0112] The system inputs the coordinates of the 3D spatial anchor point as the target attraction point into the inverse kinematics solver. The inverse kinematics solver can employ the CCD algorithm, FABRIK algorithm, Jacobi transpose algorithm, or Jacobi pseudo-inverse algorithm. The solution chain can start from the shoulder joint, pass through the elbow joint, and end at the wrist joint or the end of the hand. The solver calculates the corrected joint rotation amount that guides the digital human's hand end to the 3D spatial anchor point coordinates based on the target attraction point. To avoid unnatural arm posture, the solver can incorporate joint angle constraints, such as a shoulder abduction range of 0 to 120 degrees, an elbow flexion and extension range of 0 to 150 degrees, and a wrist rotation range of -60 degrees to +60 degrees. The corrected rotation amount per frame can also be limited to keep the hand movement speed between 0.1 m / s and 2.0 m / s.

[0113] Inverse kinematics solvers are a mature technology in the fields of digital human animation and robot motion control. This invention does not aim to improve upon the inverse kinematics algorithm itself. The purpose of using an inverse kinematics solver in this invention is to convert the 3D spatial anchor point coordinates obtained from the layout analysis model into joint rotations that can be executed by the end effector, so that spatial semantics such as tables, hierarchies, and processes in the knowledge base document can be expressed by the gestures of the digital human.

[0114] After obtaining the corrected joint rotation, the system integrates this correction into the initial skeletal motion trajectory to update it. This integration can employ a weighted mixing method, with weights determined based on the damping coefficient, layout analysis confidence, and the current speech pause position. When there is high damping, high confidence, and the speech is near an emphasis word or pause, the hand pointing motion has a larger weight; when there is low damping or low layout analysis confidence, the hand pointing motion has a smaller weight. The integrated trajectory is then fed into the physics engine along with the updated physical-mechanical damping coefficient. This ensures that the digital human maintains a restrained posture while its hand extremities accurately point to spatial anchor points in the knowledge base document, preventing the complete loss of narration due to high damping.

[0115] In another embodiment, such as Figure 5As shown, the system includes a text sequence acquisition module, an information entropy processing module, a damping increment acquisition module, a damping update module, and an action calculation output module.

[0116] The text sequence acquisition module receives natural language input from the user, inputs it into the large language model, obtains the text sequence to be played from the output of the large language model, and converts the text sequence into an audio waveform. This module can also record external knowledge base documents referenced by the large language model for subsequent spatial anchoring processing.

[0117] The information entropy processing module is used to input the text sequence to be broadcast into a causal language model for word segmentation, obtain the probability distribution of each word after segmentation, and calculate the word-level information entropy. Based on the word-level information entropy, it obtains the moving average information entropy within the current time window and calculates the information entropy difference between the moving average information entropy and the historical baseline entropy value. This module can maintain historical baseline entropy values ​​under different business scenarios, enabling the digital human to have different action style baselines in customer service, teaching, broadcasting, and professional consultation scenarios.

[0118] The damping increment acquisition module is used to input the information entropy difference into a damping mapping model pre-trained based on reinforcement learning, and obtain the damping adjustment increment output by the damping mapping model. This module can be deployed on the server side or on the client side for local inference to meet the needs of real-time animation generation.

[0119] The damping update module is used to accumulate the damping adjustment increment to the current physical-mechanical damping coefficients of key joints in the digital human 3D skeletal animation system, resulting in updated physical-mechanical damping coefficients. Key joints include the spinal joint, shoulder joint, and elbow joint. This module can also perform damping coefficient clipping, inter-frame smoothing, and anomaly rollback.

[0120] The motion calculation output module is used to input audio waveforms into the motion generation network, obtain the initial skeletal motion trajectory output by the motion generation network, and input the updated physical mechanical damping coefficients and the initial skeletal motion trajectory into the physics engine of the digital human 3D skeletal animation system to perform dynamic calculations and output the target motion animation. The target motion animation can include torso posture, shoulder and arm movements, hand pointing movements, head micro-movements, and eye movements, and can also be rendered synchronously with lip-sync animation and facial expression animation.

[0121] In a further embodiment, the system also includes a pre-judgment module, an eye control module, a knowledge base layout analysis module, and a spatial anchor point motion module. The pre-judgment module calculates the information entropy of the predicted statement before the natural language input enters the large language model and outputs a Boolean classification result through the judgment model. The eye control module extracts the current gaze coordinates and current rotation matrix of the eye skeleton when the Boolean classification result is true, generates eye deflection commands and micro-sagittation commands, and cancels these commands after obtaining the text sequence to be played. The knowledge base layout analysis module parses external knowledge base documents to obtain two-dimensional spatial layout metadata when the damping coefficient exceeds the damping activation threshold. The spatial anchor point motion module converts the two-dimensional spatial layout metadata into three-dimensional spatial anchor point coordinates, obtains corrected joint rotations through an inverse kinematics solver, and integrates the corrected joint rotations into the initial skeleton motion trajectory.

[0122] Through the above implementation methods, the present invention can form a clear and executable overall technical solution. The system calculates text information entropy using the lexical probability distribution output by the causal language model, and then uses the damping mapping model obtained by reinforcement learning to convert the information entropy difference into the physical and mechanical damping adjustment amount of key joints; it uses the information entropy of predicted sentences to drive eye deflection and micro-salivation, making the knowledge base retrieval waiting process appear as a biologically realistic thinking state; it uses a layout analysis model and an inverse kinematics solver to transform the two-dimensional spatial structure in the knowledge base documents into the three-dimensional pointing actions of the digital human, so that the digital human can be both steady and restrained when giving professional explanations, and can accurately express the spatial semantics in tables, charts and hierarchical structures.

Claims

1. A knowledge-based enhanced digital human dialogue and action collaboration method, characterized in that, include: Receive natural language input from the user, input the natural language input into the large language model, obtain the text sequence to be played output by the large language model, and convert the text sequence to be played into an audio waveform; The text sequence to be broadcast is input into a causal language model for word segmentation. The probability distribution of each word after segmentation is obtained and the word-level information entropy is calculated. Based on the word-level information entropy, the moving average information entropy within the current time window is obtained, and the information entropy difference between the moving average information entropy and the preset historical baseline entropy value is calculated. The information entropy difference is input into a damping mapping model pre-trained based on reinforcement learning to obtain the damping adjustment increment output by the damping mapping model. The damping adjustment increment is accumulated and added to the current physical and mechanical damping coefficient of the key joint preset in the digital human 3D skeletal animation system to obtain the updated physical and mechanical damping coefficient. The audio waveform is input into the motion generation network to obtain the initial skeletal motion trajectory output by the motion generation network. The updated physical mechanical damping coefficient and the initial skeletal motion trajectory are then input into the physics engine of the digital human 3D skeletal animation system to perform dynamic calculations and output the target motion animation.

2. The knowledge-based enhanced digital human dialogue and action collaboration method according to claim 1, characterized in that, The damping mapping model is a machine learning model trained using a reinforcement learning algorithm. The training process of the damping mapping model includes: Obtain the set of environmental states containing historical entropy differences; The historical information entropy difference in the environmental state set is input into the initial machine learning model, and the predicted damping adjustment increment output by the initial machine learning model is obtained as the action output. Obtain a preset smoothness target, and calculate the reward function value between the predicted damping adjustment increment and the smoothness target; The gradient ascent algorithm is used to maximize the reward function value to update the model parameters of the initial machine learning model, and the damping mapping model is obtained when the iteration termination condition is met.

3. The knowledge-based enhanced digital human dialogue and action collaboration method according to claim 1, characterized in that, The key joints include the spinal joint, shoulder joint, and elbow joint in the digital human 3D skeletal animation system. The steps to obtain the moving average information entropy within the current time window include: The text sequence to be broadcast is divided into a time window by a preset number of word units after word segmentation according to the chronological order. Extract the word-level information entropy of all words within the same time window and calculate the arithmetic mean. Use the arithmetic mean as the moving average information entropy within the current time window.

4. The knowledge-based enhanced digital human dialogue and action collaboration method according to claim 1, characterized in that, The steps of inputting the updated physical mechanical damping coefficient and the initial skeletal motion trajectory into the physics engine of the digital human 3D skeletal animation system to perform dynamic calculations and output the target motion animation include: The initial skeletal motion trajectory is converted into the expected rotational angular velocity of the key joint; The updated physical-mechanical damping coefficient is introduced into the physics engine to generate a physical damping torque in the opposite direction to the expected rotational angular velocity; The physical damping torque is superimposed on the kinematic equation of the key joint to obtain the actual rotational angular velocity of the key joint, and the key joint is driven by the actual rotational angular velocity to generate the target motion animation.

5. The knowledge-based enhanced digital human dialogue and action collaboration method according to claim 1, characterized in that, Before feeding the natural language input into the large language model, the method further includes: The natural language input is fed into the predictive language model, and the predicted sentence information entropy of the natural language input is obtained from the output of the predictive language model. The predicted statement information entropy is input into a preset judgment model to obtain the Boolean classification result output by the judgment model; Under the condition that the Boolean classification result is true, extract the current gaze point coordinates and the current rotation matrix of the eyeball skeleton in the digital human 3D skeletal animation system; The current gaze point coordinates are modified to a pre-defined lateral coordinate using a preset nonlinear interpolation function to generate an eye deflection command; A white noise signal of a preset frequency is superimposed onto the current rotation matrix to generate a micro-scanning command; The eye-turning command and the micro-sagittation command are input together into the digital human 3D skeletal animation system to control the movement of the eye skeleton, and when the text sequence to be broadcast is obtained from the output of the large language model, the eye-turning command and the micro-sagittation command are cancelled.

6. The knowledge-based enhanced digital human dialogue and action collaboration method according to claim 5, characterized in that, The determination model is a binary classification model, and the training process of the determination model includes: Obtain a judgment sample set containing the information entropy of historical prediction statements and manually labeled true and false classification tags; Input the historical prediction statement information entropy from the judgment sample set into the initial classification model to obtain the predicted true / false labels output by the initial classification model; Calculate the cross-entropy loss value between the predicted true / false labels and the manually labeled true / false classification labels; The network weights of the initial classification model are updated using the cross-entropy loss value, and the decision model is obtained when the cross-entropy loss value converges.

7. The knowledge-based enhanced digital human dialogue and action collaboration method according to claim 1, characterized in that, Before inputting the updated physical-mechanical damping coefficient and the initial skeletal motion trajectory into the physics engine of the digital human 3D skeletal animation system to perform dynamic calculations, the method further includes: Extract the updated physical-mechanical damping coefficient and compare it with a preset damping opening threshold. Under the condition that the updated physical and mechanical damping coefficient is greater than the damping opening threshold, the external knowledge base documents referenced by the large language model when generating the text sequence to be broadcast are parsed. The external knowledge base document is input into the layout analysis model to obtain the two-dimensional spatial layout metadata output by the layout analysis model; The two-dimensional spatial layout metadata is converted into three-dimensional spatial anchor point coordinates in a preset virtual world coordinate system; The coordinates of the three-dimensional spatial anchor point are input into the inverse kinematics solver as the target attraction point. The inverse kinematics solver is used to obtain the corrected joint rotation amount that guides the end of the digital human's hand to the coordinates of the three-dimensional spatial anchor point. The corrected joint rotation is incorporated into the initial bone motion trajectory to update the initial bone motion trajectory.

8. The knowledge-based enhanced digital human dialogue and action collaboration method according to claim 7, characterized in that, The steps for obtaining the two-dimensional spatial layout metadata output by the layout analysis model include: Extract the chart row and column boundary coordinates and node hierarchy relationships output by the layout analysis model, and use the chart row and column boundary coordinates and node hierarchy relationships as the two-dimensional spatial layout metadata; The steps for converting the two-dimensional spatial layout metadata into three-dimensional spatial anchor point coordinates in a preset virtual world coordinate system include: The pelvic bone points in the digital human 3D skeletal animation system are set as the origin of the coordinate system to construct a local 3D coordinate system. Replace the row and column boundary coordinates of the chart with the horizontal axis coordinates of the local three-dimensional coordinate system; Replace the node hierarchy relationship with the vertical axis coordinate in the local three-dimensional coordinate system; Obtain the preset depth axis coordinates, and concatenate the depth axis coordinates with the horizontal axis coordinates and the vertical axis coordinates to generate the three-dimensional space anchor point coordinates.

9. The knowledge-based enhanced digital human dialogue and action collaboration method according to claim 7, characterized in that, The layout analysis model was trained using the following steps: Obtain a document dataset containing document images and manually annotated boundary coordinates and hierarchical labels; The document images in the document dataset are input into the initial graph neural network to obtain the predicted boundary coordinates and predicted level labels output by the initial graph neural network; Calculate the coordinate error between the predicted boundary coordinates and the manually labeled boundary coordinates, and calculate the classification error between the predicted hierarchical labels and the manually labeled hierarchical labels; The parameters of the initial graph neural network are updated by combining the coordinate error value and the classification error value, and the layout analysis model is obtained after completing a preset number of training cycles.

10. A knowledge-based enhanced digital human dialogue and action collaboration system for implementing the method of claim 1, characterized in that, include: The text sequence acquisition module is configured to receive natural language input from the user, input the natural language input into a large language model, obtain the text sequence to be played output by the large language model, and convert the text sequence to be played into an audio waveform. The information entropy processing module is configured to input the text sequence to be broadcast into a causal language model for word segmentation, obtain the probability distribution of each word after segmentation and calculate the word-level information entropy, obtain the moving average information entropy within the current time window based on the word-level information entropy, and calculate the information entropy difference between the moving average information entropy and the preset historical benchmark entropy value. The damping increment acquisition module is configured to input the information entropy difference into a damping mapping model pre-trained based on reinforcement learning, and acquire the damping adjustment increment output by the damping mapping model; The damping update module is configured to add the damping adjustment increment to the current physical-mechanical damping coefficient of the preset key joint in the digital human 3D skeletal animation system to obtain the updated physical-mechanical damping coefficient. The motion calculation output module is configured to input the audio waveform into the motion generation network, obtain the initial skeletal motion trajectory output by the motion generation network, input the updated physical mechanical damping coefficient and the initial skeletal motion trajectory into the physics engine of the digital human 3D skeletal animation system to perform dynamic calculation, and output the target motion animation.

Images