Method and system for predicting semantics and preemptively scheduling digital human interaction, electronic device and storage medium

Abstract

This application provides a digital human interaction method, system, electronic device, and storage medium based on predictive semantics and preemptive scheduling. The method includes: during streaming dialogue generation, the server monitors the text stream output by a large language model and its generation probability in real time; before audio synthesis begins, the server predicts and sends non-linguistic behavior instructions to the client based on the generation probability and a semantic lookahead window; when a subsequent text stream is detected to constitute a semantic inversion, the server sends a cancellation instruction for the already sent instruction; the client receives and parses the instruction containing scheduling metadata, and performs preemptive scheduling of actions based on the metadata, including interruption, parallel execution, or smooth transition; in response to the cancellation instruction, the client executes a decay cancellation algorithm to smoothly transition the target action to a terminated state. This application reduces perceptual latency and achieves dynamic behavior correction by decoupling visual feedback from audio generation and introducing a predictive cancellation mechanism.

Description

Technical Field

[0001] This application relates to the interdisciplinary fields of artificial intelligence, computer graphics and human-computer interaction, and in particular to a digital human interaction method, system, electronic device and storage medium for predictive semantics and preemptive scheduling. Background Technology

[0002] Current mainstream digital human interaction systems typically follow a sequential processing flow: first, user speech is converted into text using Automatic Speech Recognition (ASR); then, a Large Language Model (LLM) generates response text; next, text-to-speech (TTS) technology converts the text into audio; and finally, the audio drives the digital human's facial animations and body movements. In this architecture, the digital human's visual feedback, such as facial expressions, lip movements, and gestures, heavily relies on the audio stream generated by the speech synthesis step and its timestamp information.

[0003] Although some systems employ streaming technology to output audio in segments, the triggering of visual actions still lags behind the actual generation of semantics. This tightly coupled "voice-driven vision" model introduces several insurmountable drawbacks: First, visual actions must wait for the audio data packets to be generated before they can begin, resulting in the user's perception that the digital human "thought it out, spoke it, and then started moving." Visual feedback lags significantly behind the dialogue content, causing high perceptual latency and severely undermining the real-time feel of the interaction. Second, the deep emotions, tone, and intentions contained in the text stream generated by the large language model are greatly simplified or lost during transmission to the client, leading to stiff actions that are disconnected from the dialogue content and the user's emotions. More importantly, in streaming generation scenarios, the large language model may first output a preliminary judgment and then correct it. Once the existing system drives an affirmative action based on the first half of the sentence, it cannot gracefully undo or correct the action, ultimately presenting bizarre behavior that defies common sense and damages the realism of the interaction. Summary of the Invention

[0004] The main purpose of this application is to provide a digital human interaction method, system, electronic device, and storage medium for predictive semantics and preemptive scheduling. It aims to solve the problems in existing digital human interaction systems, such as high interaction latency, disconnect between emotional expression and content, and inability to dynamically correct semantic changes during the streaming generation process, which are caused by the strongly coupled architecture of visual feedback relying strictly on audio stream generation, thus impairing the real-time performance and realism of the interaction.

[0005] To achieve the above objectives, this application provides a digital human interaction method for predicting semantics and preemptive scheduling, comprising the following steps: Monitoring the text stream: During the streaming dialogue generation process, the server monitors the text stream output by the large language model and the generation probability corresponding to the text stream in real time; Issuing behavioral instructions: Based on the generation probability and a dynamically sliding semantic look-ahead window, the server predicts non-verbal behavioral instructions associated with the current semantic segment before audio synthesis begins and issues them to the client; Detecting semantic inversion and issuing a reversal instruction: When a subsequent text stream is detected to constitute a semantic inversion with the semantic segment of the issued instruction, the server immediately generates and issues a reversal instruction for the issued instruction; Parsing the instruction: The client receives and parses the non-verbal behavioral instruction and the reversal instruction, wherein the non-verbal behavioral instruction contains scheduling metadata indicating the action execution method; Preemptive scheduling: The client performs preemptive scheduling on the currently executing action and the action indicated by the received instruction based on the scheduling metadata, wherein the preemptive scheduling includes priority-based interruption, part-based decoupling parallel execution, and smooth transition based on fusion parameters; Attenuation cancellation: In response to receiving the cancellation instruction, the client locates the target instruction and executes the attenuation cancellation algorithm to smoothly transition the action corresponding to the target instruction to the terminated state, while connecting it with the action indicated by subsequent instructions.

[0006] Optionally, the generation probability is the log probability output by the large language model. The server calculates the generation confidence of the current keyword element based on the log probability and sets a dynamic threshold. When the confidence is lower than the dynamic threshold, the semantic segment is marked as low certainty, and the generated action command is either reduced in intensity or neutral in emotion, or the transmission is postponed.

[0007] Optionally, the dynamic threshold is adaptively adjusted based on the confusion level of user feedback in historical interaction data. The confusion level is determined based on the user's explicit feedback or implicit behavioral data. The implicit behavioral data includes at least one of the following: the number of times the user asks a question repeatedly, the number of user dialogue rounds, the duration of pauses in the user's speech, and the user's facial expression features.

[0008] Optionally, in the step of parsing the instruction, the scheduling metadata includes at least one of the following: instruction validity period, interruption level, priority, target skeletal location, fusion mode, and fusion duration.

[0009] Optionally, in the preemptive scheduling step, the client establishes independent rendering clock and audio clock dual timelines, and classifies instructions into immediate execution type and audio-synchronized type according to the timing triggering method of the instructions; the immediately executed type instructions are triggered immediately based on the rendering clock, and the audio-synchronized type instructions are triggered synchronously with the phoneme timestamps in the subsequent TTS audio stream; or, in the preemptive scheduling step, the client maintains a part mutual exclusion lock management mechanism, divides the digital human skeleton into logical layers, specifies the target logical layer for the instruction, and the scheduler independently locks each layer to achieve parallel execution of actions of different parts; or, the priority-based interruption is a hard cutoff, when the interruption level of the new instruction is interruptible and the priority is higher than the currently executed instruction, the client suspends the current action and starts the new action within 0.1 seconds; or, the smooth transition based on fusion parameters is a weighted fusion, when the interruption level of the new instruction is only fusion or the difference between the priority and the currently executed instruction is less than a preset threshold, the client performs linear interpolation or spherical linear interpolation on the two actions within 0.3 to 0.5 seconds to achieve a smooth transition.

[0010] Optionally, the method further includes: in the step of monitoring the text stream, the server also receives the emotion embedding vector extracted from the user's audio in real time, and fuses the emotion embedding vector with the text embedding of the current semantic window to predict the non-verbal behavior instruction; and / or, in the step of performing decay reversal, the decay reversal algorithm is an exponential decay algorithm, which decays the amplitude of the target action to zero exponentially within a specified duration.

[0011] Optionally, the method further includes: in the step of issuing the behavioral instruction, the semantic look-ahead window is a dynamically sliding window, and the length of the semantic look-ahead window is predetermined based on a balance between response speed and semantic integrity; and / or, in the step of issuing the behavioral instruction, the server uses a hierarchical prediction model to predict non-verbal behavioral instructions, the hierarchical prediction model including: a fast reflection layer based on rule and keyword matching to handle high-frequency, low-latency requirements, and a deep sentiment and intent layer that uses a Transformer model to fuse text embedding and user audio sentiment embedding, and outputs sentiment tags, sentiment intensity, and behavioral intent tags; and / or, in the step of detecting semantic reversal and issuing a revocation instruction, the detection of semantic reversal includes based on a preset transition conjunction rule. The lightweight model can detect transitional conjunctions or semantic negation patterns in the text stream in real time; and / or, in the step of detecting semantic reversal and issuing a revocation command, the payload of the revocation command includes a target command identifier, revocation reason, decay algorithm type, and decay duration; and / or, in the step of parsing the command, the payload of the non-verbal behavior command includes a text fragment, confidence score, sentiment label and intensity, and command list, wherein each command in the command list includes a command identifier, target layer, action type, fusion mode, fusion duration, intensity, temporal trigger type, interruption level, and priority; and / or, in the preemptive scheduling step, the smooth transition based on the fusion parameters is achieved through at least one of linear interpolation, spherical linear interpolation, or exponential decay.

[0012] This application also provides a digital human interaction system for predictive semantics and preemptive scheduling, including a server and a client, for executing the method described in any of the preceding claims, wherein: the server includes a listening module, a behavior instruction module, and a detection module; the listening module is configured to execute the step of listening to the text stream, the behavior instruction module is configured to execute the step of issuing instructions, and the detection module is configured to execute the step of detecting semantic inversion and issuing a revocation instruction; the client includes a parsing module, a preemptive scheduling module, and a decay revocation module; the parsing module is configured to execute the step of parsing instructions, the preemptive scheduling module is configured to execute the step of preemptive scheduling, and the decay revocation module is configured to execute the step of decay revocation.

[0013] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method as described in any of the foregoing embodiments.

[0014] This application also provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the method as described in any of the preceding claims.

[0015] The technical solution provided in this application decouples the timing of visual feedback from audio generation by performing predictive analysis on the text stream output by a large language model on the server side and issuing non-verbal behavior commands in advance before audio synthesis. Simultaneously, by establishing a semantic inversion detection and command cancellation mechanism, the system is given the ability to dynamically correct executed actions. The client side, through preemptive scheduling and decay cancellation algorithms, ensures smooth fusion and natural transition of multiple commands.

[0016] This application has the following beneficial effects:

[0017] 1. Reduced perceptual latency: By predicting and issuing non-verbal behavioral commands in advance on the server side, which are then executed immediately by the client, the visual feedback of the digital human no longer waits for the audio stream to be ready. The response is almost synchronous with the semantic generation, thereby reducing the perceptual latency from hundreds of milliseconds to tens of milliseconds and improving the real-time feel of the interaction.

[0018] 2. Achieve dynamic behavior correction: The "prediction-cancellation" mechanism allows the system to gracefully cancel or correct actions that have already been executed when it detects a change in the semantics of the large language model output. This enables the digital human to simulate the human cognitive process of "thinking-reacting-correcting," avoiding illogical and bizarre behavior and improving the realism of the interaction.

[0019] 3. Ensure smooth integration of multiple actions: The client's preemptive scheduling mechanism effectively solves the conflict and visual jump problems when multiple action instructions are executed in parallel by finely managing the priority, application part and integration mode of the instructions. This ensures that the digital human's actions remain smooth, natural and harmonious in complex interactive scenarios. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of the architecture of a digital human interaction system according to an embodiment of this application.

[0022] Figure 2 This is a schematic diagram of the digital human interaction method for predictive semantics and preemptive scheduling according to an embodiment of this application.

[0023] Figure 3 This is a schematic diagram of the module interaction sequence in a semantic inversion scenario according to an embodiment of this application. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0026] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0027] (1) Preemptive scheduling: refers to the process by which the client makes intelligent decisions and schedules the currently executing action and the action indicated by a newly received instruction based on the scheduling metadata contained in the instruction. This process includes at least: deciding whether to interrupt the current action based on the instruction priority (interruption); allowing the actions of different body parts to be executed in parallel without causing physical conflicts (parallel execution based on body part decoupling); and achieving a smooth visual transition when switching between two actions through interpolation calculation (smooth transition based on fusion parameters).

[0028] (2) Semantic reversal: refers to the situation in which the subsequently generated text content (such as conjunctions of transition and negation) logically or emotionally overturns or significantly modifies the meaning of the previously generated text fragments during the streaming text generation process. For example, a large language model first outputs "Okay, no problem", and then outputs "...but think about it carefully, there is a risk", the second half of the sentence constitutes a semantic reversal of the first half of the sentence.

[0029] (3) Decaying cancellation: In response to the cancellation command, the client does not immediately terminate the target action, but initiates a smooth transition process. Within a preset duration, the magnitude and influence weight of the action are gradually reduced to zero through a specific algorithm (such as exponential decay) to avoid visual abruptness and simulate the natural human action termination process.

[0030] Please see Figure 2This application provides a digital human interaction method based on predictive semantics and preemptive scheduling, aiming to solve the technical problems in existing digital human interaction systems, such as high perceptual latency, disconnect between emotional expression and content, and inability to dynamically correct semantic changes during streaming generation, due to the strong dependence of visual feedback on audio streams. This method decouples the temporal dependence of visual feedback and audio generation by performing forward prediction on the server side and preemptive scheduling on the client side, thereby improving the real-time performance and realism of the interaction. Figure 2 This is a flowchart illustrating the method, showing the main steps from start to finish.

[0031] In a basic implementation, the method first performs step S201, which involves monitoring the text stream during the streaming dialogue generation process. The server monitors the text stream output by the large language model and the corresponding generation probability of the text stream in real time. Specifically, as shown... Figure 1 As shown, during the streaming dialogue between the user and the digital human, the Large Language Model (LLM) inside the server generates text streams (tokens) in real time as responses to user input. Simultaneously, a predictive streaming enhancer monitors each text stream output by the LLM and its corresponding generation probability in real time. This step obtains information from the very source of data generation, providing a foundation for subsequent predictions. By monitoring in real time, the system can capture every subtle dynamic in semantic formation, rather than waiting for the entire sentence or paragraph to be generated, which is a prerequisite for achieving low-latency interaction.

[0032] Next, in step S202, the server executes the step of issuing behavioral instructions. Based on the generation probability and the dynamically sliding semantic look-ahead window, the server predicts non-verbal behavioral instructions associated with the current semantic segment before audio synthesis begins and sends them to the client. The server's predictive streaming enhancer analyzes the currently forming semantic segment based on the generation probability it monitors and a preset semantic look-ahead window. This semantic look-ahead window can be understood as a buffer that holds several newly generated tokens. By analyzing the text and probability distribution within this window, the predictive streaming enhancer can predict the non-verbal behaviors (such as nodding, smiling, or confused expressions or gestures) most likely associated with the current semantics before the TTS module begins audio synthesis, and encapsulates them into non-verbal behavioral instructions, sending them to the client through an independent, low-latency channel. This method of predicting and issuing instructions in advance solves the latency problem in traditional architectures where visual actions must wait for audio generation, achieving advance visual feedback.

[0033] While the server issues commands, a parallel monitoring process is also ongoing: detecting semantic inversion and issuing reversal commands. Specifically, when a subsequent text stream is detected to have a semantic inversion with the semantic fragment of the already issued command, the server immediately generates and issues a reversal command for the already issued command. Figure 2 As shown in the parallel monitoring section, the predictive streaming enhancer continuously analyzes the subsequently generated text stream and executes step S205, which detects semantic reversals. If newly generated text (such as words like "but" or "however") is detected to constitute a logical or emotional reversal with the semantic fragment upon which the previously issued instruction was based, i.e., the detection result in step S205 is "yes," then step S206 (issuing a reversal instruction) is executed. The server immediately generates a reversal instruction for the previously issued instruction and immediately sends it to the client. This step gives the system the ability to dynamically correct itself, solving the logical contradictions such as "smiling refusal" caused by the inability to revoke actions in the background technology, making the digital human's reactions more in line with human thinking and communication patterns.

[0034] After receiving the instruction from the server (i.e., step S207), the client executes the instruction parsing step. The client receives and parses the non-verbal behavior instruction and the revocation instruction. The non-verbal behavior instruction contains scheduling metadata indicating the action execution method. Specifically, the client's internal multimodal rendering scheduler receives and parses these non-verbal behavior instructions and possible revocation instructions (corresponding to...). Figure 2 (Steps S207 and S208 in the process). The non-verbal behavioral instructions contain crucial information: scheduling metadata. This metadata provides clear guidance on how the client should execute and schedule actions, forming the foundation for managing complex actions. By parsing these instructions, the client can understand the intent, priority, and execution method of each action.

[0035] Subsequently, the client executes preemptive scheduling step S204. Based on the scheduling metadata, the client preemptively schedules the currently executing action and the action indicated by the received instruction. This preemptive scheduling includes priority-based interruption, location-based decoupling parallel execution, and smooth transition based on fusion parameters. The multimodal rendering scheduler intelligently preemptively schedules the currently executing action and the action indicated by the newly received instruction based on the parsed scheduling metadata. This scheduling is multi-dimensional; it may immediately interrupt the current low-priority action upon receiving a high-priority instruction; it may allow parallel execution when two instructions act on different parts of the body (such as the face and hands); or it may perform a smooth fusion transition when two actions need to switch. This preemptive scheduling mechanism solves the action conflict and visual jump problems that easily occur when multiple instructions are executed in parallel in the background technology, ensuring the continuity and harmony of the digital human's actions.

[0036] When the client receives a revocation command (step S207), it executes step S208, a decaying revocation. In response to receiving the revocation command, the client locates the target command and executes a decaying revocation algorithm to smoothly transition the action corresponding to the target command to a terminated state, while seamlessly connecting it with the action indicated by subsequent commands. The multimodal rendering scheduler locates the target command to be revoked based on the information in the revocation command and initiates a decaying revocation algorithm. This algorithm does not simply stop the action abruptly, but rather controls the amplitude or influence weight of the target action to smoothly decay to zero within a short, preset time, while seamlessly connecting it with the action indicated by any subsequent possible new commands. This decaying revocation method improves the naturalness of the action correction process, avoiding the visual abruptness caused by sudden action cessation, making the digital human's behavior correction process appear more like a thoughtful, natural reaction. If no semantic inversion is detected, the process returns to the step of listening to the text stream.

[0037] Please see Figure 3 This diagram illustrates the typical interaction process between the user, server, and client in a semantic inversion scenario, presented as a sequence diagram. After the user initiates a conversation, the server issues a prediction command based on the initially generated text. Subsequently, when the server detects a semantic inversion, it immediately issues a reversal command, and the client executes the corresponding decay reversal algorithm to smoothly terminate the previous actions. This process is described in more detail in the comprehensive embodiment described later.

[0038] Furthermore, in a preferred embodiment, to make the server-side predictions more accurate and reliable, the generation probabilities are specifically the logarithmic probabilities (Logprobs) output by the Large Language Model (LLM). The server-side predictive streaming enhancer calculates the generation confidence of key words (such as sentiment words and intent words) in the current semantic segment based on these logprobs. Simultaneously, the system sets a dynamic threshold, for example, an initial value of 0.8. When the calculated confidence is lower than this dynamic threshold, the system determines that the current LLM output has high uncertainty and marks the semantic segment as low-deterministic. This low-determinism means that the LLM may overturn the current conclusion in subsequent generation. Therefore, the predictive streaming enhancer adopts a more conservative strategy, such as generating an action instruction with significantly reduced intensity or neutral sentiment, or even suspending the sending of any instructions until more tokens are received and the semantics become clearer. By introducing confidence-based judgments, this solution solves the problem of erroneous actions that may result from blind prediction, improving the accuracy and reliability of the prediction behavior.

[0039] In another preferred embodiment, to enable the system to adapt to different interaction scenarios and user habits, the dynamic threshold is adaptively adjusted based on the confusion level of user feedback in historical interaction data. In one specific embodiment, the confusion level of user feedback can be quantified in various ways. For example, the system can collect implicit behavioral data from users, including but not limited to: the number of repeated questions asked by the user during interaction with the digital human, the total number of rounds in a single dialogue, and the duration of abnormal pauses in voice input. This data can be normalized and used as a measure of confusion level. In another embodiment, the system can also use the front-facing camera of the client device to capture the user's facial image and output the user's confusion confidence score through a pre-trained micro-expression recognition model. Furthermore, the system also supports receiving explicit user feedback, such as user ratings of the experience or "dislike" clicks after the interaction ends; this explicit feedback can also be used to dynamically adjust the threshold. By combining multiple feedback sources, the system can more accurately assess the user's satisfaction with the digital human's responses, thereby achieving precise adaptive adjustment of the threshold.

[0040] Based on the aforementioned quantitative indicators, the system can dynamically adjust the thresholds. For example, if the system detects that certain predictive actions of the digital human frequently cause user confusion (such as actions that are too premature or inconsistent with the final semantics), it automatically increases the dynamic confidence threshold to make predictions more "cautious." Conversely, if predictive actions receive positive feedback from users, the threshold can be appropriately lowered to pursue faster response times. Through this adaptive adjustment mechanism, the system achieves intelligent self-optimization, continuously improving the interaction quality and effectiveness in specific application scenarios.

[0041] Furthermore, to enable fine-grained action scheduling by the client, the scheduling metadata during instruction parsing can include at least one of the following: instruction validity period, interruption level, priority, target skeletal location, fusion mode, and fusion duration. The instruction validity period defines the lifespan of the instruction, preventing outdated instructions from being executed due to network latency. The interruption level and priority together determine how new and old instructions are handled when they meet. The target skeletal location specifies the body area affected by the action, forming the basis for parallel execution. The fusion mode and fusion duration guide the client on how to perform a smooth transition. This specific metadata concretizes the abstract scheduling concept into executable parameters, providing the necessary data support for implementing complex preemptive scheduling.

[0042] In one optional implementation, the preemptive scheduling can be implemented in various ways. One implementation involves the client's multimodal rendering scheduler establishing independent rendering and audio clocks, forming a dual timeline. Instructions are categorized into immediately executed and audio-synchronized types based on their timing. Immediately executed instructions (such as a raised eyebrow expressing surprise) are triggered immediately based on the rendering clock to achieve visual feedback with minimal latency. Audio-synchronized instructions (such as gestures synchronized with specific words) wait for subsequent TTS audio streams and are triggered synchronously with the phoneme timestamps in the audio to ensure accurate audio-visual synchronization. Through this timing decoupling, the system balances response speed and accuracy.

[0043] In another alternative implementation, the client's multimodal rendering scheduler maintains a part-specific mutex lock management mechanism. This mechanism divides the skeletal model of the digital human into multiple logical layers, such as the "face layer," "upper body layer," "left hand layer," and "right hand layer." Each issued instruction can specify its target logical layer. When executing an instruction, the scheduler attempts to acquire the "lock" of the corresponding logical layer. Since the locks of different layers are independent, instructions acting on different layers (such as "smiling" and "waving") can be executed in parallel without interference. This part-specific decoupling enhances the digital human's ability to perform multiple tasks simultaneously, making its behavior richer and more natural.

[0044] In another optional implementation, the priority-based interruption can be specifically implemented as a hard cutoff. For example, the state machine can include states such as idle, running, and weighted fusion. When the multimodal rendering scheduler is in the running state, if it receives a new instruction whose interruptibility level is marked as "interruptible" and whose priority is significantly higher than the currently executing instruction, the scheduler will perform a hard cutoff. This means that the client will forcibly suspend its current action within a very short time (e.g., 0.1 seconds) and immediately start the action indicated by the new instruction. This mechanism is suitable for scenarios that require rapid response to urgent or important signals. For example, if a digital human is performing a smooth introductory action and suddenly receives a high-priority "alert" instruction, it needs to immediately switch to the alert action through a hard cutoff, which helps to ensure the timely delivery of key information.

[0045] In another optional implementation, the smooth transition based on fusion parameters can be specifically implemented as weighted fusion. For example, when the interruption level of a new instruction is marked as "fusion only," or the difference between its priority and the currently executed instruction is less than a preset threshold, the scheduler will transition from the execution state to the weighted fusion state. In this state, the client will perform a weighted average of the skeletal animation data of the old and new actions within a preset fusion duration (e.g., 0.3 to 0.5 seconds). For example, linear interpolation (Lerp) or spherical linear interpolation (Slerp) algorithms can be used. During the transition, the weight of the old action smoothly decays from 1 to 0, while the weight of the new action smoothly increases from 0 to 1. After the fusion is completed, the state machine returns to the execution state and fully executes the new action. This weighted fusion method avoids visual jumps during action switching, making the action transition smooth and natural.

[0046] In an alternative implementation, the smooth transition based on fusion parameters can also be achieved through exponential decay. Specifically, when transitioning from an old action to a new action is required, the client can treat the pose difference between the two actions (e.g., the rotational difference of skeletal joints) as a quantity that needs to be decayed. During the fusion duration, the current pose can continuously approximate the target pose according to an exponential function, the formula of which can be expressed as: ,in yes The posture of the moment It is the initial posture. It is the target posture. It is the attenuation coefficient. This method is particularly suitable for transition scenes that require a quick start and a gentle end, thereby achieving a smooth transition effect with a specific dynamic aesthetic and enriching the means of motion fusion.

[0047] Furthermore, to enhance the emotional expression of the digital human, a preferred embodiment of this application further includes, in the step of monitoring the text stream, the server receiving in real time an emotion embedding vector extracted from the user's original audio. Specifically, the server's ASR module or a parallel audio analysis module, when processing the user's input audio, utilizes a pre-trained emotion recognition model (e.g., a Wav2Vec2-based model) to extract a feature vector that represents the user's current emotion (e.g., happiness, sadness, anxiety). This emotion embedding vector is then sent to a predictive streaming enhancer and fused with the text embedding of the current semantic window. In this way, when generating nonverbal behavioral instructions, the predictive model's decision-making is not solely based on the literal meaning of the text, but also includes the perception of the user's emotions. For example, even if the user says "I'm fine," but their tone is tired and frustrated, the predictive model, fused with the emotion vector, may generate an action instruction expressing "concern" rather than "happiness," thereby improving the consistency of emotional interaction. In a feasible embodiment, the emotion embedding vector is extracted in real time from the user's original audio by a pre-trained emotion recognition model.

[0048] Furthermore, to make the undo process more natural, the decay undo algorithm can be specifically implemented as an exponential decay algorithm. When the client's multimodal rendering scheduler receives the undo command, it initiates an exponential decay function to process the target action. Simultaneously with the decay undo, the client initiates the action indicated by the new command, and smoothly transitions to the new action during the decay process. Assuming the rotation or displacement value of the target action on each skeletal joint is A, the decay process will calculate the motion amplitude for each frame within a specified duration T according to the formula A(t) = A * exp(-k*t), where t is the time since the decay began and k is the decay coefficient. This causes the motion amplitude to decay rapidly at the beginning, then gradually slow down, and finally smoothly converge to zero. This non-linear decay method is very consistent with physical inertia and human movement habits, resulting in a smoother and more natural visual effect compared to linear decay.

[0049] In a more specific implementation, the server-side prediction and detection mechanism can be further refined. For example, the semantic look-ahead window can be a dynamically sliding window, the length of which (e.g., N = 3 to 5 tokens) is determined experimentally in advance based on the balance between response speed and semantic integrity. Furthermore, the server can employ a hierarchical prediction model to predict nonverbal behavioral instructions. This model includes a fast-reaction layer that handles high-frequency, low-latency needs (such as greetings and confirmations) based on rule and keyword matching, with extremely short response times; it also includes a deep sentiment and intent layer that uses a lightweight Transformer model, fusing text embeddings and user audio sentiment embeddings to output more complex sentiment labels, sentiment intensity, and behavioral intent. This hierarchical structure balances efficiency and depth.

[0050] Similarly, the process of detecting semantic inversion can be further specified. For example, a predictive streaming enhancer can incorporate a fast detector based on preset transitional conjunction rules (such as detecting "but," "however," "however," etc.), or use a lightweight text classification model to detect transitional or semantic negation patterns in the text stream in real time. Such specialized detectors are more efficient than relying on general LLMs for judgment.

[0051] To support the complex logic described above, the data protocol between the server and client also needs to be precisely designed. For example, the payload of nonverbal behavior instructions can include text fragments, confidence scores, sentiment labels and intensity, and a detailed list of instructions. Each instruction in the list includes a unique instruction identifier, target layer (e.g., FACE), action type (e.g., nod_firm), fusion mode, fusion duration, intensity, timing trigger type (IMMEDIATE / SYNC), interruption level, and priority. Correspondingly, the payload of the revocation instruction includes the identifier of the target instruction, the revocation reason (e.g., SEMANTIC_REVERSAL), the decay algorithm type (e.g., EXPONENTIAL), and the decay duration. This structured data protocol helps the system operate efficiently and accurately. The nonverbal behavior instructions and revocation instructions can be encapsulated in JSON format.

[0052] This application also provides a digital human interaction system based on predictive semantics and preemptive scheduling. Please refer to... Figure 1The system includes a server and a client for executing the methods described in any of the foregoing embodiments. The server includes a listening module, a behavior instruction module, and a detection module. The functions of these modules can be integrated into a predictive streaming enhancer. Specifically, the listening module is configured to listen to the text stream and monitor the output of the Large Language Model (LLM) in real time. The behavior instruction module is configured to issue behavior instructions, generating and sending predictive instructions based on the analysis results. The detection module is configured to detect semantic inversion and issue reversal instructions, achieving dynamic correction.

[0053] The client includes a parsing module, a preemptive scheduling module, and a decay cancellation module. The functions of these modules can be integrated into a multimodal rendering scheduler. Specifically, the parsing module is configured to execute the parsing instruction step, understanding the instruction content and metadata from the server. The preemptive scheduling module is configured to execute the preemptive scheduling step, interrupting, parallelizing, or merging new and old actions based on the metadata, and handing over the final rendering task to the rendering engine to generate the final digital human output. The decay cancellation module is configured to execute the decay cancellation step, smoothly aborting the target action upon receiving a cancellation instruction. Through this modular collaborative work, the server and client together constitute the specific implementation of the technical solution of this application.

[0054] In a comprehensive embodiment that integrates a specific technical approach of this application, the entire workflow will exhibit high intelligence and smoothness. Imagine a scenario: after a complex project presentation, a user says to the digital assistant with a slightly sarcastic and weary tone, "Well done, you almost succeeded."

[0055] First, the server-side ASR module converts speech to text, while a parallel audio sentiment analysis module extracts sentiment embedding vectors for "disappointment" and "fatigue." The deep sentiment and intent layer of the predictive streaming enhancer receives the text "Well done!" along with its corresponding low-confidence Logprobs and a strong negative sentiment vector. The model determines that this is an ironic scenario, so instead of generating a conventional "nod" or "smile" instruction, it generates a small, tentative action instruction A with emotions of "doubt" and "concern" (e.g., slightly tilting the head, slightly furrowing the eyebrows), and sends it to the client.

[0056] The client's multimodal rendering scheduler receives instruction A and executes it immediately. The digital human displays an expression as if trying to understand the user's true intentions. Immediately afterwards, the LLM generates the second half of the sentence, "...almost succeeded." At the same time, when an external high-priority event (such as a new email arriving) is triggered, the server issues a high-priority instruction B, instructing the digital human to "point its right hand to the email icon in the lower right corner of the screen."

[0057] At this point, the client's multimodal rendering scheduler is facing multitasking. It detects that the target skeletal part of instruction B is the "right hand," which does not conflict with the currently executing instruction A (which mainly affects the "face" and "head"). Therefore, using a part mutex mechanism, it allows instruction B to be executed in parallel, and the digital human immediately reaches out to point. Almost simultaneously, after seeing the digital human's reaction, the user clarifies, "Just kidding, actually the result is better than expected."

[0058] The server-side semantic inversion detector detected the clear signal of "just kidding" and immediately issued a reversal instruction C for the tentative action instruction A. This instruction specified the use of an exponential decay algorithm and was to be completed within 200 milliseconds. At the same time, the predictive streaming enhancer generated a new action instruction D based on the new positive semantics and the user's softened tone. This instruction included "smiling" and "nodding" actions and specified a "weighted fusion" mode.

[0059] Ultimately, the client's multimodal rendering scheduler demonstrated its scheduling capabilities: while maintaining the action of pointing the right hand at the email icon (executed in parallel), it executed the undo instruction C, causing the "concerned and worried" expressions on the head and face to disappear smoothly through exponential decay. Following the requirements of instruction D, it seamlessly transitioned to the final expressions and actions of "smiling" and "nodding" through weighted fusion. Throughout the process, the digital human demonstrated the ability to listen, understand, correct, multitask, and respond to the user's emotions, exhibiting coherent, natural, and logically clear reactions.

[0060] This embodiment achieves beneficial overall technical effects by combining all the aforementioned preferred technical features. Compared to implementing only the basic solution, it has lower response latency, improved accuracy and depth of emotional interaction, and can handle complex interaction scenarios involving semantic inversion, multi-task parallelism, and action conflicts. The synergistic effect of each technical feature ultimately achieves a highly efficient, accurate, realistic, and responsive digital human interaction effect.

[0061] The technical solutions provided in this application can be widely applied to various practical scenarios. In the field of intelligent customer service, this method enables virtual customer service representatives to respond more quickly and express emotions more appropriately, understanding the user's true intentions rather than merely responding to literal questions, thereby improving service quality and the effectiveness of interaction. In the field of virtual companions or digital healing, this method endows digital humans with the ability to dynamically correct and empathize, enabling them to provide more emotionally engaging companionship. In the gaming field, NPCs (non-player characters) using this method will no longer respond to fixed programs, but will be able to exhibit rich, natural, and logical reactions based on real-time dialogue with players, enhancing the immersion of the game. In online education and virtual training, virtual teachers or coaches can provide immediate and appropriate feedback based on students' words and emotions, improving the effectiveness of teaching interaction. In emerging applications such as digital human live streaming, this method can also make the interaction between virtual anchors and viewers more vivid and natural, enhancing the liveliness of interactive content.

[0062] This application also provides an electronic device, which can be a server-side or client-side device, such as a smartphone, tablet computer, personal computer, server, etc. The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it can implement the predictive semantics and preemptive scheduling digital human interaction method described in any of the foregoing embodiments.

[0063] This application also provides a computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program can implement the predictive semantics and preemptive scheduling digital human interaction method described in any of the foregoing embodiments. The computer-readable storage medium can be any type of volatile or non-volatile storage device or a combination thereof, such as read-only memory (ROM), random access memory (RAM), solid-state drive (SSD), optical disk, or magnetic disk.

[0064] The above description is merely a preferred embodiment of this application and is not intended to limit this application. It should be understood that the predictive semantics and preemptive scheduling digital human interaction method, system, electronic device, and storage medium provided in this application achieve temporal decoupling of visual feedback and audio generation through real-time prediction of the semantic stream by the server and preemptive scheduling of instructions by the client, and support dynamic correction of semantic inversion. Those skilled in the art can implement corresponding digital human interaction systems in different hardware platforms, software environments, or application scenarios based on the technical content disclosed in this application. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this application (such as adjusting the length of the semantic lookahead window, changing the specific structure of the hierarchical prediction model, or adopting other equivalent attenuation algorithms), should be included within the protection scope of this application. Furthermore, the reference numerals in the accompanying drawings are for illustrative purposes only and should not be construed as limiting this application. The protection scope of this application should be determined by the appended claims.

Claims

1. A method for predicting digital human interaction with pre-emptive scheduling of semantics, characterized by, Includes the following steps, Listening to the text stream: During the streaming dialogue generation process, the server listens to the text stream output by the large language model and the generation probability corresponding to the text stream in real time; Sending behavioral instructions: Based on the generation probability and the dynamically sliding semantic look-ahead window, the server predicts non-linguistic behavioral instructions associated with the current semantic segment before audio synthesis starts and sends them to the client; Detect semantic reversal and issue revocation command: When it is detected that the semantic reversal between the subsequent text stream and the semantic segment of the issued command constitutes a semantic reversal, the server immediately generates and issues a revocation command for the issued command. In the step of detecting semantic reversal and issuing revocation command, the detection of semantic reversal includes real-time detection of transitional conjunctions or semantic negation patterns in the text stream based on preset transitional conjunction rules or lightweight models. Parsing instructions: The client receives and parses the non-verbal behavior instructions and the undo instructions. The non-verbal behavior instructions include scheduling metadata for indicating the action execution method, and the scheduling metadata includes at least one of the following: instruction validity period, interruption level, priority, target skeletal part, fusion mode, and fusion duration. Preemptive scheduling: The client performs preemptive scheduling on the currently executing action and the action indicated by the received instruction based on the scheduling metadata. The preemptive scheduling includes priority-based interruption, parallel execution based on part decoupling, and smooth transition based on fusion parameters. The smooth transition is a weighted fusion. When the interruption level of the new instruction is only fusion or the difference between the priority and the currently executing instruction is less than a preset threshold, the client performs linear interpolation or spherical linear interpolation on the two actions within 0.3 seconds to 0.5 seconds to achieve a smooth transition. Attenuation cancellation: In response to receiving the cancellation instruction, the client locates the target instruction and executes the attenuation cancellation algorithm to smoothly transition the action corresponding to the target instruction to the terminated state, while connecting it with the action indicated by subsequent instructions.

2. The method according to claim 1, characterized in that, The generation probability is the logarithmic probability output by the large language model. The server calculates the generation confidence of the current keyword element based on the logarithmic probability and sets a dynamic threshold. When the confidence is lower than the dynamic threshold, the semantic segment is marked as low certainty, and the generation of action instructions with reduced intensity or neutral emotion is postponed.

3. The method according to claim 2, characterized in that, The dynamic threshold is adaptively adjusted based on the confusion level of user feedback in historical interaction data. The confusion level is determined based on the user's explicit feedback or implicit behavioral data. The implicit behavioral data includes at least one of the following: the number of times the user asks a question repeatedly, the number of user dialogue rounds, the duration of pauses in the user's speech, and the user's facial expression features.

4. The method according to claim 1, characterized in that, The method further includes: In the preemptive scheduling step, the client establishes independent rendering clock and audio clock dual timelines, and classifies instructions into immediately executed type and audio-synchronized type according to the timing triggering method; the immediately executed type instructions are triggered immediately based on the rendering clock, and the audio-synchronized type instructions are triggered synchronously with the phoneme timestamps in the subsequent TTS audio stream; or, In the preemptive scheduling step, the client-maintained part mutex lock management mechanism divides the digital human skeleton into logical layers, with instructions specifying the target logical layer. The scheduler independently locks each layer to achieve parallel execution of actions in different parts; or, The priority-based interrupt is a hard cutoff. When the interruption level of the new instruction is interruptible and its priority is higher than that of the currently executed instruction, the client will suspend the current action and start the new action within 0.1 seconds.

5. The method according to claim 1, characterized in that, The method further includes: In the step of monitoring the text stream, the server also receives the emotion embedding vector extracted from the user's audio in real time, and fuses the emotion embedding vector with the text embedding of the current semantic window to predict the non-verbal behavior instruction; and / or, In the step of performing decay cancellation, the decay cancellation algorithm is an exponential decay algorithm, which decays the magnitude of the target action to zero exponentially over a specified duration.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: In the step of issuing the action instruction, the length of the semantic look-ahead window is predetermined based on a balance between response speed and semantic integrity; and / or, In the step of issuing behavioral instructions, the server uses a hierarchical prediction model to predict non-verbal behavioral instructions. This hierarchical prediction model includes: a fast reflection layer that handles high-frequency, low-latency requirements based on rule and keyword matching; and a deep sentiment and intent layer that uses a Transformer model to fuse text embeddings and user audio sentiment embeddings, outputting sentiment tags, sentiment intensity, and behavioral intent tags; and / or, In the step of detecting semantic inversion and issuing a revocation command, the payload of the revocation command includes a target command identifier, a revocation reason, a decay algorithm type, and a decay duration; and / or, In the step of parsing the instructions, the payload of the non-verbal behavior instructions includes text fragments, confidence scores, sentiment labels and intensity, and a list of instructions. Each instruction in the list of instructions includes an instruction identifier, target layer, action type, fusion mode, fusion duration, intensity, timing trigger type, interruption level, and priority.

7. A digital human interaction system based on predictive semantics and preemptive scheduling, characterized in that, Includes a server and a client for performing the method as described in any one of claims 1 to 6, wherein: The server includes a listening module, a behavior instruction module, and a detection module; the listening module is configured to execute the listening text stream step, the behavior instruction module is configured to execute the issuing instruction step, and the detection module is configured to execute the detecting semantic reversal and issuing a cancellation instruction step. The client includes a parsing module, a preemptive scheduling module, and a decay cancellation module; the parsing module is configured to execute the parsing instruction step, the preemptive scheduling module is configured to execute the preemptive scheduling step, and the decay cancellation module is configured to execute the decay cancellation step.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 6.

Images