Intelligent Digital Human Interaction System and Method Based on Multi-Module Collaboration

Abstract

This invention discloses an intelligent digital human interaction system and method based on multi-module collaboration. By synchronously generating speech synthesis prosodic annotations in the language large model module and directly transmitting them to the real-time digital human driving module, the prosodic benchmark of text-speech-action is unified. This ensures that the opening and closing of the digital human's mouth and the micro-movement trajectory of the head can perfectly match the stress, pauses, and emotional fluctuations of the speech, eliminating the awkwardness of lip-syncing and speech asynchrony and greatly enhancing the naturalness and anthropomorphism of the interaction. Secondly, by introducing a knowledge base module for real-time retrieval gain and fact verification, it can effectively suppress factual errors that may occur in the language large model during the generation process, enhancing the accuracy and professionalism of the answers. In addition, the four modules are decoupled and communicate through a preset asynchronous message bus, allowing each module to operate independently without blocking each other, effectively ensuring the continuity and stability of the interaction process.

Description

Technical Field

[0001] This invention relates to the field of digital human interaction technology, and in particular to an intelligent digital human interaction system and method based on multi-module collaboration. Background Technology

[0002] With the rapid development of artificial intelligence and computer graphics technology, intelligent digital humans, as an important carrier of human-computer interaction, are gradually evolving from traditional pre-recorded videos or simple rule-driven approaches to having highly human-like characteristics and natural interaction capabilities.

[0003] Existing intelligent digital human interaction systems typically aim to achieve real-time dialogue and interaction with users through technologies such as speech recognition, natural language processing, and motion generation. However, they still face many challenges in terms of interactive experience: (1) In existing systems, speech synthesis (TTS) and digital human-driven (Animation) are often two separate processes. The speech module only focuses on the conversion of text to audio, while the motion module only focuses on the mapping of text to posture. The two lack a unified prosodic benchmark, which leads to the phenomenon of "mechanical feeling" when digital humans speak, such as lip movements not being synchronized with speech, stiff head movements, or lack of emotional fluctuations. This greatly reduces the realism and immersion of the interaction; (2) Existing systems lack effective external knowledge enhancement mechanisms and cannot call authoritative knowledge bases for fact verification in real time during the generation process. This results in the digital human performing poorly in professional field question answering and is difficult to meet users' needs for information accuracy. Summary of the Invention

[0004] In view of this, the present invention proposes an intelligent digital human interaction system and method based on multi-module collaboration, which can effectively solve the defects of existing technologies, such as low realism and immersion in interaction, and difficulty in meeting users' needs for information accuracy.

[0005] The technical solution of this invention is implemented as follows:

[0006] An intelligent digital human interaction system based on multi-module collaboration includes:

[0007] The voice interaction module is used to receive users' voice commands, extract acoustic features and make preliminary determination of intent, and generate text commands;

[0008] The language model module is communicatively connected to the voice interaction module and is used to receive text commands and generate target response text and corresponding speech synthesis prosodic annotations based on contextual semantics.

[0009] The knowledge base module, which is communicatively connected to the language model module, is used to provide domain knowledge retrieval gains and fact verification during the generation of the target answer text.

[0010] The real-time digital human driving module is communicatively connected to the language large model module. It is used to receive the target answer text and the corresponding speech synthesis prosodic annotation, and synchronously generate matching lip movement sequences and body posture instructions to drive the digital human model to perform anthropomorphic display.

[0011] The voice interaction module, language big model module, knowledge base module, and real-time digital human driving module communicate decoupledly through a preset asynchronous message bus.

[0012] As a further optional solution for the intelligent digital human interaction system based on multi-module collaboration, the voice interaction module adopts a streaming speech recognition engine, which outputs intermediate recognition results in real time during the user's voice input process, and transmits the recognition results to the language big model module in advance through the intent slot pre-filling mechanism.

[0013] As a further optional solution to the aforementioned intelligent digital human interaction system based on multi-module collaboration, the step of pre-passing the recognition results to the language large model module through an intent slot pre-filling mechanism specifically includes:

[0014] During the streaming speech recognition process, the voice interaction module performs local intent inference on the recognized text segments with a preset window granularity, and extracts the core intent tags and key slot candidate values.

[0015] When the confidence level of the local intent inference reaches the preset trigger threshold, the core intent tag and key slot candidate value are encapsulated into a pre-filled context structure, and the pre-filled context structure is transmitted to the language large model module in advance via the asynchronous message bus in a bypass push manner.

[0016] As a further optional solution for the intelligent digital human interaction system based on multi-module collaboration, the knowledge base module adopts a dual-path retrieval architecture, which includes a semantic retrieval path based on vector similarity and a keyword matching path based on inverted index. The retrieval results from the two paths are weighted and integrated by a cross-modal attention fusion layer and then input into the generation process of the language large model module.

[0017] As a further optional solution to the aforementioned intelligent digital human interaction system based on multi-module collaboration, the real-time digital human driving module includes a prosodic feature analysis unit and a mapping unit. The prosodic feature analysis unit is used to perform prosodic feature analysis on the target response text and the corresponding speech synthesis prosodic annotation, and extract stress positions and pause nodes. The mapping unit is used to map the extracted stress positions and pause nodes into the mouth opening curve and head micro-movement trajectory of the digital human model.

[0018] As a further optional solution to the aforementioned intelligent digital human interaction system based on multi-module collaboration, the step of performing prosodic feature analysis on the target response text and its corresponding speech synthesis prosodic annotation, and extracting stress positions and pause nodes, specifically includes:

[0019] Based on the pitch baseline, duration distribution, and energy envelope in the prosodic annotation of the speech synthesis, prosodic feature analysis is performed. The stress positions and pause nodes corresponding to the target response text are extracted frame by frame, and the stress positions and pause nodes are output to the mapping unit in the form of a timestamped event sequence.

[0020] As a further optional solution to the aforementioned intelligent digital human interaction system based on multi-module collaboration, the step of mapping the extracted accent positions and pause nodes to the lip opening curve and head micro-movement trajectory of the digital human model specifically includes:

[0021] Based on a pre-built prosodic-action mapping lookup table, the stress positions are mapped one by one to the peak parameters of mouth opening and closing of the digital human model at the corresponding time points, and adjacent peak points are connected by cubic spline interpolation to generate continuous mouth opening and closing curves.

[0022] The pause node is mapped to the head posture holding command of the digital human model, and a head micro-movement downward trajectory of a preset amplitude is superimposed at the start of the pause, and a head micro-lifting back trajectory is superimposed at the end of the pause.

[0023] A method for intelligent digital human interaction based on multi-module collaboration, specifically including:

[0024] It receives the user's voice commands, performs acoustic feature extraction and preliminary intent determination, and generates text commands.

[0025] Based on text instructions and combined with contextual semantics, a target answer text and corresponding speech synthesis prosodic annotations are generated. In the process of generating the target answer text, retrieval gains and fact verification of domain knowledge are obtained from the knowledge base.

[0026] Based on the target response text and the corresponding speech synthesis prosodic annotation, a matching lip-sync sequence and body posture instructions are generated to drive the digital human model to perform anthropomorphic display.

[0027] A computing device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described intelligent digital human interaction method based on multi-module collaboration.

[0028] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described intelligent digital human interaction method based on multi-module collaboration.

[0029] The beneficial effects of this invention are as follows: By synchronously generating speech synthesis prosodic annotations in the language model module and directly transmitting them to the real-time digital human driving module, the prosodic benchmark of "text-speech-action" is unified. This ensures that the digital human's mouth opening and closing, head micro-movement trajectory, and speech stress, pauses, and emotional fluctuations are perfectly matched, eliminating the awkwardness of lip-syncing and speech asynchrony, greatly enhancing the naturalness and anthropomorphism of the interaction, and thus significantly improving the user's immersion. Secondly, by introducing a knowledge base module for real-time retrieval gain and fact verification, it is possible to effectively suppress factual errors that may occur in the language model module during the generation process, ensuring that the digital human provides accurate and authoritative information in professional field Q&A, enhancing the accuracy and professionalism of the answers. In addition, the four modules are decoupled and communicate through a preset asynchronous message bus, allowing each module to operate independently without blocking each other. Even if a certain module experiences high load or latency, it will not cause the entire system to crash, ensuring the continuity and stability of the interaction process. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 This is a schematic diagram illustrating the composition of an intelligent digital human interaction system based on multi-module collaboration according to the present invention.

[0032] Figure 2 This is a flowchart illustrating an intelligent digital human interaction method based on multi-module collaboration according to the present invention.

[0033] Figure 3 This is a schematic diagram of the composition of a computing device according to the present invention. Detailed Implementation

[0034] The technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0035] refer to Figures 1 to 3 A multi-module collaborative intelligent digital human interaction system includes a voice interaction module, a language large model module, a knowledge base module, and a real-time digital human driving module. The voice interaction module, language large model module, knowledge base module, and real-time digital human driving module communicate decoupledly via a preset asynchronous message bus.

[0036] The voice interaction module is used to receive the user's voice commands, extract acoustic features and make preliminary intent judgments, and generate text commands. In some embodiments, the voice interaction module adopts a streaming speech recognition engine, which outputs intermediate recognition results in real time during the user's voice input process, and transmits the recognition results to the language big model module in advance through an intent slot pre-filling mechanism.

[0037] Specifically, the voice interaction module adopts a streaming speech recognition engine based on an end-to-end (E2E) architecture. When a user issues a voice command (e.g., "Check the weather in Beijing tomorrow"), the system does not wait for the user to finish speaking, but extracts acoustic features in real time as the voice signal is input. The specific process is as follows:

[0038] Streaming recognition and intermediate result output: When the user says "Please check tomorrow's information for me", the recognition engine has already converted it into a text fragment and output the intermediate recognition result;

[0039] Intent slot pre-filling: The system performs rapid semantic analysis on the above intermediate text through a preset intent slot pre-filling mechanism, identifies the core intent as "weather query", and extracts the key slot information "time = tomorrow" and "location = Beijing";

[0040] Advance delivery: Before the user finishes saying "weather", the system has already passed a pre-filled context structure containing "intent = weather query", "time = tomorrow", and "location = Beijing" to the language big model module in advance via the asynchronous message bus;

[0041] Parallel processing: After receiving the pre-filled information, the language big model module immediately begins to build the query logic for "Tomorrow's Beijing weather" without waiting for the complete voice command to end.

[0042] Thus, through the "listen-and-transmit" mechanism, the large model can begin warm-up inference in the middle and later stages of voice input. Under this mechanism, the average first character generation latency of the system can be reduced by more than 30%, achieving human-like instant response and greatly improving the smoothness of interaction. Secondly, by transmitting the core intent and slot in advance, the large model can use contextual semantics to make real-time corrections and additions to the recognition results, avoiding the deviation in understanding the whole sentence caused by single character recognition errors.

[0043] In some embodiments, the step of pre-passing the recognition results to the language large model module through the intent slot pre-filling mechanism specifically includes:

[0044] During the streaming speech recognition process, the voice interaction module performs local intent inference on the recognized text segments with a preset window granularity, and extracts the core intent tags and key slot candidate values.

[0045] When the confidence level of the local intent inference reaches the preset trigger threshold, the core intent tag and key slot candidate value are encapsulated into a pre-filled context structure, and the pre-filled context structure is transmitted to the language large model module in advance via the asynchronous message bus in a bypass push manner.

[0046] Specifically, during user voice input, the voice interaction module performs real-time analysis on the recognized text segments with a preset window granularity of 500 milliseconds (ms). For example, when the segment "Help me check tomorrow" is recognized, the system immediately starts the local intent inference algorithm, determines the core intent label as "Weather_Query", and extracts the key slot candidate values: Time_Slot = "Tomorrow", Location_Slot = "Beijing".

[0047] The system uses a built-in lightweight intent classification model to score the aforementioned local text fragments in real time. The specific calculation logic is as follows:

[0048] Feature extraction: Extract semantic vector features from text fragments and perform cosine similarity comparison with a pre-defined standard intent library (containing standard intent templates such as "Weather_Query", "Calendar_Set", and "Music_Play");

[0049] Confidence generation: The similarity score obtained from the comparison is used as the base confidence score. At the same time, the system introduces a slot integrity verification factor. (If the number of extracted slots reaches the minimum number of slots required for this purpose, then) =1, otherwise <1);

[0050] Final confidence score calculation: The final confidence score for local intent inference is calculated by combining all relevant data. ;

[0051] In this example, because the intent of "help me check Beijing tomorrow" highly matches the intent of "weather query," and the two key slots "tomorrow" and "Beijing" have been extracted, the system calculates... It quickly reached the preset trigger threshold (e.g., 90%).

[0052] When detected When the value reaches the preset trigger threshold, it is determined that the current intent is clear enough and there is no need to wait for subsequent voice input. At this time, the system encapsulates the extracted core intent label (Weather_Query) and key slot candidate values ​​(Tomorrow, Beijing) into a standardized "pre-filled context structure".

[0053] The pre-filled context structure is not passed through the regular main processing queue, but is directly and preferentially passed through to the language big model module via the asynchronous message bus in a bypass push manner. At this time, the user may still be saying "the weather" or pausing at the end of a sentence, but the big model module has already received the instruction containing complete query elements in advance.

[0054] Thus, through local intent inference and confidence triggering mechanisms, the system breaks the traditional serial limitation of "speak first, then listen." The large model module can obtain the core instruction elements before the user finishes speaking, thereby eliminating the psychological gap of waiting for the system to respond and achieving truly low-latency interaction. Secondly, by using window granularity for local inference, the system can handle changes in the user's speaking speed or occasional slips of the tongue. Even if there are slight deviations between the subsequent recognition results and the pre-filled slots, the large model can self-correct based on the strong intent signals received in advance, avoiding the failure of understanding the entire sentence due to single-point recognition errors. In addition, by using a bypass push method to directly deliver high-priority pre-filled information to the large model, the system avoids these key information from queuing in the regular message queue, ensuring zero-packet-loss transmission of key instructions, reducing the burden on the main processing flow, and improving the stability of the system in high-concurrency scenarios.

[0055] The language model module is communicatively connected to the voice interaction module. It is used to receive text commands and generate target response text and corresponding speech synthesis prosodic annotations based on contextual semantics.

[0056] Specifically, the language big data model module merges the received pre-filled structure with the currently recognized complete speech text ("Could you check the weather in Beijing tomorrow?"). The big data model uses the strong intent signal from the pre-filled structure to quickly pinpoint the domain of the answer and combines it with historical dialogue records (such as the user previously asking "the weather in Shanghai") to generate a logically sound target answer text. An example of the generated text is: "Okay, here's the weather in Beijing for you tomorrow. According to the latest meteorological data, tomorrow Beijing will be mainly sunny turning cloudy, with a high of 25 degrees Celsius and a low of 12 degrees Celsius. The air quality is excellent, making it perfect for travel."

[0057] While generating the target answer text, the large model module calls the built-in prosody prediction model to perform word-by-word prosodic analysis on the generated text:

[0058] Accent marking: Identify key information points such as "Beijing", "sunny to cloudy", and "25 degrees Celsius" and mark them as accents.

[0059] Pause marking: Identify the natural breathing pause after "very suitable for travel" at the end of the sentence, and the logical pause after "according to the latest weather data", and mark them as pause nodes (Pause).

[0060] Emotion and speech rate annotation: Based on the semantics of "sunny turning cloudy" and "suitable for travel", the tone is annotated as "light and cheerful" and a moderate speech rate is set.

[0061] The generated target response text and the prosodic annotations of speech synthesis are packaged into a standard data package and sent to the real-time digital human driving module via an asynchronous message bus. At the same time, the plain text is sent to the speech synthesis (TTS) engine for audio generation.

[0062] In this way, by directly generating prosodic annotations through the large model, the perfect correspondence between semantic emphasis (such as "Beijing" and "25 degrees") and speech stress is ensured, making the digital human's tone more natural and in line with human speaking habits. Secondly, by combining contextual semantics to generate answers, the large model no longer answers the current question in isolation, but can understand the dialogue flow. For example, if the user then asks "What about Shanghai?", the large model can directly reuse the previous "check the weather" intent without the user having to repeat themselves, greatly enhancing the intelligence of the interaction. In addition, the generated prosodic annotations not only serve speech synthesis, but also directly guide the digital human's actions as a script. For example, the "stress" in the annotation corresponds to the digital human's "wide opening of the mouth" and "emphasis on head movement", and the "pause" in the annotation corresponds to the digital human's "silencing" and "eye movement". This end-to-end prosodic transmission is the core foundation for achieving highly immersive interaction.

[0063] The knowledge base module, which is communicatively connected to the language model module, is used to provide domain knowledge retrieval gain and fact verification during the generation of the target answer text. In some embodiments, the knowledge base module adopts a dual-path retrieval architecture, which includes a semantic retrieval path based on vector similarity and a keyword matching path based on inverted index. The retrieval results from the two paths are weighted and integrated by a cross-modal attention fusion layer before being input into the generation process of the language model module.

[0064] Specifically, when the language model module starts generating the answer text for "Tomorrow's Beijing weather," to ensure the accuracy of the information, the system will simultaneously trigger the knowledge base module to perform dual-path retrieval and fusion. The specific process is as follows:

[0065] Semantic retrieval pathway:

[0066] Vectorization: The system first converts the draft text generated by the large model (such as "Tomorrow in Beijing, sunny turning cloudy, 25 degrees Celsius") and the user's original query intent into high-dimensional dense vectors through a pre-trained text encoder (such as BERT or RoBERTa model). These vectors capture the deep semantic information of the text.

[0067] Approximate Nearest Neighbor Search (ANN Search): The generated query vector is input into a vector database (such as Milvus or Faiss). The system uses the HNSW (Hierarchical Navigable Small World) algorithm to perform an approximate nearest neighbor search, quickly finding the Top-K records with the most similar semantics in a massive historical meteorological corpus.

[0068] Results Acquisition: The search results return authoritative meteorological record fragments that are highly semantically related to "Beijing Weather Tomorrow", such as: "Official Forecast: Beijing will be cloudy turning sunny tomorrow, with a high of 24-26 degrees Celsius". This step mainly addresses the issues of "understanding intent" and "obtaining general information".

[0069] Keyword matching pathway:

[0070] Entity Extraction and Index Building: The system uses Named Entity Recognition (NER) technology to accurately extract key entity words from the text, such as "Beijing" (location), "tomorrow" (time), and "temperature" (attribute). These keywords are built into an inverted index and stored in a dedicated keyword database, establishing a mapping relationship between "keywords and document IDs".

[0071] Boolean logic retrieval: When a query is received, the system extracts entity keywords as search terms and performs exact matching in the inverted index. For example, the system executes the logical query: Location="Beijing" AND Time="Tomorrow";

[0072] Results Acquisition: This pathway returns structured data or document fragments containing precise entity matches, such as: "Beijing Meteorological Bureau issues: Air quality will be excellent tomorrow, wind force level 2". This step mainly addresses the issues of "precise positioning" and "acquiring structured facts".

[0073] Cross-modal attention fusion:

[0074] The system inputs the results from the two retrieval paths ("temperature range" from the vector retrieval and "wind speed and air quality" from the inverted index) into the cross-modal attention fusion layer:

[0075] The fusion layer calculates the weights of the two results. For example, for factual data such as "temperature", the semantic search result is given a higher weight; for auxiliary information such as "wind force", the keyword matching result is given a higher weight.

[0076] The final result is an enhanced prompt: "According to the latest authoritative data, tomorrow's weather in Beijing will be cloudy turning sunny, with temperatures ranging from 24 to 26 degrees Celsius, excellent air quality, and a level 2 wind."

[0077] Injection generation process:

[0078] The enhanced prompt words after fusion are injected back into the language big model module in real time during the generation process. Based on this, the big model corrects the inaccurate expression "25 degrees" that may have existed in the original text, and generates the final target answer text that has been verified by facts.

[0079] Thus, by introducing real-time verification from an external knowledge base, the system successfully corrected the potentially ambiguous temperature generated by the large model to a precise "24 to 26 degrees Celsius," and supplemented it with key facts such as "excellent air quality," greatly enhancing the credibility of the digital human's answers. Secondly, the dual-path architecture enables the system to understand the user's ambiguous intent through semantic retrieval (e.g., when a user asks "How's the weather?", the system can retrieve the description "partly cloudy turning sunny"), and to ensure the absolute accuracy of key entities (e.g., place names, times) through keyword matching. This complementary mechanism makes the knowledge base both semantically savvy and factually accurate. Furthermore, through cross-modal attention fusion, the system can intelligently integrate multi-source heterogeneous information (e.g., numerical temperature and descriptive air quality), making the generated answers not only accurate but also informative and logically coherent.

[0080] A real-time digital human driving module, communicatively connected to the language large model module, is used to receive the target response text and the corresponding speech synthesis prosodic annotations, and synchronously generate matching lip movement sequences and body posture instructions to drive the digital human model to perform anthropomorphic display. In some embodiments, the real-time digital human driving module includes a prosodic feature analysis unit and a mapping unit. The prosodic feature analysis unit is used to perform prosodic feature analysis on the target response text and the corresponding speech synthesis prosodic annotations to extract stress positions and pause nodes. The mapping unit is used to map the extracted stress positions and pause nodes to the mouth opening curve and head micro-movement trajectory of the digital human model.

[0081] Specifically, when the real-time digital human driving module receives the target response text ("Tomorrow in Beijing, cloudy turning sunny, temperature 24 to 26 degrees Celsius") and the corresponding speech synthesis prosodic annotations (emphasis on "Beijing" and "24 to 26 degrees Celsius", with a pause at the end of the sentence) from the language large model module, the specific execution process is as follows:

[0082] The prosodic feature analysis unit performs in-depth analysis on the received prosodic annotation data:

[0083] Accent extraction: The system identifies the accented locations in the text. For example, in the time period of "24 to 26 degrees", the system marks high-energy accented nodes.

[0084] Pause extraction: The system identifies long pauses after the word "degree" at the end of a sentence, as well as short pauses at logical transitions within the sentence.

[0085] The mapping unit transforms the extracted abstract prosodic features into specific 3D motion commands:

[0086] Mouth opening curve generation: For stressed syllables of "24 to 26 degrees", the mapping unit generates a curve of "large mouth opening and dramatic lip shape changes"; for sentence-end pauses, a curve of "closed lips and stillness" is generated.

[0087] Head micro-movement trajectory generation: For stressed syllable nodes, the mapping unit generates a trajectory of "head making emphasis movements in accordance with tone"; for long pauses, it generates a micro-movement trajectory of "head returning to center and eyes moving naturally".

[0088] The generated lip-sync sequence and body posture instructions are sent to the digital human rendering engine in real time. The digital human model is driven accordingly to present a human-like display effect where the lips open and close naturally when speaking, the head moves slightly when emphasizing words, and there is a natural breathing sensation at the end of sentences.

[0089] Thus, by directly mapping speech prosody to body movements (lip shapes and head postures), this system completely solves the problems of misaligned lip shapes or stiff movements in traditional digital humans. Each stress in the digital human is accompanied by corresponding head emphasis movements, and each pause is accompanied by natural breathing postures, greatly enhancing the visual realism. Secondly, the micro head movement trajectories make the digital human look like having a real conversation with eye contact and postures with the user, rather than mechanical reading, significantly improving the user's immersion. In addition, by directly using the prosody annotations output by the large model for mapping, the step of the driving module re - performing speech analysis is omitted, further shortening the end - to - end latency from text generation to action display.

[0090] In some embodiments, performing prosody feature analysis on the target answer text and the corresponding speech synthesis prosody annotations, and extracting stress positions and pause nodes specifically includes:

[0091] Performing prosody feature analysis based on the pitch baseline, duration distribution, and energy envelope in the speech synthesis prosody annotations, extracting the stress positions and pause nodes corresponding to the target answer text frame by frame, and outputting the stress positions and pause nodes to the mapping unit in the form of an event sequence with timestamps.

[0092] Specifically, when the real - time digital human driving module receives the target answer text ("Tomorrow, Beijing will be cloudy turning to sunny, with temperatures ranging from 24 to 26 degrees") generated by the language large model module and the corresponding speech synthesis prosody annotation data packet, the prosody feature analysis unit executes the following frame - by - frame analysis process:

[0093] Performing in - depth calculation on three core dimensions in the speech synthesis prosody annotations:

[0094] Pitch Contour analysis: The system reads the pitch curve in the annotation, identifies the upward pitch trend corresponding to the key information segment "24 to 26 degrees", and determines this as the highlight moment of the intonation.

[0095] Duration analysis: The system analyzes the preset duration of each word, identifies that the two digital syllables "24" and "26" are assigned longer duration weights, while the word "to" has a shorter duration.

[0096] Energy Envelope analysis: The system reads the energy curve and finds that the energy value corresponding to the interval "24 to 26 degrees" is significantly higher than that of "Tomorrow, Beijing" at the beginning of the sentence, indicating stronger pronunciation intensity here.

[0097] Frame - by - frame event sequence extraction:

[0098] Based on the above analysis, the system scans frame by frame with millisecond-level precision, generating an event sequence with accurate timestamps:

[0099] T = 1200ms: It is detected that both the pitch and energy reach their peaks simultaneously, marked as the start of a stress event, corresponding to the text "24";

[0100] T = 1350ms: It is detected that the energy drops suddenly and the duration is extremely short, marked as a pause / weak reading event, corresponding to the text "至";

[0101] T = 1500ms: It is detected that the pitch climbs again, marked as the peak of a stress event, corresponding to the text "26";

[0102] T = 1650ms: It is detected that the energy returns to zero and the duration exceeds 200ms, marked as an end-of-sentence pause event, corresponding to the breathing position after the text "度".

[0103] Timestamp sequence output:

[0104] The system encapsulates the above-extracted stress positions and pause nodes into an event sequence containing accurate timestamps (Timestamp) (such as [{Event: Stress, Time: 1200ms}, {Event: Pause, Time: 1650ms}]), and outputs it to the mapping unit in real time.

[0105] In this way, through frame-by-frame extraction and the timestamp sequence, the lip opening and closing of the digital human and the energy burst points of the speech (such as stress) are pixel-level aligned on the time axis, completely eliminating the sense of disharmony where the mouth doesn't move when speaking or the mouth moves after the speech has stopped; secondly, by analyzing the pitch baseline and energy envelope, the system can not only identify where the stress is, but also distinguish the type of stress (whether it is an increase in pitch or an increase in energy), which enables the digital human to express an emphasized tone delicately when saying "24至26度", rather than just opening the mouth mechanically, greatly improving the anthropomorphic degree; in addition, for complex answers containing long sentences, clauses or numerical enumerations, the event sequence based on timestamps can clearly disassemble every tiny pause and stress change, ensuring that the digital human still maintains a natural sense of breathing and rhythm when processing long and difficult sentences.

[0106] In some embodiments, mapping the extracted stress positions and pause nodes to the lip opening and closing degree curve and the micro-movement trajectory of the head of the digital human model specifically includes:

[0107] Based on a pre-constructed prosody-action mapping lookup table, each of the stress positions is mapped to the peak parameter of the lip opening and closing degree of the digital human model at the corresponding time point, and the adjacent peak points are connected by cubic spline interpolation to generate a continuous lip opening and closing degree curve;

[0108] The pause node is mapped to the head posture holding command of the digital human model, and a head micro-movement downward trajectory of a preset amplitude is superimposed at the start of the pause, and a head micro-lifting back trajectory is superimposed at the end of the pause.

[0109] Specifically, the generation of the lip-sync curve for accented syllables:

[0110] Lookup table mapping: The system first queries the pre-built prosody-action mapping lookup table. For the accent event with T=1200ms, the system retrieves the corresponding mouth shape opening peak parameter as the maximum opening (1.0).

[0111] Cubic Spline Interpolation: In order to generate a smooth transition, the system does not use discrete points directly, but uses the cubic spline interpolation method. The system connects T=1200ms (peak) with the adjacent low opening nodes to generate a continuous, smooth and ergonomic mouth opening curve. This makes the process of the digital human's lips moving from closing to opening and back to closing as smooth as silk, without any jagged edges.

[0112] Generation of the head micro-motion trajectory of the pause node:

[0113] Attitude hold command: For a pause event of T=1650ms, the system maps it to a head attitude hold command, that is, locks the head position during this time period to avoid unnecessary shaking;

[0114] Micro-motion trajectory superposition: Initial downward pressure: At the beginning of the pause (T=1650ms), the system superimposes a preset amplitude head micro-motion downward pressure trajectory to simulate the slight nod of confirmation that accompanies the tone of voice after a human finishes speaking; Ending return to center: At the end of the pause (T=1650ms+pause duration), the system superimposes a head slight lift return to center trajectory, so that the head smoothly returns to a natural eye-level state, preparing for the next sentence.

[0115] Thus, by using cubic spline interpolation to connect adjacent peaks, the system ensures that the changes in lip shape during the digital human's pronunciation are continuous and conform to physical laws. This solves the problem of the strong mechanical feel in the speech of digital humans in existing technologies, making the lip movements look very natural and smooth. Secondly, by superimposing the pressure and return trajectories at the pause nodes, the digital human is no longer rigidly staring at the screen. After finishing a sentence, the digital human will make a natural nod or return eye movement. This subtle body language greatly enhances the realism and affinity of the interaction, making users feel like they are talking to a real person. In addition, based on the pre-built mapping table and precise timestamps, the system ensures that the lip opening corresponding to each stressed syllable is precisely timed to the moment of the burst of speech energy, and that every slight head movement is precisely aligned with the pauses in the speech flow, achieving a highly unified immersive experience of audiovisuals.

[0116] A method for intelligent digital human interaction based on multi-module collaboration, specifically including:

[0117] It receives the user's voice commands, performs acoustic feature extraction and preliminary intent determination, and generates text commands.

[0118] Based on text instructions and combined with contextual semantics, a target answer text and corresponding speech synthesis prosodic annotations are generated. In the process of generating the target answer text, retrieval gains and fact verification of domain knowledge are obtained from the knowledge base.

[0119] Based on the target response text and the corresponding speech synthesis prosodic annotation, a matching lip-sync sequence and body posture instructions are generated to drive the digital human model to perform anthropomorphic display.

[0120] A computing device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described intelligent digital human interaction method based on multi-module collaboration.

[0121] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described intelligent digital human interaction method based on multi-module collaboration.

[0122] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An intelligent digital human interaction system based on multi-module collaboration, characterized in that, include: The voice interaction module is used to receive users' voice commands, extract acoustic features and make preliminary determination of intent, and generate text commands; The language model module is communicatively connected to the voice interaction module and is used to receive text commands and generate target response text and corresponding speech synthesis prosodic annotations based on contextual semantics. The knowledge base module, which is communicatively connected to the language model module, is used to provide domain knowledge retrieval gains and fact verification during the generation of the target answer text. The real-time digital human driving module is communicatively connected to the language large model module. It is used to receive the target answer text and the corresponding speech synthesis prosodic annotation, and synchronously generate matching lip movement sequences and body posture instructions to drive the digital human model to perform anthropomorphic display. The voice interaction module, language big model module, knowledge base module, and real-time digital human driving module communicate decoupledly through a preset asynchronous message bus.

2. The intelligent digital human interaction system based on multi-module collaboration according to claim 1, characterized in that, The voice interaction module uses a streaming speech recognition engine to output intermediate recognition results in real time during the user's voice input process, and transmits the recognition results to the language big model module in advance through an intent slot pre-filling mechanism.

3. The intelligent digital human interaction system based on multi-module collaboration according to claim 2, characterized in that, The step of pre-filling the intent slots to transmit the recognition results to the language model module specifically includes: During the streaming speech recognition process, the voice interaction module performs local intent inference on the recognized text segments with a preset window granularity, and extracts the core intent tags and key slot candidate values. When the confidence level of the local intent inference reaches the preset trigger threshold, the core intent tag and key slot candidate value are encapsulated into a pre-filled context structure, and the pre-filled context structure is transmitted to the language large model module in advance via the asynchronous message bus in a bypass push manner.

4. The intelligent digital human interaction system based on multi-module collaboration according to claim 1, characterized in that, The knowledge base module adopts a dual-path retrieval architecture, which includes a semantic retrieval path based on vector similarity and a keyword matching path based on inverted index. The retrieval results from the two paths are weighted and integrated by a cross-modal attention fusion layer before being input into the generation process of the language big model module.

5. The intelligent digital human interaction system based on multi-module collaboration according to claim 1, characterized in that, The real-time digital human driving module includes a prosodic feature analysis unit and a mapping unit. The prosodic feature analysis unit is used to perform prosodic feature analysis on the target answer text and the corresponding speech synthesis prosodic annotation, and extract the stress position and pause node. The mapping unit is used to map the extracted stress position and pause node into the mouth opening curve and head micro-movement trajectory of the digital human model.

6. The intelligent digital human interaction system based on multi-module collaboration according to claim 5, characterized in that, The step of performing prosodic feature analysis on the target answer text and its corresponding speech synthesis prosodic annotation, and extracting stress positions and pause nodes, specifically includes: Based on the pitch baseline, duration distribution, and energy envelope in the prosodic annotation of the speech synthesis, prosodic feature analysis is performed. The stress positions and pause nodes corresponding to the target response text are extracted frame by frame, and the stress positions and pause nodes are output to the mapping unit in the form of a timestamped event sequence.

7. The intelligent digital human interaction system based on multi-module collaboration according to claim 6, characterized in that, The process of mapping the extracted accent positions and pause nodes to the lip opening curve and head micro-movement trajectory of the digital human model specifically includes: Based on a pre-built prosodic-action mapping lookup table, the stress positions are mapped one by one to the peak parameters of mouth opening and closing of the digital human model at the corresponding time points, and adjacent peak points are connected by cubic spline interpolation to generate continuous mouth opening and closing curves. The pause node is mapped to the head posture holding command of the digital human model, and a head micro-movement downward trajectory of a preset amplitude is superimposed at the start of the pause, and a head micro-lifting back trajectory is superimposed at the end of the pause.

8. A method for intelligent digital human interaction based on multi-module collaboration, characterized in that, Specifically, it includes: It receives the user's voice commands, performs acoustic feature extraction and preliminary intent determination, and generates text commands. Based on text instructions and combined with contextual semantics, a target answer text and corresponding speech synthesis prosodic annotations are generated. In the process of generating the target answer text, retrieval gains and fact verification of domain knowledge are obtained from the knowledge base. Based on the target response text and the corresponding speech synthesis prosodic annotation, a matching lip-sync sequence and body posture instructions are generated to drive the digital human model to perform anthropomorphic display.

9. A computing device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the intelligent digital human interaction method based on multi-module collaboration as described in claim 8.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the steps of the intelligent digital human interaction method based on multi-module collaboration as described in claim 8.

Images