Multimodal digital human interaction system and method supporting instant hot update

Abstract

The invention discloses a multi-modal digital human interaction system and method supporting instant hot update, and relates to the technical field of multi-modal digital human interaction.The method comprises the steps that a multi-modal interaction instruction input by a user is received, the multi-modal interaction instruction is analyzed, and semantic description of a target digital human is obtained; synchronously acquiring the voice, the facial micro-expression and the physiological sensing signal of the user to generate an emotional state index; while keeping real-time interaction of a main rendering channel of the current digital person uninterrupted, calling a lightweight generation model in a background channel according to semantic description to construct three-dimensional appearance assets of a target digital person, and establishing a mapping relationship between phonemes corresponding to the target digital person and expression bases; and after monitoring that the assets of the target digital person in the background independent thread are ready, extracting historical voices and the expression weight sequence corresponding to the historical voices from the cross-modal memory cache, and performing expression-driven pre-calculation on the target digital person in combination with the future phoneme sequence of the current voice stream and the mapping relationship.

Description

Technical Field

[0001] This invention relates to the field of multimodal digital human interaction technology, and in particular to a multimodal digital human interaction system and method that supports real-time hot updates. Background Technology

[0002] Multimodal digital human interaction technology refers to a technological system that uses artificial intelligence-driven virtual digital humans as its core, integrating multiple input and output channels such as voice, text, vision (facial expressions, eye contact, body movements), and physiological signals to achieve natural, fluent, and human-like two-way interaction with human users. Therefore, how to utilize advanced technologies to improve the intelligence level and security of multimodal digital human interaction has become one of the most pressing issues to be addressed.

[0003] The existing system requires preloading multiple complete digital human models, which consumes a lot of video memory and bandwidth. When users make temporary requests, it cannot respond immediately due to the lack of corresponding pre-loaded packages, or it may take several seconds to load, which disrupts the smoothness of interaction. Even if the switch is completed, the new digital human lacks a mouth shape driving state that is aligned with the current voice. In the first frame, the mouth often opens / closes suddenly or remains still, causing obvious audio-visual desynchronization, which seriously affects the sense of immersion and credibility. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a multimodal digital human interaction method that supports real-time hot updates. This solves the problems of existing systems requiring the pre-loading of multiple complete digital human models, which consumes a lot of video memory and bandwidth; when users make temporary requests, the system cannot respond in time due to the lack of corresponding pre-stored packages, or it may take several seconds to load, which disrupts the smoothness of the interaction; and even if the switch is completed, the new digital human lacks a mouth shape driving state that is aligned with the current speech, and the mouth often suddenly opens / closes or remains still in the first frame, causing obvious audio-visual asynchrony, which seriously affects the immersion and credibility.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: In a first aspect, the present invention provides a multimodal digital human interaction method supporting real-time hot updates, comprising: It receives multimodal interaction commands from users, parses the multimodal interaction commands to obtain the semantic description of the target digital human, and simultaneously acquires the user's voice, facial micro-expressions and physiological sensor signals to generate an emotional state index. While maintaining uninterrupted real-time interaction in the current main rendering channel of the digital human, a lightweight generation model is called in the background channel to construct the three-dimensional appearance assets of the target digital human based on semantic description, and a mapping relationship between the phonemes and expression bases corresponding to the target digital human is established. After detecting that the target digital human asset in the background independent thread is ready, historical speech and its corresponding expression weight sequence are extracted from the cross-modal memory cache. Combined with the future phoneme sequence of the current speech stream and the mapping relationship, the expression-driven pre-calculation of the target digital human is performed to generate initial expression weights that are continuous with the expression of the last frame of the current digital human. Based on the decay characteristics of the soft landing interpolation function configured according to the emotional state index, an expression transition strategy for interrupting the response is obtained; When a gap in the vertical synchronization signal of the display device is detected, the main rendering channel is switched to the target digital human through atomic operations, and the initial expression weights and expression transition strategies are loaded synchronously. After the switch is completed, the lip-sync consistency is calculated based on the voice and the screen. If the synchronization deviation exceeds the threshold, the mapping relationship is corrected online, and the generated asset parameters and interaction context are updated to the local cache and cross-modal memory cache.

[0007] As a preferred embodiment of the multimodal digital human interaction method supporting real-time hot updates described in this invention, the specific steps are as follows: Based on semantic description, a lightweight generation model is invoked in the background channel to construct the 3D appearance asset of the target digital human, and a mapping relationship is established between the phonemes and facial expression bases corresponding to the target digital human. The semantic description input by the user is fed into a 3D generative model that has been knowledge distilled and compressed. The 3D generative model outputs the vertex coordinate set, triangle index set, texture map, and expression base set of the target digital human. For each phoneme in the standard phoneme set, drive the target digital human to perform lip-sync simulation and record the elicited facial expression weight vector. Organize all phonemes and their corresponding expression basis weight vectors into a mapping table; The vertex coordinate set, triangle face index set, texture map, expression base set and mapping table are encapsulated into the target digital human asset package and uploaded to the graphics processor's video memory; Wherein, the standard phoneme set is assumed to include Each phoneme, the target digital human possesses If there are 1 expression base, then the mapping table is represented as a matrix. , matrix number Travelogue , indicating the first The expression basis weight vector excited by each phoneme.

[0008] As a preferred embodiment of the multimodal digital human interaction method supporting real-time hot updates described in this invention, the steps of extracting historical speech and its corresponding expression weight sequence from the cross-modal memory cache, combining the future phoneme sequence of the current speech stream with the mapping relationship, performing expression-driven pre-calculation on the target digital human, and generating initial expression weights continuous with the expression of the last frame of the current digital human are as follows: The cross-modal memory cache stores the interaction data within the most recent time window using a double-buffered circular queue structure. Each item contains a speech frame, the corresponding phoneme label, and the current digital human's facial expression weight vector. After the target digital human asset package is loaded, the last segment is read from the secondary buffer. Each historical phoneme and its corresponding expression weight; Finding the future in speech synthesis engines One predicted phoneme; By piecing together historical and future phonemes, a complete contextual phoneme sequence is formed; Using mapping tables Convert the complete contextual phoneme sequence into a facial expression weight sequence for the target digital human; The initial target facial weight at the switching moment is calculated using a motion trend alignment algorithm. It satisfies the following formula: ; in, For the target digital human at the switching moment The corresponding facial expression weights are determined by the mapping table. Query the current phoneme to obtain, The weight of the expression from the previous moment. This is the motion continuity coefficient, used to maintain the continuity of facial expression speed.

[0009] As a preferred embodiment of the multimodal digital human interaction method supporting real-time hot updates described in this invention, the specific steps for obtaining an expression transition strategy for interrupting responses by configuring the decay characteristics of the soft landing interpolation function based on the emotion state index are as follows: Collect the user's speech fundamental frequency, energy, and speech rate to construct a speech emotion feature vector; The micro-expression intensity vector is output through the facial action unit detector; If the system is connected to a physiological sensor, the heart rate variability index is acquired; otherwise, the index is set to zero. The emotional state index is obtained by normalizing the voice emotion feature vector, micro-expression intensity vector, and heart rate variability index, and then summing them by weight. The expression is: ; in, For speech emotion feature vectors, Denote its Euclidean norm. For the first Activation intensity of each facial motion unit The total number of action units. As an indicator of heart rate variability, , , Preset weighting coefficients are used to adjust the contribution ratio of different signal sources to emotion judgment; Set the total soft landing time to Define the two control points of the Bessel interpolation function as: Generate a nonlinear decay curve ; When the interruption event is triggered, the weights of any active facial expressions are... Perform interpolation operations; the expression is: ; in, To interrupt the moment of occurrence The weight value of each expression base. It is a smooth function that decreases monotonically over time, and its decay rate is determined by the sentiment state index. Dynamic adjustment.

[0010] As a preferred embodiment of the multimodal digital human interaction method supporting real-time hot updates described in this invention, the steps are as follows: When a gap in the vertical synchronization signal of the display device is detected, the main rendering channel is switched to the target digital human through atomic operations, and the initial expression weights and expression transition strategies are loaded synchronously. After the backend channel completes the loading of the target digital human assets, pre-calculation of expressions, and configuration of expression transition strategies, it sends a ready flag to the atomic-level state synchronization controller. The atomic-level state synchronization controller continuously monitors the vertical synchronization interrupt signal issued by the graphics processor; Within the vertical blank gap, perform a single-instruction atomic swap operation to change the pointer of the digit man instance in the main rendering channel from the current digit man to the target digit man; At the same time, the initial expression weights will be adjusted. Write the status register of the vision-driven engine and the Bezier decay curve A visual processing unit that is bound to the audiovisual decoupling and blocking controller; Reset the time base of the audio synthesis pipeline to ensure that its playback start time is strictly aligned with the rendering time of the first frame of the new character.

[0011] As a preferred embodiment of the multimodal digital human interaction method supporting real-time hot updates described in this invention, the following steps are taken: After the switching is completed, lip-sync consistency is calculated based on voice and image. If the synchronization deviation exceeds a threshold, the mapping relationship is corrected online, and the generated asset parameters and interaction context are updated to the local cache and cross-modal memory cache. Before the switch Within a given time frame, extract the key point locations of the target digital human's lips frame by frame; Force phoneme alignment is performed on the currently playing speech to obtain a time-aligned phoneme sequence; The minimum cumulative distance between the lip trajectory and the reference lip shape template library is calculated using a dynamic time warping algorithm. ; like Greater than the preset threshold Then the phoneme category with the largest identification deviation will be identified. ; This phoneme The corresponding mapping row is fine-tuned using gradient descent, and the updated formula is as follows: ; in, Indicates the corresponding phoneme in the mapping table The expression basis weight vector; The learning rate controls the correction step size; The gradient of the lip shape reconstruction loss function with respect to the expression weights reflects the direction of deviation between the current lip shape and the ideal lip shape; Write the corrected mapping table, the hash digest of the target digital human asset package, and the original semantic description text into the local asset cache; New digital human The time-weighted sequence of facial expressions is appended to the main buffer of the cross-modal memory cache.

[0012] As a preferred embodiment of the multimodal digital human interaction method supporting real-time hot updates described in this invention, the cross-modal memory cache adopts a double-buffered ring structure, with the main buffer used to write the current interaction data in real time and the secondary buffer used to read historical data in the background. Each time a hot update is triggered, the system locks the contents of the secondary buffer for use in pre-calculation of facial expressions, while the primary buffer continues to receive new data. After a hot update is completed, the roles of the primary and secondary buffers are swapped to ensure that historical context reading and real-time data writing do not block each other. Among them, the dual buffering mechanism uses memory barriers and read-write locks to work together to ensure that the contents of the secondary buffer remain stable at any time, while the primary buffer can be written to without locks; When a new hot update request arrives, the system swaps the pointers of the primary and secondary buffers. The original primary buffer becomes the new secondary buffer for the next round of pre-computation, and the original secondary buffer is cleared and then becomes the new primary buffer to receive real-time streaming data.

[0013] Secondly, the present invention provides a multimodal digital human interaction system that supports real-time hot updates, comprising: Multimodal instruction parsing module, semantic-driven asset generation module, cross-modal pre-driven calculation module, emotion-adaptive soft landing module, atomic-level hot-switching execution module, and online synchronous correction module; The multimodal instruction parsing module is used to receive multimodal interaction instructions input by the user, parse them to obtain the semantic description of the target digital human, and simultaneously collect the user's voice signal, facial micro-expression data and physiological sensor signals, and fuse them to generate an emotional state index. The semantically driven asset generation module is used to construct the vertex coordinate set, triangle index set, texture map and expression base set of the target digital human by calling the lightweight 3D generation model in the background channel according to the semantic description, and to establish a mapping table between phonemes and expression base by performing lip-syncing simulation on the standard phoneme set, and to encapsulate it into the target digital human asset package and load it into the graphics processor memory. The cross-modal pre-driven calculation module is used to extract historical speech frames and their corresponding expression weight sequences from the cross-modal memory cache, combine them with the future phoneme sequence output by the speech synthesis engine and the mapping table, perform expression-driven pre-calculation on the target digital human, and generate initial expression weights that are continuous with the expression movement trend of the current digital human's last frame. The emotion-adaptive soft landing module is used to dynamically configure the control points of the Bessel interpolation function according to the emotion state index, generate a nonlinear decay curve, and drive the target digital human's expression weight to smoothly transition to a silent state according to the curve when an interruption event is detected, so as to realize the anthropomorphic visual feedback of emotion perception. The atomic-level hot-switching execution module is used to monitor the vertical synchronization signal of the display device, switch the digital human instance pointer of the main rendering channel to the target digital human through a single instruction atomic operation within the vertical blank gap, load the initial expression weight and the nonlinear decay curve, and reset the time base of the audio synthesis pipeline to align with the audiovisual start time. The online synchronization correction module is used to calculate lip-sync consistency based on the lip key point trajectory and speech phoneme sequence after the switch is completed. If the synchronization deviation exceeds the threshold, gradient descent fine-tuning is performed on the phoneme row with the largest deviation in the mapping table, and the corrected asset parameters and the new interaction context are written to the local asset cache and cross-modal memory cache, respectively.

[0014] Thirdly, the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, it implements any step of the multimodal digital human interaction method supporting real-time hot updates as described in the first aspect of the present invention.

[0015] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the multimodal digital human interaction method supporting real-time hot updates as described in the first aspect of the present invention.

[0016] The beneficial effects of this invention are as follows: Through innovative mechanisms such as semantically driven on-demand asset generation, cross-modal memory-assisted expression pre-driving, emotion-adaptive soft landing strategy, atomic-level vertical synchronization switching, and online lip-sync correction, it achieves truly real-time hot updates for multimodal digital humans in full-duplex interaction scenarios. This not only completely eliminates audio-visual asynchrony caused by pre-loading delays but also ensures continuous facial expressions, human-like interruption responses, and high audio-visual synchronization during image switching. At the same time, the system supports generating new images based on arbitrary semantic descriptions without the need for pre-storing a large amount of assets, effectively reducing storage and distribution costs. Furthermore, it achieves a personalized interactive experience through physiological and behavioral signal perception, enhancing the usability, naturalness, and immersion of digital humans in high-real-time and high-emotional-demand scenarios such as virtual customer service, metaverse social interaction, and intelligent education. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of the multimodal digital human interaction method that supports real-time hot updates in Example 1.

[0019] Figure 2 This is a schematic diagram of the multimodal digital human interaction system that supports real-time hot updates in Example 1. Detailed Implementation

[0020] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0021] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0022] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0023] Example 1, referring to Figure 1 and Figure 2 As one embodiment of the present invention, this embodiment provides a multimodal digital human interaction method that supports real-time hot updates, comprising the following steps: S1. Receive multimodal interaction commands input by the user, parse the multimodal interaction commands to obtain the semantic description of the target digital human, and simultaneously acquire the user's voice, facial micro-expressions and physiological sensor signals to generate an emotional state index.

[0024] Furthermore, automatic speech recognition is performed on the speech portion of the multimodal interaction commands to obtain a text sequence, and the text sequence is input into a large language model to extract the semantic description of the target digital human. Simultaneously, fundamental frequency, energy, and speech rate features are extracted from the audio stream to form a speech emotion feature vector; The system captures a video stream of the user's face using a camera and outputs micro-expression intensity values ​​that include at least AU4 (frowning), AU12 (turning up the corners of the mouth), and AU45 (blinking) using an action unit detection algorithm. If the system is connected to a physiological sensor, it reads the time-domain index of heart rate variability; The voice emotion feature vector, micro-expression intensity value and heart rate variability index are normalized and then summed according to preset weights to generate an emotion state index with values ​​in the range [0,1].

[0025] It should be noted that by integrating speech recognition, semantic extraction from large language models, and collaborative analysis of multi-source emotional signals (speech prosody, facial micro-expression action units, and heart rate variability), this approach not only accurately interprets the user's intention regarding the target digital human's image but also simultaneously constructs a high-dimensional emotional context representation, providing a personalized basis for subsequent interruption responses. This method avoids the problems of rigid image generation and mechanical interactive feedback caused by relying solely on text commands in traditional solutions, effectively improving the system's perception accuracy of user intentions and emotional states, and laying the foundation for achieving human-like, context-adaptive digital human interaction.

[0026] S2. While maintaining uninterrupted real-time interaction in the current main rendering channel of the digital human, the lightweight generation model is called in the background channel to construct the three-dimensional appearance assets of the target digital human based on the semantic description, and the mapping relationship between the phonemes and expression bases corresponding to the target digital human is established.

[0027] Furthermore, the semantic description input by the user is fed into a 3D generative model that has been knowledge distilled and compressed. The 3D generative model outputs the vertex coordinate set, triangle index set, texture map, and expression base set of the target digital human. For each phoneme in the standard phoneme set, drive the target digital human to perform lip-sync simulation and record the elicited facial expression weight vector. Organize all phonemes and their corresponding expression basis weight vectors into a mapping table; The vertex coordinate set, triangle face index set, texture map, expression base set and mapping table are encapsulated into the target digital human asset package and uploaded to the graphics processor's video memory; Wherein, the standard phoneme set is assumed to include Each phoneme, the target digital human possesses If there are 1 expression base, then the mapping table is represented as a matrix. , matrix number Travelogue , indicating the first The expression basis weight vector excited by each phoneme.

[0028] It should be noted that the 3D generative model using knowledge distillation compression significantly reduces computational overhead while ensuring generation quality, enabling asset generation to be completed in milliseconds; it automatically constructs a phoneme-expression base mapping table through lip-sync simulation, eliminating the need for manual annotation or offline calibration, ensuring that any newly generated digital person has native lip-sync capability; it encapsulates geometry, materials, expressions, and mapping relationships into a unified whole and preloads them into video memory, providing a complete and independent rendering unit for seamless switching, fundamentally solving the problem of audio-visual asynchrony caused by resource loading delays or missing mappings.

[0029] S3. After detecting that the target digital human asset in the background independent thread is ready, extract the historical speech and its corresponding expression weight sequence from the cross-modal memory cache, combine the future phoneme sequence of the current speech stream and the mapping relationship, perform expression-driven pre-calculation on the target digital human, and generate initial expression weights that are continuous with the expression of the last frame of the current digital human.

[0030] Furthermore, the cross-modal memory cache stores the interaction data within the most recent time window using a double-buffered circular queue structure. Each item contains a speech frame, the corresponding phoneme label, and the current digital human's facial expression weight vector. After the target digital human asset package is loaded, the last segment is read from the secondary buffer. Each historical phoneme and its corresponding expression weight; Finding the future in speech synthesis engines One predicted phoneme; By piecing together historical and future phonemes, a complete contextual phoneme sequence is formed; Using mapping tables Convert the complete contextual phoneme sequence into a facial expression weight sequence for the target digital human; The initial target facial weight at the switching moment is calculated using a motion trend alignment algorithm. It satisfies the following formula: ; in, For the target digital human at the switching moment The corresponding facial expression weights are determined by the mapping table. Query the current phoneme to obtain, The weight of the expression from the previous moment. This is the motion continuity coefficient, used to maintain the continuity of facial expression speed.

[0031] It should be noted that the introduction of cross-modal memory caching and the combination of historical and future phoneme context for expression pre-driving effectively overcomes the abrupt changes in lip shape in the first frame caused by insufficient instantaneous audio features (such as silence segments and consonant transitions); the initial expression weights generated by the motion trend alignment algorithm... It not only matches the current voice content, but also inherits the facial expression movement speed of the old digital human to achieve visual motion continuity; the double-buffered circular queue structure ensures that the background pre-calculation and the front-end real-time interaction do not block each other, ensuring low latency and high determinism in the hot update process.

[0032] S4. Based on the decay characteristics of the soft landing interpolation function configured according to the emotional state index, an expression transition strategy for interrupting the response is obtained.

[0033] Furthermore, the user's speech fundamental frequency, energy, and speech rate are collected to form a speech emotion feature vector; The micro-expression intensity vector is output through the facial action unit detector; If the system is connected to a physiological sensor, the heart rate variability index is acquired; otherwise, the index is set to zero. The emotional state index is obtained by normalizing the voice emotion feature vector, micro-expression intensity vector, and heart rate variability index, and then summing them by weight. The expression is: ; in, For speech emotion feature vectors, Denote its Euclidean norm. For the first Activation intensity of each facial motion unit The total number of action units. As an indicator of heart rate variability, , , Preset weighting coefficients are used to adjust the contribution ratio of different signal sources to emotion judgment; Set the total soft landing time to Define the two control points of the Bessel interpolation function as: Generate a nonlinear decay curve ; When the interruption event is triggered, the weights of any active facial expressions are... Perform interpolation operations; the expression is: ; in, To interrupt the moment of occurrence The weight value of each expression base. It is a smooth function that decreases monotonically over time, and its decay rate is determined by the sentiment state index. Dynamic adjustment.

[0034] It should be noted that, based on the decay characteristics of the dynamic adjustment soft landing interpolation function of the emotional state index, the interruption response is no longer a fixed animation template, but rather the convergence rate is adaptively adjusted according to the user's current emotional state: quickly calming anxious users to show respect, and gently transitioning calm users to show gentleness; this mechanism effectively reduces the uncanny valley effect caused by abrupt cessation of facial expressions, improves the naturalness of interaction and emotional resonance, and is a key breakthrough for multimodal digital humans to move from functional usability to emotional credibility.

[0035] S5. When a gap in the vertical synchronization signal of the display device is detected, the main rendering channel is switched to the target digital human through atomic operations, and the initial expression weight and expression transition strategy are loaded synchronously.

[0036] Furthermore, after the backend channel completes the loading of the target digital human assets, pre-calculation of expressions, and configuration of expression transition strategies, it sends a ready flag to the atomic-level state synchronization controller. The atomic-level state synchronization controller continuously monitors the vertical synchronization interrupt signal issued by the graphics processor; Within the vertical blank gap, perform a single-instruction atomic swap operation to change the pointer of the digit man instance in the main rendering channel from the current digit man to the target digit man; At the same time, the initial expression weights will be adjusted. Write the status register of the vision-driven engine and the Bezier decay curve A visual processing unit that is bound to the audiovisual decoupling and blocking controller; Reset the time base of the audio synthesis pipeline to ensure that its playback start time is strictly aligned with the rendering time of the first frame of the new character.

[0037] It should be noted that atomic pointer swapping is performed using vertical synchronization gaps to ensure that the main rendering channel switching occurs at the frame boundary, completely avoiding screen tearing or stuttering; initial facial expression weights and facial expression transition strategies are injected synchronously, and the audio time base is reset to achieve hardware-level alignment at the start of audiovisual events; this operation is completed within a single CPU cycle, and the switching process is completely transparent to the user, truly achieving the zero-perceptible latency and seamless experience required for instant hot updates.

[0038] S6. After the handover is completed, the lip-sync consistency is calculated based on the voice and screen. If the synchronization deviation exceeds the threshold, the mapping relationship is corrected online, and the generated asset parameters and interaction context are updated to the local cache and cross-modal memory cache.

[0039] Furthermore, before the switch Within a given time frame, extract the key point locations of the target digital human's lips frame by frame; Force phoneme alignment is performed on the currently playing speech to obtain a time-aligned phoneme sequence; The minimum cumulative distance between the lip trajectory and the reference lip shape template library is calculated using a dynamic time warping algorithm. ; like Greater than the preset threshold Then the phoneme category with the largest identification deviation will be identified. ; This phoneme The corresponding mapping row is fine-tuned using gradient descent, and the updated formula is as follows: ; in, Indicates the corresponding phoneme in the mapping table The expression basis weight vector; The learning rate controls the correction step size; The gradient of the lip shape reconstruction loss function with respect to the expression weights reflects the direction of deviation between the current lip shape and the ideal lip shape; Write the corrected mapping table, the hash digest of the target digital human asset package, and the original semantic description text into the local asset cache; New digital human The time-weighted expression sequence is appended to the main buffer of the cross-modal memory cache; The cross-modal memory cache adopts a dual-buffered ring structure, with the main buffer used to write current interaction data in real time and the secondary buffer used to read historical data in the background. Each time a hot update is triggered, the system locks the contents of the secondary buffer for use in pre-calculation of facial expressions, while the primary buffer continues to receive new data. After a hot update is completed, the roles of the primary and secondary buffers are swapped to ensure that historical context reading and real-time data writing do not block each other. Among them, the dual buffering mechanism uses memory barriers and read-write locks to work together to ensure that the contents of the secondary buffer remain stable at any time, while the primary buffer can be written to without locks; When a new hot update request arrives, the system swaps the pointers of the primary and secondary buffers. The original primary buffer becomes the new secondary buffer for the next round of pre-computation, and the original secondary buffer is cleared and then becomes the new primary buffer to receive real-time streaming data.

[0040] It should be noted that by fine-tuning the mapping relationship through online lip-sync evaluation and gradient-driven optimization, the system has the ability to continuously optimize model accuracy and overcome long-term synchronization degradation caused by generalization error of the generated model or phoneme mapping deviation. The correction results and interaction context are written back to the local cache and cross-modal memory cache to form a closed-loop mechanism of generation-verification-learning-reuse. With the dual-buffered ring cache architecture, the system can ensure the stability of historical data reading and support high-throughput real-time writing, providing efficient and reliable context management support for high-frequency hot update scenarios.

[0041] This embodiment also provides a multimodal digital human interaction system that supports real-time hot updates, including: Multimodal instruction parsing module, semantic-driven asset generation module, cross-modal pre-driven calculation module, emotion-adaptive soft landing module, atomic-level hot-switching execution module, and online synchronous correction module; The multimodal instruction parsing module is used to receive multimodal interaction instructions input by the user, parse them to obtain the semantic description of the target digital human, and simultaneously collect the user's voice signal, facial micro-expression data and physiological sensor signals, and fuse them to generate an emotional state index. The semantic-driven asset generation module is used to construct the vertex coordinate set, triangle index set, texture map and expression base set of the target digital human by calling the lightweight 3D generation model in the background channel according to the semantic description. It also establishes a mapping table between phonemes and expression base by performing lip-syncing simulation on the standard phoneme set, and encapsulates it into the target digital human asset package and loads it into the graphics processor memory. The cross-modal pre-driven calculation module is used to extract historical speech frames and their corresponding expression weight sequences from the cross-modal memory cache, and combine them with the future phoneme sequence and mapping table output by the speech synthesis engine to perform expression-driven pre-calculation on the target digital human, generating initial expression weights that are continuous with the expression movement trend of the current digital human's last frame. The emotion-adaptive soft landing module is used to dynamically configure the control points of the Bezier interpolation function based on the emotion state index, generate a non-linear decay curve, and drive the target digital human's expression weight to smoothly transition to a silent state according to the curve when an interruption event is detected, so as to realize the anthropomorphic visual feedback of emotion perception. The atomic-level hot-switching execution module is used to monitor the vertical synchronization signal of the display device. Within the vertical blank gap, it switches the pointer of the digital human instance in the main rendering channel to the target digital human through a single instruction atomic operation. At the same time, it loads the initial expression weights and non-linear decay curves, and resets the time base of the audio synthesis pipeline to align with the audiovisual start time. The online synchronization correction module is used to calculate lip-sync consistency based on the lip key point trajectory and speech phoneme sequence after the switch is completed. If the synchronization deviation exceeds the threshold, gradient descent fine-tuning is performed on the phoneme row with the largest deviation in the mapping table, and the corrected asset parameters and the new interaction context are written to the local asset cache and cross-modal memory cache, respectively.

[0042] This embodiment also provides a computer device suitable for supporting a multimodal digital human interaction method that supports real-time hot updates, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the multimodal digital human interaction method that supports real-time hot updates as proposed in the above embodiment.

[0043] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0044] This embodiment also provides a storage medium storing a computer program that, when executed by a processor, implements the multimodal digital human interaction method supporting real-time hot updates as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0045] In summary, this invention achieves truly real-time hot updates for multimodal digital humans in full-duplex interaction scenarios through innovative mechanisms such as semantically driven on-demand asset generation, cross-modal memory-assisted facial expression pre-driving, emotion-adaptive soft-landing strategy, atomic-level vertical synchronization switching, and online lip-syncing. This not only completely eliminates audio-visual asynchrony caused by pre-loading delays but also ensures continuous facial expressions, human-like interruption responses, and high audio-visual synchronization during image switching. Furthermore, the system supports generating new images based on arbitrary semantic descriptions without the need for pre-storing large amounts of assets, effectively reducing storage and distribution costs. It also achieves personalized interactive experiences through physiological and behavioral signal perception, enhancing the usability, naturalness, and immersion of digital humans in high-real-time and high-emotional-demand scenarios such as virtual customer service, metaverse social interaction, and intelligent education.

[0046] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A multimodal digital human interaction method supporting real-time hot updates, characterized in that: include: It receives multimodal interaction commands from users, parses the multimodal interaction commands to obtain the semantic description of the target digital human, and simultaneously acquires the user's voice, facial micro-expressions and physiological sensor signals to generate an emotional state index. While maintaining uninterrupted real-time interaction in the current main rendering channel of the digital human, a lightweight generation model is called in the background channel to construct the three-dimensional appearance assets of the target digital human based on semantic description, and a mapping relationship between the phonemes and expression bases corresponding to the target digital human is established. After detecting that the target digital human asset in the background independent thread is ready, historical speech and its corresponding expression weight sequence are extracted from the cross-modal memory cache. Combined with the future phoneme sequence of the current speech stream and the mapping relationship, the expression-driven pre-calculation of the target digital human is performed to generate initial expression weights that are continuous with the expression of the last frame of the current digital human. Based on the decay characteristics of the soft landing interpolation function configured according to the emotional state index, an expression transition strategy for interrupting the response is obtained; When a gap in the vertical synchronization signal of the display device is detected, the main rendering channel is switched to the target digital human through atomic operations, and the initial expression weights and expression transition strategies are loaded synchronously. After the switch is completed, the lip-sync consistency is calculated based on voice and screen. If the synchronization deviation exceeds the threshold, the mapping relationship is corrected online, and the generated asset parameters and interaction context are updated to the local cache and cross-modal memory cache.

2. The multimodal digital human interaction method supporting real-time hot updates as described in claim 1, characterized in that: The steps involve constructing the 3D appearance asset of the target digital human by calling a lightweight generation model in the background channel based on semantic description, and establishing a mapping relationship between the phonemes and expression bases corresponding to the target digital human. The semantic description input by the user is fed into a 3D generative model that has been knowledge distilled and compressed. The 3D generative model outputs the vertex coordinate set, triangle index set, texture map, and expression base set of the target digital human. For each phoneme in the standard phoneme set, drive the target digital human to perform lip-sync simulation and record the elicited facial expression weight vector. Organize all phonemes and their corresponding expression basis weight vectors into a mapping table; The vertex coordinate set, triangle face index set, texture map, expression base set and mapping table are encapsulated into the target digital human asset package and uploaded to the graphics processor's video memory; Wherein, the standard phoneme set is assumed to include Each phoneme, the target digital human possesses If there are 1 expression base, then the mapping table is represented as a matrix. , matrix number Travelogue , indicating the first The expression basis weight vector excited by each phoneme.

3. The multimodal digital human interaction method supporting real-time hot updates as described in claim 2, characterized in that: The steps are as follows: extracting historical speech and its corresponding expression weight sequence from the cross-modal memory cache, combining the future phoneme sequence of the current speech stream with the mapping relationship, performing expression-driven pre-calculation on the target digital human, and generating initial expression weights that are continuous with the expression of the last frame of the current digital human. The cross-modal memory cache stores the interaction data within the most recent time window using a double-buffered circular queue structure. Each item contains a speech frame, the corresponding phoneme label, and the current digital human's facial expression weight vector. After the target digital human asset package is loaded, the last segment is read from the secondary buffer. Each historical phoneme and its corresponding expression weight; Finding the future in speech synthesis engines One predicted phoneme; By piecing together historical and future phonemes, a complete contextual phoneme sequence is formed; Using mapping table Convert the complete contextual phoneme sequence into a facial expression weight sequence for the target digital human; The initial target facial weight at the switching moment is calculated using a motion trend alignment algorithm. It satisfies the following formula: ; in, For the target digital human at the switching moment The corresponding facial expression weights are determined by the mapping table. Query the current phoneme to obtain, The weight of the expression from the previous moment. This is the motion continuity coefficient, used to maintain the continuity of facial expression speed.

4. The multimodal digital human interaction method supporting real-time hot updates as described in claim 3, characterized in that: The decay characteristics of the soft landing interpolation function are configured based on the emotional state index to obtain the facial expression transition strategy for interrupting the response. The specific steps are as follows: Collect the user's speech fundamental frequency, energy, and speech rate to construct a speech emotion feature vector; The micro-expression intensity vector is output through the facial action unit detector; If the system is connected to a physiological sensor, the heart rate variability index is acquired; otherwise, the index is set to zero. The emotional state index is obtained by normalizing the voice emotion feature vector, micro-expression intensity vector, and heart rate variability index, and then summing them by weight. The expression is: ; in, For speech emotion feature vectors, Denote its Euclidean norm. For the first Activation intensity of each facial motion unit The total number of action units. As an indicator of heart rate variability, , , Preset weighting coefficients are used to adjust the contribution ratio of different signal sources to emotion judgment; Set the total soft landing time to Define the two control points of the Bessel interpolation function as: Generate nonlinear decay curves ; When the interruption event is triggered, the weights of any active facial expressions are... Perform interpolation operations; the expression is: ; in, To interrupt the moment of occurrence The weight value of each expression base. It is a smooth function that decreases monotonically over time, and its decay rate is determined by the sentiment state index. Dynamic adjustment.

5. The multimodal digital human interaction method supporting real-time hot updates as described in claim 4, characterized in that: When a gap in the vertical synchronization signal of the display device is detected, the main rendering channel is switched to the target digital human through atomic operations, and the initial expression weights and expression transition strategies are loaded synchronously. The specific steps are as follows: After the backend channel completes the loading of the target digital human assets, pre-calculation of expressions, and configuration of expression transition strategies, it sends a ready flag to the atomic-level state synchronization controller. The atomic-level state synchronization controller continuously monitors the vertical synchronization interrupt signal issued by the graphics processor; Within the vertical blank gap, perform a single-instruction atomic swap operation to change the pointer of the digit man instance in the main rendering channel from the current digit man to the target digit man; At the same time, the initial expression weights will be adjusted. Write the status register of the vision-driven engine and the Bezier decay curve A visual processing unit that is bound to the audiovisual decoupling and blocking controller; Reset the time base of the audio synthesis pipeline to ensure that its playback start time is strictly aligned with the rendering time of the first frame of the new character.

6. The multimodal digital human interaction method supporting real-time hot updates as described in claim 5, characterized in that: After the switch is completed, lip-sync consistency is calculated based on voice and image. If the synchronization deviation exceeds a threshold, the mapping relationship is corrected online, and the generated asset parameters and interaction context are updated to the local cache and cross-modal memory cache. The specific steps are as follows: Before switching Within a given time frame, extract the key point locations of the target digital human's lips frame by frame; Force phoneme alignment is performed on the currently playing speech to obtain a time-aligned phoneme sequence; The minimum cumulative distance between the lip trajectory and the reference lip shape template library is calculated using a dynamic time warping algorithm. ; like Greater than the preset threshold Then the phoneme category with the largest identification deviation will be identified. ; This phoneme The corresponding mapping row is fine-tuned using gradient descent, and the updated formula is as follows: ； in, Indicates the corresponding phoneme in the mapping table The expression basis weight vector; The learning rate controls the correction step size; The gradient of the lip shape reconstruction loss function with respect to the expression weights reflects the direction of deviation between the current lip shape and the ideal lip shape; Write the corrected mapping table, the hash digest of the target digital human asset package, and the original semantic description text into the local asset cache; New digital human The time-weighted sequence of facial expressions is appended to the main buffer of the cross-modal memory cache.

7. The multimodal digital human interaction method supporting real-time hot updates as described in claim 6, characterized in that: The cross-modal memory cache adopts a dual-buffered ring structure, with the main buffer used to write current interaction data in real time and the secondary buffer used to read historical data in the background. Each time a hot update is triggered, the system locks the contents of the secondary buffer for use in pre-calculation of facial expressions, while the main buffer continues to receive new data. After a hot update is completed, the roles of the primary and secondary buffers are swapped to ensure that historical context reading and real-time data writing do not block each other. Among them, the dual buffering mechanism uses memory barriers and read-write locks to work together to ensure that the contents of the secondary buffer remain stable at any time, while the primary buffer can be written to without locks; When a new hot update request arrives, the system swaps the pointers of the primary and secondary buffers. The original primary buffer becomes the new secondary buffer for the next round of pre-computation, and the original secondary buffer is cleared and then becomes the new primary buffer to receive real-time streaming data.

8. A multimodal digital human interaction system supporting real-time hot updates, based on the multimodal digital human interaction method supporting real-time hot updates as described in any one of claims 1 to 7, characterized in that: include: Multimodal instruction parsing module, semantic-driven asset generation module, cross-modal pre-driven calculation module, emotion-adaptive soft landing module, atomic-level hot-switching execution module, and online synchronous correction module; The multimodal instruction parsing module is used to receive multimodal interaction instructions input by the user, parse them to obtain the semantic description of the target digital human, and simultaneously collect the user's voice signal, facial micro-expression data and physiological sensor signals, and fuse them to generate an emotional state index. The semantically driven asset generation module is used to construct the vertex coordinate set, triangle index set, texture map and expression base set of the target digital human by calling the lightweight 3D generation model in the background channel according to the semantic description, and to establish a mapping table between phonemes and expression base by performing lip-syncing simulation on the standard phoneme set, and to encapsulate it into the target digital human asset package and load it into the graphics processor memory. The cross-modal pre-driven calculation module is used to extract historical speech frames and their corresponding expression weight sequences from the cross-modal memory cache, combine them with the future phoneme sequence output by the speech synthesis engine and the mapping table, perform expression-driven pre-calculation on the target digital human, and generate initial expression weights that are continuous with the expression movement trend of the current digital human's last frame. The emotion-adaptive soft landing module is used to dynamically configure the control points of the Bessel interpolation function according to the emotion state index, generate a nonlinear decay curve, and drive the target digital human's expression weight to smoothly transition to a silent state according to the curve when an interruption event is detected, so as to realize the anthropomorphic visual feedback of emotion perception. The atomic-level hot-switching execution module is used to monitor the vertical synchronization signal of the display device, switch the pointer of the digital human instance in the main rendering channel to the target digital human through a single instruction atomic operation within the vertical blank gap, load the initial expression weight and the nonlinear decay curve, and reset the time base of the audio synthesis pipeline to align with the audiovisual start time. The online synchronization correction module is used to calculate lip-sync consistency based on the lip key point trajectory and speech phoneme sequence after the switch is completed. If the synchronization deviation exceeds the threshold, gradient descent fine-tuning is performed on the phoneme row with the largest deviation in the mapping table, and the corrected asset parameters and the new interaction context are written to the local asset cache and cross-modal memory cache, respectively.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the multimodal digital human interaction method supporting real-time hot updates as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the multimodal digital human interaction method supporting real-time hot updates as described in any one of claims 1 to 7.

Images