Digital human live broadcast interaction processing method and device fusing large language model, equipment and medium

Abstract

This invention discloses a method, apparatus, device, and medium for digital human live streaming interactive processing that integrates a large language model. It relates to the technical field of digital human live streaming interactive processing, including multimodal data acquisition and preprocessing steps, sentiment analysis and semantic understanding steps, response content generation and personalization enhancement steps, and multimodal interactive output steps. By constructing a multimodal data acquisition and preprocessing mechanism, it integrates input information from three modalities: voice, text, and video / image, achieving comprehensive perception of the audience's interactive intent in the live streaming scenario. Based on this, it utilizes a large language model for deep semantic understanding and contextual awareness, and combines a multimodal sentiment analysis module to comprehensively infer the audience's tone, speech rate, facial micro-expressions, and bullet screen text, enabling the digital human to accurately capture the audience's fine-grained emotional state changes, thereby generating highly human-like, context-appropriate response content.

Description

Technical Field

[0001] This invention relates to the technical field of digital human live streaming interactive processing, specifically to digital human live streaming interactive processing methods, apparatus, devices, and media that integrate large language models. Background Technology

[0002] With the rapid development of the live streaming industry and the continuous advancement of artificial intelligence technology, the application of digital human anchors in live streaming scenarios is becoming increasingly widespread. Digital human live streaming interaction systems typically involve the collection and processing of multimodal information, including audience voice input, bullet screen text, and video image information. During the interaction process, the system needs to comprehensively analyze this multimodal data, understand the audience's semantic intent and emotional state, and generate corresponding responses. Ultimately, real-time interaction between the digital human and the audience is achieved through speech synthesis and motion-driven technologies. In recent years, large language models have demonstrated powerful capabilities in natural language understanding and generation, enabling deep semantic parsing and anthropomorphic text generation based on context. Simultaneously, multimodal sentiment analysis technology, by integrating features from multiple modalities such as voice tone, facial expressions, and text semantics, can more comprehensively infer the user's true emotional state. In the field of motion generation, generative models such as diffusion models can generate digital human facial micro-expressions and body postures in real time based on text descriptions or sentiment tags, making the nonverbal behavior of digital humans more vivid and natural.

[0003] However, existing digital human live streaming interaction systems generally have a flaw: they lack the ability to identify and remember the identities of viewers across sessions. This makes it impossible for digital humans to actively identify viewers they have interacted with before, or to call up historical interaction records to make personalized responses. As a result, each interaction is in a "strange" state, and it is impossible to establish a personalized interactive experience with continuity and relationship depth. Summary of the Invention

[0004] The purpose of this invention is to provide a digital human live streaming interactive processing method, apparatus, device and medium that integrates a large language model, so as to solve the problem that the existing technology lacks the ability to identify and remember the audience's identity across sessions, and each interaction is in a "strange" state, making it impossible to establish a personalized interactive experience with continuity and relationship depth.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a digital human live interactive processing method integrating a large language model, comprising the following steps:

[0006] S1: Multimodal data acquisition and preprocessing steps, including:

[0007] S11: Collect multimodal input data in the live streaming scenario, including audience voice, bullet screen text, and live streaming video images;

[0008] S12: Preprocess the multimodal input data to extract speech feature vectors respectively. Text feature vectors and video image feature vectors ;

[0009] S2: Sentiment analysis and semantic understanding steps, using a large language model to perform deep semantic understanding and context awareness on the preprocessed multimodal features, and combining the sentiment analysis module to identify the audience's emotional state, generating semantic understanding results containing sentiment tags;

[0010] S3: Response content generation and personalization enhancement steps, including:

[0011] S31: Based on the semantic understanding results and the audience's emotional state, generate anthropomorphic response text;

[0012] S32: Introduce a long-term memory module across sessions to retrieve the current audience's historical interaction records, and use the retrieval results to personalize and enhance the response text, generating personalized response text;

[0013] S4: Multimodal interactive output step, the personalized response text is generated into synchronous speech through a speech synthesis model, and control signals that drive the lip movements, facial micro-expressions and body movements of the digital human are generated through an action generation model to achieve multimodal interactive output with synchronized audio and video.

[0014] In step S3, the long-term memory module, when retrieving historical interaction records, establishes an association between the current viewer and historical interaction records based on the viewer identity information identified by the video image features collected in step S1, so as to achieve personalized interaction across sessions.

[0015] Furthermore, the sentiment analysis module in step S2 further includes:

[0016] S21: Extract intonation features from the audience's speech. and speech rate characteristics ;

[0017] S22: Combining the live stream video images collected in step S1, extract the facial micro-expression features of the viewers. ;

[0018] S23: Perform multimodal sentiment fusion of the tone features, speech rate features, facial micro-expression features, and sentiment features of the bullet screen text, and generate a fine-grained comprehensive sentiment inference result according to the following formula. As the emotional state of the audience:

[0019]

[0020] in, This represents the sentiment feature vector of the bullet screen text. , , , The weight matrix is ​​a learnable matrix. For bias terms, This is the activation function.

[0021] Furthermore, in step S32, the long-term memory module retrieves historical interaction records using the following similarity matching formula:

[0022]

[0023] in, The video image feature vector of the current viewer. An identity vector is embedded for each historical viewer. When the similarity exceeds a preset threshold τ, the viewer is identified as the same viewer, and their historical interaction records are used to generate personalized response text.

[0024] Furthermore, the action generation model in step S4 adopts a diffusion model, which generates a digital human's facial micro-expression sequence and body posture sequence in real time based on the semantic content of the personalized response text and the subtle changes in the audience's emotional state identified in step S2; the conditional generation process of the diffusion model is expressed as follows:

[0025]

[0026] in, Let k be the number of diffusion steps, k = 1, 2, ..., K. Let be the action pose vector after the k-th step diffusion. For the first The action posture vector of the step. For conditional control signals, Indicates a Gaussian distribution. and They are respectively from parameters A defined mean function and variance function;

[0027] The condition control signal Semantic embedding vector containing the personalized response text The fusion features with the audience's emotional state ,Right now , where Concat(x) represents the vector concatenation operation.

[0028] Furthermore, in step S23, a fine-grained comprehensive sentiment inference result is generated. Then, this result is used as a condition input into the action generation model in step S4 to dynamically adjust the generation intensity and type of digital human facial micro-expressions; wherein, the micro-expression intensity adjustment factor Calculate using the following formula:

[0029]

[0030] in Represents a vector Perform a transpose operation. For learnable weight vectors, For bias terms;

[0031] The generated micro-expression control signals are represented as :

[0032]

[0033] in, These are the basic micro-expression parameters for semantic generation. These are neutral expression parameters.

[0034] Furthermore, the live stream video images collected in step S1 include images from the broadcaster's side and images from the audience's side; the sentiment analysis module in step S2 combines the broadcaster's facial expression features from the broadcaster's side image. Audience facial expression features in the audience-side image A two-way emotional state analysis was conducted, and the audience's emotional state was corrected according to the following formula:

[0035]

[0036] in, To correct the audience's emotional state, For adjustment coefficients, This is the interaction weight matrix. This represents the vector concatenation operation. tanh(x) is the hyperbolic tangent activation function, used to map interactive features to the (−1,1) interval to control the correction magnitude.

[0037] Furthermore, step S4 also includes:

[0038] S41: Detect the timing deviation between the synchronized speech generated by the speech synthesis model and the control signal generated by the action generation model. ;

[0039] S42: Based on the timing deviation, the control signal is dynamically interpolated and adjusted to achieve fine calibration of audio-visual synchronization; the dynamic interpolation adjustment is expressed as:

[0040]

[0041] in, The original motion control signal, For smooth interpolation coefficients, This is the calibrated motion control signal.

[0042] A digital human live-streaming interactive processing device integrating a large language model, comprising the following steps for implementing the above method:

[0043] The multimodal acquisition and preprocessing module includes a voice acquisition unit, a text acquisition unit, a video acquisition unit, and a feature extraction unit. It is used to acquire multimodal input data in live streaming scenarios and extract voice feature vectors, text feature vectors, and video image feature vectors respectively.

[0044] The sentiment analysis and semantic understanding module includes a large language model inference unit and a sentiment analysis unit. The large language model inference unit is used to perform deep semantic understanding and context awareness on the preprocessed multimodal features. The sentiment analysis unit is used to identify the audience's emotional state by combining multimodal features and generate semantic understanding results containing sentiment tags.

[0045] The response generation and personalization enhancement module includes a response text generation unit and a long-term memory unit. The response text generation unit is used to generate anthropomorphic response text based on the semantic understanding results and the audience's emotional state. The long-term memory unit is used to retrieve the current audience's historical interaction records and use the retrieval results to personalize and enhance the response text, generating personalized response text.

[0046] The multimodal interactive output module includes a speech synthesis unit and an action generation unit. The speech synthesis unit is used to generate synchronized speech from the personalized response text, and the action generation unit is used to generate control signals to drive the lip movements, facial micro-expressions and body movements of the digital human, so as to realize multimodal interactive output with synchronized audio and video.

[0047] When retrieving historical interaction records, the long-term memory unit establishes a connection between the current viewer and historical interaction records based on the viewer identity information identified by the video image features extracted by the multimodal acquisition and preprocessing module, so as to realize personalized interaction across sessions.

[0048] A digital human live interactive processing device integrating a large language model includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the above method.

[0049] A digital human live interactive processing medium that integrates a large language model, wherein the computer program, when executed by a processor, implements the steps of the above method.

[0050] Compared with existing technologies, this invention constructs a multimodal data acquisition and preprocessing mechanism, integrating input information from three modalities: voice, text, and video / image, to achieve comprehensive perception of audience interaction intentions in live streaming scenarios. Based on this, it utilizes a large language model for deep semantic understanding and contextual awareness, and combines a multimodal sentiment analysis module to comprehensively infer the audience's tone, speaking speed, facial micro-expressions, and bullet screen text. This enables the digital human to accurately capture fine-grained changes in the audience's emotional state, thereby generating highly human-like, context-appropriate responses.

[0051] By introducing a long-term memory module across sessions and establishing a link between the current viewer and historical interaction records based on viewer identity information identified from video image features, the digital human achieves seamless identity recognition and personalized memory retrieval for specific viewers. When the same viewer re-enters the live stream, the digital human can proactively identify their identity, retrieve historical interaction records to generate personalized greetings or recommended content, ensuring continuity and depth of interaction. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0053] Figure 1 This is a schematic diagram of the overall process provided for an embodiment of the present invention. Detailed Implementation

[0054] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0055] As attached Figure 1 As shown:

[0056] Example 1:

[0057] This invention provides a method for processing live interactive digital human data using a fusion of large language models, comprising the following steps:

[0058] S1: Multimodal data acquisition and preprocessing steps, including:

[0059] S11: Collect multimodal input data in the live streaming scenario. The multimodal input data includes audience voice, bullet screen text, and live streaming video images.

[0060] S12: Preprocess the multimodal input data and extract speech feature vectors respectively. Text feature vectors and video image feature vectors ;

[0061] S2: Sentiment analysis and semantic understanding steps, using a large language model to perform deep semantic understanding and context awareness on the preprocessed multimodal features, and combining the sentiment analysis module to identify the audience's emotional state, generating semantic understanding results containing sentiment tags;

[0062] S3: Response content generation and personalization enhancement steps, including:

[0063] S31: Generate anthropomorphic response text based on semantic understanding results and audience emotional state;

[0064] S32: Introduce a long-term memory module across sessions to retrieve the current audience's historical interaction records and use the retrieval results to personalize and enhance the response text, generating personalized response text;

[0065] S4: Multimodal interactive output step, which generates synchronized speech from personalized response text through a speech synthesis model, and generates control signals to drive the lip movements, facial micro-expressions and body movements of the digital human through an action generation model, so as to realize multimodal interactive output with synchronized audio and video.

[0066] In step S3, the long-term memory module, when retrieving historical interaction records, establishes a connection between the current viewer and historical interaction records based on the viewer identity information identified by the video image features collected in step S1, so as to achieve personalized interaction across sessions.

[0067] The sentiment analysis module in step S2 further includes:

[0068] S21: Extract intonation features from the audience's speech and speech rate characteristics ;

[0069] S22: Combine the live stream video images collected in step S1 to extract the facial micro-expression features of the viewers. ;

[0070] S23: Multimodal sentiment fusion is performed by integrating intonation features, speech rate features, facial micro-expression features, and sentiment features of the bullet screen text, generating fine-grained comprehensive sentiment inference results according to the following formula. As for the audience's emotional state:

[0071]

[0072] in, This represents the sentiment feature vector of the bullet screen text. , , , The weight matrix is ​​a learnable matrix. For bias terms, For activation functions;

[0073] The expanded form is,

[0074] .

[0075] Furthermore, in step S32, the long-term memory module retrieves historical interaction records using the following similarity matching formula:

[0076]

[0077] Right now,

[0078]

[0079] in, The video image feature vector of the current viewer. An identity vector is embedded for each historical viewer. When the similarity exceeds a preset threshold τ, the viewer is identified as the same viewer, and their historical interaction records are used to generate personalized response text.

[0080] Specifically, the motion generation model in step S4 employs a diffusion model, generating real-time sequences of facial micro-expressions and body postures of the digital human based on the semantic content of the personalized response text and subtle changes in the audience's emotional state identified in step S2. The conditional generation process of the diffusion model is represented as follows:

[0081]

[0082] in, Let k be the number of diffusion steps, k = 1, 2, ..., K. Let be the action pose vector after the k-th step diffusion. For the first The action posture vector of the step. For conditional control signals, Indicates a Gaussian distribution. and They are respectively from parameters A defined mean function and variance function;

[0083] Conditional control signals Semantic embedding vectors containing personalized response text Features of integration with the audience's emotional state ,Right now , where Concat(x) represents the vector concatenation operation.

[0084] It should be noted that step S23 generates fine-grained comprehensive sentiment inference results. Then, this result is used as a condition input into the action generation model in step S4 to dynamically adjust the generation intensity and type of digital human facial micro-expressions; among which, the micro-expression intensity adjustment factor Calculate using the following formula:

[0085]

[0086] in Represents a vector Perform a transpose operation. For learnable weight vectors, For bias terms;

[0087] The generated micro-expression control signals are represented as :

[0088]

[0089] Right now,

[0090]

[0091] in, These are the basic micro-expression parameters for semantic generation. These are neutral expression parameters.

[0092] Furthermore, the live stream video images collected in step S1 include images from the broadcaster's side and images from the audience's side; the sentiment analysis module in step S2 combines the broadcaster's facial expression features from the broadcaster's side image. Audience facial features compared to audience-side images A two-way emotional state analysis was conducted, and the audience's emotional state was corrected using the following formula:

[0093]

[0094] in, To correct the audience's emotional state, For adjustment coefficients, This is the interaction weight matrix. The vector concatenation operation is represented by tanh(x), which is the hyperbolic tangent activation function used to map interactive features to the (−1,1) interval to control the correction magnitude.

[0095] Furthermore, step S4 also includes:

[0096] S41: Detect the timing deviation between the synchronized speech generated by the speech synthesis model and the control signal generated by the action generation model. ;

[0097] S42: Based on timing deviation, the control signal is dynamically interpolated and adjusted to achieve fine calibration of audio-visual synchronization; the dynamic interpolation adjustment is expressed as:

[0098]

[0099] Right now,

[0100]

[0101] in, The original motion control signal, For smooth interpolation coefficients, The calibrated motion control signal;

[0102] By using the comprehensive emotion inference results as conditional input to a diffusion model or a streaming generation model, dynamic collaborative generation of digital human facial micro-expressions and body movements with response content and audience emotions is achieved. The micro-expression intensity adjustment factor can adjust the intensity of the digital human's facial expressions in real time according to subtle changes in audience emotions, ensuring a high degree of consistency between the digital human's nonverbal behavior and verbal content, thus creating emotional resonance. Simultaneously, a dynamic interpolation adjustment mechanism is used to finely calibrate the temporal deviation between speech synthesis and action generation, ensuring the quality of multimodal output with synchronized audio and video.

[0103] By organically integrating visual identity recognition, long-term memory retrieval, multimodal emotion perception, and anthropomorphic action generation, a complete technological loop is formed, from "identifying the viewer's identity" to "perceiving current emotions" and then to "generating personalized multimodal responses." This dual-driven interaction mechanism enables digital humans not only to remember "who you are" but also to perceive "your feelings at this moment" and respond in a way that conforms to historical relationships and the current context. This achieves a qualitative leap from one-way response to two-way emotional interaction, providing a complete technical solution for digital human live streaming interaction with high continuity, personalization, and emotional resonance capabilities.

[0104] First, the system performs multimodal data acquisition and preprocessing steps. In a live streaming scenario, the system simultaneously acquires input data from three modalities: viewer speech, bullet screen text, and live stream video images. For the acquired viewer speech, the system extracts intonation feature vectors and speech rate feature vectors; for the bullet screen text, the system extracts text feature vectors; and for the video images, the system extracts video image feature vectors. The video image feature vectors include the viewer's facial micro-expression features, as well as the broadcaster's facial expression features from the broadcaster's side image and the viewer's facial expression features from the viewer's side image.

[0105] Subsequently, the system proceeds to the sentiment analysis and semantic understanding steps. A large language model is used to perform deep semantic understanding and context awareness on the preprocessed multimodal features, while the sentiment analysis module identifies the audience's emotional state. During sentiment analysis, the system performs multimodal sentiment fusion by integrating extracted audience speech intonation, speech rate, and facial micro-expression features with the sentiment features of the bullet screen text. These features are weighted and summed using a learnable weight matrix, and after adding a bias vector, processed by an activation function to generate a fine-grained comprehensive sentiment inference result, which serves as the audience's emotional state.

[0106] Building upon this foundation, the system generates and personalizes response content. First, it generates anthropomorphic response text based on semantic understanding results and the audience's emotional state. Simultaneously, the system introduces a cross-session long-term memory module, retrieving the current audience's historical interaction records using a similarity matching formula. This formula calculates the similarity value by performing a dot product operation between the current audience's video image feature vector and the identity embedding vector of historical audiences, and then dividing by the product of their magnitudes. When the similarity exceeds a preset threshold, it is determined to be the same audience member, and the system uses their historical interaction records to personalize the anthropomorphic response text, generating a personalized response text.

[0107] Furthermore, the system combines the facial expression features of the anchor from the anchor's side image with the facial expression features of the audience from the audience's side image to perform bidirectional sentiment analysis. The anchor's and audience's facial expression features are concatenated as vectors, linearly transformed using an interaction weight matrix, mapped by a hyperbolic tangent activation function, multiplied by an adjustment coefficient, and then superimposed onto the comprehensive sentiment inference result to obtain the corrected audience sentiment state, further optimizing the accuracy of sentiment recognition.

[0108] Finally, the system executes the multimodal interactive output step. Personalized response text is used to generate synchronized speech via a speech synthesis model, while a motion generation model generates control signals to drive the digital human's lip movements, facial micro-expressions, and body movements. The motion generation model employs a diffusion model, generating real-time facial micro-expression sequences and body posture sequences based on a conditional control signal fused from the semantic embedding vector of the personalized response text and the corrected audience emotional state. The micro-expression intensity adjustment factor is calculated by weighting and summing the corrected audience emotional state and applying a sigmoid activation function. This factor dynamically adjusts the micro-expression control signal; that is, it linearly interpolates the basic micro-expression parameters based on semantic generation with neutral expression parameters to achieve real-time adaptation of micro-expression intensity to audience emotion. During output, the system detects the timing deviation between the synchronized speech generated by speech synthesis and the control signal generated by the motion generation model. Based on this deviation, the control signal is dynamically interpolated and adjusted. The original motion control signal and the delayed motion control signal are weighted and fused using smoothing interpolation coefficients to generate a calibrated motion control signal, ensuring synchronized audio-visual multimodal interactive output.

[0109] Example 2:

[0110] This embodiment is basically the same as the previous embodiment, except that the digital human live interactive processing device that integrates a large language model is used to implement the steps of the above method, including a multimodal acquisition and preprocessing module, which includes a voice acquisition unit, a text acquisition unit, a video acquisition unit and a feature extraction unit, used to acquire multimodal input data in the live scene and extract voice feature vectors, text feature vectors and video image feature vectors respectively.

[0111] The sentiment analysis and semantic understanding module includes a large language model inference unit and a sentiment analysis unit. The large language model inference unit is used to perform deep semantic understanding and context awareness on the preprocessed multimodal features, while the sentiment analysis unit is used to identify the audience's emotional state by combining multimodal features and generate semantic understanding results containing sentiment tags.

[0112] The response generation and personalization enhancement module includes a response text generation unit and a long-term memory unit. The response text generation unit is used to generate anthropomorphic response text based on semantic understanding results and audience emotional state. The long-term memory unit is used to retrieve the current audience's historical interaction records and use the retrieval results to personalize and enhance the response text, generating personalized response text.

[0113] The multimodal interactive output module includes a speech synthesis unit and an action generation unit. The speech synthesis unit is used to generate synchronized speech from personalized response text, and the action generation unit is used to generate control signals to drive the lip movements, facial micro-expressions and body movements of the digital human, so as to realize multimodal interactive output with synchronized audio and video.

[0114] When retrieving historical interaction records, the long-term memory unit establishes a connection between the current viewer and historical interaction records based on the viewer identity information identified by the video image features extracted by the multimodal acquisition and preprocessing module, so as to achieve personalized interaction across sessions.

[0115] The multimodal acquisition and preprocessing module is responsible for the real-time acquisition and feature extraction of multimodal data in the live streaming scenario. The voice acquisition unit continuously captures the audience's voice input, the text acquisition unit simultaneously acquires the bullet screen text information, and the video acquisition unit captures the live streaming video images, including images from the broadcaster's side and images from the audience's side. The feature extraction unit preprocesses the acquired multimodal data, extracting voice feature vectors, text feature vectors, and video image feature vectors, respectively. The video image feature vectors include the audience's facial micro-expression features, as well as the broadcaster's facial expression features and the audience's facial expression features.

[0116] The sentiment analysis and semantic understanding module receives preprocessed multimodal features and performs deep semantic understanding and sentiment recognition. The large language model inference unit utilizes a large language model to perform deep semantic understanding and context awareness of the multimodal features, constructing the semantic context of the current interactive scenario. The sentiment analysis unit combines multimodal features to identify the audience's emotional state, fusing the audience's speech intonation, speech rate, and facial micro-expression features with the sentiment features of the bullet screen text. A learnable weight matrix is ​​used to weight and sum these features, and after activation function processing, a fine-grained comprehensive sentiment inference result is generated. Simultaneously, the sentiment analysis unit also performs bidirectional sentiment state analysis by combining the anchor's facial expression features and the audience's facial expression features. The two are concatenated as vectors and linearly transformed using an interaction weight matrix. After mapping with a hyperbolic tangent activation function and multiplying by an adjustment coefficient, the results are superimposed on the comprehensive sentiment inference result to obtain the corrected audience emotional state.

[0117] The response generation and personalization enhancement module generates personalized response content based on semantic understanding results and emotional states. The response text generation unit generates anthropomorphic response text based on the semantic understanding results output by the large language model inference unit and the audience's emotional state identified by the sentiment analysis unit. The long-term memory unit is responsible for cross-session personalization enhancement. It retrieves the current audience's historical interaction records using a similarity matching formula. This formula calculates the similarity value by performing a dot product operation between the current audience's video image feature vector and the identity embedding vector of historical audiences, and then dividing by the product of their magnitudes. When the similarity exceeds a preset threshold, it is determined that they are the same audience, and their historical interaction records are used to personalize the anthropomorphic response text, generating personalized response text.

[0118] The multimodal interactive output module is responsible for converting the generated response content into synchronous multimodal interactive output. The speech synthesis unit synthesizes personalized response text into synchronous speech. The action generation unit uses a diffusion model to generate control signals that drive the digital human's lip movements, facial micro-expressions, and body movements in real time, based on the conditional control signal formed by fusing the semantic embedding vector of the personalized response text with the corrected audience emotional state. During micro-expression generation, the action generation unit uses a micro-expression intensity adjustment factor to weight and sum the corrected audience emotional state and calculates the intensity coefficient using a sigmoid activation function. This coefficient is used to linearly interpolate the basic micro-expression parameters and neutral expression parameters based on semantic generation, achieving real-time adaptation of micro-expression intensity to audience emotion. Regarding output synchronization, the multimodal interactive output module detects the timing deviation between the synchronous speech generated by the speech synthesis unit and the control signal generated by the action generation unit. Based on this deviation, it dynamically interpolates and adjusts the control signal, and uses a smooth interpolation coefficient to weightedly fuse the original action control signal with the delayed action control signal to generate a calibrated action control signal, ultimately achieving synchronized audio-visual multimodal interactive output.

[0119] Throughout the process, there is a collaborative interaction between the long-term memory unit and the multimodal acquisition and preprocessing module. When retrieving historical interaction records, the long-term memory unit establishes the association between the current audience and historical interaction records based on the audience identity information identified by the video image features extracted by the multimodal acquisition and preprocessing module. Thus, personalized interaction across sessions can be achieved solely based on visual information without the audience having to actively log in or speak. This enables the digital human to continuously accumulate relational memories with specific audiences and provide personalized services with continuity and depth.

[0120] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. A method for processing digital human live interaction of a large language model fusion, characterized in that, Includes the following steps: S1: Multimodal data acquisition and preprocessing steps, including: S11: Collect multimodal input data in the live streaming scenario, including audience voice, bullet screen text, and live streaming video images; S12: Preprocess the multimodal input data to extract speech feature vectors respectively. Text feature vectors and video image feature vectors ; S2: Sentiment analysis and semantic understanding steps, using a large language model to perform deep semantic understanding and context awareness on the preprocessed multimodal features, and combining the sentiment analysis module to identify the audience's emotional state, generating semantic understanding results containing sentiment tags; S3: Response content generation and personalization enhancement steps, including: S31: Based on the semantic understanding results and the audience's emotional state, generate anthropomorphic response text; S32: Introduce a long-term memory module across sessions to retrieve the current audience's historical interaction records, and use the retrieval results to personalize and enhance the response text, generating personalized response text; S4: Multimodal interactive output step, the personalized response text is generated into synchronous speech through a speech synthesis model, and control signals that drive the lip movements, facial micro-expressions and body movements of the digital human are generated through an action generation model to achieve multimodal interactive output with synchronized audio and video. In step S3, the long-term memory module, when retrieving historical interaction records, establishes an association between the current viewer and historical interaction records based on the viewer identity information identified by the video image features collected in step S1, so as to achieve personalized interaction across sessions.

2. The digital human live-streaming interactive processing method integrating a large language model according to claim 1, characterized in that, The sentiment analysis module in step S2 further includes: S21: Extract intonation features from the audience's speech. and speech rate characteristics ; S22: Combining the live stream video images collected in step S1, extract the facial micro-expression features of the viewers. ; S23: Perform multimodal sentiment fusion of the tone features, speech rate features, facial micro-expression features, and sentiment features of the bullet screen text, and generate a fine-grained comprehensive sentiment inference result according to the following formula. As the emotional state of the audience: in, This represents the sentiment feature vector of the bullet screen text. , , , The weight matrix is ​​a learnable matrix. For bias terms, This is the activation function.

3. The digital human live interactive processing method integrating a large language model according to claim 1, characterized in that, In step S32, the long-term memory module retrieves historical interaction records using the following similarity matching formula: in, The video image feature vector of the current viewer. An identity vector is embedded for each historical viewer. When the similarity exceeds a preset threshold τ, the viewer is identified as the same viewer, and their historical interaction records are used to generate personalized response text.

4. The digital human live-stream interactive processing method integrating a large language model according to claim 1, characterized in that, The action generation model in step S4 adopts a diffusion model, which generates a digital human's facial micro-expression sequence and body posture sequence in real time based on the semantic content of the personalized response text and the subtle changes in the audience's emotional state identified in step S2; the conditional generation process of the diffusion model is expressed as follows: in, Let k be the number of diffusion steps, k = 1, 2, ..., K. Let be the action pose vector after the k-th step diffusion. For the first The action posture vector of the step. For conditional control signals, Indicates a Gaussian distribution. and They are respectively from parameters A defined mean function and variance function; The condition control signal Semantic embedding vector containing the personalized response text The fusion features with the audience's emotional state ,Right now , where Concat(x) represents the vector concatenation operation.

5. The digital human live-stream interactive processing method integrating a large language model according to claim 1, characterized in that, In step S23, a fine-grained comprehensive sentiment inference result is generated. Then, this result is used as a condition input into the action generation model in step S4 to dynamically adjust the generation intensity and type of digital human facial micro-expressions; wherein, the micro-expression intensity adjustment factor Calculate using the following formula: in Represents a vector Perform a transpose operation. For learnable weight vectors, For bias terms; The generated micro-expression control signals are represented as : in, These are the basic micro-expression parameters for semantic generation. These are neutral expression parameters.

6. The digital human live-stream interactive processing method integrating a large language model according to claim 1, characterized in that, The live stream video images collected in step S1 include images from the broadcaster's side and images from the audience's side; the sentiment analysis module in step S2 combines the broadcaster's facial expression features from the broadcaster's side image. Audience facial expression features in the audience-side image A two-way emotional state analysis was conducted, and the audience's emotional state was corrected according to the following formula: in, To correct the audience's emotional state, For adjustment coefficients, This is the interaction weight matrix. This represents the vector concatenation operation. tanh(x) is the hyperbolic tangent activation function, used to map interactive features to the (−1,1) interval to control the correction magnitude.

7. The digital human live interactive processing method integrating a large language model according to claim 1, characterized in that, Step S4 also includes: S41: Detect the timing deviation between the synchronized speech generated by the speech synthesis model and the control signal generated by the action generation model. ; S42: Based on the timing deviation, the control signal is dynamically interpolated and adjusted to achieve fine calibration of audio-visual synchronization; the dynamic interpolation adjustment is expressed as: in, The original motion control signal, For smooth interpolation coefficients, This is the calibrated motion control signal.

8. A digital human live-streaming interactive processing device integrating a large language model, used to implement the steps of the method according to any one of claims 1 to 7, characterized in that, include: The multimodal acquisition and preprocessing module includes a voice acquisition unit, a text acquisition unit, a video acquisition unit, and a feature extraction unit. It is used to acquire multimodal input data in live streaming scenarios and extract voice feature vectors, text feature vectors, and video image feature vectors respectively. The sentiment analysis and semantic understanding module includes a large language model inference unit and a sentiment analysis unit. The large language model inference unit is used to perform deep semantic understanding and context awareness on the preprocessed multimodal features. The sentiment analysis unit is used to identify the audience's emotional state by combining multimodal features and generate semantic understanding results containing sentiment tags. The response generation and personalization enhancement module includes a response text generation unit and a long-term memory unit. The response text generation unit is used to generate anthropomorphic response text based on the semantic understanding results and the audience's emotional state. The long-term memory unit is used to retrieve the current audience's historical interaction records and use the retrieval results to personalize and enhance the response text, generating personalized response text. The multimodal interactive output module includes a speech synthesis unit and an action generation unit. The speech synthesis unit is used to generate synchronized speech from the personalized response text, and the action generation unit is used to generate control signals to drive the lip movements, facial micro-expressions and body movements of the digital human, so as to realize multimodal interactive output with synchronized audio and video. When retrieving historical interaction records, the long-term memory unit establishes a connection between the current viewer and historical interaction records based on the viewer identity information identified by the video image features extracted by the multimodal acquisition and preprocessing module, so as to realize personalized interaction across sessions.

9. A digital human live-streaming interactive processing device integrating a large language model, characterized in that: It includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the method according to any one of claims 1 to 7.

10. A digital human live-streaming interactive processing medium integrating a large language model, characterized in that: When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images