A digital human live broadcast interaction method based on multi-language dynamic driving

Abstract

The application relates to the technical field of digital human interaction, in particular to a digital human live broadcast interaction method based on multilingual dynamic driving, which comprises the following steps: constructing a digital human live broadcast knowledge base containing general knowledge and personalized knowledge; acquiring multilingual barrage streams of a live broadcast room in real time, uniformly processing the barrage streams in terms of language, and extracting semantic features; performing semantic aggregation processing on the barrage streams by using the semantic features, so as to extract a plurality of representative interactive topics; acquiring a voice interactive question set; inputting the voice interactive question set into an intelligent question and answer model based on fine tuning of the knowledge base, and generating corresponding interactive texts; if a target question contains multilingual mixed information, generating interactive texts mixed with multiple languages; generating voice and digital human action video streams based on the interactive texts, and returning to the live broadcast room for real-time playing. The application greatly improves the effectiveness of live broadcast content and audience experience.

Description

Technical Field

[0001] This application relates to the field of digital human interaction technology, specifically to a digital human live streaming interaction method based on multilingual dynamic driving. Background Technology

[0002] With the development of artificial intelligence technology, digital human live streaming, with its advantages of low cost and all-weather operation, has been widely used. However, significant interaction bottlenecks still exist when facing globalized, multilingual live streaming scenarios. Existing digital human bullet screen processing relies heavily on traditional keyword matching technology. When faced with massive, highly fragmented comment information containing multiple language expressions, it is difficult to effectively aggregate semantically similar interactive questions, leading to the dilution and omission of users' genuine questions.

[0003] At the same time, the live broadcast room inevitably contains a lot of low-value interference information. In addition, the network latency or the large number of repeated bullet comments caused by following the trend makes it impossible for the existing technology to accurately assess the real timeliness and popularity of interactive questions. This can easily lead to the digital human repeatedly answering outdated and redundant questions or causing system blockage due to the indiscriminate processing of multilingual replies.

[0004] Therefore, how to overcome the interference caused by differences in multilingual expressions, useless noise, and network latency, accurately aggregate the semantics of bullet comments, and dynamically evaluate the real-time popularity of interactive questions, so as to achieve efficient and non-blocking multilingual interactive scheduling and response, is a technical problem that digital human live streaming interaction urgently needs to solve. Summary of the Invention

[0005] To address the aforementioned technical problems, this application provides a multilingual, dynamically driven digital human live-streaming interaction method to resolve existing issues.

[0006] This application proposes a multilingual, dynamically driven digital human live-streaming interaction method, which employs the following technical solution: One embodiment of this application provides a method for live streaming interaction of digital humans based on multilingual dynamic driving, the method comprising the following steps: Construct a digital human live-streaming knowledge base that includes both general and personalized knowledge; The system acquires multilingual bullet comments in the live stream in real time, and extracts semantic features after unifying the language of the bullet comments. The semantic features are used to perform semantic aggregation processing on the bullet screen stream to extract several representative interactive topics; for each representative interactive topic, its dynamic interactive features at different time granularities are evaluated to generate a comprehensive popularity index, and target questions with interactive value are selected based on the comprehensive popularity index to generate a set of voice interaction questions. The set of voice interaction questions is input into the intelligent question-answering model fine-tuned based on the knowledge base to generate corresponding interactive text; if the target question contains multilingual information, multilingual interactive text is generated. The system generates audio and digital human motion video streams based on interactive text and transmits them back to the live streaming room for real-time playback.

[0007] In one embodiment, the step of extracting semantic features after unifying the language of the bullet screen stream includes: Identify the original language of each bullet comment message in the bullet comment stream; When the original language is not a preset standard language, a machine translation model is invoked to translate it into the preset standard language; The semantic features are obtained by vectorizing the bullet screen messages after unifying the language using a natural language processing model.

[0008] In one embodiment, before performing semantic aggregation processing on the bullet screen stream using the semantic features, the method further includes: Calculate the semantic similarity between the semantic features of each bullet screen message and the semantic features of the text stored in the knowledge base. If the semantic similarity is less than a preset threshold, it is determined to be irrelevant information and the bullet screen message is deleted. The cleaned bullet screen stream is then used for semantic aggregation processing.

[0009] In one embodiment, the extraction of several representative interactive topics includes: The semantic features of the bullet comments in the bullet comment stream are clustered to obtain each cluster. The bullet comment corresponding to the semantic features in each cluster is a representative interactive topic.

[0010] In one embodiment, the process of generating the comprehensive popularity index is as follows: The historical interaction frequency of each representative interactive topic within the current interactive question statistical period is counted to calculate the first popularity. The current interactive question statistical period is divided into several local time periods. The rate of change of interaction frequency in each local time period is statistically analyzed to calculate the second popularity. The first and second popularity metrics are positively merged to generate the comprehensive popularity index.

[0011] In one embodiment, the overall popularity index is the product of the first popularity and the second popularity.

[0012] In one embodiment, the filtering of target questions with interactive value includes: All representative interactive topics are sorted from high to low according to the comprehensive popularity index, and the representative interactive topics with the first preset number of comprehensive popularity indexes are selected as target questions with interactive value.

[0013] In one embodiment, generating the corresponding interactive text includes: Based on the target question in the set of voice interaction questions, the finely tuned intelligent question answering model is controlled to perform autoregressive inference and generate the corresponding interactive text.

[0014] In one embodiment, generating multilingual interactive text includes: Analyze the proportion of each language information in the cluster to which the target question belongs; The intelligent question-answering model is controlled to alternately use the corresponding language to generate response statements according to the stated proportion.

[0015] In one embodiment, the generation of voice and digital human motion video streams based on interactive text includes: The interactive text is input into the text-to-speech module to synthesize an audio stream that matches the digital human image; The acoustic features of the audio stream are extracted to drive the digital human to generate video frames of facial expressions and body movements synchronized with lip movements; The audio stream and video frames are time-aligned and rendered to obtain a digital human motion video stream.

[0016] This application has at least the following beneficial effects: This application overcomes the limitations of traditional keyword matching by combining unified language processing with semantic feature aggregation, significantly improving the accuracy of intent recognition in multilingual environments. Addressing the issues of highly fragmented and synonymous multilingual expressions in live-streaming chat, this application unifies the translation of multilingual chat messages, extracts semantic features, and then performs clustering and aggregation. This breaks the strong dependence of traditional keyword matching on specific words and fixed languages, accurately grouping issues with different languages ​​and expressions but the same core intent into the same representative interactive topic. Simultaneously, it effectively filters out low-value spam, providing a precise data foundation for high-quality interaction.

[0017] By evaluating dynamic interaction characteristics at different time granularities to generate a comprehensive popularity index, this approach effectively resists interference from network latency and trending comments, ensuring the real-time nature and high value of interaction. Addressing the issues of delayed and repetitive calculations of comments due to network latency in live streaming, and the susceptibility of traditional simple counting methods to being misled by short-term bursts of information, this approach achieves dynamic quantification of the true popularity of comments by evaluating dynamic interaction characteristics at different time granularities. It accurately avoids the repetitive answering of already addressed questions by digital bots, accurately captures the real hot topics and demands of viewers in the current live stream, and greatly improves the effectiveness of live stream content and viewer experience. Attached Figure Description

[0018] Figure 1 A flowchart illustrating the steps of a multilingual, dynamically driven digital human live-streaming interaction method provided in this application. Detailed Implementation

[0019] The following description, in conjunction with the accompanying drawings, details a specific scheme for a multilingual, dynamically driven digital human live-streaming interaction method provided in this application.

[0020] This application provides an embodiment of a multilingual dynamically driven digital human live streaming interaction method. Specifically, it provides the following multilingual dynamically driven digital human live streaming interaction method. Please refer to [link to relevant documentation]. Figure 1 The method includes the following steps: S1: Construct a digital human live streaming knowledge base that includes general knowledge and personalized knowledge; S2: Real-time acquisition of multilingual bullet comments in the live stream, and extraction of semantic features after language unification processing of the bullet comments; S3: Use the semantic features to perform semantic aggregation processing on the bullet screen stream to extract several representative interactive topics; evaluate the dynamic interactive features of each representative interactive topic at different time granularities to generate a comprehensive popularity index, and select target questions with interactive value based on the comprehensive popularity index to generate a set of voice interaction questions. S4: Input the set of voice interaction questions into the intelligent question answering model fine-tuned based on the knowledge base to generate corresponding interactive text; if the target question contains multilingual mixed information, then generate multilingual mixed interactive text. S5: Generates audio and digital human motion video streams based on interactive text and transmits them back to the live streaming room for real-time playback.

[0021] During live streaming of digital humans, it is first necessary to provide factual basis and dialogue boundaries for the voice interaction of digital humans. Therefore, building a knowledge base for live streaming of digital humans can also serve as a semantic benchmark for filtering out the massive noise in the live streaming room.

[0022] The digital human live streaming knowledge base is constructed specifically based on the current live streaming type and the information of the live streaming subject. Its content deeply integrates general knowledge in the field with personalized knowledge in specific scenarios.

[0023] S101: Gather general knowledge in the field of digital human live streaming.

[0024] General knowledge covers common script templates for live streaming openings and transitions for digital humans, as well as answers to frequently asked questions such as logistics and after-sales service, to ensure the standardization and natural coherence of digital human interactions.

[0025] S102: Collect personalized knowledge about the field of digital human live streaming.

[0026] Personalized knowledge is tailored to the specific live stream audience, including detailed information such as product price, specifications, functions, or specific attractions, to ensure the digital human's answers are highly targeted and accurate.

[0027] S103: Preprocess general knowledge and personalized knowledge and store them in the digital human live streaming knowledge base.

[0028] The collected general and personalized knowledge is segmented into multiple text blocks with complete semantics. A pre-trained text embedding model is used to encode and extract each text block, and the dense vector corresponding to each text block is obtained, which is denoted as the semantic feature vector. The original text data of the text block and its corresponding semantic feature vector are used to construct a key-value pair mapping matrix, which is then stored in the digital human live broadcast knowledge base to build a vector index.

[0029] Preferably, in this embodiment, the text segmentation process specifically adopts a segmentation method based on punctuation rules. The pre-trained text embedding model includes, but is not limited to, the BERT model and the Sentence-BERT model. Specifically, the text block is input into the pre-trained text embedding model, the [CLS] flag feature vector in the model output layer is extracted, and the obtained vector is used as the semantic feature vector corresponding to the text block.

[0030] S104: Train an intelligent question-answering model using a digital human live-streaming knowledge base.

[0031] Preferably, this embodiment uses the open-source ChatGLM3-6B model as the intelligent question-answering model, and employs LoRA (Low-Rank Adaptation) fine-tuning technology combined with a digital human live-streaming knowledge base for fine-tuning. Specifically: Step A: Fine-tuning dataset construction; extract general and personalized knowledge from the digital human live streaming knowledge base, use common audience questions in the live streaming room as instructions, take the corresponding product information and context as input, and use common dialogue templates and common question answer templates as output to construct a triplet fine-tuning dataset for supervised fine-tuning.

[0032] Step B: Tokenization processing of basic data; call the BPE (Byte-Pair Encoding) tokenizer supporting the ChatGLM3-6B model to perform text tokenization and digital encoding on the constructed triple fine-tuning dataset. The maximum input sequence length of the model is set to 1024. Text sequences exceeding this length are truncated, and sequences shorter than this length are padded with padding identifiers to obtain formatted feature tensors.

[0033] Step C: Low-Rank Adaptation (LoRA) fine-tuning configuration; freeze the backbone network weights of 6 billion parameters in the ChatGLM3-6B pre-trained model, and inject LoRA modules only into the branches beside the key-value query projection layers and the feedforward network layers of the attention mechanism module in the Transformer architecture of the model. Specifically, the rank r of the LoRA module is set to 8, and the scaling factor is set to 16. At this time, the number of trainable parameters involved in gradient updates only accounts for about 0.1% of the total parameters, which greatly reduces the training video memory pressure.

[0034] Step D: Cross-entropy loss calculation and weight update; input the formatted feature tensors into the above configured model in batches (for example, BatchSize=4). Calculate the cross-entropy loss of the autoregressive task for the generated output sequence. Use the AdamW optimizer for parameter update, set the initial learning rate to 2e-5, adopt the cosine learning rate decay strategy, set the number of training epochs to 3, and stop the training after the loss function of the model converges.

[0035] Step E: Fine-tuned model merging; after the training is completed, multiply the dimension reduction matrix A and the dimension increase matrix B learned in the LoRA module, add and fuse the result with the frozen basic weight matrix of the backbone network, and export the complete fine-tuned intelligent question-answering model.

[0036] The steps of extracting semantic features after performing unified language processing on the multilingual bullet comment stream in the live broadcast room are as follows: S201, read the multilingual bullet comment stream in the live broadcast room.

[0037] First, divide the digital human live broadcast timeline into consecutive interactive question statistics cycles, and define the duration of each statistics cycle as T seconds. This time window is used for aggregation analysis of all bullet comment messages falling into it, so as to filter out the set of voice interaction questions that require the digital human to generate voice and images for response within this statistics cycle.

[0038] The value range of T is set to [20, 60], and the initial configuration in this embodiment is T=30. In order to cope with the fluctuation of the popularity of the live room, a dynamic adjustment mechanism of statistical duration is introduced: the number of concurrent bullet comments per unit time in the live room is monitored in real time. When the number of concurrent bullet comments is greater than the preset high popularity threshold (e.g., 50 comments / second), the value of T is reduced to speed up the frequency of digital human response; when the number of concurrent bullet comments is lower than the preset low popularity threshold (e.g., 10 comments / second), the value of T is appropriately increased to accumulate a sufficient number of bullet comments for analysis; the specific values ​​of the above thresholds can be pre-calibrated by the system according to the average traffic level of the connected live streaming platform.

[0039] Specifically, the first interactive question statistics period after the live stream begins (i.e., the first T seconds) is designated as the cold start phase. During this phase, due to insufficient accumulated bullet screen data, bullet screen-based voice interaction is not triggered. Instead, the opening remarks from the digital human live stream knowledge base are directly extracted to drive the digital human to perform transitional broadcasts. At the same time, the background process continues to collect and analyze bullet screen data for the first T seconds, providing data support for entering the second interactive question statistics period.

[0040] During the interactive issue statistics period, the real-time audience interaction data stream in the live room is obtained by calling the open API interface of the connected live streaming platform; the audience interaction data stream is parsed to extract the original multilingual text data contained in each bullet comment and the corresponding sending timestamp, and the original bullet comment message set is constructed.

[0041] S202, extract semantic features after unifying the language of the bullet screen stream.

[0042] For the The set of raw bullet screen messages obtained within a statistical period for each interactive question is represented as follows: , ,in, Indicates the first The number of bullet comments is as follows: for The total number of bullet screen comments.

[0043] First of all, Each bullet comment is identified by language. If a non-standard language is identified (in this embodiment, the standard language is preset to be Chinese), a multilingual translation model based on the Transformer architecture is called to translate it into standard language text to unify subsequent semantic processing. At the same time, the original language type identified for that bullet comment is used as a language tag and attached to the attribute of the translated standard text for future use in multilingual dynamic replies.

[0044] Subsequently, using the same pre-trained text embedding model as when constructing the digital human live-streaming knowledge base, the text was processed... Semantic features are extracted from all bullet comment information after language standardization. Specifically, the bullet comment text is input into a text embedding model, and the corresponding feature vector of the output layer is extracted as the bullet comment. The semantic feature vector, denoted as By employing the same text embedding model and extraction method, we ensured that the semantic feature vectors of the bullet screen comment information maintained strict consistency with the semantic feature vectors of each text block in the knowledge base in terms of feature space and data dimension.

[0045] S203, filter the bullet comments in the original bullet comment message set.

[0046] During live streaming with digital humans, it's crucial to extract key questions from the massive amounts of text messages sent by viewers and generate corresponding interactive answers based on a knowledge base. Precise message processing, filtering out irrelevant information and extracting trending topics, is fundamental to achieving high-quality interaction with digital humans.

[0047] Traditional bullet screen comment processing methods primarily rely on fixed keyword matching techniques for question retrieval. However, in complex live streaming scenarios targeting multilingual users, keyword matching struggles to accurately aggregate semantically similar interactive questions and is easily affected by a large amount of irrelevant spam, lacking the ability to discern contextual information. It also fails to consider the impact of network transmission latency and viewer-driven spamming, easily mistaking previously answered, repetitive messages for currently trending questions, leading to redundant answers from the digital human. This severely reduces the accuracy and real-time nature of the digital human's interaction with real user needs, ultimately resulting in poor overall interactive performance in multilingual live streaming scenarios.

[0048] Therefore, this embodiment will The semantic feature vectors of all bullet screen comments are matched against the semantic feature vectors of text blocks in the digital human live streaming knowledge base. The matching search employs an approximate nearest neighbor search algorithm (e.g., the KD-Tree algorithm). To ensure consistency between the distance retrieval algorithm and the similarity metric, all semantic feature vectors are L2 normalized before retrieval. Then, the cosine similarity between the normalized bullet screen semantic feature vectors and the knowledge base text block semantic feature vectors is calculated using KD-Tree. The knowledge base text block with the highest semantic similarity to each bullet screen comment is extracted. If this maximum cosine similarity is less than a set filtering threshold... If the comment is deemed irrelevant to the current live stream topic, it will be removed from the original comment message set to eliminate the interference of low-value, meaningless spam comments on subsequent popularity statistics.

[0049] Filtering threshold The value range is [-1, 1], and its specific value can be preset by the implementer according to the strictness of filtering spam information. The higher the set value, the stricter the filtering. In this embodiment, The setting value is 0.1.

[0050] S301 extracts representative interactive topics from the bullet comment stream.

[0051] After eliminating spam comments, the first The set of bullet screen comment information within a statistical period for each interactive question is represented as: , ,in, for The total number of bullet comments. Specifically, if... If no further statistical processing is performed, a set of voice interaction questions is obtained. At this point, the digital human is directly driven to explain and describe the live broadcast topic using familiar language.

[0052] During live-streamed interactions with digital humans, factors such as network latency and viewers' tendency to flood the chat with comments can cause a large number of questions that were answered in the previous statistical period to reappear in the current period, resulting in delayed and repetitive comments. Directly using the absolute number of these delayed and repetitive messages in the current period's trending topics statistics would lead to the digital human repeatedly explaining the same questions, severely impacting the viewing experience.

[0053] To address the aforementioned issues, this application constructs a first popularity index by comparing the number of similar questions in the current statistical period with that in the previous statistical period, thereby dynamically reducing the weight of duplicate questions. The specific process is as follows: First of all, All bullet comment messages are semantically clustered. This embodiment uses the DBSCAN clustering algorithm. Euclidean distance is used as the distance metric during clustering, and the neighborhood radius parameter of DBSCAN clustering is... The minimum number of core samples (MinPts) is set to 3, with a value of 0.5. After clustering, I clusters are obtained. The i-th cluster is denoted as [i-th cluster]. , Each The included bullet comments all belong to a highly similar category of questions, which can serve as a representative interactive topic, allowing the digital human to provide a concentrated answer through a single interaction.

[0054] Similarly, for the set of the previous statistical period Semantic clustering is performed using the same method, resulting in J clusters. The j-th cluster is denoted as . , Subsequently, the mean of the semantic feature vectors of all bullet comments within each cluster is calculated, and this is used as the representative semantic vector of that cluster.

[0055] S302, Calculate the historical interaction frequency of each representative interactive topic within the current interactive question statistical period.

[0056] For any cluster in the current statistical period Calculate the cosine similarity between its representative semantic vector and the representative semantic vectors of all J clusters in the previous statistical period; select the cluster with the largest cosine similarity, and whose maximum value is greater than a set threshold. Clusters from the previous statistical period ,Will and Defined as a pair of similar bullet screen clusters. The set threshold Its specific size is set by the implementer according to the similarity judgment criteria. In this embodiment, The value is 0.8. If there is no value greater than 0.8... The similarity is then used to determine This is a newly added cluster of issues for the current statistical period.

[0057] After determining the similarity relationships among clusters within the current statistical period, the first popularity of each cluster containing similar bullet screen clusters within the current statistical period is calculated. Specifically, this is done by calculating the number of bullet screen messages belonging to that cluster within the current statistical period, representing a percentage of the total number of bullet screen comments in the current statistical period. The proportion of the total number of bullet comments is used as the basic popularity ratio of the cluster; then, the basic popularity ratio is multiplied by the popularity decay factor corresponding to the cluster to obtain the first popularity.

[0058] The determination of the heat decay factor is as follows: First, it is determined whether the cluster of the current statistical period simultaneously meets the following three preconditions: First, the current statistical period is not the first statistical period after the live broadcast starts; Second, the current cluster has a corresponding similar cluster in the previous period; Third, the similar cluster in the previous statistical period has been selected into the voice interaction question set of the previous statistical period, and has been answered by the digital human.

[0059] If any of the above three preconditions are not met, it means that the cluster belongs to an unsolved new problem or is in the cold start stage. In this case, the heat decay factor is directly set to 1.

[0060] If all three prerequisites are met, the ratio of the number of bullet comments in the current statistical period to the number of bullet comments in the similar clusters of the previous statistical period is further calculated: if the number in the current statistical period is less than the number in the previous statistical period, the ratio is used as the popularity decay factor; if the number in the current statistical period is greater than or equal to the number in the previous statistical period, the popularity decay factor is set to 1.

[0061] It's important to note that a high base popularity percentage for a particular type of question indicates high apparent popularity. However, if this type of question was already answered by the digital human in the previous statistical period, and the number of such questions in the current statistical period is decreasing compared to the previous period, it suggests that these comments are likely delayed duplicates due to network latency or the tail effect of viewers following the trend, indicating a decline in genuine audience interaction. In this case, using a popularity decay factor less than 1 to reduce weighting can prevent the digital human from repeatedly answering the same question, leading to redundancy in the live stream. Conversely, if the question has been answered but the number of such questions in the current statistical period has not decreased but rather increased or remained the same, it indicates a continuous influx of new viewers with similar needs into the live stream. In this case, a popularity decay factor of 1 is used without weighting, thus avoiding missing genuine high-popularity user demands.

[0062] S303, Calculate the rate of change of interaction frequency within each local time period of the current interaction problem statistical period.

[0063] In live streaming interactive scenarios, the hot topics that viewers pay attention to have dynamic characteristics over time. If only the total number of bullet comments within the entire interactive question statistics period is counted, it can only reflect the static total popularity over a period of time, and cannot reflect the real-time changing trend of viewers' attention. It is also very susceptible to short-term bursts or network delays, which can lead to evaluation bias.

[0064] To accurately capture the changing trends of user attention over time, the current statistical period for interactive issues is divided into multiple consecutive local time segments. The duration of each local time segment is set to t seconds, and the total duration T is divisible by t. The value of t ranges from [2, 6]. In this embodiment, t is set to 5 seconds, thus dividing the total duration into T / t local time segments.

[0065] Based on the chronological order of the bullet comments' timestamps, the number of bullet comments for each cluster within each local time period is counted, and this number is used to calculate the cluster's second popularity. First, all adjacent local time periods are traversed chronologically, and the difference in the number of bullet comments between the two periods is calculated. Second, this difference is divided by the total number of bullet comments for the cluster in the current statistical period to obtain the relative change rate between adjacent time periods. Then, this relative change rate is subjected to an exponential function with the natural constant e as its base to obtain the trend index value for each pair of adjacent time periods. Finally, the sum of the trend index values ​​for all pairs of adjacent time periods is calculated, and the average is obtained by dividing by the number of adjacent time period pairs. This average is used as the second popularity of the cluster.

[0066] It should be noted that the characteristics of the exponential function are used to quantify the trend. When the number of bullet comments in a cluster continuously increases within adjacent time periods, the relative rate of change is positive, and the exponential result is greater than 1; when the number of bullet comments declines, the relative rate of change is negative, and the exponential result is less than 1; when the number remains unchanged, the exponential result equals 1. By calculating the average of the trend exponential values ​​of all adjacent time periods, if the final second popularity score is greater than 1, it indicates that the issue is at the peak of audience attention, having a positive gain effect; if the second popularity score is less than 1, it indicates that audience attention is declining, having a deweighting effect; if it equals 1, no adjustment is made to the popularity score.

[0067] S304, evaluate the dynamic interaction characteristics of representative interactive topics at different time granularities to generate a comprehensive popularity index.

[0068] Multiply the first heat index of a cluster by the second heat index to obtain the overall heat index of the cluster.

[0069] S305: Filter target questions with interactive value and generate a set of voice interaction questions.

[0070] The same method was used to calculate the comprehensive popularity index of all clusters in the current statistical period, and all clusters were sorted in descending order of comprehensive popularity index.

[0071] The top K clusters based on the overall popularity index are selected, where K is a positive integer pre-defined by the implementer; in this embodiment, K is set to 5. For each selected cluster, the cosine similarity between the semantic feature vectors of all bullet comment messages within that cluster and the representative semantic vector of that cluster is calculated. The original bullet comment with the highest cosine similarity, i.e., the bullet comment text closest to the cluster center, is selected as the typical question text representing the intent of that cluster, indicating the target question with interactive value.

[0072] The typical question texts corresponding to the top K clusters are extracted to form the voice interaction question set for the current interaction question statistical period.

[0073] S4, the process of generating multilingual interactive text is as follows: for each target question in the set of voice interaction questions, count the original language tags carried by all bullet comments in its cluster, calculate the proportion of each language information, and sort them in descending order according to the proportion.

[0074] To prevent excessive long-tail languages ​​from causing long single-broadcast times for the digital human and thus leading to system interaction blockage, a language truncation mechanism is introduced: a preset language generation truncation threshold M is set (for example, M=2 in this embodiment), and the top M languages ​​are selected as the target response languages, while language requests ranked outside the top M are directly discarded.

[0075] Subsequently, the selected target response language is used as the response instruction, and the target question is used as the input information, which is then fed into the intelligent question-answering model, which has been fine-tuned using a digital human live-streaming knowledge base. The fine-tuned intelligent question-answering model performs autoregressive inference and outputs multilingual interactive text data that is consistent with the style of the live-streaming dialogue and is concatenated in order of the proportion of the selected languages.

[0076] S501 synthesizes an audio stream that matches the digital human avatar.

[0077] The interactive text data is input into a text-to-speech (TTS) module that supports multilingual speech. The TTS module synthesizes a multilingual mixed audio stream that matches the digital human's preset image and voiceprint features based on the language distribution in the text.

[0078] S502 drives the generation of video frames of facial expressions and body movements that are synchronized with lip movements for digital human generation.

[0079] Furthermore, acoustic features (e.g., Mel spectrum features) of the audio stream are extracted. These acoustic features are then input into a pre-trained digital human face-driving model (e.g., the Wav2Lip model), and a deep learning network is used to predict lip movements, generating facial expression video frames that are strictly synchronized with the current lip movements. Simultaneously, the digital human motion engine loops through regular live-stream body movement video frames from a pre-set general motion library, based on the duration of the synthesized audio stream.

[0080] S503 generates a complete video stream of digital human motion.

[0081] Finally, the synthesized audio stream, generated facial expression video frames, and body movement video frames are time-aligned and fused on the timeline to generate a complete digital human motion video stream, which is then transmitted back to the live broadcast room for real-time playback via a streaming media transmission protocol, completing the digital human live broadcast interaction.

Claims

1. A method for live streaming interaction of digital humans based on multilingual dynamic driving, characterized in that, The method includes the following steps: Construct a digital human live-streaming knowledge base that includes both general and personalized knowledge; The system acquires multilingual bullet comments in the live stream in real time, and extracts semantic features after unifying the language of the bullet comments. The semantic features are used to perform semantic aggregation processing on the bullet screen stream to extract several representative interactive topics; for each representative interactive topic, its dynamic interactive features at different time granularities are evaluated to generate a comprehensive popularity index, and target questions with interactive value are selected based on the comprehensive popularity index to generate a set of voice interaction questions. The set of voice interaction questions is input into the intelligent question-answering model fine-tuned based on the knowledge base to generate corresponding interactive text; if the target question contains multilingual information, multilingual interactive text is generated. The system generates audio and digital human motion video streams based on interactive text and transmits them back to the live streaming room for real-time playback.

2. The method for live streaming interaction of digital humans based on multilingual dynamic driving as described in claim 1, characterized in that, The extraction of semantic features after unifying the language of the bullet screen stream includes: Identify the original language of each bullet comment message in the bullet comment stream; When the original language is not a preset standard language, a machine translation model is invoked to translate it into the preset standard language; The semantic features are obtained by vectorizing the bullet screen messages after unifying the language using a natural language processing model.

3. The method for live streaming interaction of digital humans based on multilingual dynamic driving as described in claim 2, characterized in that, Before performing semantic aggregation processing on the bullet screen stream using the semantic features, the process also includes: Calculate the semantic similarity between the semantic features of each bullet screen message and the semantic features of the text stored in the knowledge base. If the semantic similarity is less than a preset threshold, it is determined to be irrelevant information and the bullet screen message is deleted. The cleaned bullet screen stream is then used for semantic aggregation processing.

4. The method for live streaming interaction of digital humans based on multilingual dynamic driving as described in claim 3, characterized in that, The extracted representative interactive topics include: The semantic features of the bullet comments in the bullet comment stream are clustered to obtain each cluster. The bullet comment corresponding to the semantic features in each cluster is a representative interactive topic.

5. The method for live streaming interaction of digital humans based on multilingual dynamic driving as described in claim 1, characterized in that, The process of generating the comprehensive heat index is as follows: The historical interaction frequency of each representative interactive topic within the current interactive question statistical period is counted to calculate the first popularity. The current interactive question statistical period is divided into several local time periods. The rate of change of interaction frequency in each local time period is statistically analyzed to calculate the second popularity. The first and second popularity metrics are positively merged to generate the comprehensive popularity index.

6. The method for live streaming interaction of digital humans based on multilingual dynamic driving as described in claim 5, characterized in that, The comprehensive popularity index is the product of the first popularity index and the second popularity index.

7. The method for live streaming interaction of digital humans based on multilingual dynamic driving as described in claim 1, characterized in that, The selection of target questions with interactive value includes: All representative interactive topics are sorted from high to low according to the comprehensive popularity index, and the representative interactive topics with the first preset number of comprehensive popularity indexes are selected as target questions with interactive value.

8. The method for live streaming interaction of digital humans based on multilingual dynamic driving as described in claim 1, characterized in that, The generation of the corresponding interactive text includes: Based on the target question in the set of voice interaction questions, the finely tuned intelligent question answering model is controlled to perform autoregressive inference and generate the corresponding interactive text.

9. The method for live streaming interaction of digital humans based on multilingual dynamic driving as described in claim 4, characterized in that, The generation of multilingual interactive text includes: Analyze the proportion of each language information in the cluster to which the target question belongs; The intelligent question-answering model is controlled to alternately use the corresponding language to generate response statements according to the stated proportion.

10. The method for live streaming interaction of digital humans based on multilingual dynamic driving as described in claim 1, characterized in that, The video stream generated based on interactive text, including voice and digital human motion, includes: The interactive text is input into the text-to-speech module to synthesize an audio stream that matches the digital human image; The acoustic features of the audio stream are extracted to drive the digital human to generate video frames of facial expressions and body movements synchronized with lip movements; The audio stream and video frames are time-aligned and rendered to obtain a digital human motion video stream.

Images