Digital human explanation and display control method based on voice recognition driving

Abstract

The invention discloses a digital human explanation and display control method based on voice recognition driving, which relates to the technical field of voice recognition and comprises the following steps of: receiving an original voice signal input by a user, preprocessing the original voice signal, and sending the preprocessed voice signal into a voice recognition path and a style analysis path in parallel; generating a user question text and a real style embedding vector; integrating the structured knowledge data, the prospective explanation theme and the matching style parameter set through a multi-modal fusion mechanism, generating an explanation text sequence, and synthesizing a voice audio stream; and inputting the voice audio stream and the matching style parameter set into a digital person for driving, generating a non-linguistic behavior sequence, rendering the non-linguistic behavior sequence in real time through a content synchronization mechanism and a multi-terminal linkage protocol, and outputting a digital person explanation video stream through display trigger logic. According to the method, voice recognition and style analysis are processed in parallel, and a dynamic feature interaction gating mechanism is adopted, so that deep optimization of semantic content and style features in voice signals is realized.

Description

Technical Field

[0001] This invention relates to the field of speech recognition technology, and in particular to a speech recognition-driven method for controlling the explanation and demonstration of digital humans. Background Technology

[0002] With the rapid development of artificial intelligence technology, speech recognition can now achieve high-accuracy speech-to-text conversion. Meanwhile, digital human modeling and driving technologies are maturing, capable of generating realistic virtual avatars and performing basic interactive functions such as lip-syncing and facial expression display. In applications such as intelligent explanation and virtual tours, voice-interactive digital humans are gradually replacing traditional human guides. By combining knowledge base retrieval and natural language generation technologies, a certain degree of adaptive content output can be achieved.

[0003] However, existing digital human explanations fall short in terms of naturalness and personalization in voice interaction. Speech recognition and style analysis are often treated as independent, sequential tasks, processing only the recognized text content while ignoring the rich paralinguistic information (such as prosody, rhythm, and emotional tone) within the speech signal and its inherent connection to semantic content. This results in a lack of deep synergy between the generated digital human's behavior and the speech content, making it difficult to achieve a personalized interactive experience that matches the user's style, and rendering the explanation process mechanical and template-based. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a speech recognition-driven digital human explanation and demonstration control method to solve the problems of unnatural interaction and lack of personalized adaptation caused by the separation of speech recognition and style analysis in existing digital human explanations.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: This invention provides a speech recognition-driven digital human explanation and display control method, which includes: receiving raw speech signals input by the user and preprocessing them; sending the preprocessed speech signals in parallel to a speech recognition path and a style analysis path to generate user question text and a real style embedding vector; concatenating the user question text with historical dialogue text records to form a dialogue context sequence; analyzing the logical connections and intent classification in the dialogue context sequence through a causal inference model to output a prospective explanation topic; inputting the prospective explanation topic into a historical knowledge base for retrieval to obtain structured knowledge data, while simultaneously inputting the real style embedding vector into a digital human explanation style generator to generate a digital human explanation style parameter set; performing adversarial matching between the digital human explanation style parameter set and a digital human explanation style discriminator to output a matched style parameter set; integrating the structured knowledge data, the prospective explanation topic, and the matched style parameter set through a multimodal fusion mechanism to generate an explanation text sequence and synthesizing a speech audio stream; inputting the speech audio stream and the matched style parameter set into the digital human for driving to generate a non-verbal behavior sequence; rendering the non-verbal behavior sequence in real time through a content synchronization mechanism and a multi-terminal linkage protocol; and outputting a digital human explanation video stream through a display trigger logic.

[0007] As a preferred embodiment of the speech recognition-driven digital human explanation and demonstration control method of the present invention, the specific steps for generating user question text and realistic style embedding vectors are as follows: The original speech signal is denoised, pre-emphasized, and framed and windowed to generate a pre-processed speech signal. The deep feature extraction network in the speech recognition model is used to extract multi-scale acoustic features from the preprocessed speech signal and output a high-dimensional feature tensor. The high-dimensional feature tensor is input in parallel into the encoder of the speech recognition path and the encoder of the style analysis path to separate the content features and style features, and output the content feature sequence and style feature sequence. The content feature sequence and style feature sequence are collaboratively optimized by the dynamic feature interaction gating mechanism in the speech recognition model to generate the collaboratively optimized content feature sequence and style feature sequence. The co-optimized content feature sequence is input into the decoder of the speech recognition path to be converted into user question text. The co-optimized style feature sequence is input into the feature aggregation layer of the style analysis path to generate a true style embedding vector.

[0008] As a preferred embodiment of the speech recognition-driven digital human explanation and demonstration control method of the present invention, the specific steps for outputting the forward-looking explanation topic are as follows: The user's question text and historical dialogue text records are encoded and concatenated in chronological order to generate a dialogue context sequence. The dialogue context sequence is input into the causal inference model for intent classification and latent causal graph structure learning, generating a latent causal graph adjacency matrix; The PageRank algorithm is applied to calculate the PageRank center score of each topic node in the adjacency matrix of the latent causal graph; The topic nodes are sorted according to their PageRank center scores, and the topic node with the highest score is selected as the topic for forward-looking explanation.

[0009] As a preferred embodiment of the speech recognition-driven digital human explanation and demonstration control method of the present invention, the specific steps for acquiring structured knowledge data are as follows: The forward-looking topic is converted into a high-dimensional query vector using a language model, and the relevance score between the high-dimensional query vector and the knowledge vector in the historical knowledge base is calculated. Based on relevance scores, the most relevant knowledge points are selected from the historical knowledge base, and the corresponding structured knowledge data is retrieved.

[0010] As a preferred embodiment of the speech recognition-driven digital human explanation and demonstration control method of the present invention, the specific steps for generating the digital human explanation style parameter set are as follows: The real style is embedded into a vector and input into the digital human to explain the style generator. The latent style distribution is learned through the built-in encoder network to generate a latent style vector. The latent style vectors are input into the built-in decoder network to generate a set of digital human explanation style parameters.

[0011] As a preferred embodiment of the speech recognition-driven digital human explanation and demonstration control method of the present invention, the specific steps for outputting the style parameter set are as follows: The digital human's narration style parameter set and the real style are embedded into vector inputs to a deep neural network discriminator to calculate the distribution difference between the narration style and the real style; Based on the distribution differences, the parameter set of the digital human's explanation style is optimized in reverse, and the discrimination ability of the deep neural network discriminator is updated synchronously. When the deep neural network discriminator cannot effectively distinguish between the real style and the narration style, it outputs an optimized set of digital human narration style parameters as the matching style parameter set.

[0012] As a preferred embodiment of the speech recognition-driven digital human explanation and demonstration control method of the present invention, the specific steps for generating the explanation text sequence and synthesizing the speech audio stream are as follows: By using a content generation model, structured knowledge data and forward-looking explanation topics are dynamically synthesized to generate explanation content; By using a cross-modal attention mechanism, the content being explained is deeply integrated with the matching style parameter set to generate weighted stylistic features; The weighted stylized features are decoded to generate a sequence of explanatory text. The explanatory text sequence is then processed by neural speech synthesis to synthesize a speech audio stream.

[0013] As a preferred embodiment of the speech recognition-driven digital human explanation and demonstration control method of the present invention, the specific steps for generating the non-verbal behavior sequence are as follows: Extract prosodic feature vectors from speech audio streams and extract non-linguistic behavior sequence style parameters from a set of matching style parameters; Behavioral sequence modeling is performed on prosodic feature vectors and non-verbal behavior sequence style parameters to generate non-verbal behavior sequences.

[0014] As a preferred embodiment of the speech recognition-driven digital human explanation and display control method of the present invention, the non-verbal behavior sequence includes lip-syncing animation sequence, facial expression sequence and body movement sequence.

[0015] As a preferred embodiment of the speech recognition-driven digital human explanation and demonstration control method of the present invention, the steps of rendering non-verbal behavior sequences in real time through a content synchronization mechanism and a multi-terminal linkage protocol, and outputting a digital human explanation video stream through a display trigger logic, are as follows: The non-verbal behavior sequences are dynamically prioritized and synchronized using a content synchronization mechanism to generate synchronization instructions. Synchronization instructions are distributed to each rendering terminal through a multi-terminal linkage protocol to render non-verbal behavior sequences in real time and generate digital human frame sequences. By encapsulating the digital human frame sequence and audio stream through the display trigger logic, a digital human narration video stream is output.

[0016] The beneficial effects of this invention are as follows: By parallel processing of speech recognition and style analysis, and by adopting a dynamic feature interaction gating mechanism, deep optimization of semantic content and style features in speech signals is achieved; accurate text information and subtle style embedding vectors can be extracted synchronously from the same speech source, rather than simple concatenation processing; through dynamic interaction and coupling optimization at the feature level, the generated style vectors not only retain the user's unique prosody and rhythm and other paralinguistic features, but also form an intrinsic connection with the semantic content, providing a highly adaptable personalized parameter foundation for digital human-driven operation, and improving the naturalness, emotional expressiveness and user style matching of digital human explanation behavior. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart for a voice recognition-driven digital human explanation and demonstration control method.

[0019] Figure 2 A flowchart for generating user question text and realistic style embedding vectors.

[0020] Figure 3 A flowchart for outputting forward-looking explanations of the topic.

[0021] Figure 4 This is a flowchart for outputting the matching style parameter set. Detailed Implementation

[0022] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0023] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0024] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0025] Reference Figures 1-4 This is one embodiment of the present invention, which provides a voice recognition-driven digital human explanation and demonstration control method, including the following steps: S1. Receive the raw speech signal input by the user and preprocess it. Then, send the preprocessed speech signal in parallel into the speech recognition path and style analysis path to generate the user's question text and the real style embedding vector.

[0026] S1.1: Perform noise reduction, pre-emphasis, and frame windowing on the original speech signal to generate a preprocessed speech signal; It should be noted that the original speech signal includes fundamental frequency information, formant structure, spectral envelope, temporal energy distribution, temporal structure, and noise components; Fundamental frequency information reflects the pitch of speech and corresponds to the frequency of vocal cord vibration; formant structure reflects the characteristics of the articulatory cavity and determines the timbre of vowels; spectral envelope reflects the spectral characteristics of speech; temporal energy distribution reflects the loudness and intensity changes of speech; temporal structure includes phoneme boundaries, syllable duration, phoneme duration, word boundaries, prosodic word duration, pause intervals, and prosodic boundary features; noise components include environmental noise, equipment circuit noise, transmission channel noise, airborne noise, power supply interference signals, and electromagnetic interference signals.

[0027] Specifically, spectral subtraction is applied to the original speech signal for noise reduction. Spectral subtraction involves determining the noise power spectrum in the silent segments of the original speech signal, subtracting the noise power spectrum from the power spectrum of the noisy speech signal, reconstructing the original speech signal through inverse Fourier transform to generate a denoised speech signal, applying a first-order high-pass filter to the denoised speech signal for pre-emphasis, enhancing high-frequency components to generate a pre-emphasized speech signal, dividing the pre-emphasized speech signal into a fixed-length frame sequence with overlapping frames, and applying a Hamming window function to weight each frame to suppress the amplitude at both ends of each frame to generate a pre-processed speech signal.

[0028] S1.2: Multi-scale acoustic feature extraction is performed on the preprocessed speech signal through a deep feature extraction network in the speech recognition model, and a high-dimensional feature tensor is output. It should be noted that the pre-training process of the deep feature extraction network is as follows: Pre-training is implemented based on a large-scale historical speech corpus. The original speech signals in the large-scale historical speech corpus undergo noise reduction, pre-emphasis, and frame-segmentation windowing operations to generate pre-trained speech signals. The pre-trained speech signals are fed into the deep feature extraction network in the speech recognition model. In the supervised learning framework of pre-training, the high-dimensional feature tensor generated by the deep feature extraction network is compared with the correct transcribed text corresponding to the speech signal. The internal parameters of the deep feature extraction network are adjusted through optimization algorithms so that the high-dimensional feature tensor can more accurately correspond to the transcribed text, thereby completing the parameter learning of the deep feature extraction network and obtaining the pre-trained deep feature extraction network.

[0029] Specifically, the preprocessed speech signal is input into the deep feature extraction network in the speech recognition model. The deep feature extraction network is based on a one-dimensional convolutional neural network architecture, which contains multiple one-dimensional convolutional layers. Each one-dimensional convolutional layer is equipped with convolutional kernels of different sizes, which can capture acoustic features of different time ranges from the preprocessed speech signal. Each one-dimensional convolutional layer is followed by a ReLU activation function and a pooling layer. The ReLU activation function enables the one-dimensional convolutional neural network to have non-linear characteristics and generate feature maps. The pooling layer compresses the feature dimension of the feature maps. The feature maps generated by multiple one-dimensional convolutional layers are pooled and then enter a fully connected layer. The fully connected layer transforms the pooled feature maps into high-dimensional feature tensors.

[0030] S1.3: Input the high-dimensional feature tensor into the encoder of the speech recognition path and the encoder of the style analysis path in parallel, separate the content features and style features, and output the content feature sequence and style feature sequence. It should be noted that the encoder for the speech recognition path is a neural network component based on the Transformer architecture, specifically designed to handle time-series dependencies in high-dimensional feature tensors; the encoder for the style analysis path is a neural network component based on the Transformer architecture, focusing on handling prosody and timbre features in high-dimensional feature tensors.

[0031] Specifically, the encoder for the speech recognition path is based on the Transformer architecture and uses a self-attention mechanism to model the temporal dependencies in the high-dimensional feature tensor. This self-attention mechanism enables the encoder for the speech recognition path to focus on acoustic patterns related to speech content, thereby deriving a content feature sequence. Similarly, the encoder for the style analysis path is also based on the Transformer architecture and uses a self-attention mechanism to model the prosody and timbre features in the high-dimensional feature tensor. This self-attention mechanism enables the encoder for the style analysis path to focus on acoustic attributes related to speaking style, thereby deriving a style feature sequence.

[0032] It should be noted that the content feature sequence mainly carries the textual information of speech, while the style feature sequence mainly carries the individual characteristics of speech.

[0033] S1.4: The content feature sequence and style feature sequence are jointly optimized through the dynamic feature interaction gating mechanism in the speech recognition model to generate the jointly optimized content feature sequence and the jointly optimized style feature sequence; Specifically, the dynamic feature interaction gating mechanism in the speech recognition model is based on a gated loop structure, which includes an update gate and a reset gate. The update gate generates information proportion weights based on the current states of the content feature sequence and the style feature sequence. These information proportion weights are used to adjust the proportion of information flowing from the style feature sequence to the content feature sequence. The reset gate generates influence weights based on the current states of the content feature sequence and the style feature sequence. These influence weights are used to control the degree of influence of the style feature sequence on the formation of new states in the content feature sequence. Through the interaction of the update gate and the reset gate, the content feature sequence absorbs relevant prosodic information from the style feature sequence, while the style feature sequence is adjusted with reference to the contextual structure (the correlation and dependency between features at different time points) in the content feature sequence. The interaction process generates a co-optimized content feature sequence and a co-optimized style feature sequence.

[0034] S1.5: Input the co-optimized content feature sequence into the decoder of the speech recognition path to convert it into user question text, and input the co-optimized style feature sequence into the feature aggregation layer of the style analysis path to generate a true style embedding vector.

[0035] Specifically, the co-optimized content feature sequence is input into the decoder of the speech recognition path. The decoder of the speech recognition path is based on the Transformer architecture and consists of multiple stacked decoding layers. Each decoding layer contains a self-attention mechanism and a feedforward network. The self-attention mechanism models the relationships between elements in the content feature sequence, and the feedforward network enhances the feature representation. The decoder focuses on the internal dependencies of the sequence through the self-attention mechanism and associates the content features with the encoder-decoder attention mechanism to gradually generate the word probability distribution. Through sequence generation methods (such as greedy word selection), the user's question text is formed. At the same time, the co-optimized style feature sequence is input into the feature aggregation layer of the style analysis path. The feature aggregation layer of the style analysis path is a global average pooling layer. The global average pooling layer integrates all data points of the style feature sequence in the time dimension and transforms the style feature sequence into a dense vector of fixed dimensions. The dense vector is the true style embedding vector.

[0036] S2. The user's question text is concatenated with the historical dialogue text records to form a dialogue context sequence. The logical connections in the dialogue context sequence are analyzed through a causal inference model, and a forward-looking explanation topic is output.

[0037] S2.1: Encode the user's question text and historical dialogue text records and concatenate them in chronological order to generate a dialogue context sequence; Specifically, by querying each word in the user's question text and historical dialogue text records through a preset word embedding table, word vectors of the question text and dialogue text records are obtained. The word vectors of the question text and dialogue text records are then input into a bidirectional recurrent neural network structure. The bidirectional recurrent neural network structure scans the word vectors in two directions: from front to back and from back to front. The final state vectors of the two directions are connected to form semantic vectors of the question text and dialogue text records. The semantic vectors of the dialogue text records are arranged in chronological order of the dialogue, with the semantic vectors of the question text placed after them. These are then connected in sequence to form a dialogue context sequence.

[0038] It should be noted that the preset word embedding table refers to a pre-built data structure that stores a one-to-one correspondence between words and fixed-dimensional real vectors.

[0039] S2.2: Input the dialogue context sequence into the causal inference model to perform intent classification and latent causal graph structure learning, and generate the latent causal graph adjacency matrix; It should be noted that the pre-training process of the causal inference model is as follows: Historical dialogue context sequences are used to pre-train the causal inference model. During pre-training, the causal inference model, based on the NOTEARS method, learns the causal dependencies between topic nodes in the historical dialogue context sequences through an optimization mechanism. The optimization mechanism aims to ensure that the generated latent causal graph adjacency matrix can both fit the historical dialogue context sequences well and satisfy the structural constraints of a directed acyclic graph. Pre-training exposes the causal inference model to a large number of historical dialogue context sequences iteratively, gradually adjusting the internal weights of the causal inference to ensure that the generated latent causal graph adjacency matrix can accurately capture the logical connections in the dialogue. After pre-training is completed, a trained causal inference model is obtained.

[0040] Specifically, the causal inference model derives intent categories from the dialogue context sequence. The intent categories are combined with each semantic vector in the dialogue context sequence to generate a set of intent semantic vectors. The causal inference model maps each intent semantic vector in the set of intent semantic vectors to a topic node. The topic node is associated with the intent category. Based on the statistical dependencies between topic nodes and the intent category, the causal inference model constructs a directed graph structure. The directed edges between topic nodes in the directed graph represent the potential causal influence direction. The connection strength between topic nodes in the directed graph is represented by the corresponding weight value. All weight values ​​are organized in a matrix, where the rows and columns of the matrix correspond to topic nodes. The value at each position in the matrix represents the causal influence strength weight from the corresponding node in the row to the corresponding node in the column, forming a potential causal graph adjacency matrix.

[0041] It should be noted that the dialogue context sequence refers to a structured text sequence used to capture complete contextual information of the dialogue, including the current user question and historical dialogue background; Intent categories refer to the classification results of users' potential motivations or knowledge needs, which are used to guide the generation of forward-looking explanation topics, including users' knowledge query intent, comparison intent, and explanation request intent.

[0042] S2.3: Calculate the PageRank center score of each topic node in the adjacency matrix of the latent causal graph using the PageRank algorithm. The expression is:

[0043] In the formula, Indicates the current topic node PageRank center score, Indicates the index of the current topic node. Indicates the damping coefficient. Indicates the total number of topic nodes. Indicates the index of other topic nodes. Represents the adjacency matrix of the latent causal graph From other topic nodes To the current topic node The strength of causal influence Indicates other topic nodes PageRank center score, Indicates other topic nodes All topic node indexes outside of, Represents the adjacency matrix of the latent causal graph From other topic nodes To other topic nodes The causal influence strength of all topic nodes outside of the main topic.

[0044] It should be noted that the damping coefficient comes from the original concept of the PageRank algorithm. The example value is 0.85, which is based on the fact that it can better simulate the behavior pattern of web users who follow links with an 85% probability and randomly jump to new pages with a 15% probability, thereby ensuring the stability and rationality of the score calculation.

[0045] S2.4: Sort each topic node according to the PageRank center score, and select the topic node with the highest score as the topic for forward-looking explanation.

[0046] Specifically, by comparing the PageRank center scores of all topic nodes, all topic nodes are arranged in descending order of score to form an ordered list. In the ordered list, the topic node at the top is the topic node with the highest score. The semantic content represented by the topic node originates from the topic information contained in the dialogue context sequence and is directly identified as the forward-looking topic for explanation.

[0047] It should be noted that topic information refers to conceptual descriptions derived from the semantic vectors of the dialogue context sequence that can characterize the core content or focus of the discussion.

[0048] A superior approach analyzes the logical connections in the dialogue context sequence through a causal inference model, outputting forward-looking explanation topics. This contrasts with conventional topic recommendation methods based on keyword matching (such as TF-IDF) or shallow semantic similarity (such as Word2Vec cosine similarity). Conventional techniques can only match topics based on word co-occurrence or static semantic distance, and cannot parse the causal logic between questions in the dialogue. By constructing a latent causal graph and identifying core topic nodes, logically related unmentioned topics can be inferred from the evolution of the dialogue, making the explanation topic recommendation logically forward-looking and improving the coherence and depth of the explanation content.

[0049] S3. Input the forward-looking explanation topic into the historical knowledge base for retrieval to obtain structured knowledge data. At the same time, embed the real style into the vector and input it into the digital human explanation style generator to generate a digital human explanation style parameter set.

[0050] S3.1: Convert the forward-looking topic into a high-dimensional query vector using a language model, and calculate the relevance score between the high-dimensional query vector and the knowledge vector in the historical knowledge base; It should be noted that the pre-training process of the language model is as follows: Text sentences from historical and forward-looking explanatory topics are randomly masked, i.e., parts of the words are randomly covered to form incomplete sentences. These masked sentences are then input into the Transformer encoder architecture. The goal of pre-training is to predict the correct masked words through the encoder's understanding of the context. During this process, the encoder adjusts its internal weights, learns the contextual relationships and semantic rules between words, and gradually acquires the ability to convert text sequences into high-quality vector representations by exposing them to a large number of historical and forward-looking explanatory topics and completing prediction tasks. After pre-training is complete, a pre-trained language model is obtained.

[0051] Specifically, the prospective topic is input into the language model, which is based on the Transformer encoder architecture. The encoder architecture consists of multiple stacked encoding layers. Each encoding layer contains a self-attention mechanism and a feedforward network. The self-attention mechanism models the word relationships in the prospective topic, and the feedforward network enhances the feature representation and generates a word sequence. Multiple encoding layers transform the word sequence layer by layer, and the last encoding layer generates a fixed-dimensional vector representation, which is the high-dimensional query vector.

[0052] The relevance score between the high-dimensional query vector and the knowledge vectors in the historical knowledge base is calculated using the following expression:

[0053] In the formula, Representing a high-dimensional query vector With the first in the historical knowledge base The relevance score of the knowledge vector corresponding to each knowledge point; Represents a high-dimensional query vector. Indicates the first in the historical knowledge base The knowledge vector corresponding to each knowledge point Representing a high-dimensional query vector The Euclidean norm, Indicates the first in the historical knowledge base The Euclidean norm of the knowledge vector corresponding to each knowledge point.

[0054] S3.2: Based on the relevance score, select the most relevant knowledge points from the historical knowledge base and retrieve the structured knowledge data corresponding to the knowledge points; Specifically, the knowledge vectors corresponding to all knowledge points are sorted in descending order according to their respective relevance scores to form an ordered relevance list. The knowledge point ranked first in the ordered relevance list, that is, the knowledge point with the highest relevance score, is determined as the most relevant knowledge point. Based on the unique identifier of the most relevant knowledge point in the historical knowledge base, the storage record corresponding to the most relevant knowledge point is located and accessed, and the associated structured knowledge data is extracted from the storage record.

[0055] S3.3: Embed the real style into the vector input of the digital human to explain the style generator, learn the latent style distribution through the built-in encoder network, and generate the latent style vector; Specifically, the built-in encoder network of the digital human style generator is used to input the real style embedding vector. The built-in encoder network consists of multiple fully connected layers connected in sequence. The first fully connected layer transforms the dimension of the real style embedding vector and enhances its robustness through the ReLU activation function. This process is repeated in the remaining fully connected layers and ReLU activation function. Each layer transforms and enhances the features of the real style embedding vector. After the last fully connected layer, a new vector with fixed dimensions is generated, which is the latent style vector.

[0056] S3.4: Input the latent style vector into the built-in decoder network to generate a set of digital human explanation style parameters.

[0057] Specifically, the latent style vector is input into the built-in decoder network of the digital human narration style generator. The built-in decoder network consists of multiple fully connected layers connected sequentially. The first fully connected layer transforms the dimension of the latent style vector, and the robustness of the latent style vector is enhanced by the ReLU activation function. This process is repeated in the remaining fully connected layers and the ReLU activation function. Each layer transforms and enhances the features of the latent style vector. After the last fully connected layer, a set of parameters including speech rate, tone, facial expression intensity, and gesture frequency is generated, which is the digital human narration style parameter set.

[0058] It should be noted that the encoder-decoder collaborative mechanism compresses the real style embedding vector into a latent style vector through the encoder network, thereby achieving feature abstraction and dimensionality reduction, reducing data noise and redundancy, and avoiding overfitting to specific details in the training data; the decoder network reconstructs parameters based on the latent style vector to generate specific behavioral parameters such as speech rate and tone, ensuring a smooth transformation of style features from abstract representation to specific output. By preserving essential features and enhancing generalization ability, it improves the accuracy and personalized adaptability of parameter generation, supporting highly natural and consistent digital human narration behavior.

[0059] S4. Perform adversarial matching between the digital human narration style parameter set and the digital human narration style discriminator, and output the matching style parameter set.

[0060] S4.1: Embed the digital human's narration style parameter set and the real style into a vector input deep neural network discriminator, and calculate the distribution difference between the narration style and the real style. The expression is:

[0061] In the formula, Indicating the distribution of explanation styles Distribution of real styles The distribution differences between them Indicates the true style distribution. This indicates the distribution of teaching styles. Indicating the distribution of explanation styles Distribution of real styles The intermediate distribution, Represents the true style distribution With intermediate distribution KL divergence between them Indicating the distribution of explanation styles With intermediate distribution The KL divergence between them.

[0062] It should be noted that the narration style distribution is obtained by probabilistically modeling the narration style parameter set of the digital human, transforming the narration style parameter set of the digital human into a statistical distribution that reflects the characteristics of the generated style; the real style distribution is obtained by probabilistically representing the real style embedding vector, generating a statistical distribution that represents the user's real speech style; the intermediate distribution is obtained by arithmetically averaging the real style distribution and the narration style distribution, generating a balanced reference distribution.

[0063] S4.2: Optimize the digital human explanation style parameter set in reverse based on the distribution differences, and update the discrimination ability of the deep neural network discriminator in sync; Specifically, based on the distribution differences, the values ​​of the digital human's narration style parameter set are adjusted to reduce the numerical value of the distribution differences. At the same time, the connection weights inside the deep neural network discriminator are also adaptively updated to enhance the discriminator's ability to distinguish between true and false styles. The adjustment of the digital human's narration style parameter set and the update of the deep neural network discriminator's weights alternate. In each round of alternation, the digital human's narration style parameter set is first fixed, and the weights of the deep neural network discriminator are updated based on the discriminator's judgment. The weights of the deep neural network discriminator are then fixed, and the digital human's narration style parameter set is adjusted according to the current distribution differences. This alternating optimization continues, allowing the digital human's narration style parameter set to gradually approach the true style distribution, while the discriminative ability of the deep neural network discriminator is also improved in tandem.

[0064] S4.3: When the deep neural network discriminator cannot effectively distinguish between the real style and the narration style, the optimized digital human narration style parameter set is output as the matching style parameter set.

[0065] Specifically, the alternating optimization process continues. The accuracy of the deep neural network discriminator in judging the real style embedding vector and the digital human narration style parameter set is continuously evaluated. When the accuracy of the deep neural network discriminator drops to the preset accuracy threshold, it indicates that the deep neural network discriminator can no longer effectively distinguish between the real style and the narration style. At this time, the optimization process terminates, and the digital human narration style parameter set obtained after the current round of optimization is the matching style parameter set.

[0066] It should be noted that the preset accuracy threshold is set based on the theoretical failure point concept when the discriminator cannot effectively distinguish between real style and generated style during adversarial training. The value ranges from 50% to 60%, and the basis for this value is that 50% accuracy is used as the benchmark for completely random guessing, while 60% accuracy is used as the critical value at which the discriminant ability begins to fail, so as to ensure the optimization balance of the generated style parameter set. Below 50% will cause the discriminator to completely fail and the generation quality to get out of control, while above 60% will cause the training to terminate prematurely and result in insufficient optimization of the generated style parameter set.

[0067] A superior approach optimizes the digital human's narration style parameter set through an adversarial matching mechanism. In contrast to conventional methods based on preset rules or static style templates, conventional techniques (such as matching with a predefined style parameter template library) generate fixed and discrete style parameters that are difficult to adapt to continuously changing real speaking styles. Through the dynamic game between the digital human narration style generator and the discriminator, the generated style parameter set is made to infinitely approximate the real style embedding vector in terms of distribution, thereby producing a more expressive and highly personalized narration style. This solves the problem of overly stiff styles and differences from real human styles in traditional methods.

[0068] S5. By integrating structured knowledge data, forward-looking explanation topics, and matching style parameter sets through a multimodal fusion mechanism, a sequence of explanation texts is generated, and a speech audio stream is synthesized.

[0069] S5.1: Dynamically synthesize structured knowledge data and forward-looking explanation topics through a content generation model to generate explanation content; It should be noted that the pre-training process of the content generation model is as follows: Historical structured knowledge data and historical forward-looking explanation topics are subjected to random masking, i.e., partially masking words to form incomplete sentence sequences. These masked sentence sequences are input into a content generation model based on the Transformer architecture. The encoder part of the content generation model encodes the context of the sentence sequence, while the decoder part attempts to predict the correct masked words based on the encoded context information. The content generation model learns the semantic relationships and grammatical structures between words, enabling it to generate coherent text based on a given topic and contextual knowledge. This allows for dynamic content synthesis of structured knowledge data and forward-looking explanation topics, thus obtaining a well-trained content generation model.

[0070] Specifically, structured knowledge data and prospective explanation topics are input into the content generation model. The content generation model is based on the Transformer architecture, which includes an encoder and a decoder. The structured knowledge data is encoded by the encoder to generate a context vector. The prospective explanation topic is used as the starting signal and is input into the decoder along with the context vector generated by the encoder. The decoder focuses on the generated content through a self-attention mechanism and dynamically integrates relevant knowledge information in the context vector through an encoder-decoder attention mechanism to generate explanation content word by word.

[0071] S5.2: Deeply integrate the content being explained with the matching style parameter set through a cross-modal attention mechanism to generate weighted stylistic features; Specifically, the text representation of the explanatory content is used as the query vector, and the matching style parameter set is used as both the key vector and the value vector. The cross-modal attention mechanism generates a set of attention weights by comparing the similarity between the query vector and the key vector. This set of attention weights is used for the value vector, i.e., the matching style parameter set, and weighted combination. The result of the weighted combination is a new feature representation. The new feature representation contains both the original explanatory content and the weighted style features. The new feature representation is the weighted stylized feature.

[0072] S5.3: Decode the weighted stylized features to generate a sequence of explanatory texts, and then perform neural speech synthesis on the explanatory text sequence to synthesize it into a speech audio stream.

[0073] Specifically, weighted stylistic features are input into a text decoder based on the Transformer architecture. The text decoder transforms the weighted stylistic features through a self-attention mechanism. The self-attention mechanism focuses on the relationships between different parts within a feature and generates words one by one in an autoregressive manner based on the relationships between different parts. The words are combined in sequence to form a narration text sequence. The narration text sequence is passed to a neural speech synthesizer based on a sequence-to-sequence architecture. The neural speech synthesizer converts the narration text sequence into an intermediate acoustic representation, namely a Mel spectrogram. The Mel spectrogram is converted into a time-domain waveform signal by a waveform generator. The time-domain waveform signal drives the speaker to vibrate, generating a speech audio stream.

[0074] S6. Input the voice audio stream and matching style parameter set into the digital human to drive it, generate a non-verbal behavior sequence, render the non-verbal behavior sequence in real time through a content synchronization mechanism and a multi-terminal linkage protocol, and output the digital human's narration video stream through the display trigger logic.

[0075] S6.1: Nonverbal behavior sequences include lip-syncing sequences, facial expression sequences, and body movement sequences.

[0076] It should be noted that lip-sync animation sequence refers to the visual movement of the mouth area of ​​a digital human face opening, closing, and changing shape in sync with the speech content; expression sequence refers to the facial muscle movements of the digital human face to express emotions such as joy, anger, sorrow, and happiness; and body movement sequence refers to the posture and movement of the digital human's head, torso, arms, and hands.

[0077] S6.2: Extract prosodic feature vectors from the speech audio stream and extract non-linguistic behavior style parameters from the matching style parameter set; Specifically, the speech audio stream undergoes a short-time Fourier transform to obtain a spectrogram. Fundamental frequency contour, energy envelope, and syllable duration information are extracted from the spectrogram and combined into a prosodic feature vector. The matching style parameter set includes parameters such as facial expression intensity and gesture frequency. Parameters from the matching style parameter set are directly selected as nonverbal behavior style parameters.

[0078] S6.3: Perform behavioral sequence modeling on prosodic feature vectors and nonverbal behavior style parameters to generate nonverbal behavior sequences; It should be noted that the pre-training of the Long Short-Term Memory (LSTM) network is based on a labeled dataset containing phonological prosody and corresponding digital human action sequences. The phonological segments in the labeled dataset are converted into prosodic feature vectors, and the corresponding limb action sequences are converted into nonverbal behavior sequence codes. The prosodic feature vectors and nonverbal behavior sequence codes are concatenated to form a fusion feature vector sequence for pre-training. The goal of pre-training is to predict the correct nonverbal behavior sequence code for the next time step based on the current and historical fusion feature vectors at each time step. In this way, the LSM network learns the temporal mapping rules from fusion features to nonverbal behavior sequences. The LSM network has the ability to generate accurate and coherent nonverbal behavior sequences based on the new fusion feature vectors, thereby obtaining a well-trained LSM network.

[0079] Specifically, the prosodic feature vector is concatenated with the nonverbal behavior style parameters to form a fused feature vector. The fused feature vector is then input into a long short-term memory network. Based on the fused features at the current moment and the historical state of the memory within the long short-term memory network, the long short-term memory network predicts the nonverbal behavior state that the digital human should have at the current moment. The nonverbal behavior state is represented in an encoded form. The long short-term memory network predicts frame by frame in chronological order to generate a nonverbal behavior sequence.

[0080] S6.4: Dynamically prioritize and synchronize non-verbal behavior sequences through a content synchronization mechanism to generate synchronization instructions; It should be noted that the content synchronization mechanism is a coordination method based on dynamic priority synchronization processing. By assigning a real-time rendering order to non-verbal behavior sequences (including lip animation sequences, expression sequences, and body movement sequences), synchronization instructions are generated to ensure the precise alignment of the digital human behavior and the voice audio stream in a multi-terminal environment.

[0081] Specifically, the content synchronization mechanism receives non-verbal behavior sequences. Based on the inherent characteristics of each behavior in the non-verbal behavior sequence (for example, lip animation sequences are often given a higher priority because they need to maintain lip synchronization with the voice audio stream, and expression sequences and body movement sequences are adjusted according to the scene), it determines the dynamic priority synchronization to assign a real-time rendering order. Referring to the timestamps and type tags of the non-verbal behavior sequence, it determines the priority order to ensure that lip animation is rendered first. After completing the assignment of the real-time rendering order and determining the priority order, the content synchronization mechanism forms synchronization instructions. The synchronization instructions include the rendering timing sequence of non-verbal behaviors and terminal coordination information, which are used for multi-terminal linkage to ensure the precise alignment of the rendering of non-verbal behavior sequences and the voice audio stream, and support the smooth display of digital human explanations.

[0082] S6.5: Distribute the synchronization instructions to each rendering terminal through a multi-terminal linkage protocol, and perform real-time rendering on the non-verbal behavior sequence to generate a digital human frame sequence; It should be noted that the multi-terminal linkage protocol is a distributed communication mechanism using the publish-subscribe mode, which is used to reliably distribute synchronization instructions to each rendering terminal to ensure data synchronization and consistency among multiple terminals; the publish-subscribe mode is a distributed communication paradigm based on a message broker, in which the message sender (publisher) classifies messages into specific topics without directly specifying the recipient, and the message recipient (subscriber) asynchronously receives relevant messages by subscribing to the topics of interest, realizing decoupled and scalable multi-terminal data distribution; The rendering terminal is a computing device equipped with hardware-accelerated rendering (such as WebGL), which performs real-time rendering on the non-verbal behavior sequence according to the synchronization instructions to generate a digital human frame sequence.

[0083] Specifically, the multi-terminal linkage protocol receives the synchronization instructions and starts the distribution process. The multi-terminal linkage protocol encapsulates the synchronization instructions into a standardized message packet and broadcasts it to all rendering terminals (such as the large-screen terminal in the control hall and the mobile guide screen terminal) through the publish-subscribe mode. After each rendering terminal receives the standardized message packet, it returns an acknowledgment signal. The multi-terminal linkage protocol verifies the distribution reliability based on the majority confirmation principle to avoid data loss; based on the priority information of the synchronization instructions, the rendering terminal calls hardware-accelerated rendering (such as a WebGL renderer) to perform frame-by-frame rendering on the non-verbal behavior sequence. During the rendering process, the timing sequence of the non-verbal behavior sequence is aligned with the voice audio stream to ensure that the lip animation sequence is processed first to match the voice rhythm, and the rendering terminal generates a digital human frame sequence.

[0084] It should be noted that the digital human frame sequence contains the visual motion state of the digital human, providing a basis for encapsulating the digital human's narration video stream.

[0085] S6.6: By displaying the trigger logic, the digital human frame sequence and the audio stream are encapsulated to output the digital human narration video stream.

[0086] It should be noted that the display triggering logic is a coordination mechanism based on external events (such as user interaction signals or timed triggers). By listening to specific triggering conditions (such as the voice command "start narration"), it automatically encapsulates the digital human frame sequence and audio stream into a standardized video format (such as MP4), outputs the digital human narration video stream, and realizes the synchronous display and on-demand distribution of audiovisual content.

[0087] Specifically, the demonstration trigger logic starts encapsulation based on the digital human frame sequence and the audio stream. The demonstration trigger logic ensures that each frame of the digital human frame sequence is precisely matched with the corresponding time point of the audio stream through a timestamp alignment mechanism (e.g., lip-sync animation frames are synchronized with speech syllables). The video encapsulator is called to integrate the aligned digital human frame sequence and audio stream into a standardized container format (e.g., MP4 format) to support streaming media transmission. During the encapsulation process, the demonstration trigger logic activates the output stage based on external events (such as user interaction signals or timer triggers) to generate a digital human narration video stream.

[0088] It should be noted that the digital human's narration video stream can be distributed to the display device via network protocols. The entire process achieves seamless integration of audiovisual data through the coordination of display trigger logic, ensuring the real-time nature and completeness of the digital human's narration content.

[0089] In summary, this invention achieves deep optimization of semantic content and style features in speech signals by: parallel processing of speech recognition and style analysis, and employing a dynamic feature interaction gating mechanism; it can simultaneously extract accurate text information and nuanced style embedding vectors from the same speech source, rather than simply concatenating them; through dynamic interaction and coupling optimization at the feature level, the generated style vectors not only retain the user's unique prosody and rhythmic features, but also form an intrinsic connection with the semantic content, providing a highly adaptable personalized parameter foundation for digital human-driven operations, and improving the naturalness, emotional expressiveness, and user style matching of digital human narration.

[0090] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A digital human explanation and demonstration control method based on speech recognition, characterized in that: include, The system receives the raw speech signal input by the user and performs preprocessing. The preprocessed speech signal is then fed into the speech recognition path and style analysis path in parallel to generate the user's question text and the real style embedding vector. The user's question text is concatenated with the historical dialogue text record to form a dialogue context sequence. The logical connections and intent classification in the dialogue context sequence are analyzed through a causal inference model to output a forward-looking explanation topic. The forward-looking topic is input into the historical knowledge base for retrieval to obtain structured knowledge data. At the same time, the real style is embedded into the vector and input into the digital human narration style generator to generate a set of digital human narration style parameters. The digital human narration style parameter set is adversarially matched with the digital human narration style discriminator, and the matched style parameter set is output. By integrating structured knowledge data, forward-looking explanation topics, and matching style parameter sets through a multimodal fusion mechanism, a sequence of explanation texts is generated and synthesized into a speech audio stream; The audio stream and matching style parameter set are input into the digital human to drive the generation of non-verbal behavior sequences. The non-verbal behavior sequences are rendered in real time through a content synchronization mechanism and a multi-terminal linkage protocol, and the digital human narrates a video stream by display trigger logic.

2. The digital human explanation and demonstration control method based on speech recognition as described in claim 1, characterized in that: The specific steps for generating the user-generated question text and the realistic style embedding vector are as follows. The original speech signal is denoised, pre-emphasized, and framed windowed to generate a pre-processed speech signal. The deep feature extraction network in the speech recognition model is used to extract multi-scale acoustic features from the preprocessed speech signal and output a high-dimensional feature tensor. The high-dimensional feature tensor is input in parallel into the encoder of the speech recognition path and the encoder of the style analysis path to separate the content features and style features, and output the content feature sequence and style feature sequence. The content feature sequence and style feature sequence are collaboratively optimized by the dynamic feature interaction gating mechanism in the speech recognition model to generate the collaboratively optimized content feature sequence and style feature sequence. The co-optimized content feature sequence is input into the decoder of the speech recognition path to be converted into user question text. The co-optimized style feature sequence is input into the feature aggregation layer of the style analysis path to generate a true style embedding vector.

3. The digital human explanation and demonstration control method based on speech recognition as described in claim 1, characterized in that: The specific steps for providing forward-looking explanations of the topic are as follows. The user's question text and historical dialogue text records are encoded and concatenated in chronological order to generate a dialogue context sequence. The dialogue context sequence is input into the causal inference model for intent classification and latent causal graph structure learning, generating a latent causal graph adjacency matrix; The PageRank algorithm is applied to calculate the PageRank center score of each topic node in the adjacency matrix of the latent causal graph; The topic nodes are sorted according to their PageRank center scores, and the topic node with the highest score is selected as the topic for forward-looking explanation.

4. The digital human explanation and demonstration control method based on speech recognition as described in claim 1, characterized in that: The specific steps for obtaining structured knowledge data are as follows. The forward-looking topic is converted into a high-dimensional query vector using a language model, and the relevance score between the high-dimensional query vector and the knowledge vector in the historical knowledge base is calculated. Based on relevance scores, the most relevant knowledge points are selected from the historical knowledge base, and the corresponding structured knowledge data is retrieved.

5. The digital human explanation and demonstration control method based on speech recognition as described in claim 1, characterized in that: The specific steps for generating the digital human's narration style parameter set are as follows: The real style is embedded into a vector and input into the digital human to explain the style generator. The latent style distribution is learned through the built-in encoder network to generate a latent style vector. The latent style vectors are input into the built-in decoder network to generate a set of digital human explanation style parameters.

6. The digital human explanation and demonstration control method based on speech recognition as described in claim 1, characterized in that: The specific steps for matching the output style parameter set are as follows. The digital human's narration style parameter set and the real style are embedded into vector inputs to a deep neural network discriminator to calculate the distribution difference between the narration style and the real style; Based on the distribution differences, the parameter set of the digital human's explanation style is optimized in reverse, and the discrimination ability of the deep neural network discriminator is updated synchronously. When the deep neural network discriminator cannot effectively distinguish between the real style and the narration style, it outputs an optimized set of digital human narration style parameters as the matching style parameter set.

7. The digital human explanation and demonstration control method based on speech recognition as described in claim 1, characterized in that: The specific steps for generating the explanatory text sequence and synthesizing the speech audio stream are as follows. By using a content generation model, structured knowledge data and forward-looking explanation topics are dynamically synthesized to generate explanation content; By using a cross-modal attention mechanism to deeply integrate the content being explained with the matching style parameter set, weighted stylistic features are generated. The weighted stylized features are decoded to generate a sequence of explanatory text. The explanatory text sequence is then processed by neural speech synthesis to synthesize a speech audio stream.

8. The digital human explanation and demonstration control method based on speech recognition as described in claim 1, characterized in that: The specific steps for generating the non-linguistic behavior sequence are as follows. Extract prosodic feature vectors from speech audio streams and extract non-linguistic behavior style parameters from a set of matching style parameters; Behavioral sequence modeling is performed on prosodic feature vectors and nonverbal behavioral style parameters to generate nonverbal behavioral sequences.

9. The digital human explanation and demonstration control method based on speech recognition as described in claim 8, characterized in that: The nonverbal behavior sequences include lip-syncing animation sequences, facial expression sequences, and body movement sequences.

10. The digital human explanation and demonstration control method based on speech recognition as described in claim 1, characterized in that: The process involves real-time rendering of non-verbal behavior sequences through a content synchronization mechanism and a multi-terminal linkage protocol, followed by outputting a digital human-guided narration video stream via a display trigger logic. The specific steps are as follows: The non-verbal behavior sequences are dynamically prioritized and synchronized using a content synchronization mechanism to generate synchronization instructions. Synchronization instructions are distributed to each rendering terminal through a multi-terminal linkage protocol to render non-verbal behavior sequences in real time and generate digital human frame sequences. By encapsulating the digital human frame sequence and audio stream through the display trigger logic, a digital human narration video stream is output.

Images