Digital human explanation and display control method based on voice recognition driving

Abstract

The application discloses a digital human explanation and display control method based on voice recognition driving, relates to the technical field of voice recognition, and comprises the following steps: receiving original voice signals input by a user and performing pretreatment, feeding the pretreated voice signals into a voice recognition path and a style analysis path in parallel, and generating user question text and a real style embedding vector; integrating structured knowledge data, a forward-looking explanation theme and a matching style parameter set through a multi-modal fusion mechanism, generating an explanation text sequence, and synthesizing a voice audio stream; inputting the voice audio stream and the matching style parameter set into a digital human for driving, generating a non-verbal behavior sequence, real-time rendering the non-verbal behavior sequence through a content synchronization mechanism and a multi-terminal linkage protocol, and outputting a digital human explanation video stream through display triggering logic. The application realizes the deep optimization of semantic content and style features in voice signals by processing voice recognition and style analysis in parallel and adopting a dynamic feature interaction gate mechanism.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech recognition, in particular to a digital human explanation and display control method based on speech recognition driving. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, speech recognition has been able to realize high-accuracy speech-to-text conversion, while digital human modeling and driving technology has also become mature, which can generate realistic virtual images and complete basic interactive functions such as lip synchronization and expression display. In the application scenarios of intelligent explanation and virtual guide, digital humans based on voice interaction gradually replace traditional human explanation, and through the combination of knowledge base retrieval and natural language generation technology, a certain degree of content adaptive output can be realized.

[0003] However, the existing digital human explanation has deficiencies in the naturalness and personalization of voice interaction. Speech recognition and style analysis are usually treated as independent serial tasks, only the recognized text content is processed, and the internal relationship between the rich paralinguistic information (such as prosody, rhythm and emotional color) in the speech signal and the semantic content is ignored, resulting in a lack of depth coordination between the generated digital human behavior and the speech content, making it difficult to achieve personalized interaction experience matching the user's style, and making the explanation process appear mechanical and templated. SUMMARY

[0004] In view of the above existing problems, the present application is proposed.

[0005] Therefore, the present application provides a digital human explanation and display control method based on speech recognition driving to solve the problem of unnatural interaction and lack of personalized adaptation caused by the fragmentation of speech recognition and style analysis in the existing digital human explanation.

[0006] To solve the above technical problems, the present application provides the following technical solutions:

[0007] The application provides a digital human explanation and display control method based on voice recognition driving, which comprises the following steps: receiving original voice signals input by a user and performing preprocessing, sending the preprocessed voice signals into a voice recognition path and a style analysis path in parallel, and generating user question text and a real style embedding vector; splicing the user question text and historical dialogue text records into a dialogue context sequence, analyzing logical association and intent classification in the dialogue context sequence through a causal inference model, and outputting a forward-looking explanation theme; inputting the forward-looking explanation theme into a historical knowledge base for retrieval, obtaining structured knowledge data, and simultaneously inputting the real style embedding vector into a digital human explanation style generator to generate a digital human explanation style parameter set; performing adversarial matching on the digital human explanation style parameter set and a digital human explanation style discriminator to output a matching style parameter set; integrating the structured knowledge data, the forward-looking explanation theme and the matching style parameter set through a multi-modal fusion mechanism to generate an explanation text sequence, and synthesizing a voice audio stream; inputting the voice audio stream and the matching style parameter set into a digital human for driving to generate a non-verbal behavior sequence, and performing real-time rendering on the non-verbal behavior sequence through a content synchronization mechanism and a multi-terminal linkage protocol, and outputting a digital human explanation video stream through display triggering logic.

[0008] As a preferred scheme of the digital human explanation and display control method based on voice recognition driving, the method comprises the following steps:

[0009] The original voice signals are denoised, pre-emphasized and framed and windowed to generate preprocessed voice signals;

[0010] The preprocessed voice signals are subjected to multi-scale acoustic feature extraction through a deep feature extraction network in the voice recognition model to output a high-dimensional feature tensor;

[0011] The high-dimensional feature tensor is input into an encoder of the voice recognition path and an encoder of the style analysis path in parallel to separate content features and style features, and output content feature sequences and style feature sequences;

[0012] The content feature sequences and the style feature sequences are subjected to collaborative optimization through a dynamic feature interaction gating mechanism in the voice recognition model to generate collaboratively optimized content feature sequences and collaboratively optimized style feature sequences;

[0013] The collaboratively optimized content feature sequences are input into a decoder of the voice recognition path to be converted into user question text, and the collaboratively optimized style feature sequences are input into a feature aggregation layer of the style analysis path to generate a real style embedding vector.

[0014] As a preferred scheme of the digital human explanation and display control method based on voice recognition driving provided in the application, the specific steps of outputting the forward-looking explanation theme are as follows,

[0015] The user question text and the historical dialogue text record are encoded and spliced in chronological order to generate a dialogue context sequence.

[0016] The dialogue context sequence is input into a causal inference model to perform intent classification and potential causal graph structure learning, and a potential causal graph adjacency matrix is generated.

[0017] The PageRank center score of each theme node in the potential causal graph adjacency matrix is calculated by applying the PageRank algorithm.

[0018] The theme nodes are sorted according to the PageRank center score, and the theme node with the highest score is selected as the forward-looking explanation theme.

[0019] As a preferred scheme of the digital human explanation and display control method based on voice recognition driving provided in the application, the specific steps of obtaining structured knowledge data are as follows,

[0020] The forward-looking explanation theme is converted into a high-dimensional query vector by a language model, and the relevance score of the high-dimensional query vector and the knowledge vector in the historical knowledge base is calculated.

[0021] The knowledge point with the highest relevance is selected from the historical knowledge base based on the relevance score, and the structured knowledge data corresponding to the knowledge point is retrieved.

[0022] As a preferred scheme of the digital human explanation and display control method based on voice recognition driving provided in the application, the specific steps of generating the digital human explanation style parameter set are as follows,

[0023] The real style embedding vector is input into the digital human explanation style generator, the latent style distribution is learned through the built-in encoder network, and the latent style vector is generated.

[0024] The latent style vector is input into the built-in decoder network to generate the digital human explanation style parameter set.

[0025] As a preferred scheme of the digital human explanation and display control method based on voice recognition driving provided in the application, the specific steps of outputting the matching style parameter set are as follows,

[0026] The digital human explanation style parameter set and the real style embedding vector are input into the deep neural network discriminator to calculate the distribution difference between the explanation style and the real style.

[0027] According to the distribution difference reverse optimization digital human explanation style parameter set, the discrimination ability of the deep neural network discriminator is updated synchronously.

[0028] When the deep neural network discriminator cannot effectively distinguish the real style from the explanation style, the optimized digital human explanation style parameter set is output as the matching style parameter set.

[0029] As a preferred scheme of the digital human explanation and display control method based on speech recognition driving provided in the application, the specific steps of generating the explanation text sequence and synthesizing the speech audio stream are as follows,

[0030] The structured knowledge data and the forward-looking explanation theme are dynamically content-synthesized by the content generation model to generate the explanation content.

[0031] The explanation content and the matching style parameter set are deeply fused by the cross-modal attention mechanism to generate weighted stylized features.

[0032] The weighted stylized features are text-decoded to generate the explanation text sequence, and the explanation text sequence is processed by neural speech synthesis to be synthesized into a speech audio stream.

[0033] As a preferred scheme of the digital human explanation and display control method based on speech recognition driving provided in the application, the specific steps of generating the non-verbal behavior sequence are as follows,

[0034] The prosodic feature vector is extracted from the speech audio stream, and the non-verbal behavior sequence style parameter is extracted from the matching style parameter set.

[0035] The prosodic feature vector and the non-verbal behavior sequence style parameter are modeled into a non-verbal behavior sequence.

[0036] As a preferred scheme of the digital human explanation and display control method based on speech recognition driving provided in the application, the non-verbal behavior sequence includes a mouth shape animation sequence, an expression sequence, and a limb action sequence.

[0037] As a preferred scheme of the digital human explanation and display control method based on speech recognition driving provided in the application, the specific steps of real-time rendering the non-verbal behavior sequence by the content synchronization mechanism and the multi-terminal linkage protocol, and outputting the digital human explanation video stream by the display triggering logic are as follows,

[0038] The non-verbal behavior sequence is dynamically prioritized by the content synchronization mechanism to generate a synchronization instruction.

[0039] The synchronization instruction is distributed to each rendering terminal by the multi-terminal linkage protocol to real-time render the non-verbal behavior sequence and generate a digital human frame sequence.

[0040] The digital human explanation video stream is output by encapsulating the digital human frame sequence with the voice audio stream through the display trigger logic.

[0041] The present application has the beneficial effects that: by processing voice recognition and style analysis in parallel, and using a dynamic feature interaction gating mechanism, the semantic content and style features in the voice signal are deeply optimized; accurate text information and delicate style embedding vectors can be extracted from the same voice source simultaneously, rather than being simply concatenated; through dynamic interaction and coupling optimization at the feature level, the generated style vector not only retains the user's unique prosody and rhythm and other paralanguage features, but also forms an internal correlation with the semantic content, providing a highly adaptive personalized parameter basis for digital human driving, and improving the naturalness, emotional expressiveness and user style matching degree of digital human explanation behavior. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0043] Fig. 1 The flowchart of the digital human explanation and display control method based on voice recognition driving.

[0044] Fig. 2 The flowchart of generating user question text and real style embedding vectors.

[0045] Fig. 3 The flowchart of outputting forward-looking explanation topics.

[0046] Fig. 4 The flowchart of outputting matching style parameter sets. DETAILED DESCRIPTION

[0047] In order to make the above-mentioned purposes, features and advantages of the present application more apparent and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the drawings of the specification.

[0048] In the following description, many specific details are set forth in order to provide a thorough understanding of the present application, but the present application can also be implemented in other ways different from those described herein, and those skilled in the art can make similar generalizations without departing from the connotation of the present application, therefore the present application is not limited to the specific embodiments disclosed below.

[0049] Second, the "one embodiment" or "an embodiment" referred to herein can include a particular feature, structure, or characteristic. The various embodiments appearing at different places in this specification are not necessarily all cumulative or mutually exclusive of each other.

[0050] Referring to Figs. 1-4 For one embodiment of the present application, the embodiment provides a digital human explanation and display control method based on speech recognition driving, comprising the following steps:

[0051] S1, receiving the original speech signal input by the user and pre-processing, the pre-processed speech signal is sent into the speech recognition path and the style analysis path in parallel, and the user question text and the real style embedding vector are generated.

[0052] S1.1: denoising, pre-emphasis and frame windowing are performed on the original speech signal to generate the pre-processed speech signal;

[0053] It should be noted that the original speech signal includes fundamental frequency information, formant structure, spectral envelope, time domain energy distribution, time sequence structure and noise component;

[0054] The fundamental frequency information reflects the pitch of the speech, corresponding to the frequency of vocal cord vibration; the formant structure reflects the characteristics of the sound cavity, determining the timbre of the vowel; the spectral envelope reflects the spectral characteristics of the speech; the time domain energy distribution reflects the loudness and intensity changes of the speech; the time sequence structure includes phoneme boundary, syllable duration, phoneme duration, word boundary, prosodic word duration, pause interval and prosodic boundary features; the noise component includes environmental noise, device circuit noise, transmission channel noise, air conduction noise, power interference signal and electromagnetic interference signal.

[0055] Specifically, the spectral subtraction is applied to the original speech signal for noise reduction, which includes determining the noise power spectrum in the silent segment of the original speech signal, subtracting the noise power spectrum from the power spectrum of the noisy speech signal, reconstructing the original speech signal through inverse Fourier transform, producing the noise-reduced speech signal, applying a first-order high-pass filter to pre-emphasize the noise-reduced speech signal, the first-order high-pass filter enhances the high-frequency component, produces the pre-emphasized speech signal, divides the pre-emphasized speech signal into a fixed time length frame sequence, the frames overlap, applies a Hamming window function to each frame of speech signal for weighting to suppress the amplitude at both ends of each frame of speech signal, and produces the pre-processed speech signal.

[0056] S1.2: multi-scale acoustic feature extraction is performed on the pre-processed speech signal through the deep feature extraction network in the speech recognition model, and a high-dimensional feature tensor is output;

[0057] It should be noted that the pre-training process of the deep feature extraction network is as follows: the pre-training is implemented based on a historical large-scale speech corpus, the original speech signal in the historical large-scale speech corpus is subjected to noise reduction, pre-emphasis and frame windowing operations to generate a pre-trained speech signal, the pre-trained speech signal is sent to the deep feature extraction network in the speech recognition model, in a pre-training supervised learning framework, the high-dimensional feature tensor generated by the deep feature extraction network is compared with the correct transcription text corresponding to the speech signal, the internal parameters of the deep feature extraction network are adjusted through an optimization algorithm, so that the high-dimensional feature tensor can more accurately correspond to the transcription text, thereby completing the parameter learning of the deep feature extraction network and obtaining a pre-trained deep feature extraction network.

[0058] Specifically, the preprocessed speech signal is input into the deep feature extraction network in the speech recognition model, the deep feature extraction network is based on a one-dimensional convolutional neural network architecture, the one-dimensional convolutional neural network comprises a plurality of one-dimensional convolutional layers, the one-dimensional convolutional layers are equipped with convolution kernels of different sizes and can capture acoustic features of different time ranges from the preprocessed speech signal, a ReLU activation function and a pooling layer are connected behind each one-dimensional convolutional layer, the ReLU activation function enables the one-dimensional convolutional neural network to have nonlinear characteristics and generate a feature map, and the pooling layer compresses the feature dimension of the feature map, the feature maps generated by the plurality of one-dimensional convolutional layers are converged and then input into a fully connected layer, and the fully connected layer converts the converged feature maps into a high-dimensional feature tensor.

[0059] S1.3: The high-dimensional feature tensor is input into the encoder of the speech recognition path and the encoder of the style analysis path in parallel to separate the content features and the style features, and output the content feature sequence and the style feature sequence;

[0060] It should be noted that the encoder of the speech recognition path is a neural network component based on the Transformer architecture and is specially used for processing the time sequence dependency relationship in the high-dimensional feature tensor; the encoder of the style analysis path is a neural network component based on the Transformer architecture and is focused on processing the prosody and timbre features in the high-dimensional feature tensor.

[0061] Specifically, the encoder of the speech recognition path is based on the Transformer architecture and models the time sequence dependency relationship in the high-dimensional feature tensor through a self-attention mechanism, the self-attention mechanism enables the encoder of the speech recognition path to focus on the acoustic patterns related to the speech content, thereby deriving the content feature sequence; the encoder of the style analysis path is also based on the Transformer architecture and models the prosody and timbre features in the high-dimensional feature tensor through a self-attention mechanism, the self-attention mechanism enables the encoder of the style analysis path to focus on the acoustic attributes related to the speaking style, thereby deriving the style feature sequence.

[0062] It should be noted that the content feature sequence mainly carries the textual information of speech, while the style feature sequence mainly carries the individual characteristics of speech.

[0063] S1.4: The content feature sequence and style feature sequence are jointly optimized through the dynamic feature interaction gating mechanism in the speech recognition model to generate the jointly optimized content feature sequence and the jointly optimized style feature sequence;

[0064] Specifically, the dynamic feature interaction gating mechanism in the speech recognition model is based on a gated loop structure, which includes an update gate and a reset gate. The update gate generates information proportion weights based on the current states of the content feature sequence and the style feature sequence. These information proportion weights are used to adjust the proportion of information flowing from the style feature sequence to the content feature sequence. The reset gate generates influence weights based on the current states of the content feature sequence and the style feature sequence. These influence weights are used to control the degree of influence of the style feature sequence on the formation of new states in the content feature sequence. Through the interaction of the update gate and the reset gate, the content feature sequence absorbs relevant prosodic information from the style feature sequence, while the style feature sequence is adjusted with reference to the contextual structure (the correlation and dependency between features at different time points) in the content feature sequence. The interaction process generates a co-optimized content feature sequence and a co-optimized style feature sequence.

[0065] S1.5: Input the co-optimized content feature sequence into the decoder of the speech recognition path to convert it into user question text, and input the co-optimized style feature sequence into the feature aggregation layer of the style analysis path to generate a true style embedding vector.

[0066] Specifically, the co-optimized content feature sequence is input into the decoder of the speech recognition path. The decoder of the speech recognition path is based on the Transformer architecture and consists of multiple stacked decoding layers. Each decoding layer contains a self-attention mechanism and a feedforward network. The self-attention mechanism models the relationships between elements in the content feature sequence, and the feedforward network enhances the feature representation. The decoder focuses on the internal dependencies of the sequence through the self-attention mechanism and associates the content features with the encoder-decoder attention mechanism to gradually generate the word probability distribution. Through sequence generation methods (such as greedy word selection), the user's question text is formed. At the same time, the co-optimized style feature sequence is input into the feature aggregation layer of the style analysis path. The feature aggregation layer of the style analysis path is a global average pooling layer. The global average pooling layer integrates all data points of the style feature sequence in the time dimension and transforms the style feature sequence into a dense vector of fixed dimensions. The dense vector is the true style embedding vector.

[0067] S2, splice the user question text and the historical dialogue text record into a dialogue context sequence, analyze the logical association in the dialogue context sequence through a causal inference model, and output a forward-looking explanation theme.

[0068] S2.1: encode the user question text and the historical dialogue text record and splice them in chronological order to generate a dialogue context sequence;

[0069] Specifically, the preset word embedding table is used to query each word in the user question text and the historical dialogue text record to obtain a question text word vector and a dialogue text record word vector. The question text word vector and the dialogue text record word vector are input into a bidirectional recurrent neural network structure. The bidirectional recurrent neural network structure scans the word vector from front to back and from back to front. The final state vectors in the two directions are connected to form a question text semantic vector and a dialogue text record semantic vector. The dialogue text record semantic vector is arranged in chronological order in front, and the question text semantic vector is placed behind the dialogue text record semantic vector. They are sequentially connected and combined into a dialogue context sequence.

[0070] It should be noted that the preset word embedding table refers to a pre-constructed data structure that stores a one-to-one correspondence between words and fixed-dimensional real vectors.

[0071] S2.2: input the dialogue context sequence into the causal inference model for intent classification and potential causal graph structure learning to generate a potential causal graph adjacency matrix;

[0072] It should be noted that the pre-training process of the causal inference model is as follows: the historical dialogue context sequence is called to pre-train the causal inference model. During the pre-training process, the causal inference model learns the causal dependence relationship between the theme nodes in the historical dialogue context sequence based on the NOTEARS method through an optimization mechanism. The optimization mechanism aims to make the generated potential causal graph adjacency matrix not only fit the historical dialogue context sequence well, but also satisfy the structure constraint of a directed acyclic graph. The pre-training exposes the causal inference model to a large number of historical dialogue context sequences in an iterative manner, gradually adjusts the internal weights of the causal inference, and makes the generated potential causal graph adjacency matrix accurately capture the logical association in the dialogue. After the pre-training is completed, a trained causal inference model is obtained.

[0073] Specifically, the causal inference model derives intent categories from the dialogue context sequence. The intent categories are combined with each semantic vector in the dialogue context sequence to generate a set of intent semantic vectors. The causal inference model maps each intent semantic vector in the set of intent semantic vectors to a topic node. The topic node is associated with the intent category. Based on the statistical dependencies between topic nodes and the intent category, the causal inference model constructs a directed graph structure. The directed edges between topic nodes in the directed graph represent the potential causal influence direction. The connection strength between topic nodes in the directed graph is represented by the corresponding weight value. All weight values ​​are organized in a matrix, where the rows and columns of the matrix correspond to topic nodes. The value at each position in the matrix represents the causal influence strength weight from the node corresponding to the row to the node corresponding to the column, forming a potential causal graph adjacency matrix.

[0074] It should be noted that the dialogue context sequence refers to a structured text sequence used to capture complete contextual information of the dialogue, including the current user question and historical dialogue background;

[0075] Intent categories refer to the classification results of users' potential motivations or knowledge needs, which are used to guide the generation of forward-looking explanation topics, including users' knowledge query intent, comparison intent, and explanation request intent.

[0076] S2.3: Calculate the PageRank center score of each topic node in the adjacency matrix of the latent causal graph using the PageRank algorithm. The expression is:

[0077]

[0078] In the formula, Indicates the current topic node PageRank center score, Indicates the index of the current topic node. Indicates the damping coefficient. Indicates the total number of topic nodes. Indicates the index of other topic nodes, Represents the adjacency matrix of the latent causal graph From other topic nodes To the current topic node The strength of causal influence Indicates other topic nodes PageRank center score, Indicates other topic nodes All topic node indexes outside of, Represents the adjacency matrix of the latent causal graph From other topic nodes To other topic nodes The causal influence strength of all topic nodes outside the scope of the study.

[0079] It should be noted that the damping coefficient is derived from the original idea of PageRank algorithm, and the example value is 0.85, which is based on the behavior pattern that network users can better simulate the behavior pattern of following the link browsing with 85% probability and randomly jumping to a new page with 15% probability, so as to ensure the stability and rationality of score calculation.

[0080] S2.4: According to the PageRank central score, the theme nodes are sorted, and the theme node with the highest score is selected as the forward-looking explanation theme.

[0081] Specifically, by comparing the numerical values of the PageRank central scores of all theme nodes, all theme nodes are arranged in order from high to low according to the scores, forming an ordered list, and in the ordered list, the theme node at the top is the theme node with the highest score. The semantic content represented by the theme node, which is derived from the theme information contained in the dialogue context sequence, is directly determined as the forward-looking explanation theme.

[0082] It should be noted that the theme information refers to the conceptual description that can represent the core content or discussion focus of the dialogue derived from the semantic vector of the dialogue context sequence.

[0083] Preferably, the logical association in the dialogue context sequence is analyzed by the causal inference model to output the forward-looking explanation theme. Compared with the conventional theme recommendation method based on keyword matching (such as TF-IDF) or shallow semantic similarity (such as Word2Vec cosine similarity), the conventional technology can only match the theme according to the word co-occurrence or static semantic distance, and cannot analyze the causal logic between the problems in the dialogue. By constructing a latent causal graph and identifying the core theme node, the logically related unmentioned theme can be inferred from the dialogue evolution, so that the explanation theme recommendation has logical foresight, and the coherence and depth of the explanation content are improved.

[0084] S3, input the forward-looking explanation theme into the historical knowledge base for retrieval to obtain structured knowledge data, and input the real style embedding vector into the digital human explanation style generator to generate a set of digital human explanation style parameters.

[0085] S3.1: Convert the forward-looking explanation theme into a high-dimensional query vector through a language model, and calculate the relevance score of the high-dimensional query vector and the knowledge vector in the historical knowledge base;

[0086] It should be noted that the pre-training process of the language model is as follows: randomly mask the text sentences in the historical forward explanation theme, that is, randomly cover part of the words to form incomplete sentences, input the masked sentences into the encoder architecture of the Transformer, the pre-training target is to predict the correct words covered by the encoder through the understanding of the context, in this process, the encoder adjusts the weights inside the encoder, learns the context association and semantic rules between words, through exposure to a large number of historical forward explanation themes and completion of the prediction task, the encoder gradually obtains the ability to convert text sequences into high-quality vector representations. After pre-training, a pre-trained language model is obtained.

[0087] Specifically, the forward explanation theme is input into the language model, and the language model is based on the encoder architecture of the Transformer; the encoder architecture is stacked by multiple encoding layers, each encoding layer contains a self-attention mechanism and a feedforward network, the self-attention mechanism models the relationship between the words in the forward explanation theme, and the feedforward network enhances the feature representation to generate a word sequence, multiple encoding layers convert the word sequence layer by layer, and the last encoding layer generates a fixed-dimensional vector representation, that is, a high-dimensional query vector.

[0088] The correlation score of the high-dimensional query vector and the knowledge vector in the historical knowledge base is calculated, and the expression is:

[0089]

[0090] In the formula, represents the high-dimensional query vector , and the correlation score of the high-dimensional query vector corresponding to the knowledge vector in the historical knowledge base. represents the high-dimensional query vector, represents the knowledge vector corresponding to the knowledge point in the historical knowledge base, represents the high-dimensional query vector , and the Euclidean norm of the high-dimensional query vector . represents the Euclidean norm of the knowledge vector corresponding to the knowledge point in the historical knowledge base.

[0091] S3.2: Select the knowledge point with the highest correlation from the historical knowledge base based on the correlation score, and retrieve the structured knowledge data corresponding to the knowledge point;

[0092] ​Specifically, the knowledge vectors corresponding to all knowledge points are arranged in descending order according to respective correlation score values to form a correlation ordered list, a knowledge point ranked first in the correlation ordered list, i.e., a knowledge point with the highest correlation score value, is determined as a knowledge point with the highest correlation, and a storage record corresponding to the knowledge point with the highest correlation is located and accessed according to a unique identifier of the knowledge point with the highest correlation in the historical knowledge base, and structured knowledge data associated in the storage record is extracted.

[0093] S3.3: input the real style embedding vector into the digital human explanation style generator, learn the latent style distribution through the built-in encoder network, and generate a latent style vector;

[0094] Specifically, the real style embedding vector is input into the built-in encoder network of the digital human explanation style generator, the built-in encoder network is composed of multiple fully connected layers connected in sequence, the dimension of the real style embedding vector is converted through the first fully connected layer, and the robustness of the real style embedding vector is enhanced through the ReLU activation function, and the conversion and enhancement of the features of the real style embedding vector are repeated in the remaining fully connected layers and ReLU activation functions, each layer converts and enhances the features of the real style embedding vector, and after the last fully connected layer, a new and fixed-dimension vector is generated, which is the latent style vector.

[0095] S3.4: input the latent style vector into the built-in decoder network to generate a digital human explanation style parameter set.

[0096] Specifically, the latent style vector is input into the built-in decoder network of the digital human explanation style generator, the built-in decoder network is composed of multiple fully connected layers connected in sequence, the dimension of the latent style vector is converted through the first fully connected layer, and the robustness of the latent style vector is enhanced through the ReLU activation function, and the conversion and enhancement of the features of the latent style vector are repeated in the remaining fully connected layers and ReLU activation functions, each layer converts and enhances the features of the latent style vector, and after the last fully connected layer, a set containing the parameters of speech speed, tone, expression intensity and gesture frequency is generated, which is the digital human explanation style parameter set.

[0097] It should be noted that the encoding-decoding collaborative mechanism compresses the real style embedding vector into the latent style vector through the encoder network, realizes feature abstraction and dimension reduction, thereby reducing data noise and redundancy and avoiding overfitting to specific details in the training data; the decoder network reconstructs parameters based on the latent style vector to generate specific behavior parameters such as speech speed and tone, ensures smooth conversion from abstract representation to specific output of style features, enhances the accuracy and personalized adaptability of parameter generation by preserving essential features and enhancing generalization ability, and supports high naturalization and consistency of digital human explanation behavior.

[0098] S4, perform adversarial matching on the digital human explanation style parameter set and the digital human explanation style discriminator, and output a matching style parameter set.

[0099] S4.1: input the digital human explanation style parameter set and the real style embedding vector into the deep neural network discriminator, calculate the distribution difference between the explanation style and the real style, and the expression is:

[0100]

[0101] In the formula, represents the distribution difference between the explanation style distribution and the real style distribution represents the real style distribution, represents the explanation style distribution, represents the intermediate distribution of the explanation style distribution and the real style distribution represents the KL divergence between the real style distribution and the intermediate distribution represents the KL divergence between the explanation style distribution and the intermediate distribution.

[0102] It should be noted that the explanation style distribution is obtained by probabilistic modeling on the digital human explanation style parameter set, which converts the digital human explanation style parameter set into a statistical distribution reflecting the generated style characteristics; the real style distribution is obtained by probabilistic representation on the real style embedding vector, which generates a statistical distribution representing the user's real voice style; the intermediate distribution is obtained by arithmetic averaging the real style distribution and the explanation style distribution, which generates a balanced reference distribution.

[0103] S4.2: reverse optimize the digital human explanation style parameter set according to the distribution difference, and update the discrimination ability of the deep neural network discriminator synchronously;

[0104] ​​​​Specifically, according to the distribution difference, the numerical value of the digital human explanation style parameter set is adjusted, and the adjustment direction is to reduce the numerical value of the distribution difference. At the same time, the connection weight inside the deep neural network discriminator is also adaptively updated to enhance the ability of the deep neural network discriminator to distinguish the authenticity of the style. The adjustment of the digital human explanation style parameter set and the weight update of the deep neural network discriminator are alternately performed. In each round of alternation, the digital human explanation style parameter set is fixed first, and the weight of the deep neural network discriminator is updated according to the judgment of the deep neural network discriminator. The weight of the deep neural network discriminator is fixed, and the digital human explanation style parameter set is adjusted according to the current distribution difference. The alternating optimization continues to be carried out, so that the digital human explanation style parameter set gradually approaches the real style distribution, and at the same time, the discrimination ability of the deep neural network discriminator is also improved.

[0105] S4.3: When the deep neural network discriminator cannot effectively distinguish the real style from the explanation style, output the optimized digital human explanation style parameter set as the matching style parameter set.

[0106] Specifically, the alternating optimization process continues, and the judgment accuracy of the deep neural network discriminator on the real style embedding vector and the digital human explanation style parameter set is continuously evaluated. When the judgment accuracy of the deep neural network discriminator decreases to a preset accuracy threshold, it indicates that the deep neural network discriminator has been unable to effectively distinguish the real style from the explanation style. At this time, the optimization process is terminated, and the digital human explanation style parameter set obtained after the current round of optimization is the matching style parameter set.

[0107] It should be noted that the preset accuracy threshold is based on the theoretical failure point concept when the discriminator cannot effectively distinguish the real style from the generated style in the adversarial training. The value range is 50% to 60%, and the value basis is to take 50% accuracy as the benchmark for completely random guessing and 60% accuracy as the critical value for the beginning of the failure of discrimination ability, to ensure the optimization balance of the generated style parameter set; below 50% will lead to complete failure of the discriminator and make the generation quality out of control, and above 60% will cause the training to terminate too early and result in insufficient optimization of the generated style parameter set.

[0108] Preferably, the digital human explanation style parameter set is optimized through the adversarial matching mechanism, which is in contrast to the conventional method based on preset rules or static style templates. The generated style parameters of the conventional technology (such as pre-defined style parameter template library matching) are fixed and discrete, which is difficult to adapt to the continuous change of real speaking style. Through the dynamic game between the digital human explanation style generator and the discriminator, the generated style parameter set is infinitely close to the real style embedding vector in distribution, thereby generating a more expressive and highly personalized explanation style, solving the problem of excessive rigidity of traditional methods and the difference between real human style.

[0109] S5. By integrating structured knowledge data, forward-looking explanation topics, and matching style parameter sets through a multimodal fusion mechanism, a sequence of explanation texts is generated, and a speech audio stream is synthesized.

[0110] S5.1: Dynamically synthesize structured knowledge data and forward-looking explanation topics through a content generation model to generate explanation content;

[0111] It should be noted that the pre-training process of the content generation model is as follows: Historical structured knowledge data and historical forward-looking explanation topics are subjected to random masking, i.e., partially masking words to form incomplete sentence sequences. These masked sentence sequences are input into a content generation model based on the Transformer architecture. The encoder part of the content generation model encodes the context of the sentence sequence, while the decoder part attempts to predict the correct masked words based on the encoded context information. The content generation model learns the semantic relationships and grammatical structures between words, enabling it to generate coherent text based on a given topic and contextual knowledge. This allows for dynamic content synthesis of structured knowledge data and forward-looking explanation topics, thus obtaining a well-trained content generation model.

[0112] Specifically, structured knowledge data and prospective explanation topics are input into the content generation model. The content generation model is based on the Transformer architecture, which includes an encoder and a decoder. The structured knowledge data is encoded by the encoder to generate a context vector. The prospective explanation topic is used as the starting signal and is input into the decoder along with the context vector generated by the encoder. The decoder focuses on the generated content through a self-attention mechanism and dynamically integrates relevant knowledge information in the context vector through an encoder-decoder attention mechanism to generate explanation content word by word.

[0113] S5.2: Deeply integrate the content being explained with the matching style parameter set through a cross-modal attention mechanism to generate weighted stylistic features;

[0114] Specifically, the text representation of the explanatory content is used as the query vector, and the matching style parameter set is used as both the key vector and the value vector. The cross-modal attention mechanism generates a set of attention weights by comparing the similarity between the query vector and the key vector. This set of attention weights is used for the value vector, i.e., the matching style parameter set, and weighted combination. The result of the weighted combination is a new feature representation. The new feature representation contains both the original explanatory content and the weighted style features. The new feature representation is the weighted stylized feature.

[0115] S5.3: Decode the weighted stylized features to generate a sequence of explanatory texts, and then perform neural speech synthesis on the explanatory text sequence to synthesize it into a speech audio stream.

[0116] Specifically, the weighted stylized features are input into a text decoder based on a Transformer architecture, which converts the weighted stylized features through a self-attention mechanism that focuses on the relationship between different parts of the features and generates words one by one in a self-recursive manner according to the relationship between different parts of the features, and the words are combined in order to form a sequence of explanation texts, which are passed to a neural speech synthesizer based on a sequence-to-sequence architecture, which converts the sequence of explanation texts into an intermediate acoustic representation, i.e. a mel-spectrogram, which is converted into a time-domain waveform signal via a waveform generator, which drives the speaker to vibrate and produce a stream of speech audio.

[0117] S6. The speech audio stream and the matching style parameter set are input into the digital human to drive the generation of a non-verbal behavior sequence, which is rendered in real time through a content synchronization mechanism and a multi-terminal linkage protocol, and a digital human explanation video stream is output through a display trigger logic.

[0118] S6.1: The non-verbal behavior sequence includes a mouth animation sequence, an expression sequence, and a body movement sequence.

[0119] It should be noted that the mouth animation sequence refers to the visual movement of the digital human's face and mouth region opening and closing and shape changing in synchronization with the speech content; the expression sequence refers to the facial muscle movement of the digital human's facial features presenting emotional states such as joy, anger, sadness, etc.; and the body movement sequence refers to the posture and movement of the digital human's head, torso, arms, and hands.

[0120] S6.2: Extracting a prosody feature vector from the speech audio stream and extracting non-verbal behavior style parameters from the matching style parameter set;

[0121] Specifically, the speech audio stream is subjected to a short-time Fourier transform to obtain a frequency spectrum, from which the fundamental frequency contour, energy envelope, and syllable duration information are extracted to form a prosody feature vector; the matching style parameter set includes parameters such as expression intensity and gesture frequency, which are directly selected as non-verbal behavior style parameters.

[0122] S6.3: Behavior sequence modeling of the prosody feature vector and the non-verbal behavior style parameters to generate a non-verbal behavior sequence;

[0123] It should be noted that the pre-training of the long short-term memory network is based on a labeled data set containing speech prosody and corresponding digital human action sequences, the speech segments in the labeled data set are converted into prosodic feature vectors, the corresponding limb action sequences are converted into non-verbal behavior sequence encodings, the prosodic feature vectors and the non-verbal behavior sequence encodings are spliced into a fusion feature vector sequence for pre-training, and the pre-training target is to predict the correct non-verbal behavior sequence encoding at the next moment according to the current and historical fusion feature vectors at each time step. In this way, the long short-term memory network learns the time sequence mapping rule from the fusion features to the non-verbal behavior sequence, and the long short-term memory network has the ability to generate accurate and coherent non-verbal behavior sequences according to new fusion feature vectors, thereby obtaining a trained long short-term memory network.

[0124] Specifically, the prosodic feature vectors and the non-verbal behavior style parameters are spliced to form fusion feature vectors, and the fusion feature vectors are input into the long short-term memory network. The long short-term memory network predicts the non-verbal behavior state that the digital human should have according to the fusion feature at the current moment and the historical state of the internal memory of the long short-term memory network. The non-verbal behavior state is represented in the form of an encoding. The long short-term memory network predicts frame by frame in time sequence to generate a non-verbal behavior sequence.

[0125] S6.4: Perform dynamic priority synchronization processing on the non-verbal behavior sequence through a content synchronization mechanism to generate a synchronization instruction;

[0126] It should be noted that the content synchronization mechanism is a coordination method based on dynamic priority synchronization processing. By assigning real-time rendering orders to the non-verbal behavior sequence (including mouth shape animation sequence, expression sequence, and limb action sequence), a synchronization instruction is generated to ensure that the digital human behavior and the speech audio stream are accurately aligned in a multi-terminal environment.

[0127] Specifically, the content synchronization mechanism receives the non-verbal behavior sequence, determines the dynamic priority synchronization assignment real-time rendering order according to the inherent characteristics of each behavior in the non-verbal behavior sequence (for example, the mouth shape animation sequence is often given a higher priority because it needs to be synchronized with the speech audio stream, and the expression sequence and the limb action sequence are adjusted according to the scene), determines the priority order by referring to the timestamps and type labels of the non-verbal behavior sequence, ensures that the mouth shape animation is rendered first, and after the real-time rendering order is assigned and the priority order is determined, the content synchronization mechanism forms a synchronization instruction. The synchronization instruction contains non-verbal behavior rendering timing and terminal coordination information, which is used for multi-terminal linkage to ensure that the rendering of the non-verbal behavior sequence is accurately aligned with the speech audio stream, and to support smooth display of the digital human explanation.

[0128] S6.5: Distribute the synchronization instruction to each rendering terminal through a multi-terminal linkage protocol to render the non-verbal behavior sequence in real time and generate a digital human frame sequence;

[0129] It should be noted that the multi-terminal linkage protocol is a distributed communication mechanism using a publish-subscribe mode, which is used to reliably distribute synchronization instructions to each rendering terminal to ensure data synchronization and consistency between multiple terminals; the publish-subscribe mode is a distributed communication paradigm based on a message broker, in which a message sender (publisher) classifies messages to a specific topic without directly specifying a receiver, and a message receiver (subscriber) asynchronously receives related messages by subscribing to topics of interest, realizing decoupling and scalable multi-terminal data distribution.

[0130] The rendering terminal is a computing device equipped with hardware-accelerated rendering (such as WebGL), which renders the non-verbal behavior sequence in real time according to the synchronization instruction to generate a digital person frame sequence.

[0131] Specifically, the multi-terminal linkage protocol receives the synchronization instruction and starts the distribution process, the multi-terminal linkage protocol encapsulates the synchronization instruction into a standardized message package, and broadcasts it to all rendering terminals (such as large screen terminals and mobile guide screen terminals in the control hall) through the publish-subscribe mode, each rendering terminal returns an acknowledgement signal after receiving the standardized message package, and the multi-terminal linkage protocol verifies the reliability of the distribution according to the majority confirmation principle to avoid data loss; based on the priority information of the synchronization instruction, the rendering terminal calls the hardware-accelerated rendering (such as the WebGL renderer) to render the non-verbal behavior sequence frame by frame, and in the rendering process, the timing of the non-verbal behavior sequence is aligned with the voice audio stream, ensuring that the lip animation sequence is processed first to match the voice rhythm, and the rendering terminal generates a digital person frame sequence.

[0132] It should be noted that the digital person frame sequence contains the visual motion state of the digital person, providing a basis for digital person explanation video stream encapsulation.

[0133] S6.6: Encapsulate the digital person frame sequence and the voice audio stream through the presentation trigger logic, and output the digital person explanation video stream.

[0134] It should be noted that the presentation trigger logic is a coordination mechanism based on external events (such as user interaction signals or timing triggers), which automatically encapsulates the digital person frame sequence and the voice audio stream into a standardized video format (such as MP4) by listening to specific trigger conditions (such as the voice command "start explanation"), and outputs the digital person explanation video stream, realizing the synchronous display and on-demand distribution of audio-visual content.

[0135] Specifically, the display trigger logic initiates encapsulation based on the digital human frame sequence and the speech audio stream, ensures that each frame of the digital human frame sequence is accurately matched with the corresponding time point of the speech audio stream (e.g., the synchronization of the mouth animation frame and the speech syllable) through the timestamp alignment mechanism, calls the video encapsulator to integrate the aligned digital human frame sequence and the speech audio stream into a standardized container format (e.g., MP4 format) to support streaming transmission, and activates the output stage according to external events (such as user interaction signals or timing triggers) during the encapsulation process to generate a digital human explanation video stream.

[0136] It should be noted that the digital human explanation video stream can be distributed to the display device through the network protocol, and the seamless integration of audio-visual data is realized through the coordination of the display trigger logic, thereby guaranteeing the real-time and integrity of the digital human explanation content.

[0137] In summary, the present application realizes the deep optimization of semantic content and style features in the speech signal by parallel processing of speech recognition and style analysis and using a dynamic feature interaction gating mechanism, can simultaneously extract accurate text information and delicate style embedding vectors from the same speech source, rather than simply concatenating processing, and generates style vectors that not only retain the user's unique prosody and rhythm and other paralinguistic features, but also form an internal association with the semantic content, thereby providing a highly adaptive personalized parameter basis for digital human driving and improving the naturalness, emotional expressiveness, and user style matching degree of the digital human explanation behavior.

[0138] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and are not limiting. Although the present application has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solutions of the present application can be modified or replaced equivalently without departing from the spirit and scope of the technical solutions of the present application, and all should be covered in the scope of the claims of the present application.

Claims

1. A digital human explanation and demonstration control method based on speech recognition, characterized in that: include, The system receives the raw speech signal input from the user and preprocesses it. The preprocessed speech signal is then fed in parallel into the speech recognition path and style analysis path to generate the user's question text and a true style embedding vector. The specific steps are as follows: The original speech signal is denoised, pre-emphasized, and framed and windowed to generate a pre-processed speech signal. The deep feature extraction network in the speech recognition model is used to extract multi-scale acoustic features from the preprocessed speech signal and output a high-dimensional feature tensor. The high-dimensional feature tensor is input in parallel into the encoder of the speech recognition path and the encoder of the style analysis path to separate the content features and style features, and output the content feature sequence and style feature sequence. The content feature sequence and style feature sequence are collaboratively optimized by the dynamic feature interaction gating mechanism in the speech recognition model to generate the collaboratively optimized content feature sequence and style feature sequence. The co-optimized content feature sequence is input into the decoder of the speech recognition path and converted into user question text. The co-optimized style feature sequence is input into the feature aggregation layer of the style analysis path to generate a real style embedding vector. The user's question text is concatenated with historical dialogue text records to form a dialogue context sequence. A causal inference model is then used to analyze the logical connections and intent classifications within the dialogue context sequence, outputting a prospective explanation topic. The specific steps are as follows. The user's question text and historical dialogue text records are encoded and concatenated in chronological order to generate a dialogue context sequence. The dialogue context sequence is input into the causal inference model for intent classification and latent causal graph structure learning, generating a latent causal graph adjacency matrix; The PageRank algorithm is applied to calculate the PageRank center score of each topic node in the adjacency matrix of the latent causal graph; The topic nodes are sorted according to the PageRank center score, and the topic node with the highest score is selected as the topic for forward-looking explanation. The forward-looking topic is input into the historical knowledge base for retrieval to obtain structured knowledge data. At the same time, the real style is embedded into the vector and input into the digital human narration style generator to generate a set of digital human narration style parameters. The digital human narration style parameter set is adversarially matched with the digital human narration style discriminator, and the matched style parameter set is output. By integrating structured knowledge data, forward-looking explanation topics, and matching style parameter sets through a multimodal fusion mechanism, a sequence of explanation texts is generated and a speech audio stream is synthesized. The audio stream and matching style parameter set are input into the digital human to drive the generation of non-verbal behavior sequences. The non-verbal behavior sequences are rendered in real time through a content synchronization mechanism and a multi-terminal linkage protocol, and the digital human narrates a video stream by display trigger logic.

2. The digital human explanation and demonstration control method based on speech recognition as described in claim 1, characterized in that: The specific steps for obtaining structured knowledge data are as follows. The forward-looking topic is converted into a high-dimensional query vector using a language model, and the relevance score between the high-dimensional query vector and the knowledge vector in the historical knowledge base is calculated. Based on relevance scores, the most relevant knowledge points are selected from the historical knowledge base, and the corresponding structured knowledge data is retrieved.

3. The digital human explanation and demonstration control method based on speech recognition as described in claim 1, characterized in that: The specific steps for generating the digital human's narration style parameter set are as follows: The real style is embedded into a vector and input into the digital human to explain the style generator. The latent style distribution is learned through the built-in encoder network to generate a latent style vector. The latent style vectors are input into the built-in decoder network to generate a set of digital human explanation style parameters.

4. The digital human explanation and demonstration control method based on speech recognition as described in claim 1, characterized in that: The specific steps for matching the output style parameter set are as follows. The digital human's narration style parameter set and the real style are embedded into vector inputs to a deep neural network discriminator to calculate the distribution difference between the narration style and the real style; Based on the distribution differences, the parameter set of the digital human's explanation style is optimized in reverse, and the discrimination ability of the deep neural network discriminator is updated synchronously. When the deep neural network discriminator cannot effectively distinguish between the real style and the narration style, it outputs an optimized set of digital human narration style parameters as the matching style parameter set.

5. The digital human explanation and demonstration control method based on speech recognition as described in claim 1, characterized in that: The specific steps for generating the explanatory text sequence and synthesizing the speech audio stream are as follows. By using a content generation model, structured knowledge data and forward-looking explanation topics are dynamically synthesized to generate explanation content; By using a cross-modal attention mechanism, the content being explained is deeply integrated with the matching style parameter set to generate weighted stylistic features; The weighted stylized features are decoded to generate a sequence of explanatory text. The explanatory text sequence is then processed by neural speech synthesis to synthesize a speech audio stream.

6. The digital human explanation and demonstration control method based on speech recognition as described in claim 1, characterized in that: The specific steps for generating the non-linguistic behavior sequence are as follows. Extract prosodic feature vectors from speech audio streams and extract non-linguistic behavior style parameters from a set of matching style parameters; Behavioral sequence modeling is performed on prosodic feature vectors and nonverbal behavioral style parameters to generate nonverbal behavioral sequences.

7. The digital human explanation and demonstration control method based on speech recognition as described in claim 6, characterized in that: The nonverbal behavior sequences include lip-syncing animation sequences, facial expression sequences, and body movement sequences.

8. The digital human explanation and demonstration control method based on speech recognition as described in claim 1, characterized in that: The process involves real-time rendering of non-verbal behavior sequences through a content synchronization mechanism and a multi-terminal linkage protocol, followed by outputting a digital human-guided narration video stream via a display trigger logic. The specific steps are as follows: The non-verbal behavior sequences are dynamically prioritized and synchronized using a content synchronization mechanism to generate synchronization instructions. Synchronization instructions are distributed to each rendering terminal through a multi-terminal linkage protocol to render non-verbal behavior sequences in real time and generate digital human frame sequences. By encapsulating the digital human frame sequence and audio stream through the display trigger logic, a digital human narration video stream is output.

Images