Digital human speech generation system based on large language model

Abstract

The invention relates to the field of data processing technology, and specifically discloses a digital human speech generation system based on a large language model, which is used to solve existing problems. The system comprises a semantic analysis module, a speech generation module, a multimodal synchronization module, a data transmission module and a user feedback module. The semantic analysis module uses a large language model to perform semantic analysis and sentiment analysis on speech and text data input by a user, extracts speech information and sentiment metadata describing a context, and dynamically constructs a semantic structure and sentiment label adapted to an educational scenario according to input content based on an automatic metadata derivation technology. The invention uses a dynamic time warping algorithm and an educational knowledge graph to accurately capture semantic and sentiment changes, flexibly adjust speech speed and pitch, and improve the naturalness of digital human speech generation and the adaptability to educational scenarios.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and more specifically, to a digital human speech generation system based on a large language model. Background Art

[0002] With the rapid development of video conferencing and virtual interaction technology, digital humans have become an important tool for improving remote communication experience. However, the real-time and quality of audio and video transmission of current digital humans are difficult to guarantee in bandwidth-constrained or network-fluctuating environments, especially in multi-user interaction scenarios, resulting in low audio and video synchronization and transmission efficiency. In real-time video conferencing, delays, screen freezes, or asynchrony are prone to occur, which is more prominent. In addition, the voice and video data in video conferencing often lack deep semantic analysis and emotional expression capabilities, resulting in virtual avatars being unable to meet the needs of educational scenarios, especially in expressing supportive and motivational tones. It is difficult to quickly extract semantic and emotional features from audio and video data and generate metadata to guide the behavior of virtual avatars, which limits the accuracy and consistency of voice and animation generation, resulting in poor performance of virtual avatars. Special technical fields such as education and psychological counseling have put forward higher requirements on the system. The field of education requires digital humans to support teaching design, classroom interaction and feedback evaluation, and requires multimodal interaction capabilities, and the ability to express more complex information through emotional speech generation and facial expression animation. The current digital human audio, expression and action generation have a low degree of match between them, especially when expressing complex emotions or non-verbal information, which can easily appear stiff or unnatural, reducing the user's interactive experience. Summary of the invention

[0003] In order to overcome the above-mentioned defects of the prior art, the present invention provides a digital human speech generation system based on a large language model, which accurately captures semantic and emotional changes, flexibly adjusts speech speed and pitch, and improves the naturalness of digital human speech generation and adaptability to educational scenarios through a dynamic time warping algorithm and an educational knowledge graph.

[0004] When facing students with special educational needs, such as autism, attention deficit, etc., emotional support in remote video conferencing is particularly important. A gentle tone can be used to reduce students' emotional fluctuations and enhance classroom participation in video conferencing. When students complete their target tasks, positive affirmation and reinforcement are particularly important for these students, which can help them obtain good user experience and learning effects.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] A digital human speech generation system based on a large language model includes a semantic analysis module, a speech generation module, a multimodal synchronization module, a data transmission module and a user feedback module. The semantic analysis module uses a large language model to perform semantic analysis and sentiment analysis on the speech / text data input by the user, extracts speech information and sentiment metadata describing the context, and dynamically builds semantic structures and sentiment tags adapted to educational scenarios based on input content based on metadata automatic derivation technology. The speech generation module uses a semantic-to-speech multidimensional mapping model based on the sentiment metadata and semantic instructions generated by the semantic analysis module to generate speech output with emotional expression in real time, and dynamically adjusts the pitch, speech speed and sound intensity parameters of the speech to make the speech conform to the requirements in educational scenarios. The multimodal synchronization module synchronizes audio, expressions and actions in real time, combines timestamp alignment algorithm and prediction model to maintain multimodal data consistency under network fluctuations, and uses distributed file synchronization technology to optimize multimodal data synchronization in multi-user scenarios. The data transmission module gives priority to the transmission of voice and expression animation data based on priority sorting technology, uses dynamic compression and prediction completion algorithms to reduce the amount of data during transmission while maintaining the performance of the virtual avatar. The user feedback module submits the user interaction interface, allowing users to preview the voice and expression generation results in real time, and dynamically adjust the parameters, collect user feedback data, and optimize the voice generation and multimodal synchronization logic through a closed-loop learning mechanism.

[0007] As a further scheme of the present invention, the semantic analysis module includes a data preprocessing submodule, a speech recognition submodule, a semantic extraction submodule, a sentiment analysis submodule, a metadata generation submodule, a semantic and sentiment alignment submodule, and an output generation submodule. The data preprocessing submodule is connected to the speech recognition submodule, the semantic extraction submodule, and the sentiment analysis submodule. The semantic extraction submodule and the sentiment analysis submodule are connected to the metadata generation submodule. The metadata generation submodule is connected to the semantic and sentiment alignment submodule, and the semantic and sentiment alignment submodule is connected to the output generation submodule.

[0008] As a further solution of the present invention, the data preprocessing module distinguishes whether the user input is voice or text, calls the speech recognition submodule to convert the voice input into text, directly performs text cleaning, word segmentation and part-of-speech tagging on the text input, the semantic extraction submodule uses the GPT-4o large language model to parse the input text, the input format is user text + scene prompts, uses the multi-head attention mechanism to extract semantic points from the input, annotates core keywords and phrases, and establishes a semantic tree diagram, the semantic and sentiment alignment submodule calls the GPT context encoder to capture the previous and next correlation information in the input, uses the scene prompts to contextually align the current semantic information, and the sentiment analysis submodule uses RoBERTa's sentiment The emotion classifier classifies the input text, outputs emotion intensity and emotion category, and uses emotion embedding to quantify emotion features into an operational vector form. The semantic and emotion alignment submodule is based on the emotion modeling algorithm of dynamic time regularization, identifies the progression and change pattern of input emotions, dynamically generates time series of emotion intensity and dynamic adjustment vectors of speech rate and intonation. The metadata generation submodule calls the knowledge graph in the field of education to secondary process semantic metadata and emotion metadata, and adds suggestions for supportive and motivational intonation. The output generation submodule generates a tree-like semantic structure based on the extracted semantic points, attaches emotion metadata to the semantic structure, and outputs structured semantic information, emotion metadata and semantic structure diagram.

[0009] As a further solution of the present invention, in the semantic extraction submodule, the input word embedding sequence of the multi-head attention mechanism is ,in , , , is the word in the input word embedding sequence, is the total number of words, Reason The word embedding matrix consists of words, with a dimension of , For each word embedding dimension, The multi-head attention mechanism is used to extract the semantic key points. The semantic attention weight analysis formula is:

[0010]

[0011] Where: is the query matrix, is the key matrix, is the value matrix, , , They respectively represent the query requirements for other word relationships, the reference for matching queries, and the information transmitted after matching. is the output matrix, is the dot product operation of the query matrix and the key matrix, which represents the similarity between the query and the reference. is the dimension of the key matrix, is the normalization function, for each word The word embedding vector , is the index of the word, the query vector, key vector, and value vector are , , , , , Satisfy , , , , , are the weight matrices for query, key, and value, respectively. , , The dimensions of .

[0012] As a further solution of the present invention, in the semantic and sentiment alignment submodule, the GPT context encoder used is composed of several layers of Transformer, each layer contains multi-head attention and feedforward neural network, and the output formula of each layer is:

[0013]

[0014] Where: is the layer index of the GPT context encoder, For the The output of the layer, is the layer normalization operation, For the The output of the layer is encoded by the GPT context encoder, and the obtained context is represented as , , , , Separate words , , , Semantic representation that includes contextual information.

[0015] As a further solution of the present invention, in the semantic and sentiment alignment submodule, the scene prompt is used to perform context alignment on the current semantic information, and the formula for embedding the scene prompt with the context representation and performing prompt enhancement output is:

[0016]

[0017] Where: is the context representation after fusion of prompts, To embed the output of the scene cue into a fusion with the context representation, ,in is the embedding representation of the scene prompt, , Indicates that the input is converted into a low-dimensional vector with continuous values. It is a scenario prompt text, which includes supportive tone and motivational tone.

[0018] As a further solution of the present invention, the sentiment analysis submodule uses RoBERTa's sentiment classifier to classify the input text, outputs sentiment intensity and sentiment category, and uses sentiment embedding to quantify the sentiment features into an operable vector form. The process includes:

[0019] Step 11, context encoding: Capture the semantic and sentiment information of the input sequence through RoBERTa’s context encoder, output the context embedding of each subword, and take the position of the special token as the global semantic representation;

[0020] Step 12, sentiment classification and sentiment intensity extraction: The global semantic representation is input into a fully connected layer and a softmax activation function for sentiment classification, the sentiment categories and their respective tag numbers are counted, the category with the highest statistical number is selected as the sentiment classification result, and the sentiment intensity is extracted according to the probability distribution of the softmax output;

[0021] Step 13, emotion embedding generation: assign a learnable embedding vector to each emotion category, adjust the magnitude of the category embedding according to the emotion intensity, and output the emotion embedding vector.

[0022] In order to enhance the naturalness and adaptability of digital human in semantic and emotional expression, especially in supportive and motivational intonations, the speed and tone are adjusted dynamically. In supportive intonation, emotional expression focuses on stability and gradualness. The linear growth of speed and the smooth exponential growth of tone can convey care, support and acceptance, making the interaction process gentle and natural; the exponential growth of tone increases moderately to avoid excessive emotional expression, which conforms to the expression characteristics of supportive intonation. Incentive intonation emphasizes the dynamic increase and motivation of emotion. The quadratic curve nonlinear growth of speed can quickly convey vitality and motivation. The parabolic tone reaches the extreme value at the peak of emotional intensity, highlighting the high and inspiring effect and enhancing the motivational effect. This dynamic adjustment technical solution enables digital human to respond to emotional progressive changes more accurately, generate voice characteristics that meet the intonation requirements in combination with specific scenarios, improve the realism and expression effect of interaction, and meet the high requirements of emotional expression in scenarios such as education and psychological counseling, providing users with a better interactive experience.

[0023] As a further solution of the present invention, the semantic and sentiment alignment submodule is based on the sentiment modeling algorithm of dynamic time warping, identifies the progression and change pattern of input sentiment, and dynamically generates a time series of sentiment intensity, including:

[0024] Step 21, data input and preprocessing: obtain input sentiment features from the sentiment analysis submodule, the input sentiment features include a sentiment intensity time series consisting of the sentiment intensity of each time step and its corresponding timestamp, and normalize the input sequence and the reference sequence;

[0025] Step 22, construct a reference emotional model: The emotional intensity progressive stages of the supportive tone emotional model include stage 1, stage 2, stage 3 and stage 4, and the corresponding emotional intensity variation ranges are , , , [0.8, 1.0], the progressive stages of emotional intensity of the motivational tone emotional pattern include the initial stage, motivational stage, enhanced motivational stage, and peak stage, and the corresponding emotional intensity change ranges are , , , [0.9,1.0], and dynamically adjust the speaking speed and pitch according to the intonation category and its progressive stage of emotional intensity.

[0026] As a further solution of the present invention, in the semantic and emotional alignment submodule, the formula for dynamically adjusting the speech speed and pitch is obtained according to the intonation category and its emotional intensity progressive stage as follows:

[0027] In a supportive tone:

[0028] (1) The speaking speed increases linearly with the increase of emotional intensity. The formula for adjusting the speaking speed is:

[0029]

[0030] Where: For emotional intensity, is the initial speaking speed, To dynamically adjust the speech speed value at the current progressive stage of the emotional intensity of the supportive tone, is the speech speed growth coefficient, which is adjusted according to the current emotional intensity progressive stage. is the lower limit of the emotional intensity at the current progressive stage of the emotional intensity of the supportive tone;

[0031] (2) The pitch increases exponentially with the increase of emotional intensity. The pitch adjustment formula is:

[0032]

[0033] Where: To dynamically adjust the pitch value at the current progressive stage of the emotional intensity of the supportive tone, , are the minimum and maximum values ​​of the pitch, respectively. is the pitch growth rate adjustment factor at the current progressive stage of emotional intensity of the supportive tone, which is adjusted according to the current progressive stage of emotional intensity;

[0034] In a motivational tone:

[0035] (1) The speaking speed increases nonlinearly with the intensity of emotion in a quadratic curve. The formula for adjusting the speaking speed is:

[0036]

[0037] Where: To dynamically adjust the speech speed value at the current progressive stage of the emotional intensity of the motivational tone, is the speaking speed growth coefficient at the current progressive stage of emotional intensity of the motivational tone;

[0038] (2) The pitch increases parabolically with the intensity of emotion, reaching its extreme value at the peak of emotion. The pitch adjustment formula is:

[0039]

[0040] Where: To dynamically adjust the pitch value at the current progressive stage of the emotional intensity of the motivational tone, It is the maximum value of the emotional intensity in the current progressive stage of the emotional intensity of the motivational tone.

[0041] As a further solution of the present invention, the metadata generation submodule calls the knowledge graph in the field of education to secondary process the semantic metadata and emotional metadata, and the specific implementation process of adding supportive tone and motivational tone suggestions includes:

[0042] Step 31, inputting semantic metadata and sentiment metadata from the sentiment analysis submodule and the semantic analysis submodule;

[0043] Step 32, calling the knowledge graph: querying the knowledge graph in the field of education to obtain educational theoretical knowledge, scenario information, and emotional expression suggestions related to the current context;

[0044] Step 33, secondary processing of metadata: according to the semantic structure and context information of the knowledge graph, the semantic metadata is expanded, the associated concepts in the subgraph are added to the semantic metadata, and the associated concepts that meet the preset association degree are screened out according to the relationship weights. According to the emotion category in the emotion metadata, the corresponding emotion expression strategy in the knowledge graph is called, and the weight model is used to correct the emotion intensity;

[0045] Step 34, generating supportive and motivational tone suggestions: extracting supportive tone related suggestions from the knowledge graph, generating specific suggestions based on the subject after semantic metadata expansion, calling motivational tone related suggestions from the knowledge graph, adjusting the tone corresponding to the progressive stage of emotion, and generating motivational tone suggestions;

[0046] Step 35, output metadata: The output includes extended semantic metadata and emotional metadata, as well as supportive and motivational tone suggestions, dynamically generating semantic and emotional suggestions adapted to the educational scenario.

[0047] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention combines a dynamic time warping algorithm with an educational knowledge graph to construct a digital human speech generation system based on an emotional progressive model, so as to accurately capture the dynamic changes of input semantics and emotional characteristics, flexibly adjust the speaking speed and pitch under supportive and motivational intonations, enhance the emotional naturalness and delicacy of speech generation, and solve the problems of traditional speech generation in terms of stiff emotional expression, lack of dynamic adjustment of progressive changes, and insufficient adaptability to educational scenarios. It has significant technical advantages and practical application value in optimizing the accuracy of emotional expression, improving the human-computer interaction effect in educational scenarios, and meeting personalized emotional needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 is a system block diagram of the present invention;

[0049] Figure 2 This is a block diagram of the submodules of the semantic analysis module of the present invention;

[0050] Figure 3A flowchart of sentiment classification, sentiment intensity extraction and sentiment embedding for the sentiment analysis submodule of the present invention;

[0051] Figure 4 A flowchart for constructing a reference emotion model for the present invention;

[0052] Figure 5 The flowchart of calling the knowledge graph for secondary processing of metadata in the present invention. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] like Figure 1 As shown, the present invention proposes a digital human speech generation system based on a large language model, including a semantic analysis module, a speech generation module, a multimodal synchronization module, a data transmission module and a user feedback module. The semantic analysis module uses a large language model to perform semantic analysis and sentiment analysis on the speech / text data input by the user, extracts speech information and sentiment metadata describing the context, and dynamically constructs a semantic structure and sentiment label adapted to the educational scenario according to the input content based on the metadata automatic derivation technology. The speech generation module uses a semantic-to-speech multidimensional mapping model based on the sentiment metadata and semantic instructions generated by the semantic analysis module to generate speech output with emotional expression in real time, and dynamically adjusts the pitch, speech speed and sound intensity parameters of the speech to make the speech conform to the educational scenario. The multimodal synchronization module synchronizes audio, expressions and actions in real time, combines timestamp alignment algorithm and prediction model to maintain multimodal data consistency under network fluctuations, and uses distributed file synchronization technology to optimize multimodal data synchronization in multi-user scenarios. The data transmission module gives priority to the transmission of voice and expression animation data based on priority sorting technology, uses dynamic compression and prediction completion algorithms to reduce the amount of data during transmission while maintaining the performance of the virtual avatar. The user feedback module submits the user interaction interface, allowing users to preview the voice and expression generation results in real time, and dynamically adjust the parameters, collect user feedback data, and optimize the voice generation and multimodal synchronization logic through a closed-loop learning mechanism.

[0055] The present invention combines a dynamic time warping algorithm with an educational knowledge graph to construct a digital human speech generation system based on an emotional progression model, so as to accurately capture the dynamic changes of input semantics and emotional features, flexibly adjust the speaking speed and pitch under supportive and motivational tones, and enhance the emotional naturalness and delicacy of speech generation. Compared with the existing technology, this system solves the problems of traditional speech generation in terms of stiff emotional expression, lack of dynamic adjustment of progressive changes, and insufficient adaptability to educational scenarios. It has significant technical advantages and practical application value in optimizing the accuracy of emotional expression, improving the human-computer interaction effect in educational scenarios, and meeting personalized emotional needs.

[0056] It should be noted that the semantic analysis module includes a data preprocessing submodule, a speech recognition submodule, a semantic extraction submodule, a sentiment analysis submodule, a metadata generation submodule, a semantic and sentiment alignment submodule, and an output generation submodule. The data preprocessing submodule is connected to the speech recognition submodule, the semantic extraction submodule, and the sentiment analysis submodule. The semantic extraction submodule and the sentiment analysis submodule are connected to the metadata generation submodule. The metadata generation submodule is connected to the semantic and sentiment alignment submodule, and the semantic and sentiment alignment submodule is connected to the output generation submodule. The modules work together as follows: the data preprocessing module distinguishes whether the user input is voice or text, calls the speech recognition submodule to convert the voice input into text, directly performs text cleaning, word segmentation and part-of-speech tagging on the text input, the semantic extraction submodule uses the GPT-4o large language model to parse the input text, the input format is user text + scene prompts, uses the multi-head attention mechanism to extract semantic points from the input, annotates core keywords and phrases, and builds a semantic tree diagram, the semantic and sentiment alignment submodule calls the GPT context encoder to capture the previous and next related information in the input, uses scene prompts to contextually align the current semantic information, and the sentiment analysis submodule uses RoBERTa's sentiment The classifier classifies the input text, outputs the emotion intensity and emotion category, and uses emotion embedding to quantify the emotion features into an operational vector form. The semantic and emotion alignment submodule is based on the emotion modeling algorithm of dynamic time regularization, identifies the progression and change pattern of the input emotion, dynamically generates the time series of emotion intensity and the dynamic adjustment vectors of speech rate and intonation. The metadata generation submodule calls the knowledge graph in the field of education to secondary process the semantic metadata and emotion metadata, and adds suggestions for supportive and motivational intonation. The output generation submodule generates a tree-like semantic structure based on the extracted semantic points, attaches the emotion metadata to the semantic structure, and outputs structured semantic information, emotion metadata and semantic structure diagram.

[0057] Through close collaboration between modules, the present invention constructs a complete semantic and sentiment analysis and processing chain, from data preprocessing to speech recognition, semantic extraction, sentiment analysis, and then to context alignment and dynamic adjustment, and combines the educational knowledge graph to achieve deep processing of semantic and sentiment metadata, and finally generates semantic structured output with additional sentiment information; this system makes semantic analysis and sentiment expression more accurate and natural, which not only meets the high adaptation requirements of educational scenarios for supportive and motivational tones, but also significantly improves the intelligence and applicability of human-computer interaction, provides full-chain support and application value for emotional digital human technology, and provides more personalized emotional support for students with special needs in distance education video conferences, relieves learning anxiety, enhances self-confidence and sense of participation, thereby improving learning effects and classroom interactive experience.

[0058] It should be noted that in the semantic extraction submodule, the input word embedding sequence of the multi-head attention mechanism is ,in , , , is the word in the input word embedding sequence, is the total number of words, Reason The word embedding matrix consists of words, with a dimension of , For each word embedding dimension, The multi-head attention mechanism is used to extract the semantic key points. The semantic attention weight analysis formula is:

[0059]

[0060] Where: is the query matrix, is the key matrix, is the value matrix, , , They respectively represent the query requirements for other word relationships, the reference for matching queries, and the information transmitted after matching. is the output matrix, is the dot product operation of the query matrix and the key matrix, which represents the similarity between the query and the reference. is the dimension of the key matrix, is the normalization function, for each word The word embedding vector , is the index of the word, the query vector, key vector, and value vector are , , , , , Satisfy , , , , , are the weight matrices for query, key, and value, respectively. , , The dimensions of .

[0061] Example 1

[0062] In order to clearly express the output semantic acquisition mechanism of the multi-head attention mechanism of the semantic extraction submodule, this embodiment is used to illustrate.

[0063] The input Chinese sentence is "Students actively participate in group discussions." The sentence is segmented into , , , , , each word segment is mapped to a dimensional vector, forming a word embedding matrix , ,set up , that is, the embedding dimension of each word segment is 768.

[0064] First, construct the query, key, and value vectors: embedding for each word , through three different weight matrices , , Generate query vectors separately , key vector , value vector , , , ,set up ,but , , ; construct query matrix, key matrix and value matrix, , , .

[0065] Second, calculate the attention weight: calculate the query matrix and key matrix The dot product of , each element represents the semantic similarity between the word segments, and the dot product result is divided by , that is, divided by 8.

[0066] Again, a softmax operation is performed on the normalized results of each row to obtain the attention distribution matrix (the attention weight of a row is [0.1, 0.3, 0.4, 0.1, 0.1], indicating that the current word segmentation has the highest attention to "student").

[0067] Finally, the attention distribution matrix is ​​multiplied by the value matrix to obtain .

[0068] It should be noted that the GPT context encoder used in the semantic and sentiment alignment submodule consists of several layers of Transformer, each of which contains multi-head attention and feedforward neural networks, and the output formula of each layer is:

[0069]

[0070] Where: is the layer index of the GPT context encoder, For the The output of the layer, is the layer normalization operation, For the The output of the layer is encoded by the GPT context encoder, and the obtained context is represented as , , , , Separate words , , , Semantic representation containing contextual information. In the semantic and sentiment alignment submodule, the scene prompt is used to align the current semantic information with the context, and the formula for embedding the scene prompt with the contextual representation and performing prompt enhancement output is:

[0071]

[0072] Where: is the context representation after fusion of prompts, To embed the output of the scene cue into a fusion with the context representation, ,in is the embedding representation of the scene prompt, , Indicates that the input is converted into a low-dimensional vector with continuous values. It is a scenario prompt text, which includes supportive tone and motivational tone.

[0073] Through the multi-layer Transformer structure of the GPT context encoder combined with scene prompt embedding, the present invention realizes the efficient fusion of semantic information and scene prompts in the semantic and emotional alignment sub-module, and uses prompt enhancement technology to dynamically adjust the context representation, so that the generated context semantic representation is more in line with the supportive / motivational tone requirements of specific scenarios, thereby solving the problems of insufficient situational adaptation ability and unnatural emotional expression of traditional systems, and significantly improving the digital human's ability to accurately capture and express semantic and emotional progressive patterns in complex situations.

[0074] It should be noted that the sentiment analysis submodule uses RoBERTa's sentiment classifier to classify the input text, outputs sentiment intensity and sentiment category, and uses sentiment embedding to quantify sentiment features into an operational vector form. The process includes:

[0075] Step 11, context encoding: Capture the semantic and sentiment information of the input sequence through RoBERTa’s context encoder, output the context embedding of each subword, and take the position of the special token as the global semantic representation;

[0076] Step 12, sentiment classification and sentiment intensity extraction: The global semantic representation is input into a fully connected layer and a softmax activation function for sentiment classification, the sentiment categories and their respective tag numbers are counted, the category with the highest statistical number is selected as the sentiment classification result, and the sentiment intensity is extracted according to the probability distribution of the softmax output;

[0077] Step 13, emotion embedding generation: assign a learnable embedding vector to each emotion category, adjust the magnitude of the category embedding according to the emotion intensity, and output the emotion embedding vector.

[0078] In order to enhance the naturalness and adaptability of digital human in semantic and emotional expression, especially in supportive and motivational intonations, the speed and tone are adjusted dynamically. In supportive intonation, emotional expression focuses on stability and gradualness. The linear growth of speed and the smooth exponential growth of tone can convey care, support and acceptance, making the interaction process gentle and natural; the exponential growth of tone increases moderately to avoid excessive emotional expression, which conforms to the expression characteristics of supportive intonation. Incentive intonation emphasizes the dynamic increase and motivation of emotion. The quadratic curve nonlinear growth of speed can quickly convey vitality and motivation. The parabolic tone reaches the extreme value at the peak of emotional intensity, highlighting the high and inspiring effect and enhancing the motivational effect. This dynamic adjustment technical solution enables digital human to respond to emotional progressive changes more accurately, generate voice characteristics that meet the intonation requirements in combination with specific scenarios, improve the realism and expression effect of interaction, and meet the high requirements of emotional expression in scenarios such as education and psychological counseling, providing users with a better interactive experience.

[0079] It should be noted that the semantic and sentiment alignment submodule is based on the sentiment modeling algorithm of dynamic time warping, which identifies the progression and change pattern of input sentiment and dynamically generates the time series of sentiment intensity. The process includes:

[0080] Step 21, data input and preprocessing: obtain input sentiment features from the sentiment analysis submodule, the input sentiment features include a sentiment intensity time series consisting of the sentiment intensity of each time step and its corresponding timestamp, and normalize the input sequence and the reference sequence;

[0081] Step 22, construct a reference emotional model: The emotional intensity progressive stages of the supportive tone emotional model include stage 1, stage 2, stage 3 and stage 4, and the corresponding emotional intensity variation ranges are , , , [0.8, 1.0], the progressive stages of emotional intensity of the motivational tone emotional pattern include the initial stage, motivational stage, enhanced motivational stage, and peak stage, and the corresponding emotional intensity change ranges are , , , [0.9,1.0], and dynamically adjust the speaking speed and pitch according to the intonation category and its progressive stage of emotional intensity.

[0082] As shown in Table 1, the emotional intensity of the supportive tone emotional model is described in the progressive stages:

[0083] Table 1

[0084]

[0085] The goal of the emotional progression of supportive tone is to gradually express care and support, create a sense of security, gradually guide the other person to relax, slowly increase the intensity of emotions, and express supportive meaning in a steady tone.

[0086] As shown in Table 2, the progressive stages of emotional intensity of the motivational tone emotional model are described as follows:

[0087] Table 2

[0088]

[0089] The goal of the emotional progression of motivational tone is to dynamically express trust and encouragement, convey confidence and motivation through a positive tone, increase the intensity of emotion quickly, use tone changes moderately, and highlight the sense of energy.

[0090] Emotional intensity refers to the intensity of an emotion at a certain point in time. It is represented by quantitative features and the range of emotion intensity is set to [0,1]. The larger the value, the stronger the emotion expression. The expression characteristics of emotion intensity mainly include pitch, speech rate, waveform saturation and volume. In order to better solve the targeted problems raised in the background technology, the present invention focuses on the dynamic adjustment of pitch and volume, but does not mean ignoring the adjustment of waveform saturation and volume.

[0091] It should be noted that in the semantic and sentiment alignment submodule, the formula for dynamically adjusting the speech speed and pitch according to the intonation category and its progressive stage of sentiment intensity is as follows:

[0092] In a supportive tone:

[0093] (1) The speaking speed increases linearly with the increase of emotional intensity. The formula for adjusting the speaking speed is:

[0094]

[0095] Where: For emotional intensity, is the initial speaking speed, To dynamically adjust the speech speed value at the current progressive stage of the emotional intensity of the supportive tone, is the speech speed growth coefficient, which is adjusted according to the current emotional intensity progressive stage. is the lower limit of the emotional intensity at the current progressive stage of the emotional intensity of the supportive tone;

[0096] (2) The pitch increases exponentially with the increase of emotional intensity. The pitch adjustment formula is:

[0097]

[0098] Where: To dynamically adjust the pitch value at the current progressive stage of the emotional intensity of the supportive tone, , are the minimum and maximum values ​​of the pitch, respectively. is the pitch growth rate adjustment factor at the current progressive stage of emotional intensity of the supportive tone, which is adjusted according to the current progressive stage of emotional intensity;

[0099] In a motivational tone:

[0100] (1) The speaking speed increases nonlinearly with the intensity of emotion in a quadratic curve. The formula for adjusting the speaking speed is:

[0101]

[0102] Where: To dynamically adjust the speech speed value at the current progressive stage of the emotional intensity of the motivational tone, is the speaking speed growth coefficient at the current progressive stage of emotional intensity of the motivational tone;

[0103] (2) The pitch increases parabolically with the intensity of emotion, reaching its extreme value at the peak of emotion. The pitch adjustment formula is:

[0104]

[0105] Where: To dynamically adjust the pitch value at the current progressive stage of the emotional intensity of the motivational tone, It is the maximum value of the emotional intensity in the current progressive stage of the emotional intensity of the motivational tone.

[0106] Example 2

[0107] In order to clearly express the dynamic adjustment process of the above two intonations, this embodiment is described in detail in combination with specific setting data.

[0108] For motivational intonation, the sentiment analysis module outputs the following sentiment intensity time series and timestamps:

[0109]

[0110] At an initial speaking speed of 100 words per minute, , When , the adjustment value of its speech speed is:

[0111] against , words / minute;

[0112] against , words / minute;

[0113] against , words / minute;

[0114] against , words / minute;

[0115] For pitch adjustment, , , the growth rate adjustment factor is 2, and when the lower limit of the current emotional intensity is 0.2, the adjustment value of its tone is:

[0116] against , ;

[0117] against , ;

[0118] against , ;

[0119] against , .

[0120] For supportive intonation, the sentiment analysis module outputs the following sentiment intensity time series and timestamps:

[0121]

[0122] When the initial speaking speed is 100 words / minute, the speaking speed growth coefficient , the current lower limit of emotion intensity is 0.2, and the speed adjustment value of supportive intonation is:

[0123] against , words / minute;

[0124] against , words / minute;

[0125] against , words / minute;

[0126] against , words / minute;

[0127] For pitch adjustment, , When , the pitch adjustment value is:

[0128] against , ;

[0129] against , ;

[0130] against , ;

[0131] against , .

[0132] It should be noted that the metadata generation submodule calls the knowledge graph in the field of education to process semantic metadata and sentiment metadata for secondary processing. The specific implementation process of adding supportive and motivational tone suggestions includes:

[0133] Step 31, inputting semantic metadata and sentiment metadata from the sentiment analysis submodule and the semantic analysis submodule;

[0134] Step 32, calling the knowledge graph: querying the knowledge graph in the field of education to obtain educational theoretical knowledge, scenario information, and emotional expression suggestions related to the current context;

[0135] Step 33, secondary processing of metadata: according to the semantic structure and context information of the knowledge graph, the semantic metadata is expanded, the associated concepts in the subgraph are added to the semantic metadata, and the associated concepts that meet the preset association degree are screened out according to the relationship weights. According to the emotion category in the emotion metadata, the corresponding emotion expression strategy in the knowledge graph is called, and the weight model is used to correct the emotion intensity;

[0136] Step 34, generating supportive and motivational tone suggestions: extracting supportive tone related suggestions from the knowledge graph, generating specific suggestions based on the subject after semantic metadata expansion, calling motivational tone related suggestions from the knowledge graph, adjusting the tone corresponding to the progressive stage of emotion, and generating motivational tone suggestions;

[0137] Step 35, output metadata: The output includes extended semantic metadata and emotional metadata, as well as supportive and motivational tone suggestions, dynamically generating semantic and emotional suggestions adapted to the educational scenario.

[0138] By calling the knowledge graph in the field of education to perform secondary processing on semantic metadata and emotional metadata and generate supportive and motivational tone suggestions, this solution effectively solves the problems of insufficient semantic expression accuracy, disconnection between emotion and tone, and weak scene adaptability. It uses the semantic structure of the knowledge graph to dynamically expand semantic metadata, combines emotional categories and progressive stages to correct emotional intensity, and generates diversified and educational context-appropriate tone suggestions, enhancing the naturalness and hierarchy of semantic and emotional expression, improving the system's intelligent recommendation capabilities and interactive effects in educational scenarios, and at the same time providing technical support for personalized support and motivation in complex educational scenarios, significantly improving user experience and the effectiveness of educational interaction.

[0139] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0140] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A digital human speech generation system based on a large language model, comprising a semantic analysis module, a speech generation module, a multimodal synchronization module, a data transmission module and a user feedback module, characterized in that: The semantic analysis module uses a large language model to perform semantic parsing and sentiment analysis on the voice / text data input by the user, extracts voice information and sentiment metadata describing the context, and dynamically builds semantic structures and sentiment tags adapted to educational scenarios based on the input content based on metadata automatic derivation technology. The speech generation module uses a multidimensional mapping model from semantics to speech based on the sentiment metadata and semantic instructions generated by the semantic analysis module to generate speech output with emotional expression in real time, dynamically adjust the voice pitch, speech speed, and sound intensity parameters to make the voice conform to the supportive and motivational tone in the educational scenario. The multimodal synchronization module synchronizes audio and expressions in real time. and actions, combined with timestamp alignment algorithms and prediction models, to maintain multimodal data consistency under network fluctuations, and use distributed file synchronization technology to optimize multimodal data synchronization in multi-user scenarios. The data transmission module prioritizes the transmission of voice and expression animation data based on priority sorting technology, and uses dynamic compression and prediction completion algorithms to reduce the amount of data during transmission while maintaining the performance of the virtual avatar. The user feedback module submits a user interaction interface, allowing users to preview the voice and expression generation results in real time, and dynamically adjust the parameters, collect user feedback data, and optimize the voice generation and multimodal synchronization logic through a closed-loop learning mechanism; The semantic analysis module includes a data preprocessing submodule, a speech recognition submodule, a semantic extraction submodule, a sentiment analysis submodule, a metadata generation submodule, a semantic and sentiment alignment submodule, and an output generation submodule; The semantic and sentiment alignment submodule uses a sentiment modeling algorithm based on dynamic time warping to identify the progression and change patterns of input sentiments. The process of dynamically generating a time series of sentiment intensity includes: Step 21, data input and preprocessing: obtain input sentiment features from the sentiment analysis submodule, the input sentiment features include a sentiment intensity time series consisting of the sentiment intensity of each time step and its corresponding timestamp, and normalize the input sequence and the reference sequence; Step 22, construct a reference emotional model: The emotional intensity progressive stages of the supportive tone emotional model include stage 1, stage 2, stage 3 and stage 4, and the corresponding emotional intensity variation ranges are , , , [0.8, 1.0], the progressive stages of emotional intensity of the motivational tone emotional pattern include the initial stage, motivational stage, enhanced motivational stage, and peak stage, and the corresponding emotional intensity change ranges are , , , [0.9, 1.0], and dynamically adjust the speech speed and pitch according to the intonation category and its emotional intensity progressive stage; In the semantic and sentiment alignment submodule, the formula for dynamically adjusting the speech speed and pitch according to the intonation category and its progressive stage of sentiment intensity is as follows: In a supportive tone: (1) The speaking speed increases linearly with the increase of emotional intensity. The formula for adjusting the speaking speed is: Where: For emotional intensity, is the initial speaking speed, To dynamically adjust the speech speed value at the current progressive stage of the emotional intensity of the supportive tone, is the speech speed growth coefficient, which is adjusted according to the current emotional intensity progressive stage. is the lower limit of the emotional intensity at the current progressive stage of the emotional intensity of the supportive tone; (2) The pitch increases exponentially with the increase of emotional intensity. The pitch adjustment formula is: Where: To dynamically adjust the pitch value at the current progressive stage of the emotional intensity of the supportive tone, , are the minimum and maximum values ​​of the pitch, respectively. is the pitch growth rate adjustment factor at the current progressive stage of emotional intensity of the supportive tone, which is adjusted according to the current progressive stage of emotional intensity; In a motivational tone: (1) The speaking speed increases nonlinearly with the intensity of emotion in a quadratic curve. The formula for adjusting the speaking speed is: Where: To dynamically adjust the speech speed value at the current progressive stage of the emotional intensity of the motivational tone, is the initial speaking speed, is the lower limit of the emotional intensity in the current progressive stage of emotional intensity in the motivational tone, is the speaking speed growth coefficient at the current progressive stage of emotional intensity of the motivational tone; (2) The pitch increases parabolically with the intensity of emotion, reaching its extreme value at the peak of emotion. The pitch adjustment formula is: Where: To dynamically adjust the pitch value at the current progressive stage of the emotional intensity of the motivational tone, , are the minimum and maximum values ​​of the pitch, respectively. is the maximum value of the emotional intensity at the current progressive stage of the emotional intensity of the motivational tone; In the semantic and sentiment alignment submodule, after being encoded by the GPT context encoder, the obtained context is represented as , , , , are the words in the input word embedding sequence respectively , , , The semantic representation contains contextual information, and the current semantic information is contextually aligned using scene prompts. The scene prompt embedding is fused with the contextual representation and prompt-enhanced output is performed. The scene prompt text includes supportive and motivational tones.

2. A digital human speech generation system based on a large language model according to claim 1, characterized in that: The data preprocessing module distinguishes whether the user input is voice or text, calls the speech recognition submodule to convert the voice input into text, and directly performs text cleaning, word segmentation and part-of-speech tagging on the text input. The semantic extraction submodule uses the GPT-4o large language model to parse the input text. The input format is user text + scene prompts. The multi-head attention mechanism is used to extract semantic points from the input, annotate core keywords and phrases, and build a semantic tree diagram. The semantic and sentiment alignment submodule calls the GPT context encoder to capture the previous and next related information in the input, and uses the scene prompts to contextually align the current semantic information. The sentiment analysis submodule uses the RoBERTa sentiment classifier to classify the input. The input text is classified, and the emotion intensity and emotion category are output. The emotion embedding is used to quantify the emotion features into an operational vector form. The semantic and emotion alignment submodule is based on the emotion modeling algorithm of dynamic time regularization, which identifies the progression and change pattern of the input emotion, dynamically generates the time series of emotion intensity and the dynamic adjustment vector of speech speed and intonation. The metadata generation submodule calls the knowledge graph in the field of education to secondary process the semantic metadata and emotion metadata, and adds suggestions for supportive and motivational intonation. The output generation submodule generates a tree-like semantic structure based on the extracted semantic points, attaches the emotion metadata to the semantic structure, and outputs structured semantic information, emotion metadata and semantic structure diagram.

3. A digital human speech generation system based on a large language model according to claim 1, characterized in that: The sentiment analysis submodule uses RoBERTa's sentiment classifier to classify the input text, output the sentiment intensity and sentiment category, and uses sentiment embedding to quantify the sentiment features into an operational vector form. The process includes: Step 11, context encoding: Capture the semantic and sentiment information of the input sequence through RoBERTa’s context encoder, output the context embedding of each subword, and take the position of the special token as the global semantic representation; Step 12, sentiment classification and sentiment intensity extraction: The global semantic representation is input into a fully connected layer and a softmax activation function for sentiment classification, the sentiment categories and their respective tag numbers are counted, the category with the highest statistical number is selected as the sentiment classification result, and the sentiment intensity is extracted according to the probability distribution of the softmax output; Step 13, emotion embedding generation: assign a learnable embedding vector to each emotion category, adjust the magnitude of the category embedding according to the emotion intensity, and output the emotion embedding vector.

4. A digital human speech generation system based on a large language model according to claim 2, characterized in that: The metadata generation submodule uses the knowledge graph in the field of education to process semantic metadata and sentiment metadata. The specific implementation process of adding supportive and motivational tone suggestions includes: Step 31, inputting semantic metadata and sentiment metadata from the sentiment analysis submodule and the semantic analysis submodule; Step 32, calling the knowledge graph: querying the knowledge graph in the field of education to obtain educational theoretical knowledge, scenario information, and emotional expression suggestions related to the current context; Step 33, secondary processing of metadata: according to the semantic structure and context information of the knowledge graph, the semantic metadata is expanded, the associated concepts in the subgraph are added to the semantic metadata, and the associated concepts that meet the preset association degree are screened out according to the relationship weights. According to the emotion category in the emotion metadata, the corresponding emotion expression strategy in the knowledge graph is called, and the weight model is used to correct the emotion intensity; Step 34, generating supportive and motivational tone suggestions: extracting supportive tone related suggestions from the knowledge graph, generating specific suggestions based on the subject after semantic metadata expansion, calling motivational tone related suggestions from the knowledge graph, adjusting the tone corresponding to the progressive stage of emotion, and generating motivational tone suggestions; Step 35, output metadata: The output includes extended semantic metadata and emotional metadata, as well as supportive and motivational tone suggestions, dynamically generating semantic and emotional suggestions adapted to the educational scenario.

Images