Real-time high-fidelity voice-driven digital human system

Abstract

The invention relates to the technical field of digital people, and discloses a real-time high-fidelity voice-driven digital people system which integrates six modules of voice acquisition, voice recognition, semantic analysis, sentiment analysis, animation generation and rendering generation. According to the invention, through contrastive analysis with the input voice, determination of accent and language styles in the early stage, conversion of the voice into the text, and word segmentation and part-of-speech tagging by using advanced machine learning and deep learning technologies, intentions and demands of a user are deeply understood, and all modules of the digital human system are driven to cooperate; a series of signal processing and feature extraction methods of the sentiment analysis module are applied, and a machine learning or deep learning model is combined to accurately recognize the sentiment polarity in the user voice, so that the system can make corresponding feedback according to a sentiment analysis result, for example, in an intelligent customer service scene, a reply strategy is adjusted according to the user sentiment, and the user experience is improved. And more personalized and humanized services are provided, and emotion resonance of interaction between the user and the digital human is enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of digital human, and in particular to a real-time high-fidelity voice-driven digital human system. Background Art

[0002] With the development of artificial intelligence and the metaverse, users have put forward new requirements and expectations for the diversity of human-computer interaction. The interaction method based on virtual humans provides new ideas for human-computer interaction at this stage and has a wide range of applications.

[0003] The current digital human system has many limitations in voice-driven operation. First, it is difficult for a considerable number of existing systems to respond in real time, resulting in significant delays when digital humans interact with users. This delay seriously undermines the user's interactive experience and affects the user's satisfaction with the digital human application. The system is limited in the realism of voice-driven operation. The digital human's expressions and movements are poorly matched with the voice content it conveys, making the digital human appear stiff and unnatural as a whole, making it difficult to truly integrate into the interactive scenario with the user.

[0004] In view of the above situation, developing a digital human system that can effectively solve these problems and realize real-time and high-fidelity voice-driven has become an urgent need. Summary of the invention

[0005] The object of the present invention is to provide a real-time high-fidelity voice-driven digital human system to solve the problems raised in the above-mentioned background technology.

[0006] To achieve the above object, the present invention provides the following technical solution: a real-time high-fidelity voice-driven digital human system, the system comprising:

[0007] The voice acquisition module is used to collect the user's voice signal in real time, using a high-sensitivity microphone array to accurately obtain voice input, and perform noise reduction and gain on the collected voice signal;

[0008] Speech recognition: the speech signal acquired by the acquisition module is converted into text information, and speech with various accents, speaking speeds and language styles can be recognized;

[0009] Semantic analysis module, which analyzes text information to understand the user's intentions and needs;

[0010] The sentiment analysis module uses an audio feature extraction algorithm to extract the rhythm, speech rate, and volume features of speech signals, and combines this with a sentiment classification model to accurately identify user sentiment.

[0011] The animation generation module uses the virtual human interaction system to complete the three-dimensional face driving, and the close correlation between the voice characteristics and the movement of the mouth and nose organs of the face to form the character's expression;

[0012] The rendering generation module uses rendering technology to ensure that digital humans can be displayed smoothly on different devices and platforms.

[0013] Preferably, the voice acquisition module adopts a monophonic analog signal, and after acquisition, the voltage is converted into a quantized digital signal according to a corresponding standard, and the signal frequency is 200-1600Hz, wherein the signal frequency for boys is in the range of 400-700Hz, and that for girls is in the range of 650-1000Hz, and the signal sampling is set to 4000Hz.

[0014] Preferably, the speech recognition establishes a full-pole model, compares it with the input speech, and obtains speech prediction data after data analysis. The accent and language style of the speech are determined based on the comparison of the prediction data. The measurement formula of the prediction data is:

[0015]

[0016] Where T n Represents the predicted data, a k is the weighting coefficient, n is the input dimension, k is the number of vectors, and P is the sampling value of the point.

[0017] Preferably, the semantic analysis module performs word segmentation on the text, splits it into individual words or phrases, and marks the parts of speech, and uses machine learning methods and word vector models in deep learning to assist in analyzing the lexical structure.

[0018] Preferably, the sentiment analysis module first performs logarithmic transformation on the filter bank, compresses dynamic variables, and performs discrete cosine transformation to obtain features. The calculation formula is:

[0019]

[0020] Where S is the output vector, M is the number of filters, G is the number of coefficients, and n is the number of coefficients to be retained;

[0021] And set the eigenvector;

[0022] The eigenvector is B = {r1, r2, r3...r z , r Z};

[0023] where r z represents the feature vector of the tth time step, Z is the length of the feature sequence, and the receiving unit currently inputs the feature vector r z , the hidden state vector r at the previous moment z-1;

[0024] The features are extracted and compared with related speech to classify the emotional polarity. Finally, the feature sequence is mapped to the emotional polarity category to classify the speech emotion.

[0025] Preferably, the animation generation module is based on a model fusion method, in which the facial expressions are composed of three expression bases, and the calculation formula is:

[0026]

[0027] Where H is the comprehensive data of expression bases, H1 is the first expression base, and H t is the tth expression base, and P is the weight of the expression base, thus forming facial expression shaping.

[0028] Preferably, the rendering generation module is imported using 3Dmax software. After multiple models carrying expression bases are imported into the scene, the entity carries components, and the component model fusion method is called in the script, which can change the expression base weights to achieve the purpose of virtual human expression driving.

[0029] Preferably, the rendering generation module is combined with speech, the speech is mapped to the visual features of the face, the features of each frame of the animation are predicted by the upper and lower speech signals, and the 250ms speech signals before and after the current frame are selected, and the context speech signal is represented using linear prediction coding.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] In the present invention, by comparing and analyzing the input voice, the accent and language style are determined in advance, the voice is converted into text, and advanced machine learning and deep learning technologies are used for word segmentation and part-of-speech tagging to deeply understand the user's intentions and needs, drive the collaboration of various modules of the digital human system, and use a series of signal processing and feature extraction methods of the sentiment analysis module, combined with machine learning or deep learning models, to accurately identify the sentiment polarity in the user's voice. This enables the system to make corresponding feedback based on the sentiment analysis results. For example, in the intelligent customer service scenario, the response strategy is adjusted according to the user's emotions, providing more personalized and humane services, and enhancing the emotional resonance of the user's interaction with the digital human.

[0032] In the present invention, the animation generation module realizes rich and diverse expression driving of virtual humans based on model fusion and expression-based weight adjustment. The rendering generation module is closely integrated with the voice, and the animation frame features are predicted according to the voice context information, so that the virtual human's expression can change with the voice content and emotion in real time and accurately, and can be displayed smoothly on different devices and platforms, presenting a natural and vivid interactive effect, which greatly improves the user experience and enhances the immersion of digital humans in human-computer interaction scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A schematic diagram of the module architecture of a real-time high-fidelity voice-driven digital human system;

[0034] Figure 2 Classification flow chart for a real-time high-fidelity speech-driven digital human system. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] See also Figure 1-Figure 2 The present invention provides a technical solution: a real-time high-fidelity voice-driven digital human system, the system comprising:

[0037] The voice acquisition module is used to collect the user's voice signal in real time. It uses a high-sensitivity microphone array to accurately obtain voice input and perform noise reduction and gain on the collected voice signal. The acquisition module is based on a monophonic analog signal and converts the voltage into a quantized digital signal according to the standard. It follows the sampling theorem to prevent frequency domain aliasing. Before analog-to-digital conversion, an analog low-pass filter is used to filter out high-frequency components to avoid aliasing distortion. After digital-to-analog conversion, a low-pass filter is used to smooth the high-order harmonics of the voice signal and remove harmonic distortion, thereby providing high-quality voice guarantee for subsequent modules.

[0038] Speech recognition: The speech signal acquired by the acquisition module is converted into text information, and speech with various accents, speaking speeds and language styles is recognized. A full-pole model is established and the input speech is compared with it, including speech signal feature extraction and model parameter matching. Then, specific algorithms and methods are used to analyze the comparison data to obtain speech prediction data, and then the prediction data is compared with the ranges or features corresponding to different accents and language styles, and finally the accent and language style of the input speech are determined;

[0039] The semantic analysis module analyzes text information, understands the user's intentions and needs, and after converting the voice signal into text information, performs word segmentation on the text output by the voice recognition module, splits it into words or phrases and marks the part of speech. At the same time, it uses machine learning methods and word vector models in deep learning to assist in lexical analysis, so as to grasp the user's intentions and needs behind the text, and provide a semantic basis for subsequent sentiment analysis and digital human response;

[0040] The sentiment analysis module uses an audio feature extraction algorithm to extract the rhythm, speech rate, and volume features of speech signals, and combines it with a sentiment classification model to accurately identify user emotions. After converting speech signals into text information, it performs word segmentation and part-of-speech tagging, and uses machine learning and deep learning word vector models to assist in lexical analysis to understand user intentions and needs, providing a semantic basis for subsequent sentiment analysis and digital human responses;

[0041] The animation generation module uses the virtual human interaction system to complete the three-dimensional face driving. The voice features are closely related to the movement of the mouth and nose organs of the face to form the character's expression. The model fusion method is used to drive the virtual human expression with the three expression bases of the face. The weight is changed to affect the final expression, thereby showing various emotional states.

[0042] The rendering generation module uses rendering technology to ensure that digital humans can be displayed smoothly on different devices and platforms. The software is used to import a model containing an expression base, and components are called with the help of scripts to change the weights to drive expressions. This module is closely integrated with speech to establish a mapping relationship between speech and facial visual features. Based on the 250ms speech signals before and after each frame of the animation, linear prediction coding is used to represent the contextual speech signal for prediction, so that the virtual human's expression can be adjusted in real time according to the speech content and emotion. Considering the speech context information makes the expression prediction more accurate and coherent, and can be displayed smoothly on different devices and platforms, presenting a natural and vivid interactive effect.

[0043] The present invention will be further described below in conjunction with Example 1 to Example 3:

[0044] Embodiment 1:

[0045] The voice acquisition module uses a monophonic analog signal. After acquisition, the voltage is converted into a quantized digital signal according to the corresponding standard. The signal frequency is 200-1600Hz, of which the signal frequency of boys is in the range of 400-700Hz, and that of girls is in the range of 650-1000Hz. The signal sampling is set to 4000Hz. According to the sampling theorem, the frequency domain aliasing caused by the sampling process is avoided to ensure the fidelity of the digital-to-analog converted voice signal. In the process of voice signal processing, in order to effectively avoid aliasing distortion and minimize noise interference, an analog low-pass filter is used to filter the voice signal before analog-to-digital conversion (A / D). That is, an anti-aliasing filter is added to the voice signal. Preprocessing is performed to filter out the high-frequency components that cause aliasing distortion, ensuring that the subsequent analog-to-digital conversion can be carried out smoothly and accurately, so that the converted digital signal can reflect the characteristics of the original voice signal as realistically as possible. At the same time, after the digital-to-analog conversion (D / A) is completed, a low-pass filter is also required. The low-pass filter is mainly for the reconstructed voice signal, which can smooth the high-order harmonics in the reconstructed voice signal, thereby achieving the purpose of removing high-order harmonic distortion. When the processed voice signal is output, its quality and effect can be better guaranteed, closer to the original voice signal, and provide a high-quality voice foundation for subsequent applications such as voice playback and voice recognition.

[0046] Speech recognition, build a full-pole model, compare it with the input speech, and after data analysis, obtain speech prediction data. According to the comparison of the prediction data, determine the accent and language style of the speech. The measurement formula of the prediction data is:

[0047]

[0048] Where T n Represents the predicted data, a k is the weighting coefficient, n is the input dimension, k is the number of vectors, and P is the sampling value of the point;

[0049] The full-pole model is a mathematical modeling method for speech signals, which is used to describe the characteristics and laws of speech signals. The input speech is compared with the established full-pole model. This step involves feature extraction of speech signals and matching of model parameters. The purpose is to find the relationship and difference between the input speech and the model, and to perform data analysis. On the basis of the comparison, the data is processed and analyzed through certain algorithms and methods to obtain speech prediction data. Finally, the predicted data is compared with the calculated data to determine the accent and language style of the input speech by comparing it with the predicted data range or features corresponding to known different accents and language styles.

[0050] The semantic analysis module performs word segmentation on the text, splitting it into individual words or phrases, while marking the parts of speech, and using machine learning methods and word vector models in deep learning to assist in analyzing the lexical structure.

[0051] Embodiment 2:

[0052] The sentiment analysis module of the present invention first performs logarithmic transformation of the filter bank, compresses dynamic variables, and performs discrete cosine transformation to obtain features. The calculation formula is:

[0053]

[0054] Where S is the output vector, M is the number of filters, G is the number of coefficients, and n is the number of coefficients to be retained;

[0055] And set the eigenvector;

[0056] The eigenvector is B = {r1, r2, r3...r z , r Z};

[0057] where r z represents the feature vector of the tth time step, Z is the length of the feature sequence, and the receiving unit currently inputs the feature vector r z , the hidden state vector r at the previous moment z-1 ;

[0058] The features are extracted and compared with related speech to classify the emotional polarity. Finally, the feature sequence is mapped to the emotional polarity category to classify the speech emotion.

[0059] The animation generation module is based on a model fusion method, in which the facial expressions are composed of three expression bases, and the calculation formula is:

[0060]

[0061] Where H is the comprehensive data of expression bases, H1 is the first expression base, and H t is the tth expression base, P is the weight of the expression base, thus forming facial expression shaping;

[0062] After obtaining the feature vector, it needs to be further processed and analyzed. Some feature extraction algorithms extract more representative and discriminative features from the feature vector, and then compare these features with the relevant speech features collected and annotated in advance. The annotated speech features correspond to different emotional polarities (such as positive, negative, and neutral). The comparison can find the annotated speech that is most similar to the input speech features, thereby preliminarily determining the emotional polarity category of the input speech. Emotional polarity classification and mapping Based on the comparison, the emotional polarity of the input speech is classified using machine learning or deep learning models. The model will learn the feature patterns corresponding to different emotional categories based on the training data, so as to make accurate classification judgments on the new input speech. Finally, the feature sequence is mapped to the emotional polarity category, and the feature sequence of the entire speech signal is corresponding to the finally determined emotional polarity category, so as to facilitate subsequent application and analysis.

[0063] In a speech sentiment analysis system, when a user inputs a speech, the system goes through the above processing flow and finally outputs the sentiment polarity of the speech as "positive". It can also provide corresponding feedback or processing based on this result. For example, in an intelligent system, it can provide better services and recommendations based on the user's positive emotions.

[0064] Embodiment 3:

[0065] The rendering generation module is imported using 3Dmax software. After multiple models carrying expression bases are imported into the scene, the entity carries components. The component model fusion method is called in the script to change the expression base weight to achieve the purpose of virtual human expression driving;

[0066] The rendering generation module is combined with speech, and the speech is mapped to the visual features of the face. The features of each frame of the animation are predicted by the upper and lower speech signals, and the 250ms speech signals before and after the current frame are selected, and the context speech signal is represented using linear prediction coding;

[0067] The driving of the virtual human's expression is achieved by changing the weight of the expression base. Each expression base has a certain role in the generation of the virtual human's expression. The greater the weight, the greater the effect of the expression base on the final expression. When the weight of the smile expression base is increased, the face of the virtual human will show a more obvious smile expression. On the contrary, reducing the weight will weaken the smile. In this way, the expression of the virtual human can be flexibly controlled so that it can show various emotions and states according to needs.

[0068] By combining the rendering generation module with speech and establishing a mapping relationship between speech and facial visual features, the expression changes of the virtual person are not just random, but are associated with the input speech signal. The expression can be adjusted in real time according to the content and emotion of the speech, making the virtual person look more natural and vivid, as if he is really communicating with people.

[0069] The features of each frame of the animation are predicted by the contextual speech signal, and the 250ms speech signals before and after the current frame are selected. A method based on speech signal prediction of animation frame features is adopted, and speech signals within a certain time range before and after the current frame are selected. Taking into account the contextual information of the speech, the expression prediction is more accurate and coherent. When a word expressing surprise appears in the speech, the virtual person will not only make a surprised expression at the moment the word appears, but also make the expression of surprise more natural and logical based on the contextual information of the previous and next speech signals.

[0070] In the present invention, data preparation is performed before the device is used, and a preliminary database is formed. A high-sensitivity microphone array is used to collect user voice signals in real time. To ensure signal quality, noise reduction is first performed on the signals. The acquisition module uses a monophonic analog signal to convert the voltage into a quantized digital signal according to the standard. Frequency domain aliasing is avoided according to the sampling theorem. Before analog-to-digital conversion, an analog low-pass filter is used to filter out high-frequency components to prevent aliasing distortion. After digital-to-analog conversion, a low-pass filter is used to smooth and reconstruct the higher harmonics of the voice signal, and filtering is performed to remove harmonic distortion, thereby providing a high-quality voice foundation for subsequent modules, thereby establishing a full-pole model, and comparing the input voice with the model, which involves voice signal feature extraction and model parameter matching. The comparison data is analyzed through specific algorithms and methods to obtain voice prediction data, and the predicted data is compared with the known The range or features corresponding to different accents and language styles are compared to determine the accent and language style of the input voice, and the voice signal is converted into text information. The text output by the voice recognition module is segmented, the text is split into words or phrases, and the parts of speech are marked. With the help of machine learning methods and word vector models in deep learning, lexical analysis is assisted to understand the intentions and needs of users behind the text, and provide semantic basis for subsequent sentiment analysis and digital human response. After converting the voice signal into text information, the text output by the voice recognition module is segmented, split into words or phrases and marked with parts of speech. At the same time, machine learning methods and word vector models in deep learning are used to assist lexical analysis, so as to grasp the intentions and needs of users behind the text, and provide semantic basis for subsequent sentiment analysis and digital human response.

[0071] Based on the model fusion method, each facial expression is composed of three expression bases. The virtual human expression is driven by changing the weights of the expression bases. The weights of different expression bases determine the degree of their influence on the final expression, so that the virtual human can show various emotional states, conduct emotional polarity analysis, and compare with the analyzed data to form the final expression plan analysis. Finally, 3Dmax software is used to import the model with expression bases, and the expression base weights are changed through script calling component model fusion method to drive the virtual human expression. This module is closely integrated with speech to establish a mapping relationship between speech and facial visual features. The features of each frame of the animation are based on the 250ms speech signals before and after the current frame, and the context speech signal is represented by linear prediction coding for prediction. This enables the virtual human expression to be adjusted in real time according to the speech content and emotion. In addition, due to the consideration of speech context information, the expression prediction is more accurate and coherent, and can be displayed smoothly on different devices and platforms, presenting a natural and vivid interactive effect. The result data of the analysis is finally transmitted to the database to provide further data flow for subsequent analysis.

[0072] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A real-time high-fidelity voice-driven digital human system, characterized in that: The system comprises: The voice acquisition module is used to collect the user's voice signal in real time, using a high-sensitivity microphone array to accurately obtain voice input, and perform noise reduction and gain on the collected voice signal; Speech recognition: the speech signal acquired by the acquisition module is converted into text information, and speech with various accents, speaking speeds and language styles can be recognized; Semantic analysis module, which analyzes text information to understand the user's intentions and needs; The sentiment analysis module uses an audio feature extraction algorithm to extract the rhythm, speech rate, and volume features of speech signals, and combines this with a sentiment classification model to accurately identify user sentiment. The animation generation module uses the virtual human interaction system to complete the three-dimensional face driving, and the close correlation between the voice characteristics and the movement of the mouth and nose organs of the face to form the character's expression; The rendering generation module uses rendering technology to ensure that digital humans can be displayed smoothly on different devices and platforms.

2. A real-time high-fidelity voice-driven digital human system according to claim 1, characterized in that: The voice acquisition module adopts a monophonic analog signal, and after acquisition, the voltage is converted into a quantized digital signal according to the corresponding standard. The signal frequency is 200-1600Hz, of which the signal frequency for boys is in the range of 400-700Hz, and that for girls is in the range of 650-1000Hz. The signal sampling is set to 4000Hz.

3. The real-time high-fidelity voice-driven digital human system according to claim 1, characterized in that: The speech recognition establishes a full-pole model, compares it with the input speech, and obtains speech prediction data after data analysis. According to the comparison of the prediction data, the accent and language style of the speech are determined. The measurement formula of the prediction data is: Where T n Represents the predicted data, a k is the weighting coefficient, n is the input dimension, k is the number of vectors, and P is the sampling value of the point.

4. A real-time high-fidelity voice-driven digital human system according to claim 1, characterized in that: The semantic analysis module performs word segmentation on the text, splits it into words or phrases, and marks the parts of speech, and assists in analyzing the lexical structure based on machine learning methods and word vector models in deep learning.

5. A real-time high-fidelity voice-driven digital human system according to claim 1, characterized in that: The sentiment analysis module first performs logarithmic transformation of the filter bank, compresses dynamic variables, and performs discrete cosine transformation to obtain features. The calculation formula is: Where S is the output vector, M is the number of filters, G is the number of coefficients, and n is the number of coefficients to be retained; And set the eigenvector; The eigenvector is B = {r1, r2, r3...r z , r Z }; where r z represents the feature vector of the tth time step, Z is the length of the feature sequence, and the receiving unit currently inputs the feature vector r z , the hidden state vector r at the previous moment z-1 ; The features are extracted and compared with related speech to classify the emotional polarity. Finally, the feature sequence is mapped to the emotional polarity category to classify the speech emotion.

6. A real-time high-fidelity voice-driven digital human system according to claim 1, characterized in that: The animation generation module is based on a model fusion method, in which the facial expressions are composed of three expression bases, and the calculation formula is: Where H is the comprehensive data of expression bases, H1 is the first expression base, and H t is the tth expression base, and P is the weight of the expression base, thus forming facial expression shaping.

7. The real-time high-fidelity voice-driven digital human system according to claim 1, characterized in that: The rendering generation module is imported using 3Dmax software. After multiple models carrying expression bases are imported into the scene, the entity carries components and calls the component model fusion method in the script to change the expression base weight to achieve the purpose of virtual human expression driving.

8. A real-time high-fidelity voice-driven digital human system according to claim 7, characterized in that: The rendering generation module is combined with speech, and the speech is mapped to the visual features of the face. The features of each frame of the animation are predicted by the upper and lower speech signals, and the 250ms speech signals before and after the current frame are selected, and the context speech signal is represented by linear prediction coding.

Images