3D digital human real-time dialogue interaction system and method

Abstract

The invention discloses a 3D digital human real-time dialogue interaction system and method, and the method comprises the steps: obtaining a voice signal inputted by a user, carrying out the recognition processing of the voice signal through a voice recognition model, obtaining text data, inputting the text data into a text generation model, and outputting the voice text data, and inputting the text data into a voice synthesis model, and outputting the voice synthesis data. The method comprises the steps of inputting speech synthesis data into a 3D expression generation model, outputting 3D expression data, constructing a sound clone model according to a speech signal and the 3D expression data, outputting human voice audio data, performing fine tuning on the sound clone model, and storing fine tuning parameter data, so that a user and a digital human are in smooth and natural conversation, timeliness and continuity of communication are ensured, and user experience is improved. The user experience is greatly improved; by accessing the knowledge base, the digital person has massive knowledge reserve, valuable suggestions and guidance are provided for the user, the user can individually set the figure background of the digital person, and the emotion connection between the user and the digital person is enhanced.

Description

Technical Field

[0001] The present invention belongs to the technical field of voice data processing, and in particular relates to a 3D digital human real-time dialogue interaction system and method. Background Art

[0002] In terms of real-time conversations with 3D digital humans, existing technologies mainly focus on predicting and generating corresponding voice data and facial expressions by analyzing voice signals. Here are some key aspects: (1) Speech Recognition Traditional methods usually use technology based on hidden Markov models (HMMs). First, the input speech signal is preprocessed, including noise removal, endpoint detection and other operations to improve signal quality; then, the speech signal is divided into small segments and its acoustic features, such as Mel-frequency cepstral coefficients (MFCCs), are extracted. These feature vectors are input into the HMM and matched with the pre-trained speech model to identify the speech content. However, this method will reduce the recognition accuracy when faced with complex environmental noise, large accent differences or ambiguous pronunciation; (2) Text Generation Traditional methods are usually based on rules and statistical models. The rule-based method generates text according to pre-set grammatical rules and semantic templates, but this method lacks flexibility and is difficult to cope with the diversity of natural languages. Statistical models use a large amount of text corpus to learn the probabilistic relationship between words, such as the n-gram model. However, this type of model has the problem of data sparsity, poor processing capabilities for low-frequency words and word combinations that have never appeared, and the generated text may lack coherence and logic. (3) Speech Synthesis Traditional methods usually have two main types: parameter-based synthesis and concatenation-based synthesis. Parameter-based synthesis is to model the acoustic parameters of speech signals, such as fundamental frequency, duration, amplitude, etc., generate corresponding parameter sequences according to the text content, and then convert them into speech through a synthesis algorithm. The naturalness of the synthesized speech in this way is relatively low; concatenation-based synthesis is to select appropriate segments from a pre-recorded speech segment library and concatenate them into the target speech, but the scale and quality of the speech library will seriously affect the synthesis effect, and unnatural transitions may occur at the splicing points; (4) 3D facial data Traditional methods usually use keyframe animation technology, where animators first manually create some key facial expression postures as key frames, and then generate intermediate frames through interpolation algorithms to drive the facial expressions of three-dimensional digital people. This method requires a lot of manual production, is inefficient, and it is difficult to achieve natural, smooth and rich and diverse expression changes; in addition, there are also some methods based on the Facial Action Coding System (FACS), which decomposes facial expressions into several basic action units, but the effect is still not ideal for the expression of complex emotions and real-time emotional conversion.

[0003] In summary, the main defects of speech recognition in the prior art are as follows: Inaccurate speech recognition: error correction for dialects, foreign languages, hot words, mixed Chinese and English recognition, and speech emotion recognition; Insufficient diversity in text generation: Existing technologies are mainly based on standard QA and fixed questions and answers; they do not support the memory function of the knowledge base; Insufficient diversity in speech synthesis: Existing technologies have limited voice options, are not natural enough, lack emotion, are difficult to synthesize mixed Chinese and English, and do not support customized voices; Insufficient facial animation of virtual characters: inaccurate voice-expression matching, insufficient expression details, insufficient emotional expression, insufficient expression personalization adaptability, no support for 3D facial animation, only basic lip movements.

[0004] Therefore, there is an urgent need to provide a 3D digital human real-time dialogue interaction system and method to enable 3D digital humans to engage in smooth and natural dialogue and ensure the timeliness and continuity of communication in multiple scenarios. Summary of the invention

[0005] In view of this. The present invention provides a 3D digital human real-time dialogue interaction system and method, which can realize the real-time dialogue function of the 3D digital human, and the performance of the digital human is more vivid and natural, so that users can have a smooth and natural dialogue with the digital human at any time. Whether in information consultation, emotional companionship or entertainment interaction scenes, the timeliness and coherence of communication can be ensured, which greatly improves the user experience. The following technical solutions are specifically adopted to achieve this.

[0006] In a first aspect, the present invention provides a 3D digital human real-time dialogue interaction system, the interaction system comprising: A voice input unit, used to obtain a voice signal input by a user; A speech recognition unit, configured to obtain text data by subjecting the speech signal to recognition processing through a speech recognition model, wherein the speech recognition model uses a variety of recognition algorithms to convert the speech signal into text data, and optimizes the speech recognition model according to a scenario-based setting method; A text generation unit, used for inputting the text data into a text generation model, wherein the text generation model uses a large model as a core algorithm to output speech text data corresponding to the text data, wherein the speech text data includes text content obtained after speech recognition processing of the speech signal; A speech synthesis unit, used for inputting the text data into a speech synthesis model, and the speech synthesis model outputs speech synthesis data corresponding to the text data; A 3D expression generation unit, used for inputting the speech synthesis data into a 3D expression generation model, and the 3D expression generation model outputs 3D expression data corresponding to the speech synthesis data; A knowledge base retrieval unit, used to retrieve the text information retrieved by the RAG knowledge base retrieval model and the text information set by the system-prompt according to the personal database uploaded by the user and convert it into text vector data storage, and / or perform a knowledge base retrieval of text data according to the voice text data, and send the retrieved text data together with the text data of the user's question to the large model; The sound cloning unit is used to construct a sound cloning model according to the voice signal and the 3D expression data and output human voice audio data, fine-tune the sound cloning model and save fine-tuning parameter data corresponding to the human voice audio data.

[0007] As a preferred embodiment of the above technical solution, the interactive system further includes: The training data acquisition unit is used to obtain the training information uploaded by the user and set the preset background information for the digital human to conduct anthropomorphic dialogue and communication.

[0008] As a preferred embodiment of the above technical solution, the recognition algorithm includes at least one of an Encoder, a Predictor, a Sampler, a Decoder or a Loss function algorithm. The Encoder algorithm is used to perform preliminary feature extraction and encoding on the input speech signal. The Predictor algorithm is used to predict the number of target characters and extract the acoustic vectors corresponding to the target characters. The Sampler algorithm is used to generate a feature vector containing semantics according to the input acoustic vector and the target vector. The Decoder algorithm is used to generate corresponding text content according to preset rules based on the feature vector of semantic information, and convert the acoustic features corresponding to the speech signal into text data. The Loss function algorithm guides the model to continuously adjust parameters by calculating the difference between the predicted result and the true label. The scenario-based setting method includes scenario-based setting and differentiation of poor recognition effects, and scenario-based setting and differentiation of homophones.

[0009] As a preferred embodiment of the above technical solution, the RAG knowledge base retrieval model collects, organizes and stores various text materials to provide rich background knowledge and reference basis for text generation; The text information set by the system-prompt is used to guide the large model to generate text according to specific styles, rules, and task requirements.

[0010] As a preferred embodiment of the above technical solution, the large model combines the text information retrieved by the RAG knowledge base retrieval model and the text information set by the system-prompt to make corresponding text output, and performs preprocessing operations on the text output, wherein the preprocessing operations include merging or retaining sentences of the text information according to punctuation and word count restrictions.

[0011] As a preferred embodiment of the above technical solution, the retrieval method of the RAG knowledge base retrieval model includes that after the text generation model receives the input text, the RAG database retrieval model performs a matching search in the knowledge base according to the keyword information in the text, and uses a similarity algorithm based on a vector space model to find text fragments associated with the input text; the text information set by the system-prompt includes writing specific text content according to corresponding parameter formats and rules.

[0012] As a preferred embodiment of the above technical solution, the fine-tuning process of the sound cloning unit includes: Sound data preprocessing: receiving background clean sound and 15s voice audio data input by the user; Feature extraction: Use VQGAN to encode the human voice audio data to obtain semantic tokens; Model fine-tuning: Input the encoded semantic tokens and the corresponding text into the autoregressive language model as examples for training, and adjust the model parameters; Save fine-tuning parameters: After fine-tuning is completed, save the fine-tuning parameter data of the corresponding sound, wherein the fine-tuning parameter data includes the weight adjustment of the language model and VQGAN related parameters.

[0013] As a preferred embodiment of the above technical solution, the speech synthesis data is input into a 3D expression generation model, comprising: Extracting features from the speech synthesis data to obtain speech feature information, and inputting the speech feature information into a trained emotion classification model for training to identify the emotion classification result corresponding to the speech input data, wherein the emotion classification model is obtained by training a neural network, and the emotion classification result includes identifying the emotion category contained in the speech input data; Inputting the trained basic expression generation model according to the speech feature information and the emotion classification result to perform training to obtain expression data, wherein the basic expression generation model is obtained by training the diffusion model; The expression data and preset personalized features are fine-tuned to obtain a personalized expression generation model, and a personalized emotional expression corresponding to the voice signal is generated based on the emotion classification result and the personalized expression generation model, wherein the preset personalized features include style features extracted by a style encoder from preset audio data, and the style encoder is trained by a temporal convolutional neural network.

[0014] As a preferred embodiment of the above technical solution, the execution process of the knowledge base retrieval unit includes: In the text generation process, after the user uploads the commonly used office data, the document data is extracted and converted into text vector data storage, wherein the types of the document data include txt, pptx, docx, and the data content includes text data and picture data; When users are having a normal conversation or chatting, the knowledge base of the text data is searched based on the voice text data input by the user, and the retrieved text data is sent to the big model together with the text data of the user's question.

[0015] In a second aspect, the present invention further provides a 3D digital human real-time dialogue interaction method, which is applied to the above-mentioned 3D digital human real-time dialogue interaction system, and comprises the following steps: Obtaining the voice signal input by the user; The speech signal is recognized and processed by a speech recognition model to obtain text data, wherein the speech recognition model uses a variety of recognition algorithms to convert the speech signal into text data, and the speech recognition model is optimized according to a scenario-based setting method; The text information retrieved by the RAG knowledge base retrieval model and the text information set by the system-prompt are retrieved according to the personal database uploaded by the user and converted into text vector data for storage, and / or the knowledge base retrieval of text data is performed according to the voice text data, and the retrieved text data is sent to the large model together with the text data of the user's question; The text data is input into a text generation model, and the text generation model uses a large model as a core algorithm to output speech text data corresponding to the text data, wherein the speech text data includes text content obtained after speech recognition processing of the speech signal; Inputting the text data into a speech synthesis model, the speech synthesis model outputting speech synthesis data corresponding to the text data; Inputting the speech synthesis data into a 3D expression generation model, and the 3D expression generation model outputs 3D expression data corresponding to the speech synthesis data; A sound cloning model is constructed according to the speech signal and the 3D expression data and human voice audio data is output, the sound cloning model is fine-tuned and fine-tuning parameter data corresponding to the human voice audio data is saved.

[0016] The present invention provides a 3D digital human real-time dialogue interaction system and method, which obtains a voice signal input by a user, identifies and processes the voice signal through a voice recognition model to obtain text data, inputs the text data into a text generation model and outputs voice text data using a large model as a core algorithm, inputs the text data into a voice synthesis model and outputs voice synthesis data, inputs the voice synthesis data into a 3D expression generation model and outputs 3D expression data, constructs a sound cloning model according to the voice signal and the 3D expression data and outputs human voice audio data, fine-tunes the sound cloning model and saves the fine-tuning parameter data, so that the user and the digital human can have a smooth and natural dialogue; in multiple interactive scenes, the timeliness and coherence of communication can be ensured, greatly improving the user experience; by accessing a knowledge base, the digital human has a massive knowledge reserve, provides valuable suggestions and guidance to the user, and becomes a reliable source for the user to acquire knowledge; the user can personalize the character background of the digital human, thereby enhancing the emotional connection between the user and the digital human, and the use of voice cloning also improves the authenticity and intimacy of the digital human. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 This is a structural block diagram of the 3D digital human real-time dialogue interaction system provided by the present invention; Figure 2 A flow chart of the 3D digital human real-time dialogue interaction method provided by the present invention; Figure 3 This is an overall working process diagram of the 3D digital human real-time dialogue interaction method provided by the present invention. DETAILED DESCRIPTION

[0019] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0020] The implementation process of the present invention can be divided into cold start and hot start. In the cold start, no training information or audio information required for voice cloning training is input, and the user can directly interact with the 3D digital human. In the hot start, training information needs to be input before the dialogue interaction, personal knowledge base data needs to be uploaded, and reference audio data of the voice to be cloned needs to be uploaded. After completing the above operations, the dialogue interaction can be carried out. In this way, the 3D digital human can make a sound similar to the reference audio, and at the same time has the training information and the content of the knowledge base. The entire system is end-to-end and can generate expressions in real time.

[0021] See also Figure 1 The present invention provides a 3D digital human real-time dialogue interaction system, the interaction system comprising: A voice input unit, used to obtain a voice signal input by a user; A speech recognition unit, configured to obtain text data by subjecting the speech signal to recognition processing through a speech recognition model, wherein the speech recognition model uses a variety of recognition algorithms to convert the speech signal into text data, and optimizes the speech recognition model according to a scenario-based setting method; A text generation unit, used for inputting the text data into a text generation model, wherein the text generation model uses a large model as a core algorithm to output speech text data corresponding to the text data, wherein the speech text data includes text content obtained after speech recognition processing of the speech signal; A speech synthesis unit, used for inputting the text data into a speech synthesis model, and the speech synthesis model outputs speech synthesis data corresponding to the text data; A 3D expression generation unit, used for inputting the speech synthesis data into a 3D expression generation model, and the 3D expression generation model outputs 3D expression data corresponding to the speech synthesis data; A knowledge base retrieval unit, used to retrieve the text information retrieved by the RAG knowledge base retrieval model and the text information set by the system-prompt according to the personal database uploaded by the user and convert it into text vector data storage, and / or perform a knowledge base retrieval of text data according to the voice text data, and send the retrieved text data together with the text data of the user's question to the large model; The sound cloning unit is used to construct a sound cloning model according to the voice signal and the 3D expression data and output human voice audio data, fine-tune the sound cloning model and save fine-tuning parameter data corresponding to the human voice audio data.

[0022] In this embodiment, the interactive system further includes: a training data acquisition unit, which is used to obtain the training information uploaded by the user and set the preset background information for the digital human to conduct anthropomorphic dialogue communication. The user input voice signal is the starting point of the entire process. The input voice signal is processed by the voice recognition model, and the voice recognition model converts the user's voice signal into text data. At the same time, the voice recognition model can use the hot word library function to give priority to the recognition of poor recognition effects or homophones in specific scenarios. Users can add proper nouns and words that are easily misrecognized according to their own business scenarios to improve the accuracy of the voice recognition model. The text generation model uses the large model as the core algorithm, and outputs the corresponding text according to the user's text input. In this process, the text information retrieved by the RAG knowledge base retrieval model and the text information set by the system-prompt can also be combined and input to the large model. The large model will combine these information to make corresponding text output. In addition, the text output of the large model will be preprocessed, and the text output will be segmented into sentences according to punctuation marks (such as Chinese and English periods, commas, question marks, exclamation marks, etc.) and word segmentation restrictions. Each sentence of text data is input into the speech synthesis model, which will output the speech synthesis data corresponding to each sentence. Each segment of speech synthesis data is input into the 3D expression generation model, which will output the corresponding segment of 3D expression data.

[0023] Among them, during the text generation process, the RAG knowledge base retrieval model will automatically search according to the personal database uploaded by the user. When the user uploads the commonly used office data, the system will extract the document data and convert it into text vector data storage; when the user is in normal conversation and chat, the knowledge base of the relevant data will be retrieved according to the voice text data entered by the user, and the retrieved relevant data will be sent to the big model together with the text data of the user's question to enrich the digital person's answer content. The sound cloning unit only needs the user to input the human voice audio data with clean background sound and a duration of about 15s, and it can complete the fine-tuning of the sound cloning model within 3s and save the fine-tuning parameter data of the sound. When the user needs to use the corresponding sound clone, he only needs to select the corresponding fine-tuning parameter data to infer and output the corresponding sound in real time. Users can upload training information and set preset background information for digital people, so that digital people can play roles or become experts in solving special tasks. This setting allows digital people to better solve certain skill problems or communicate in a more humanized way.

[0024] It should be noted that during cold start, there is no need to enter any training information or audio information required for voice cloning training. Users can directly interact with 3D digital humans. This method is simple and direct and suitable for users who do not have special voice or knowledge requirements for digital humans. Hot start requires some preparation before dialogue interaction. Users need to enter training information, such as uploading personal knowledge base data, including txt, pptx, docx and other commonly used office data. These data will be automatically extracted and converted into text vector data and stored in the vector database. In addition, users also need to upload the reference audio data of the sound they want to clone. After completing these operations, users can interact with 3D digital humans. In this mode, 3D digital humans can make sounds similar to the reference audio and can communicate using the content of the knowledge base.

[0025] It should be understood that by obtaining the voice signal input by the user, the voice signal is recognized and processed by the voice recognition model to obtain text data, the text data is input into the text generation model and the voice text data is output using the large model as the core algorithm, the text data is input into the voice synthesis model and the voice synthesis data is output, the voice synthesis data is input into the 3D expression generation model and the 3D expression data is output, a sound cloning model is constructed according to the voice signal and the 3D expression data and the human voice audio data is output, the sound cloning model is fine-tuned and the fine-tuning parameter data is saved, so that the user and the digital human can have a smooth and natural conversation; in multiple interactive scenarios, the timeliness and consistency of communication can be ensured, greatly improving the user experience; by accessing the knowledge base, the digital human has a massive knowledge reserve, providing users with valuable suggestions and guidance, and becoming a reliable source of knowledge for users; users can personalize the character background of the digital human, which enhances the emotional connection between the user and the digital human, and the use of voice cloning also improves the realism and intimacy of the digital human.

[0026] Optionally, the recognition algorithm includes at least one of an Encoder, Predictor, Sampler, Decoder or Loss function algorithm, wherein the Encoder algorithm is used to perform preliminary feature extraction and encoding on the input speech signal, the Predictor algorithm is used to predict the number of target characters and extract the acoustic vector corresponding to the target characters, the Sampler algorithm is used to generate a feature vector containing semantics according to the input acoustic vector and the target vector; the Decoder algorithm is used to generate corresponding text content according to preset rules based on the feature vector of semantic information, and convert the acoustic features corresponding to the speech signal into text data; the Loss function algorithm guides the model to continuously adjust parameters by calculating the difference between the predicted result and the true label; The scenario-based setting method includes scenario-based setting and differentiation of poor recognition effects, and scenario-based setting and differentiation of homophones.

[0027] In this embodiment, the RAG knowledge base retrieval model collects, organizes, and stores various types of text materials to provide rich background knowledge and reference basis for text generation; the text information set by the system-prompt is used to guide the large model to generate text according to specific styles, rules, and task requirements. The large model combines the text information retrieved by the RAG knowledge base retrieval model and the text information set by the system-prompt to make corresponding text outputs, and performs preprocessing operations on the text output, wherein the preprocessing operations include merging or retaining sentences of the text information according to punctuation and word count restrictions. The retrieval method of the RAG knowledge base retrieval model includes that after the text generation model receives the input text, the RAG database retrieval model performs a matching search in the knowledge base based on the keyword information in the text, and uses a similarity algorithm based on a vector space model to find text fragments associated with the input text; the text information set by the system-prompt includes writing specific text content according to corresponding parameter formats and rules.

[0028] It should be noted that the specific process of converting speech signals into text data is as follows: the speech recognition model is generally composed of Encoder, Predictor, Sampler, Decoder and Loss function recognition algorithm to realize the conversion of speech to text; Encoder can adopt different network structures, such as self-attention, conformer, SAN-M, etc. Its function is to perform preliminary feature extraction and encoding of the input speech signal to prepare suitable acoustic representation for subsequent processing; Predictor is a two-layer FFN (feedforward neural network), which is based on the Continuous integrate-and-fire (CIF) mechanism to predict the number of target characters and extract the acoustic vectors corresponding to the target characters. For example, it outputs a floating point number between 0 and 1, and predicts the number of target characters by accumulating these output values, and can extract the corresponding acoustic feature vector to more accurately grasp the acoustic information related to the target characters in the speech, and assist in the subsequent text generation; Sampler is a module without learnable parameters. It produces a feature vector containing semantics based on the input acoustic vector and target vector. By sampling, it transforms the acoustic feature vector and the target character vector into a feature vector containing semantic information, and cooperates with the bidirectional Decoder to enhance the model's modeling ability for context, making the generated text more reasonable in terms of semantic coherence. The structure of the Decoder is similar to the autoregressive model, but it is a bidirectional modeling (autoregressive is a unidirectional modeling). Based on the feature vector containing semantic information provided by the previous module, it generates the corresponding text content according to certain rules and logic, thereby gradually converting the acoustic features corresponding to the speech signal into the final text data; Loss In addition to the cross entropy (CE) and MWER (minimum word error rate) discriminative optimization objectives, the loss function also includes the Predictor optimization objective MAE (mean absolute error). These loss functions play a role in the model training phase. By calculating the difference between the predicted results and the true labels, the model is guided to continuously adjust parameters and optimize the accuracy of speech-to-text conversion and other performance.

[0029] Among them, the scenario settings and distinction examples for poor recognition or homophones: In the hospital registration scenario, there are often patients with similarly pronounced names, such as "Zhang Li" and "Zhang Li", which are prone to recognition confusion; hospitals can add the names of patients who often come to the hospital for treatment (these proper nouns) to the hot word library of the speech recognition model according to their own business scenarios, and make special annotations for homophones and different meanings. In this way, when patients say their own names to register, the model will give priority to matching and recognizing according to the information added and annotated in the hot word library, reducing the misrecognition caused by homophones, and accurately identifying whether it is "Zhang Li" or "Zhang Li" to complete the registration of relevant information for the registration process. In short, the above-mentioned model modules work together to achieve speech-to-text conversion, and at the same time, with the help of functions such as hot word libraries, scenario settings are made for poor recognition and homophones to improve recognition accuracy.

[0030] Among them, the differences between the text information retrieved by the RAG knowledge base retrieval model and the text information set by the system-prompt include different sources and functions. In terms of different sources, the text information retrieved by the RAG knowledge base retrieval model: it comes from a pre-built knowledge base, which is formed by collecting, organizing, and storing various text materials, etc. It contains knowledge content from different fields and different topics, aiming to provide rich background knowledge and reference basis for text generation. For example, a knowledge base in the medical field stores many medical-related text materials such as disease descriptions, treatment methods, and drug information; the text information set by the system-prompt: it is text content set by users or developers based on specific needs, application scenarios, and expected guidance directions. It is more like a guiding and limiting instruction information used to tell the big model some pre-requirements, style preferences, or specific task instructions. For example, setting "Please reply to users' questions about pet fun in a relaxed and humorous style" is the system-prompt setting text information that clarifies the style of the reply and the general topic direction.

[0031] In different aspects of function: text information retrieved by the RAG knowledge base retrieval model: mainly to supplement text generation with more relevant factual knowledge, professional content, etc., to help the text generated by the big model to be more professional, accurate and complete. For example, when answering the question "What should patients with hypertension pay attention to in their daily diet", relevant text information about dietary taboos for hypertension retrieved from the medical knowledge base can enable the reply generated by the big model to contain correct knowledge content such as specific food taboos; text information set by system-prompt: focuses on guiding the big model to generate text according to specific styles, rules, task requirements, etc. For example, setting "the number of words in the reply content is controlled between 100-200 words" standardizes the length of the text output by the big model; or setting "explain the impact of a historical event in a rigorous academic style" so that the generated text meets the corresponding requirements in terms of tone, wording and other styles.

[0032] Specifically, the RAG knowledge base retrieval model retrieves text information in a way and with examples. Retrieval method: When the text generation model receives the input text, the RAG knowledge base retrieval model will match and search in the knowledge base based on the keywords, topics and other information in the text. It will use some information retrieval algorithms, such as similarity calculation based on the vector space model, to find text fragments that are highly relevant to the input text. For example, if the input text is "What exercises are suitable for diabetic patients", the retrieval model will extract keywords such as "diabetes" and "exercise", and then search the knowledge base for text content containing these keywords and related to exercise recommendations. Content such as "diabetic patients can appropriately perform aerobic exercises such as walking and Tai Chi. It is recommended to exercise 3-5 times a week..." may be retrieved. Example: Suppose in a financial knowledge base, the user inputs the text "How do novice stock investors choose stocks?" The RAG knowledge base retrieval model searches for keywords such as "stock investment," "novice," and "choose stocks" to find relevant text information from the knowledge base, such as "Novice investors can start with industry leaders, pay attention to company fundamentals, price-to-earnings ratio and other indicators, and refer to ratings from professional institutions..." This information is then provided to the big model to generate a more reasonable and accurate response text.

[0033] Specifically, the method and example of setting text information in system-prompt: Setting method: Usually in the relevant configuration interface or code of the text generation system, the specific text content is written according to the corresponding parameter format and rules. For example, when developing an intelligent customer service system, in response to the user's inquiry about the product return and exchange policy, the developer can set the system-prompt text information in the background to "Please respond to the user's questions about product returns and exchanges in a polite and clear language according to the company's established return and exchange terms. The number of words in the reply should be controlled within 300 words." Example: If you want to build a story generation system, and hope that the generated story is in the style of a fairy tale and revolves around small animals as the protagonists, you can set the system-prompt text information to "Please create a story in the style of a fairy tale, and the protagonists of the story are set as small animals such as bunnies and foxes. The story content should contain interesting plots and positive morals." In this way, after the big model receives this setting information, the generated text will meet the requirements of the corresponding fairy tale style and character settings. In short, these two different text information sources have different functions, and participate in the text generation process in their own ways, and jointly help the big model output text content that meets the requirements.

[0034] Specifically, in the text generation unit, the text segmentation and merging output process based on punctuation marks is as follows: 1. Split text word by word by punctuation: Since the big model outputs words one by one, the system will monitor each word in real time during the processing. Once punctuation marks appear (such as ".", ","?", "!" in Chinese, ".", ","?", "!" in English, etc.), the output content will be divided into a sentence fragment based on this punctuation mark. For example, the big model outputs the characters "today", ","", "weather", "very", "good", ".", "we" "go", "travel", "play", ".", then according to the punctuation marks, the sentence fragments "today", "the weather is very good", and "we go to travel" will be divided first.

[0035] 2. Merge or retain sentences based on word count limit: Determine the number of words in the first sentence: For the first sentence segment, count its word count (Chinese characters are counted as single characters, English letters are counted the same way, words are counted by the number of letters they consist of, etc., depending on the specific settings). If the sentence does not exceed 10 words, try to merge it with the next sentence segment, and use the merged sentence as the new first sentence to continue to determine whether its word count meets the requirements; if the first sentence itself has more than 10 words, directly output it as the first sentence. For example, for the "today" segmented earlier, the number of words is 2, which does not exceed 10 words, then look at the following sentence "the weather is very good", merge them into "the weather is very good today", and then determine the number of words in the merged sentence. Here it is 6 words, which still meets the requirements and can continue to participate in the subsequent processing flow; Determine the number of words in the subsequent sentences and process them in turn: After processing the first sentence, operate the subsequent sentences in the same way. For example, for "Let's go play", the number of words is 5 words. If it is followed by "feel", "feel", "very", "happy", "heart", ".", the sentence fragment "feeling very happy" is segmented. Because the number of words in "Let's go play" does not exceed 10 words, the two are merged into "Let's go play and feel very happy". Continue to determine whether the number of words in the merged sentence exceeds 10 words. Here it is 10 words, which just meets the requirements, and it can be output in this sentence form. If another sentence fragment is segmented in the next round, such as "hope", "hope", "next", "next", "next", "come", ".", that is, "hope to come again next time", its number of words is 6 words, not more than 10 words, it can be merged with the previous sentence again into "Let's go play, feel very happy, hope to come again next time", and then determine the overall number of words. Repeat this process until all the content output by the large model is processed according to this rule.

[0036] The example shows the entire processing process: suppose the large model outputs the following characters in sequence: "Today," we "went" to the "park," and saw "a lot" of "flowers," some red and some yellow, which were "very beautiful." "Everyone" likes it very much."

[0037] Split by punctuation: First, we get the following sentence fragments: "Today", "we went to the park", "we saw a lot of flowers", "red and yellow", "very beautiful", "everyone likes it here very much".

[0038] Processing by word count and merging rules: 1) "Today" has 2 words, less than 10 words, and is merged with the next sentence "We go to the park" to become "Today we go to the park", with 7 words, which meets the requirements; 2) "Today we go to the park" is merged with the next sentence "I saw a lot of flowers" to get "Today we went to the park and saw a lot of flowers", with 12 words, which exceeds 10 words, so "Today we went to the park and saw a lot of flowers" is output as the first sentence; 3) Next, look at "There are red and yellow", with 6 words, less than 10 words, and is merged with the next sentence "Very beautiful" to become "There are red, yellow, very beautiful", with 10 words, which meets the requirements and can be output as the second sentence; 4) Then look at "Everyone likes it here very much", with 8 words, less than 10 words. If there is no new sentence output later, it will be output as the third sentence. After preprocessing, the output sentences are: "Today we went to the park and saw a lot of flowers", "There are red and yellow ones, very beautiful", "Everyone likes it here". In this way, the sentences are segmented according to punctuation marks, and combined with the word limit to merge or output them separately, thus realizing the preprocessing operation of the large model's word-by-word output text to meet the corresponding requirements.

[0039] In the speech synthesis unit, the differences between speech synthesis data and text data include different data forms, different ways of carrying information, and different application scenarios and dependent conditions, as follows: Different data forms: Text data: information in the form of text characters, such as Chinese characters, English words and various punctuation marks, which are sequences composed of certain grammatical and semantic rules, such as "The weather is nice today." This intuitively presented text content mainly relies on vision to read and understand, and is a discrete, symbolic representation method; Speech synthesis data: usually in the form of digitized audio signals, which are obtained by sampling and quantizing the sound waveform. For example, the amplitude information of the sound at different times is collected at a specific sampling frequency (such as the common 16kHz, 44.1kHz, etc.), and it is converted into binary data that can be stored and processed by the computer, and finally presented in the form of audio files (such as .wav, .mp3, etc.). People need to perceive the content it conveys through hearing.

[0040] Different ways of carrying information: Text data: relies on the semantic function and grammatical structure of text to convey meaning, and grasps the semantic, emotional, logical and other information through the reader's understanding of words and analysis of sentence structure. For example, "she smiled happily". Through these few words, we can understand that it describes someone who is in a happy emotional state and has the behavior of laughing; Speech synthesis data: in addition to the content at the semantic level, it also incorporates many characteristics of speech to convey information. It will have changes in pitch to reflect the tone. For example, the pitch will rise when expressing questions. The speed of speech can convey different emotions or scene characteristics. For example, a rapid speaking speed may indicate anxiety, and a slow speaking speed may reflect calmness. The timbre can shape different sense of roles, such as a soft female voice timbre, a calm male voice timbre, etc. Therefore, speech synthesis data is more three-dimensional and rich when conveying information, and contains many expressive factors at the speech level.

[0041] Different application scenarios and dependent conditions: Text data: widely used in many scenarios that require reading, recording, editing, etc., such as writing articles, reading books, editing documents, etc., as long as there is a display device that can present text, it does not rely on the audio-related functions of the hardware; Speech synthesis data: more used in scenarios where information needs to be conveyed through sound, such as voice navigation, audiobooks, smart voice assistant replies, etc., relying on audio playback devices (such as speakers, headphones, etc.) to output sound for people to hear the corresponding content. The speech synthesis model does not classify and sort each sentence of text data; It simply converts text data into audio data.

[0042] Optionally, the fine-tuning process of the sound cloning unit includes: Sound data preprocessing: receiving background clean sound and 15s voice audio data input by the user; Feature extraction: Use VQGAN to encode the human voice audio data to obtain semantic tokens; Model fine-tuning: Input the encoded semantic tokens and the corresponding text into the autoregressive language model as examples for training, and adjust the model parameters; Save fine-tuning parameters: After fine-tuning is completed, save the fine-tuning parameter data of the corresponding sound, wherein the fine-tuning parameter data includes the weight adjustment of the language model and VQGAN related parameters.

[0043] In this embodiment, the human voice audio data is a real sound record, which is recorded in the form of physical sound waves and is usually stored in a specific audio format (such as MP3, WAV, etc.). It contains various acoustic features in human voice, such as pitch, timbre, volume, length of sound, and various subtle voice changes, such as tone, emotional expression, etc. For example, when a person speaks happily or sadly, the human voice audio data can record the changes in pitch, volume, etc. caused by these emotions. The voice text data is mainly text content, which is a text transcription of the voice content. It only records the words spoken by the speaker and does not contain the acoustic features of the sound itself. For example, the voice text data only records the words of the sentence "I am very happy today", regardless of whether the speaker speaks in a cheerful tone or a plain tone.

[0044] It should be noted that the implementation process of fine-tuning is as follows: sound data preprocessing: first, the user inputs human voice audio data with clean background sound and a duration of about 15s; feature extraction: use VQGAN to encode the audio and obtain semantic tokens; model fine-tuning: input the encoded semantic tokens and corresponding text into the Auto-Regressive Language Model as examples for training. This process will adjust the parameters of the model so that it can better adapt to the characteristics of specific sounds; save fine-tuning parameters: after fine-tuning is completed, save the fine-tuning parameter data of the corresponding sound. Among them, the fine-tuning parameter data includes: language model weight adjustment: the fine-tuning parameter data mainly includes the weights adjusted by the language model (Auto-Regressive Language Model) during the training process. These weights reflect the model's learning results for specific sound features; VQGAN related parameters: VQGAN is also adjusted during the fine-tuning process, so its related parameters may also be included in the fine-tuning parameter data.

[0045] Optionally, inputting the speech synthesis data into a 3D expression generation model comprises: Extracting features from the speech synthesis data to obtain speech feature information, and inputting the speech feature information into a trained emotion classification model for training to identify the emotion classification result corresponding to the speech input data, wherein the emotion classification model is obtained by training a neural network, and the emotion classification result includes identifying the emotion category contained in the speech input data; Inputting the trained basic expression generation model according to the speech feature information and the emotion classification result to perform training to obtain expression data, wherein the basic expression generation model is obtained by training the diffusion model; The expression data and preset personalized features are fine-tuned to obtain a personalized expression generation model, and a personalized emotional expression corresponding to the voice signal is generated based on the emotion classification result and the personalized expression generation model, wherein the preset personalized features include style features extracted by a style encoder from preset audio data, and the style encoder is trained by a temporal convolutional neural network.

[0046] In this embodiment, the voice input data is obtained by obtaining a voice signal input by the user and performing data preprocessing on the voice signal, the voice input data is feature extracted using the voice model HuBERT to obtain voice feature information, the voice feature information is input into a trained emotion classification model for training to identify the emotion classification result corresponding to the voice input data, the voice feature information and the emotion classification result are input into a trained basic expression generation model for training to obtain expression data, the expression data and preset personalized features are fine-tuned for personalized expression to obtain a personalized expression generation model, personalized emotional expressions corresponding to the voice signal are generated based on the emotion classification result and the personalized expression generation model, the voice input of any user can be processed, and corresponding expression parameters with emotion are generated, the voice data is processed for timbre migration, the robustness of the expression generation model to different voice timbres can be greatly improved, the existing basic expression generation model is personalized by fine-tuning, so as to learn the unique expression style and speaking habits of a specific user, and the corresponding emotional expressions can be accurately generated through voices with different emotions.

[0047] Optionally, the execution process of the knowledge base retrieval unit includes: In the text generation process, after the user uploads the commonly used office data, the document data is extracted and converted into text vector data storage, wherein the types of the document data include txt, pptx, docx, and the data content includes text data and picture data; When users are having a normal conversation or chatting, the knowledge base of the text data is searched based on the voice text data input by the user, and the retrieved text data is sent to the big model together with the text data of the user's question.

[0048] In this embodiment, three types of documents, txt, pptx and docx, are currently supported, and the data content includes text data and image data. When the user asks through voice, "I want to know who is the highest-performing employee in the office" (this is voice text data), the system will use this voice text data to retrieve the text data previously converted from common office data (such as the specific text content that records the employee's performance score), find the relevant performance data information, and then send it together with the voice text data of the user's question to the big model to improve the answer content. Among them, the text data itself is not contained in the voice text data, but the voice text data is often used as a trigger clue to guide the system to call the related text data to play a role in generating answers together.

[0049] See also Figure 2 and Figure 3 The present invention also provides a 3D digital human real-time dialogue interaction method, which is applied to the above-mentioned 3D digital human real-time dialogue interaction system, and includes the following steps: S1: Obtain the voice signal input by the user; S2: The speech signal is recognized and processed by a speech recognition model to obtain text data, wherein the speech recognition model uses a variety of recognition algorithms to convert the speech signal into text data, and the speech recognition model is optimized according to a scenario-based setting method; S3: The text information retrieved by the RAG knowledge base retrieval model and the text information set by the system-prompt are retrieved according to the personal database uploaded by the user and converted into text vector data for storage, and / or the knowledge base retrieval of text data is performed according to the voice text data, and the retrieved text data is sent to the large model together with the text data of the user's question; S4: inputting the text data into a text generation model, wherein the text generation model uses a large model as a core algorithm to output speech text data corresponding to the text data, wherein the speech text data includes text content obtained after speech recognition processing of the speech signal; S5: inputting the text data into a speech synthesis model, and the speech synthesis model outputs speech synthesis data corresponding to the text data; S6: inputting the speech synthesis data into a 3D expression generation model, and the 3D expression generation model outputs 3D expression data corresponding to the speech synthesis data; S7: constructing a sound cloning model according to the voice signal and the 3D expression data and outputting human voice audio data, fine-tuning the sound cloning model and saving fine-tuning parameter data corresponding to the human voice audio data.

[0050] In this embodiment, the above-mentioned 3D digital human real-time dialogue interaction mainly involves 3D digital human real-time dialogue, 3D digital human knowledge base, 3D digital human character background setting, 3D digital human voice cloning and 3D digital human 3D expression data. The present invention realizes the real-time dialogue function of 3D digital human, which has great advantages over traditional technologies. Traditional technologies may have delays or unsmoothness during interaction, while the present invention breaks this time limit, allowing users to have natural and smooth dialogues with digital humans at any time, whether in information query, emotional communication or entertainment scenes, to ensure the timeliness and coherence of communication, greatly improving the user experience, and making users feel like they are communicating with real people. By accessing a broad and in-depth knowledge base, digital humans can have a rich knowledge reserve. The knowledge in the knowledge base involves multiple fields such as science, history, culture, and art, which enables digital humans to answer questions more accurately and comprehensively, provide users with valuable suggestions and guidance, and become a reliable source for users to acquire knowledge.

[0051] In addition, users can customize a unique character background for the digital human according to their needs and preferences, including the digital human's growth experience, personality traits, professional identity, etc. This personalized setting gives each digital human its own unique personality, which can better meet the diverse needs of different users in different scenarios and enhance the emotional connection between users and digital humans. Users can copy their own or others' voices to digital humans, allowing digital humans to express themselves in familiar voices when communicating. This not only improves the digital human's sense of reality and intimacy, but also plays an important role in application scenarios such as role-playing and voice assistant customization, bringing users a new experience. The system can generate 3D expression data in real time, and output it to the front-end 3D digital human interaction module together with voice data for voice and expression driving, making the digital human's performance more vivid and natural.

[0052] In all examples shown and described herein, any specific values ​​should be interpreted as merely exemplary and not as limiting, and thus other examples of the exemplary embodiments may have different values.

[0053] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0054] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the present invention. It should be pointed out that for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A 3D digital human real-time dialogue interaction system, characterized in that: The interactive system comprises: A voice input unit, used to obtain a voice signal input by a user; A speech recognition unit, configured to obtain text data by subjecting the speech signal to recognition processing through a speech recognition model, wherein the speech recognition model uses a variety of recognition algorithms to convert the speech signal into text data, and optimizes the speech recognition model according to a scenario-based setting method; A text generation unit, used for inputting the text data into a text generation model, wherein the text generation model uses a large model as a core algorithm to output speech text data corresponding to the text data, wherein the speech text data includes text content obtained after speech recognition processing of the speech signal; A speech synthesis unit, used for inputting the text data into a speech synthesis model, and the speech synthesis model outputs speech synthesis data corresponding to the text data; A 3D expression generation unit, used for inputting the speech synthesis data into a 3D expression generation model, and the 3D expression generation model outputs 3D expression data corresponding to the speech synthesis data; A knowledge base retrieval unit, used to use the text information retrieved by the RAG knowledge base retrieval model and the text information set by the system-prompt to retrieve and convert the text information into text vector data storage according to the personal database uploaded by the user, and / or to perform a knowledge base retrieval of text data according to the voice text data, and send the retrieved text data together with the text data of the user's question to the large model; The sound cloning unit is used to construct a sound cloning model according to the voice signal and the 3D expression data and output human voice audio data, fine-tune the sound cloning model and save fine-tuning parameter data corresponding to the human voice audio data.

2. The 3D digital human real-time dialogue interaction system according to claim 1, characterized in that: The interactive system further comprises: The training data acquisition unit is used to obtain the training information uploaded by the user and set the preset background information for the digital human to conduct anthropomorphic dialogue and communication.

3. The 3D digital human real-time dialogue interaction system according to claim 1, characterized in that: The recognition algorithm includes at least one of an encoder, a predictor, a sampler, a decoder or a loss function algorithm. The encoder algorithm is used to perform preliminary feature extraction and encoding on the input speech signal. The predictor algorithm is used to predict the number of target characters and extract the acoustic vector corresponding to the target characters. The sampler algorithm is used to generate a feature vector containing semantics according to the input acoustic vector and the target vector. The decoder algorithm is used to generate corresponding text content according to preset rules based on the feature vector of semantic information, and convert the acoustic features corresponding to the speech signal into text data. The loss function algorithm guides the model to continuously adjust parameters by calculating the difference between the predicted result and the real label. The scenario-based setting method includes scenario-based setting and differentiation of poor recognition effect, scenario-based setting and differentiation of homophones.

4. The 3D digital human real-time dialogue interaction system according to claim 1, characterized in that: The RAG knowledge base retrieval model collects, organizes and stores various text materials to provide rich background knowledge and reference basis for text generation; The text information set by the system-prompt is used to guide the large model to generate text according to specific styles, rules, and task requirements.

5. The 3D digital human real-time dialogue interaction system according to claim 4, characterized in that: The large model combines the text information retrieved by the RAG knowledge base retrieval model and the text information set by the system-prompt to make corresponding text output, and performs preprocessing operations on the text output, wherein the preprocessing operations include merging or retaining sentences of the text information according to punctuation and word count restrictions.

6. The 3D digital human real-time dialogue interaction system according to claim 5, characterized in that: The retrieval method of the RAG knowledge base retrieval model includes: when the text generation model receives the input text, the RAG database retrieval model performs a matching search in the knowledge base according to the keyword information in the text, and uses a similarity algorithm based on a vector space model to find a text segment associated with the input text; The text information set by system-prompt includes writing specific text content according to the corresponding parameter format and rules.

7. The 3D digital human real-time dialogue interaction system according to claim 1, characterized in that: The fine-tuning process of the sound cloning unit includes: Sound data preprocessing: receiving background clean sound and 15s voice audio data input by the user; Feature extraction: Use VQGAN to encode the human voice audio data to obtain semantic tokens; Model fine-tuning: Input the encoded semantic tokens and the corresponding text into the autoregressive language model as examples for training, and adjust the model parameters; Save fine-tuning parameters: After fine-tuning is completed, save the fine-tuning parameter data of the corresponding sound, wherein the fine-tuning parameter data includes the weight adjustment of the language model and VQGAN related parameters.

8. The 3D digital human real-time dialogue interaction system according to claim 1, characterized in that: Inputting the speech synthesis data into a 3D expression generation model comprises: Extracting features from the speech synthesis data to obtain speech feature information, and inputting the speech feature information into a trained emotion classification model for training to identify the emotion classification result corresponding to the speech input data, wherein the emotion classification model is obtained by training a neural network, and the emotion classification result includes identifying the emotion category contained in the speech input data; Inputting the trained basic expression generation model according to the speech feature information and the emotion classification result to perform training to obtain expression data, wherein the basic expression generation model is obtained by training the diffusion model; The expression data and preset personalized features are fine-tuned to obtain a personalized expression generation model, and a personalized emotional expression corresponding to the voice signal is generated based on the emotion classification result and the personalized expression generation model, wherein the preset personalized features include style features extracted by a style encoder from preset audio data, and the style encoder is trained by a temporal convolutional neural network.

9. The 3D digital human real-time dialogue interaction system according to claim 1, characterized in that: The execution process of the knowledge base retrieval unit includes: In the text generation process, after the user uploads the commonly used office data, the document data is extracted and converted into text vector data storage, wherein the types of the document data include txt, pptx, docx, and the data content includes text data and picture data; When users are having a normal conversation or chatting, the knowledge base of the text data is searched based on the voice text data input by the user, and the retrieved text data is sent to the big model together with the text data of the user's question.

10. A 3D digital human real-time dialogue interaction method, characterized in that: The 3D digital human real-time dialogue interaction system according to any one of claims 1 to 9 comprises the following steps: Obtaining the voice signal input by the user; The speech signal is recognized and processed by a speech recognition model to obtain text data, wherein the speech recognition model uses a variety of recognition algorithms to convert the speech signal into text data, and the speech recognition model is optimized according to a scenario-based setting method; The text information retrieved by the RAG knowledge base retrieval model and the text information set by the system-prompt are retrieved according to the personal database uploaded by the user and converted into text vector data for storage, and / or the knowledge base retrieval of text data is performed according to the voice text data, and the retrieved text data is sent to the large model together with the text data of the user's question; The text data is input into a text generation model, and the text generation model uses a large model as a core algorithm to output speech text data corresponding to the text data, wherein the speech text data includes text content obtained after speech recognition processing of the speech signal; Inputting the text data into a speech synthesis model, the speech synthesis model outputting speech synthesis data corresponding to the text data; Inputting the speech synthesis data into a 3D expression generation model, and the 3D expression generation model outputs 3D expression data corresponding to the speech synthesis data; A sound cloning model is constructed according to the speech signal and the 3D expression data and human voice audio data is output, the sound cloning model is fine-tuned and fine-tuning parameter data corresponding to the human voice audio data is saved.

Images