Voice conversation method and device and electronic equipment

Abstract

The invention discloses a voice dialogue method and device and electronic equipment. The voice dialogue method comprises the steps of receiving voice input of a user and converting the voice input into an input text; retrieving text fragments related to user query from the internal knowledge base, and jointly providing the retrieved text fragments and the input text to a large-scale language model; generating highly related natural language answers according to the retrieved text segments and the input text data of the user; analyzing the emotion of the input text of the user through a natural language processing technology, and matching a corresponding voice emotion for an answer generated by the large-scale language model; text answers generated by a large-scale language model are converted into voice output matched with emotions to be listened by a user, and a more accurate voice dialogue solution rich in emotions is provided by combining an RAG retrieval enhancement knowledge base generation technology and an emotion recognition technology.

Description

Technical Field

[0001] The present invention belongs to the field of human-computer interaction, and in particular relates to a voice dialogue method, device and electronic equipment. Background Art

[0002] In the field of voice conversation assistants, existing technologies mainly rely on two approaches: fixed answer modes based on predefined rules or generated answers based on pre-trained language models.

[0003] 1. Fixed answer mode: This method triggers fixed answers to different questions based on preset keywords and rules. Although this method is more effective in simple scenarios, it becomes rigid when dealing with complex and open-ended questions and is difficult to meet the diverse needs of users. At the same time, since fixed rules cannot cover all situations, they cannot handle unknown questions that exceed the rules.

[0004] 2. Pre-trained language models: This type of technology relies on large pre-trained models, such as chatGPT, to generate responses that are relevant to the context of user input. However, these models do not rely on a knowledge base in a specific field, resulting in limited accuracy in specific fields, especially for voice assistants within a company, which often cannot provide professional answers based on an internal knowledge base. In addition, this type of model lacks emotion recognition and matching capabilities, resulting in the generated responses not matching the emotions of the user input, affecting the user experience.

[0005] The above existing technologies have deficiencies in answer flexibility, accuracy and emotional interaction, so this application is proposed. Summary of the invention

[0006] Purpose of the invention: In order to overcome the deficiencies in the prior art, the present invention provides a voice dialogue method, device and electronic device, which utilizes RAG retrieval enhancement to generate knowledge base technology combined with emotion recognition technology to provide a more accurate and emotional voice dialogue solution.

[0007] Technical solution: To achieve the above purpose, the technical solution of the present invention is as follows:

[0008] A voice conversation method, comprising:

[0009] Receive user's voice input and convert it into input text;

[0010] Retrieve text snippets related to the user query from the internal knowledge base, and provide the retrieved text snippets and the input text together to the large-scale language model;

[0011] Generate highly relevant natural language answers based on the retrieved text snippets and the user's input text data;

[0012] Analyze the sentiment of the user's input text through natural language processing technology, and match the corresponding voice emotion to the answer generated by the large-scale language model;

[0013] Convert text responses generated by large-scale language models into speech outputs with matching sentiment for users to listen to.

[0014] Furthermore, the text snippets related to the user query are retrieved from the internal knowledge base including:

[0015] Slice the documents in the knowledge base according to specific text length and title information;

[0016] Use the text representation model to vectorize each document slice;

[0017] Store the vector representation of each document slice in the Faiss vector database;

[0018] Vectorize the input text of the question;

[0019] The document slice that is most similar to the vector of the user's question is retrieved from the stored document slices in the Faiss database, and a similarity score value between each document slice and the question is attached.

[0020] Furthermore, vectorizing the problem includes:

[0021] The speech recognition module converts the questions input by the user through voice into input text;

[0022] The speech recognition module uses the same embedding model to vectorize the input text of the question.

[0023] Further, generating a highly relevant natural language answer based on the output text answer and the user's input text data includes:

[0024] Checking the similarity score value of the output document slice to determine the relevance of the document slice to the user question;

[0025] If the relevance is equal to or higher than a preset threshold, answering the question by using the document fragment as the question input of the large-scale language model;

[0026] If the relevance is lower than the preset threshold, the original user question is used as the question input of the large-scale language model to answer.

[0027] Furthermore, the coefficient range of the preset threshold is 0.5 to 1.0.

[0028] Furthermore, the large-scale language model module includes a VLLM reasoning framework, and the VLLM reasoning framework is used to reason about the problem.

[0029] Furthermore, the sentiment of the user's input text is analyzed by natural language processing technology including:

[0030] Identify the emotions contained in the user's input text and perform sentiment classification tasks;

[0031] The emotion classification task includes:

[0032] Pass the user's input text to the BERT model for text preprocessing and feature extraction;

[0033] Perform emotion recognition on text through classification tasks, output the probability distribution of different emotion categories, and select the most likely emotion category as the recognition result;

[0034] The identified emotion category information is cached and subsequently passed to a large-scale language model to match the corresponding speech emotion output.

[0035] Furthermore, the speech synthesis process uses the VITS model to perform speech synthesis and output, and the working process includes:

[0036] Receive text answers generated by large-scale speech models and combine them with sentiment category information;

[0037] The VITS model uses text information and emotional features to generate speech waveforms that match the text and emotion;

[0038] The final generated speech waveform is output to the user, providing a speech answer that matches the emotion.

[0039] Furthermore, a voice dialogue device comprises

[0040] A speech recognition module receives user's speech input and converts it into input text;

[0041] RAG retrieval enhancement generation module, which is used to retrieve text fragments associated with user queries from the internal knowledge base and provide them to the large-scale language model as auxiliary information;

[0042] A large-scale language model module, which is used to generate natural language answers based on the output content of the RAG retrieval enhancement module and the user's input text data;

[0043] The emotion recognition module is used to analyze the emotion of the user's input text and match the corresponding voice emotion to the answer generated by the large-scale language model;

[0044] The speech synthesis module is used to convert the text answers generated by the large-scale language model into speech output matching the emotion for users to listen to.

[0045] Furthermore, an electronic device for voice conversation includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the voice conversation method as described in any one of claims 2 to 8 when executing the program.

[0046] Beneficial effects: The present invention uses automatic speech recognition technology to convert speech input into text, and retrieves relevant text fragments from the internal knowledge base to enhance the quality of the large language model's answers. The large language model's answers are autonomously generated based on the retrieval results or its own knowledge base, and combined with emotion recognition technology to match the corresponding speech emotion output, it can provide accurate and emotional voice assistant services. It can be used as a voice assistant within the company, serving as a company commentator to introduce the company's situation; as an HR assistant to answer employee questions; or to provide support to customers in customer service to improve the voice interaction experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Attached Figure 1 It is the overall principle block diagram of the present invention;

[0048] Attached Figure 2 The figure is a schematic diagram of a voice dialogue process according to a specific embodiment of the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] The terms used in the present disclosure are for the purpose of describing specific embodiments only and are not intended to limit the present disclosure. The singular forms "a", "said" and "the" used in the present disclosure and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise.

[0051] The specific implementation of the present invention is further described in detail below in conjunction with the drawings and examples.

[0052] As attached Figure 1 As shown, a voice dialogue method includes:

[0053] Receive user's voice input and convert it into input text; retrieve text fragments related to user query from internal knowledge base, and provide the retrieved text fragments and input text to large-scale language model; generate highly relevant natural language answers based on the retrieved text fragments and user's input text data; analyze the emotion of user's input text through natural language processing technology, and match the corresponding voice emotion to the answer generated by large-scale language model; convert the text answer generated by large-scale language model into speech output matching emotion for user to listen to.

[0054] The automatic speech recognition (ASR) module receives the user's voice input and converts it into text for subsequent module processing. The existing speech recognition technology, such as the deep learning model, is used to map the speech signal into readable text data. In the present invention, the automatic speech recognition (ASR) module uses the open source Paraformer ASR model to accurately convert the user's voice input into text.

[0055] As an advanced large-scale language model, LLM has powerful understanding and generation capabilities, and can use this contextual information to generate accurate and natural answers. The use of large models improves the natural sense of language and semantic coherence of generated answers, making the answers provided by voice assistants more fluent and closer to real human conversations. This solution retrieves relevant text fragments from the internal knowledge base to provide the large language model (LLM) with rich contextual information based on actual data and professional knowledge, so that more accurate knowledge fragments and accurate answer results can be obtained. At the same time, the answers processed by the large language model in this solution not only satisfy users in terms of information, but also give users more delicate resonance in terms of emotions, improving the overall quality of the conversation and the user's emotional satisfaction.

[0056] In some more specific embodiments, retrieving text fragments related to user queries from an internal knowledge base includes: slicing documents in the knowledge base according to specific text lengths and title information; vectorizing each document slice using a text representation model (embedding); and storing the vector representation of each document slice in a Faiss vector database.

[0057] Slice the document to ensure that each document slice can independently reflect the relevant subject content, making the subsequent retrieval granularity more precise. By storing the vector representation of each document slice in the Faiss vector database, the retrieval and matching of a large number of document slices is more efficient. Faiss is a high-performance vector library for similarity search that can quickly find the most similar entries in massive vector data.

[0058] In some more specific implementations, the input text of the question is vectorized; the document slice that is most similar to the vector of the user's question is retrieved from the stored document slices through the Faiss database, and the similarity score value between each document slice and the question is attached. A text representation (embedding) model is used to vectorize each document slice. This model can be a pre-trained Transformer architecture model that maps text content to a high-dimensional vector space. In this way, the semantic information of the slice will be represented in the form of a vector to provide support for subsequent retrieval.

[0059] In some more specific implementations, vectorizing the question includes: the speech recognition module converts the question input by the user through voice into input text; the speech recognition module vectorizes the input text of the question using the same embedding model. After the user inputs the question through voice, the ASR module converts the question into text. Then, the RAG module vectorizes the question using the same embedding model to match the document slice vector related to it.

[0060] During the vector retrieval process, the efficient similarity search function of the Faiss database is used to retrieve the slices most similar to the vector of the user's question from the stored document slices. These relevant slices will be passed to the LLM module as background knowledge to assist it in generating accurate answers.

[0061] In some more specific implementations, generating a highly relevant natural language answer based on the output text answer and the user's input text data includes:

[0062] Check the similarity score of the output document slice to determine the relevance of the document slice to the user question; if the relevance is equal to or higher than the preset threshold, answer the question by using the document fragment as the question input of the large-scale language model; if the relevance is lower than the preset threshold, answer the question by using the original user question as the question input of the large-scale language model. The coefficient range of the preset threshold is 0.5 to 1.0, for example, it can be any one of 0.5, 0.6, 0.7, 0.9 and 1.0. The higher the coefficient, the greater the similarity. A coefficient of 1.0 means that they are exactly the same, which is a more extreme case. Therefore, the coefficient value is generally 0.7 to 0.8.

[0063] In some more specific implementations, after the RAG module completes the search for relevant knowledge base fragments, the LLM module is responsible for generating the final answer. The specific process is as follows:

[0064] Similarity judgment: Each knowledge base fragment retrieved by the RAG module is accompanied by a similarity score with the user's question. The LLM module first checks the similarity scores of these fragments to determine their relevance to the user's question. The threshold is set to 0.7, which means that when the similarity score is higher than 0.7, the recalled fragment will be given priority for answering.

[0065] Answer generation includes two cases:

[0066] Recalling fragment answers: If the similarity score between the retrieved fragment and the user's question is higher than 0.7, the LLM module will use the information in these relevant fragments to generate accurate answers. By generating answers from existing relevant fragments, the answer content will be more accurate and specific to meet the user's needs.

[0067] Autonomously generate answers: If the similarity score of the retrieved fragment is lower than the preset threshold (such as 0.7 in this example), the LLM module will use its generation ability to autonomously answer the user's question based on its own knowledge base and understanding ability. In this case, the model generates answers based on its semantic understanding and general knowledge to ensure that it can provide answers that meet the user's needs. And it ensures that the answer result will be output.

[0068] In some more specific embodiments, the large-scale language model module includes a VLLM reasoning framework, and the VLLM reasoning framework is used to reason about the problem. In order to improve the generation speed and efficiency, the LLM module uses the VLLM reasoning framework for reasoning. VLLM is an optimized distributed reasoning framework that can efficiently reason on a large language model. This framework speeds up the model reasoning speed by parallelizing computing, optimizing video memory usage and other technical means, while reducing video memory occupancy, ensuring the rapidity and stability of answer generation. Through this complete process, the LLM module can combine the knowledge fragments retrieved by RAG and its own generation capabilities to provide users with accurate, flexible and timely answers. The VLLM model can take into account the emotional dimension when generating text, which not only improves the emotional matching of the answer, but also makes the emotional expression more natural. The answers processed by the large model not only satisfy the user in terms of information, but also give the user a more delicate resonance in terms of emotion, improving the overall quality of the conversation and the user's emotional satisfaction.

[0069] In some more specific implementations, analyzing the sentiment of the user's input text by natural language processing technology includes: identifying the sentiment contained in the user's input text and performing a sentiment classification task;

[0070] The emotion classification task includes:

[0071] The user's input text is passed to the BERT model for text preprocessing and feature extraction; the text is subjected to emotion recognition through classification tasks, the probability distribution of different emotion categories is output, and the most likely emotion category (i.e., the one with the highest probability distribution) is selected as the recognition result; the recognized emotion category information is cached and subsequently passed to a large-scale language model to match the corresponding speech emotion output.

[0072] The goal of emotion recognition is to identify the emotions contained in the user's input text so that LLM can refer to these emotional information when generating answers. Emotion categories may include different emotional states such as anger, joy, sadness, surprise, etc., which can reflect the user's emotional intentions.

[0073] In order to perform accurate emotion recognition, the sentiment classification module uses the BERT model to complete the emotion classification task.

[0074] The BERT model is a bidirectional encoder based on Transformer. It obtains rich contextual information through pre-training and can perform deep semantic understanding of the input text. Using the fine-tuned BERT model, the text input by the user is converted into a vector representation and sentiment classification is performed.

[0075] The classification task process includes:

[0076] Input processing: The user's input text is passed to the BERT model for text preprocessing and feature extraction.

[0077] Sentiment classification: The text is classified into different categories to identify emotions, and the probability distribution of different emotion categories is output, and the most likely emotion category is selected as the recognition result.

[0078] Emotional output:

[0079] The identified emotion category information will be passed to the LLM module to match the corresponding speech emotion output.

[0080] Based on the recognized emotional information, the answers generated by LLM will be accompanied by appropriate voice emotions, providing users with more natural and emotionally consistent voice output.

[0081] By using the BERT model for emotion recognition, the emotion recognition module can accurately capture emotional features from user input and provide a more humane emotion match for the final voice answer.

[0082] In some more specific implementations, the speech synthesis process uses the VITS model to perform speech synthesis and output, and the working process includes:

[0083] Receive text answers generated by large-scale speech models and combine them with sentiment category information;

[0084] The VITS model uses text information and emotional features to generate speech waveforms that match the text and emotion;

[0085] The final generated speech waveform is output to the user, providing a speech answer that matches the emotion.

[0086] The text answer generated by LLM is converted into speech output matching the emotion for the user to listen to. Existing speech synthesis technology is used to provide speech presentation that matches the emotion characteristics for the answer.

[0087] The speech synthesis (TTS) module converts the text answers generated by LLM into speech to provide a more humane and emotional voice conversation experience. The speech synthesis module uses the VITS (Variational Inference Text-to-Speech) model as the TTS model to convert text input into natural and fluent speech output. The VITS model is an end-to-end speech synthesis model based on variational inference, which integrates the two processes of speech feature prediction and waveform generation to make the generated speech more natural. The structure and characteristics of the VITS model:

[0088] End-to-end training: VITS generates the final speech waveform directly from the text input through joint training, without the need for intermediate features such as Mel-spectrogram or other traditional speech features.

[0089] Multimodal information: The model combines text information and emotional features, and can generate speech with corresponding emotions based on the emotional information provided by the emotion recognition module.

[0090] High-quality generation: The VITS model combines the technologies of Gaussian mixture model (GMM) and generative adversarial network (GAN), and the generated speech quality is close to that of real human voice.

[0091] TTS module workflow:

[0092] Input text processing: Receive the text answer generated by LLM and combine it with the emotion category information of the emotion recognition module.

[0093] Speech generation: The VITS model uses text information and emotional features to generate speech waveforms that match the text and emotion.

[0094] Speech output: The final generated speech waveform is output to the user, providing an accurate and emotion-matching speech response.

[0095] By using the VITS model, the TTS module can generate natural, fluent, and emotional speech output while accurately restoring the text semantics, providing users with a more humane conversation experience.

[0096] As an advanced end-to-end speech synthesis technology, the VITS model can generate high-quality, emotional speech directly from text. The use of this technology ensures that the voice answer is not only excellent in voice clarity and naturalness, but also better meets the user's emotional expectations in terms of emotional expression. Through these technical means, the present invention effectively solves the shortcomings of traditional voice assistants in terms of accuracy, emotional interaction and natural interactivity, and provides users with a more intelligent, emotional and humane conversation experience.

[0097] A more specific embodiment is shown in the attached Figure 2 As shown, including:

[0098] Establish a local internal knowledge base, which specifically includes extracting text information from local documents to obtain text information, then splitting the text to form text fragments, and representing them through embedding text vectors, and then storing them in the Faiss vector database to obtain the internal knowledge base;

[0099] The user inputs voice, and ASR speech recognition is used to convert the voice into text and recognize the emotions in the voice. The text is the user's question. The ASR module embeds the text into a text vector representation, and the RAG module uses the same embedding model to vectorize the question.

[0100] In the local internal knowledge base, the user's question is input and searched, and the vectorized similarity is compared and the similarity is judged through the embedding model. If the similarity is greater than the preset threshold of 0.7, similar text fragments are directly recalled from the internal knowledge base; if the similarity is less than the preset threshold of 0.7, the text of the user's question is embedded in the Prompt template of the LLM large language model and answered through the LLM large language model;

[0101] A prompt is formed, an answer is given in the LLM, and a result is generated. The speech emotion recognized by the speech recognition module is classified and the answer emotion of the LLM large language model is determined. The text is converted into speech through the TTS speech synthesis module, and a speech answer with emotion is output.

[0102] A voice dialogue device, comprising

[0103] The speech recognition module receives the user's speech input and converts it into input text.

[0104] This module uses existing speech recognition technology, such as deep learning models, to map speech signals into readable text data. In this invention, the automatic speech recognition (ASR) module uses the open source Paraformer ASR model to accurately convert the user's speech input into text. The specific details are as follows:

[0105] Model Overview: The Paraformer ASR model is an end-to-end speech recognition model with the advantages of high efficiency and accuracy. It uses the latest neural network technology to recognize speech signals and convert them into text even in complex audio backgrounds.

[0106] Model features: Parallel generation: The Paraformer model can process input speech signals in parallel, reduce recognition latency, and increase recognition speed. High accuracy: Combining the advanced Transformer architecture and attention mechanism, Paraformer can effectively capture contextual information in speech signals and maintain a high recognition accuracy in different noise environments.

[0107] Combination with the present invention: The ASR module passes the user's voice input to the Paraformer model, generates corresponding text data, and provides it to the RAG retrieval module and LLM. The accurate and fast recognition capability of the Paraformer ASR model provides stable and timely input text for the voice dialogue assistant of the present invention. By using the Paraformer ASR model, the ASR module can efficiently and accurately convert voice signals into text, providing a reliable basis for subsequent RAG retrieval and LLM generation of answers.

[0108] The RAG retrieval enhancement generation module is used to retrieve text fragments associated with user queries from the internal knowledge base and provide them to the large-scale language model as auxiliary information.

[0109] Responsible for retrieving text snippets related to user queries from the internal knowledge base and providing them to LLM as auxiliary information. The workflow of this module is as follows:

[0110] Knowledge base retrieval: Use the retrieval submodule to obtain text snippets related to the user query from the internal knowledge base.

[0111] Answer generation: Combine the search results with the text data entered by the user as context for LLM to generate answers.

[0112] The RAG module is responsible for retrieving text snippets related to user queries from the internal knowledge base to assist LLM in generating more accurate answers.

[0113] A large-scale language model module, which is used to generate natural language answers based on the output content of the RAG retrieval enhancement module and the user's input text data;

[0114] Generate natural language answers based on the output of the RAG retrieval module and the text data entered by the user. If the RAG retrieval fails to return a suitable text fragment, the LLM generates an answer based on its own knowledge to ensure the flexibility of the system. After the RAG module completes the retrieval of the relevant knowledge base fragment, the LLM module is responsible for generating the final answer.

[0115] The emotion recognition module is used to analyze the emotion of the user's input text and match the corresponding voice emotion to the answer generated by the large-scale language model;

[0116] The speech synthesis module is used to convert the text answers generated by the large-scale language model into speech output matching the emotion for users to listen to.

[0117] An electronic device for voice conversation includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the above-mentioned voice conversation method is implemented.

[0118] It should be understood that although the terms first, second, third, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information.

[0119] In the description of the present invention, it is necessary to understand that the terms "upper", "lower", "front", "back", "left", "right", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship are based on the orientation or position relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0120] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A voice dialogue method, characterized in that: include: Receive user's voice input and convert it into input text; Retrieve text snippets related to the user query from the internal knowledge base, and provide the retrieved text snippets and the input text together to the large-scale language model; Generate highly relevant natural language answers based on the retrieved text snippets and the user's input text data; Analyze the sentiment of the user's input text through natural language processing technology, and match the corresponding voice emotion to the answer generated by the large-scale language model; Convert text responses generated by large-scale language models into speech outputs with matching sentiment for users to listen to.

2. A voice dialogue method according to claim 1, characterized in that: The text snippets relevant to the user query retrieved from the internal knowledge base include: Slice the documents in the knowledge base according to specific text length and title information; Use the text representation model to vectorize each document slice; Store the vector representation of each document slice in the Faiss vector database; Vectorize the input text of the question; The document slice that is most similar to the vector of the user's question is retrieved from the stored document slices in the Faiss database, and a similarity score value between each document slice and the question is attached.

3. A voice dialogue method according to claim 2, characterized in that: Vectorizing the problem involves: The speech recognition module converts the questions input by the user through voice into input text; The speech recognition module uses the same embedding model to vectorize the input text of the question.

4. A voice dialogue method according to claim 2, characterized in that: Generating highly relevant natural language responses based on the output text responses and the user's input text data includes: Checking the similarity score value of the output document slice to determine the relevance of the document slice to the user question; If the relevance is equal to or higher than a preset threshold, answering the question by using the document fragment as the question input of the large-scale language model; If the relevance is lower than the preset threshold, the original user question is used as the question input of the large-scale language model to answer.

5. A voice dialogue method according to claim 4, characterized in that: The coefficient range of the preset threshold is 0.5 to 1.

0.

6. A voice dialogue method according to claim 4, characterized in that: The large-scale language model module includes a VLLM reasoning framework, and the VLLM reasoning framework is used to reason about the problem.

7. A voice dialogue method according to claim 4, characterized in that: Analyzing the sentiment of user input text through natural language processing technology includes: Identify the emotions contained in the user's input text and perform sentiment classification tasks; The emotion classification task includes: Pass the user's input text to the BERT model for text preprocessing and feature extraction; Perform emotion recognition on text through classification tasks, output the probability distribution of different emotion categories, and select the most likely emotion category as the recognition result; The identified emotion category information is cached and subsequently passed to a large-scale language model to match the corresponding speech emotion output.

8. A voice dialogue method according to claim 1, characterized in that: The speech synthesis process uses the VITS model to perform speech synthesis and output, and the working process includes: Receive text answers generated by large-scale speech models and combine them with sentiment category information; The VITS model uses text information and emotional features to generate speech waveforms that match the text and emotion; The final generated speech waveform is output to the user, providing a speech answer that matches the emotion.

9. A voice dialogue device, characterized in that: include A speech recognition module receives user's speech input and converts it into input text; RAG retrieval enhancement generation module, which is used to retrieve text fragments associated with user queries from the internal knowledge base and provide them to the large-scale language model as auxiliary information; A large-scale language model module, which is used to generate natural language answers based on the output content of the RAG retrieval enhancement module and the user's input text data; The emotion recognition module is used to analyze the emotion of the user's input text and match the corresponding voice emotion to the answer generated by the large-scale language model; The speech synthesis module is used to convert the text answers generated by the large-scale language model into speech output matching the emotion for users to listen to.

10. An electronic device for voice conversation, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the voice dialogue method according to any one of claims 2 to 8 is implemented.

Images