Multi-round voice dialogue method and system based on large model
By constructing a speech dialogue model based on speech recognition and text-to-speech, the problem of insufficient Chinese language understanding and generation capabilities in multiple rounds of speech dialogue in the prior art is solved, and a more efficient and natural speech dialogue experience is achieved.
Patent Information
- Application Number
- CN202411591975.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-08
- Publication Date
- 2025-03-11
AI Technical Summary
The existing voice dialogue system lacks knowledge about dialogue context in multiple rounds of dialogue, resulting in insufficient understanding and generation ability of Chinese language, which in turn makes the dialogue lack meaning and affects the user experience.
By obtaining a large amount of Chinese dialogue data for preprocessing, a data set is built, and a voice dialogue model is created based on the speech recognition module (wav2vec2), the Gemma module and the text-to-speech module (MeloTTS), for training and deployment, setting the API interface and authentication mechanism to achieve continuous voice dialogue.
The dialogue capabilities of multiple rounds of voice conversations have been improved, the quality of voice feature extraction and speech synthesis has been improved through deep learning technology, the understanding and generation ability of Chinese language has been enhanced, and the user interaction experience has been improved.
Smart Images
Figure CN119601013A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of voice dialogue technology, and in particular to a multi-round voice dialogue method and system based on a large model. Background Art
[0002] With the advancement of technology and the increase in labor costs, intelligent voice conversations through robots have become the current trend. For example, robots can replace customer service agents to provide after-sales service for products, and robots can make phone calls for marketing, which can greatly reduce labor costs.
[0003] However, existing voice dialogue systems can only provide general responses with low information content, especially when participants try to conduct multiple rounds of voice dialogues for in-depth discussions. Due to the lack of knowledge related to the dialogue context and poor understanding and generation capabilities of the Chinese language, the dialogues often lack meaning, which makes the participants feel bored.
[0004] Therefore, how to provide a multi-round voice dialogue method and system based on a large model to improve the dialogue capability of multi-round voice dialogue has become a technical problem that needs to be solved urgently. Summary of the invention
[0005] The technical problem to be solved by the present invention is to provide a multi-round voice dialogue method and system based on a large model, so as to improve the dialogue ability of multi-round voice dialogue.
[0006] In a first aspect, the present invention provides a multi-round voice dialogue method based on a large model, comprising the following steps:
[0007] Step S1, obtaining a large amount of Chinese conversation data, and constructing a data set after preprocessing each of the Chinese conversation data;
[0008] Step S2, creating a speech dialogue model based on a speech recognition module, a Gemma module and a text-to-speech module; the speech recognition module is used to convert input speech data into input text data; the Gemma module is used to generate output text data based on the input text data; the text-to-speech module is used to convert the output text data into output speech data;
[0009] Step S3, training the speech dialogue model using the data set;
[0010] Step S4: deploy the trained voice dialogue model to a cloud server, set an API interface called by the voice dialogue model, and set an identity authentication mechanism for the API interface;
[0011] Step S5: After identity authentication based on the identity authentication mechanism, the voice dialogue model deployed by the cloud server is called through the API interface to conduct a continuous voice dialogue.
[0012] Furthermore, the step S1 is specifically as follows:
[0013] A large amount of Chinese conversation data in voice format is obtained, each Chinese conversation data is preprocessed for format conversion, a conversation text is extracted from each preprocessed Chinese conversation data, a corresponding dictionary is generated based on each conversation text, and a data set is constructed based on each dictionary and the Chinese conversation data.
[0014] Furthermore, in step S2, the speech recognition module is constructed based on wav2vec2; and the text-to-speech module is constructed based on MeloTTS.
[0015] Furthermore, the step S3 is specifically as follows:
[0016] Dividing the data set into a training set, a validation set, and a test set according to a preset ratio, training the speech dialogue model through the training set, and optimizing the model parameters of the Gemma module through the LoRA fine-tuning technology during the training process until the loss value of the loss function of the speech dialogue model is less than a preset loss threshold;
[0017] Then, the trained speech dialogue model is verified using the verification set to determine whether the accuracy of the dialogue content output by the speech dialogue model is greater than a preset accuracy threshold. If not, the training set is expanded to continue training; if yes, the verification is passed, and:
[0018] The verified speech dialogue model is tested using the test set to determine whether the confidence of the dialogue content output by the speech dialogue model is greater than a preset confidence threshold. If not, the training set is expanded to continue training; if so, the test passes and the training ends.
[0019] Furthermore, the step S4 is specifically as follows:
[0020] Build the operating environment of the voice dialogue model on the cloud server, deploy the trained voice dialogue model to the cloud server through Gradio, set the API interface called by the voice dialogue model, and set the identity authentication mechanism of the API interface.
[0021] In a second aspect, the present invention provides a multi-round voice dialogue system based on a large model, comprising the following modules:
[0022] A data set construction module, used to obtain a large amount of Chinese conversation data, and construct a data set after preprocessing each of the Chinese conversation data;
[0023] A speech dialogue model creation module, used to create a speech dialogue model based on a speech recognition module, a Gemma module and a text-to-speech module; the speech recognition module is used to convert input speech data into input text data; the Gemma module is used to generate output text data based on the input text data; the text-to-speech module is used to convert the output text data into output speech data;
[0024] A speech dialogue model training module, used for training the speech dialogue model using the data set;
[0025] A voice dialogue model deployment module, used to deploy the trained voice dialogue model to a cloud server, set an API interface called by the voice dialogue model, and set an authentication mechanism for the API interface;
[0026] The voice dialogue module is used to perform continuous voice dialogue by calling the voice dialogue model deployed by the cloud server through the API interface after identity authentication based on the identity authentication mechanism.
[0027] Furthermore, the data set construction module is specifically used for:
[0028] A large amount of Chinese conversation data in voice format is obtained, each Chinese conversation data is preprocessed for format conversion, a conversation text is extracted from each preprocessed Chinese conversation data, a corresponding dictionary is generated based on each conversation text, and a data set is constructed based on each dictionary and the Chinese conversation data.
[0029] Furthermore, in the speech dialogue model creation module, the speech recognition module is constructed based on wav2vec2; and the text-to-speech module is constructed based on MeloTTS.
[0030] Furthermore, the speech dialogue model training module is specifically used for:
[0031] Dividing the data set into a training set, a validation set, and a test set according to a preset ratio, training the speech dialogue model through the training set, and optimizing the model parameters of the Gemma module through the LoRA fine-tuning technology during the training process until the loss value of the loss function of the speech dialogue model is less than a preset loss threshold;
[0032] Then, the trained speech dialogue model is verified using the verification set to determine whether the accuracy of the dialogue content output by the speech dialogue model is greater than a preset accuracy threshold. If not, the training set is expanded to continue training; if yes, the verification is passed, and:
[0033] The verified speech dialogue model is tested using the test set to determine whether the confidence of the dialogue content output by the speech dialogue model is greater than a preset confidence threshold. If not, the training set is expanded to continue training; if so, the test passes and the training ends.
[0034] Furthermore, the voice dialogue model deployment module is specifically used to:
[0035] Build the operating environment of the voice dialogue model on the cloud server, deploy the trained voice dialogue model to the cloud server through Gradio, set the API interface called by the voice dialogue model, and set the identity authentication mechanism of the API interface.
[0036] The advantages of the present invention are:
[0037] A data set is constructed by obtaining a large amount of Chinese conversation data and preprocessing it; then a speech conversation model is created based on the speech recognition module, Gemma module and text-to-speech module, the speech conversation model is trained through the data set, the trained speech conversation model is deployed to the cloud server, the API interface called by the speech conversation model is set, the authentication mechanism of the API interface is set, and finally, after authentication based on the authentication mechanism, the speech conversation model deployed on the cloud server is called through the API interface for continuous speech conversation; that is, the speech conversation model is trained through a data set constructed from a large amount of Chinese conversation data, and the model parameters of the Gemma module are optimized through the LoRA fine-tuning technology during the training process to improve the understanding and generation capabilities of the Chinese language; the speech recognition module is constructed based on wav2vec2, which learns the general representation of speech data through self-supervised pre-training, and then Fine-tuning is performed on specific voice data recognition tasks to achieve high-precision speech-to-text functions. Compared with traditional speech recognition systems, wav2vec2 uses deep learning technology to pre-train on unlabeled data, which greatly improves the ability to extract voice features and makes the recognition process more accurate and efficient. The text-to-speech module is built on MeloTTS. MeloTTS is an end-to-end speech synthesis system that can generate high-quality speech output directly from the input text by modeling the mapping relationship between text and speech. It can not only generate natural and fluent speech, but also has expressive speech synthesis effects, effectively improving the overall interactive experience. Compared with traditional TTS systems, MeloTTS uses deep learning of speech features through the Transformer structure, which significantly improves the quality of speech synthesis, further optimizes the user experience, and ultimately greatly improves the conversation ability of multi-round voice conversations. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] The present invention will be further described below in conjunction with embodiments with reference to the accompanying drawings.
[0039] Figure 1 It is a flow chart of a multi-round voice dialogue method based on a large model of the present invention.
[0040] Figure 2 It is a structural schematic diagram of a multi-round voice dialogue system based on a large model of the present invention. DETAILED DESCRIPTION
[0041] The technical solution in the embodiments of the present application has the following overall idea: the speech dialogue model is trained by a data set constructed from a large amount of Chinese dialogue data. During the training process, the model parameters of the Gemma module are optimized by the LoRA fine-tuning technology to improve the understanding and generation capabilities of the Chinese language; the speech recognition module is built based on wav2vec2, which learns the general representation of speech data through self-supervised pre-training, and then fine-tunes it on a specific speech data recognition task, thereby achieving a high-precision speech-to-text function; the text-to-speech module is built based on MeloTTS, which is an end-to-end speech synthesis system that can generate high-quality speech output directly from the input text by modeling the mapping relationship between text and speech. It can not only generate natural and fluent speech, but also has expressive speech synthesis effects, effectively improving the overall interactive experience, thereby improving the conversation ability of multiple rounds of voice conversations.
[0042] Please refer to Figure 1 to Figure 2 As shown, a preferred embodiment of a multi-round voice dialogue method based on a large model of the present invention includes the following steps:
[0043] Step S1, obtaining a large amount of Chinese conversation data, and constructing a data set after preprocessing each of the Chinese conversation data;
[0044] Step S2, creating a speech dialogue model based on a speech recognition module, a Gemma module and a text-to-speech module; the speech recognition module is used to convert input speech data into input text data; the Gemma module is used to generate output text data based on the input text data; the text-to-speech module is used to convert the output text data into output speech data;
[0045] Step S3, training the speech dialogue model using the data set;
[0046] Step S4: deploy the trained voice dialogue model to a cloud server, set an API interface called by the voice dialogue model, and set an identity authentication mechanism for the API interface;
[0047] Step S5: After identity authentication based on the identity authentication mechanism, the voice dialogue model deployed by the cloud server is called through the API interface to conduct a continuous voice dialogue.
[0048] The step S1 is specifically as follows:
[0049] A large amount of Chinese conversation data in voice format is obtained, each Chinese conversation data is preprocessed for format conversion, a conversation text is extracted from each preprocessed Chinese conversation data, a corresponding dictionary is generated based on each conversation text, and a data set is constructed based on each dictionary and the Chinese conversation data.
[0050] In the specific implementation, you can directly select existing Chinese dialogue datasets, such as chatterbot and qingyun. The dialogue content of these two datasets is very rich and covers a wide range of topics. The descriptions of the two Chinese dialogue datasets are as follows:
[0051]
[0052] That is, each of the dictionaries contains a dialogue text, for example, data = [{'text':'xxxx'}, {'text':'xxxx'}, {'text':'xxxx'}, ...]. It is also necessary to use the method provided by the datasets.Dataset class to encapsulate the dictionary list so that it can be directly used for model training.
[0053] In step S2, the speech recognition module is constructed based on wav2vec2; and the text-to-speech module is constructed based on MeloTTS.
[0054] The Gemma module adopts a text-to-text architecture and uses only a decoder, which is a large language model. The Gemma module provides multiple variants such as open weights, pre-training, and instruction fine-tuning in the application, which can flexibly adapt to different application scenarios. The Gemma module is based on the Transformer decoder. During the training process, it can take into account context information of up to 8192 tokens. The following table shows the key parameters of the Gemma module:
[0055] parameter 2B Eigenvector 2048 Number of layers 18 Feedforward Neural Network Hidden Layer Dimensions 32768 Note the number of heads 8 Number of key-value pairs to pay attention to 1 Note the size of the head 256 Vocabulary size 256128
[0056] The step S3 is specifically as follows:
[0057] The data set is divided into a training set, a validation set and a test set according to a preset ratio, and the speech dialogue model is trained by the training set. During the training process, the model parameters of the Gemma module are optimized by the LoRA (Low-Rank Adaptation) fine-tuning technology until the loss value of the loss function of the speech dialogue model is less than a preset loss threshold;
[0058] Then, the trained speech dialogue model is verified using the verification set to determine whether the accuracy of the dialogue content output by the speech dialogue model is greater than a preset accuracy threshold. If not, the training set is expanded to continue training; if yes, the verification is passed, and:
[0059] The verified speech dialogue model is tested using the test set to determine whether the confidence of the dialogue content output by the speech dialogue model is greater than a preset confidence threshold. If not, the training set is expanded to continue training; if so, the test passes and the training ends.
[0060] LoRA fine-tuning technology aims to efficiently adapt large pre-trained language models (Gemma modules) to multiple downstream tasks. Unlike traditional fine-tuning methods, LoRA does not update all parameters of the Gemma module, but achieves adaptation by learning a small set of task-specific parameters or external modules. To optimize the performance and efficiency of the Gemma module, the following strategies are adopted:
[0061] 1. Quantization configuration: Select the nf4 quantization method to load the model with 4-bit precision; this method performs calculations under float16 precision, which can significantly reduce the model size, speed up inference, save memory usage and reduce energy consumption; 2. LoRA configuration: Set the relevant parameters of LoRA; among them, rank is set to 16, which means the rank of the optimized rank decomposition matrix. A lower rank can reduce model complexity, but may also affect performance; the alpha parameter is set to 16, which is used to control the learning rate scaling factor of the rank decomposition matrix in LoRA optimization. A larger alpha value will speed up the update of task-specific parameters, but may also cause overfitting; lora_dropout is also set to 0.1, and the appropriate dropout value is used to help the model better generalize to new data sets; 3. Training configuration: Pass the above-set peft_config and training parameters to SFTTrainer (supervised fine-tuning trainer), and set neftune_noise_alpha to 5 to add noise to the embedding layer, which can further improve the fine-tuning performance.
[0062] The step S4 is specifically as follows:
[0063] Build the operating environment of the voice dialogue model on the cloud server, deploy the trained voice dialogue model to the cloud server through Gradio, set the API interface called by the voice dialogue model, and set the identity authentication mechanism of the API interface.
[0064] The identity authentication mechanism is specifically:
[0065] Parse the conversation request to obtain ciphertext data, signature data and MAC value; the calculation process of the ciphertext data is: encrypt the account password, device ID, and the public key used to verify the signature data by the XTEA algorithm to obtain encrypted data, and map each character of the encrypted data by a preset mapping rule to obtain ciphertext data; the MAC value is obtained by performing MAC calculation on the ciphertext data and the signature data;
[0066] After the integrity of the ciphertext data and the signature data is checked by the MAC value, the ciphertext data is decrypted to obtain the account password, device ID and public key, and then the signature data is verified by the public key. The first legitimacy check is performed on the device ID through the preset device management table, and finally the second legitimacy check is performed on the account password through the preset password management table to complete identity authentication.
[0067] A preferred embodiment of a multi-round voice dialogue system based on a large model of the present invention includes the following modules:
[0068] A data set construction module, used to obtain a large amount of Chinese conversation data, and construct a data set after preprocessing each of the Chinese conversation data;
[0069] A speech dialogue model creation module, used to create a speech dialogue model based on a speech recognition module, a Gemma module and a text-to-speech module; the speech recognition module is used to convert input speech data into input text data; the Gemma module is used to generate output text data based on the input text data; the text-to-speech module is used to convert the output text data into output speech data;
[0070] A speech dialogue model training module, used for training the speech dialogue model using the data set;
[0071] A voice dialogue model deployment module, used to deploy the trained voice dialogue model to a cloud server, set an API interface called by the voice dialogue model, and set an authentication mechanism for the API interface;
[0072] The voice dialogue module is used to perform continuous voice dialogue by calling the voice dialogue model deployed by the cloud server through the API interface after identity authentication based on the identity authentication mechanism.
[0073] The data set construction module is specifically used for:
[0074] A large amount of Chinese conversation data in voice format is obtained, each Chinese conversation data is preprocessed for format conversion, a conversation text is extracted from each preprocessed Chinese conversation data, a corresponding dictionary is generated based on each conversation text, and a data set is constructed based on each dictionary and the Chinese conversation data.
[0075] In the specific implementation, you can directly select existing Chinese dialogue datasets, such as chatterbot and qingyun. The dialogue content of these two datasets is very rich and covers a wide range of topics. The descriptions of the two Chinese dialogue datasets are as follows:
[0076]
[0077] That is, each of the dictionaries contains a dialogue text, for example, data = [{'text':'xxxx'}, {'text':'xxxx'}, {'text':'xxxx'}, ...]. It is also necessary to use the method provided by the datasets.Dataset class to encapsulate the dictionary list so that it can be directly used for model training.
[0078] In the speech dialogue model creation module, the speech recognition module is built based on wav2vec2; and the text-to-speech module is built based on MeloTTS.
[0079] The Gemma module adopts a text-to-text architecture and uses only a decoder, which is a large language model. The Gemma module provides multiple variants such as open weights, pre-training, and instruction fine-tuning in the application, which can flexibly adapt to different application scenarios. The Gemma module is based on the Transformer decoder. During the training process, it can take into account context information of up to 8192 tokens. The following table shows the key parameters of the Gemma module:
[0080] parameter 2B Eigenvector 2048 Number of layers 18 Feedforward Neural Network Hidden Layer Dimensions 32768 Note the number of heads 8 Number of key-value pairs to pay attention to 1 Note the size of the head 256 Vocabulary size 256128
[0081] The speech dialogue model training module is specifically used for:
[0082] The data set is divided into a training set, a validation set and a test set according to a preset ratio, and the speech dialogue model is trained by the training set. During the training process, the model parameters of the Gemma module are optimized by the LoRA (Low-Rank Adaptation) fine-tuning technology until the loss value of the loss function of the speech dialogue model is less than a preset loss threshold;
[0083] Then, the trained speech dialogue model is verified using the verification set to determine whether the accuracy of the dialogue content output by the speech dialogue model is greater than a preset accuracy threshold. If not, the training set is expanded to continue training; if yes, the verification is passed, and:
[0084] The verified speech dialogue model is tested using the test set to determine whether the confidence of the dialogue content output by the speech dialogue model is greater than a preset confidence threshold. If not, the training set is expanded to continue training; if so, the test passes and the training ends.
[0085] LoRA fine-tuning technology aims to efficiently adapt large pre-trained language models (Gemma modules) to multiple downstream tasks. Unlike traditional fine-tuning methods, LoRA does not update all parameters of the Gemma module, but achieves adaptation by learning a small set of task-specific parameters or external modules. To optimize the performance and efficiency of the Gemma module, the following strategies are adopted:
[0086] 1. Quantization configuration: Select the nf4 quantization method to load the model with 4-bit precision; this method performs calculations under float16 precision, which can significantly reduce the model size, speed up inference, save memory usage and reduce energy consumption; 2. LoRA configuration: Set the relevant parameters of LoRA; among them, rank is set to 16, which means the rank of the optimized rank decomposition matrix. A lower rank can reduce model complexity, but may also affect performance; the alpha parameter is set to 16, which is used to control the learning rate scaling factor of the rank decomposition matrix in LoRA optimization. A larger alpha value will speed up the update of task-specific parameters, but may also cause overfitting; lora_dropout is also set to 0.1, and the appropriate dropout value is used to help the model better generalize to new data sets; 3. Training configuration: Pass the above-set peft_config and training parameters to SFTTrainer (supervised fine-tuning trainer), and set neftune_noise_alpha to 5 to add noise to the embedding layer, which can further improve the fine-tuning performance.
[0087] The voice dialogue model deployment module is specifically used for:
[0088] Build the operating environment of the voice dialogue model on the cloud server, deploy the trained voice dialogue model to the cloud server through Gradio, set the API interface called by the voice dialogue model, and set the identity authentication mechanism of the API interface.
[0089] The identity authentication mechanism is specifically:
[0090] Parse the conversation request to obtain ciphertext data, signature data and MAC value; the calculation process of the ciphertext data is: encrypt the account password, device ID, and the public key used to verify the signature data by the XTEA algorithm to obtain encrypted data, and map each character of the encrypted data by a preset mapping rule to obtain ciphertext data; the MAC value is obtained by performing MAC calculation on the ciphertext data and the signature data;
[0091] After the integrity of the ciphertext data and the signature data is checked by the MAC value, the ciphertext data is decrypted to obtain the account password, device ID and public key, and then the signature data is verified by the public key. The first legitimacy check is performed on the device ID through the preset device management table, and finally the second legitimacy check is performed on the account password through the preset password management table to complete identity authentication.
[0092] In summary, the advantages of the present invention are:
[0093] A data set is constructed by obtaining a large amount of Chinese conversation data and preprocessing it; then a speech conversation model is created based on the speech recognition module, Gemma module and text-to-speech module, the speech conversation model is trained through the data set, the trained speech conversation model is deployed to the cloud server, the API interface called by the speech conversation model is set, the authentication mechanism of the API interface is set, and finally, after authentication based on the authentication mechanism, the speech conversation model deployed on the cloud server is called through the API interface for continuous speech conversation; that is, the speech conversation model is trained through a data set constructed from a large amount of Chinese conversation data, and the model parameters of the Gemma module are optimized through the LoRA fine-tuning technology during the training process to improve the understanding and generation capabilities of the Chinese language; the speech recognition module is constructed based on wav2vec2, which learns the general representation of speech data through self-supervised pre-training, and then Fine-tuning is performed on specific voice data recognition tasks to achieve high-precision speech-to-text functions. Compared with traditional speech recognition systems, wav2vec2 uses deep learning technology to pre-train on unlabeled data, which greatly improves the ability to extract voice features and makes the recognition process more accurate and efficient. The text-to-speech module is built on MeloTTS. MeloTTS is an end-to-end speech synthesis system that can generate high-quality speech output directly from the input text by modeling the mapping relationship between text and speech. It can not only generate natural and fluent speech, but also has expressive speech synthesis effects, effectively improving the overall interactive experience. Compared with traditional TTS systems, MeloTTS uses deep learning of speech features through the Transformer structure, which significantly improves the quality of speech synthesis, further optimizes the user experience, and ultimately greatly improves the conversation ability of multi-round voice conversations.
[0094] Although the specific implementation modes of the present invention are described above, those skilled in the art should understand that the specific implementation modes described are only illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included in the scope of protection of the claims of the present invention.
Claims
1. A multi-round voice dialogue method based on a large model, characterized by: The steps include: Step S1, obtaining a large amount of Chinese conversation data, and constructing a data set after preprocessing each of the Chinese conversation data; Step S2, creating a speech dialogue model based on a speech recognition module, a Gemma module and a text-to-speech module; the speech recognition module is used to convert input speech data into input text data; the Gemma module is used to generate output text data based on the input text data; the text-to-speech module is used to convert the output text data into output speech data; Step S3, training the speech dialogue model using the data set; Step S4: deploy the trained voice dialogue model to a cloud server, set an API interface called by the voice dialogue model, and set an identity authentication mechanism for the API interface; Step S5: After identity authentication based on the identity authentication mechanism, the voice dialogue model deployed by the cloud server is called through the API interface to conduct a continuous voice dialogue.
2. The multi-round voice dialogue method based on a large model as claimed in claim 1, characterized in that: The step S1 is specifically as follows: A large amount of Chinese conversation data in voice format is obtained, each Chinese conversation data is preprocessed for format conversion, a conversation text is extracted from each preprocessed Chinese conversation data, a corresponding dictionary is generated based on each conversation text, and a data set is constructed based on each dictionary and the Chinese conversation data.
3. The multi-round voice dialogue method based on a large model as claimed in claim 1, characterized in that: In step S2, the speech recognition module is constructed based on wav2vec2; and the text-to-speech module is constructed based on MeloTTS.
4. The multi-round voice dialogue method based on a large model as claimed in claim 1, characterized in that: The step S3 is specifically as follows: Dividing the data set into a training set, a validation set, and a test set according to a preset ratio, training the speech dialogue model through the training set, and optimizing the model parameters of the Gemma module through the LoRA fine-tuning technology during the training process until the loss value of the loss function of the speech dialogue model is less than a preset loss threshold; Then, the trained speech dialogue model is verified using the verification set to determine whether the accuracy of the dialogue content output by the speech dialogue model is greater than a preset accuracy threshold; if not, the training set is expanded to continue training; If yes, the verification passes, and: Using the test set to test the verified speech dialogue model, determining whether the confidence of the dialogue content output by the speech dialogue model is greater than a preset confidence threshold, and if not, expanding the training set to continue training; If yes, the test passes and the training ends.
5. The multi-round voice dialogue method based on a large model as claimed in claim 1, characterized in that: The step S4 is specifically as follows: Build the operating environment of the voice dialogue model on the cloud server, deploy the trained voice dialogue model to the cloud server through Gradio, set the API interface called by the voice dialogue model, and set the identity authentication mechanism of the API interface.
6. A multi-round voice dialogue system based on a large model, characterized by: Includes the following modules: A data set construction module, used to obtain a large amount of Chinese conversation data, and construct a data set after preprocessing each of the Chinese conversation data; A speech dialogue model creation module, used to create a speech dialogue model based on a speech recognition module, a Gemma module and a text-to-speech module; the speech recognition module is used to convert input speech data into input text data; the Gemma module is used to generate output text data based on the input text data; the text-to-speech module is used to convert the output text data into output speech data; A speech dialogue model training module, used for training the speech dialogue model using the data set; A voice dialogue model deployment module, used to deploy the trained voice dialogue model to a cloud server, set an API interface called by the voice dialogue model, and set an authentication mechanism for the API interface; The voice dialogue module is used to perform continuous voice dialogue by calling the voice dialogue model deployed by the cloud server through the API interface after identity authentication based on the identity authentication mechanism.
7. A multi-round voice dialogue system based on a large model as claimed in claim 6, characterized in that: The data set construction module is specifically used for: A large amount of Chinese conversation data in voice format is obtained, each Chinese conversation data is preprocessed for format conversion, a conversation text is extracted from each preprocessed Chinese conversation data, a corresponding dictionary is generated based on each conversation text, and a data set is constructed based on each dictionary and the Chinese conversation data.
8. The multi-round voice dialogue system based on a large model as claimed in claim 6, characterized in that: In the speech dialogue model creation module, the speech recognition module is built based on wav2vec2; and the text-to-speech module is built based on MeloTTS.
9. The multi-round voice dialogue system based on a large model as claimed in claim 6, characterized in that: The speech dialogue model training module is specifically used for: Dividing the data set into a training set, a validation set, and a test set according to a preset ratio, training the speech dialogue model through the training set, and optimizing the model parameters of the Gemma module through the LoRA fine-tuning technology during the training process until the loss value of the loss function of the speech dialogue model is less than a preset loss threshold; Then, the trained speech dialogue model is verified using the verification set to determine whether the accuracy of the dialogue content output by the speech dialogue model is greater than a preset accuracy threshold; if not, the training set is expanded to continue training; If yes, the verification passes, and: Using the test set to test the verified speech dialogue model, determining whether the confidence of the dialogue content output by the speech dialogue model is greater than a preset confidence threshold, and if not, expanding the training set to continue training; If yes, the test passes and the training ends.
10. The multi-round voice dialogue system based on a large model as claimed in claim 6, characterized in that: The voice dialogue model deployment module is specifically used for: Build the operating environment of the voice dialogue model on the cloud server, deploy the trained voice dialogue model to the cloud server through Gradio, set the API interface called by the voice dialogue model, and set the identity authentication mechanism of the API interface.