Decision and large language model voice interaction method based on hybrid training strategy and storage medium

Abstract

The invention relates to a voice interaction method and a storage medium, in particular to a voice interaction method based on a decision of a hybrid training strategy and a large language model and a storage medium. The method aims at solving the problems that an existing large language model cannot provide accurate answers due to lack of sufficient domain knowledge, and the calculation cost is high and the response time is long due to the fact that two large language models are adopted. The method comprises the following steps: setting a specific answer format; constructing a decision data set; constructing a dialogue data set for specific scene dialogue questions and answers; performing first fine tuning on the large language model by adopting full-parameter fine tuning based on the dialogue data set to obtain a large language model after the first fine tuning; performing second fine tuning on the large language model after the first fine tuning by adopting LoRA based on the decision data set to obtain a large language model after the second fine tuning; and accessing the voice recognition module and the voice synthesis module into the large language model after secondary fine tuning, processing the to-be-tested voice problem of the user, and generating voice to interact with the user.

Description

Technical Field

[0001] The invention relates to a voice interaction method and a storage medium. Background Art

[0002] The Large Language Model (LLM) has powerful interactive capabilities and can perform natural language dialogue tasks. Through fine-tuning technology, the dialogue performance of the large language model in specific scenarios or special task contexts is further enhanced. This technical application is based on a variety of fine-tuning strategies to enhance the application of large language models in exhibition hall robots, enabling them to respond to visitors' instructions more accurately and make corresponding decisions and dialogues. In addition, by combining existing ASR (speech recognition) and TTS (text-to-speech) technologies, the transition from text interaction to voice interaction is achieved, greatly improving the interactive experience between exhibition hall robots and visitors.

[0003] First, although large language models perform well in a wide range of tasks, they often fail to provide accurate answers in some highly specialized fields due to a lack of sufficient domain knowledge. This limitation makes large language models face significant challenges when dealing with complex or specialized tasks.

[0004] Secondly, in actual application scenarios, speech recognition technology often encounters a key problem - misrecognition of background noise and meaningless conversations. If these misrecognition results are directly passed to the large language model, it will often lead to invalid or confusing conversations, affecting the user experience. To address this problem, the current mainstream solution usually uses two large language models: one is used to determine whether the user input is valuable, and the other is responsible for answering valuable input. However, although this method improves the quality of conversation to a certain extent, it also brings about significant waste of computing resources. Due to the need to run two large models, this solution has obvious deficiencies in resource utilization efficiency, especially in computing cost and response time. Summary of the invention

[0005] The purpose of the present invention is to solve the problems that the existing large language model cannot provide accurate answers due to the lack of sufficient domain knowledge, and the use of two large language models will bring high computational cost and long response time, and to propose a decision-making and large model voice interaction method and storage medium based on a hybrid training strategy.

[0006] The specific process of the voice interaction method based on the decision-making of the hybrid training strategy and the large language model is as follows:

[0007] Step 1: Set a specific answer format; build a decision dataset;

[0008] The decision data set includes prompt words, user input, and expected answers;

[0009] Step 2: Build a dialogue dataset for dialogue question and answer in specific scenarios;

[0010] The dialogue dataset includes the roles played by the large language model, user input, and corresponding answers;

[0011] Step 3: Fine-tune the large language model for the first time using all-parameter fine-tuning based on the dialogue dataset to obtain a fine-tuned large language model;

[0012] Step 4: Based on the decision data set, LoRA is used to perform a second fine-tuning on the large language model after the first fine-tuning to obtain a large language model after the second fine-tuning;

[0013] Step 5: Connect the speech recognition module and speech synthesis module to the large language model after secondary fine-tuning, process the user's speech questions to be tested, and generate speech to interact with the user.

[0014] Preferably, in step 1, a specific answer format is set; a decision data set is constructed;

[0015] The decision data set includes prompt words, user input, and expected answers;

[0016] The specific process is:

[0017] Step 11: Set a specific answer format;

[0018] Set the expected answer to the tuple output = <command:content>;

[0019] Among them, output is a two-tuple;

[0020] command is the type of command;

[0021] content is the specific content of the instruction;

[0022] Instructions include {navigation, introduction, aborted};

[0023] Navigation means navigation, inrtoduction means introduction, and aborted means invalid;

[0024] Step 12: Construct a decision dataset based on the specific answer format and user input; the specific process is as follows:

[0025] Set instruction as the prompt word;

[0026] The decision dataset is constructed based on the user question input, the prompt word instruction and the bigram output. The entry format of the decision dataset is<instruction,input,output> A collection of .

[0027] Preferably, the step of constructing a dialogue dataset for dialogue question and answer in a specific scenario;

[0028] The dialogue dataset includes the roles played by the large language model, user input, and corresponding answers;

[0029] The specific process is:

[0030] Step 21: Collect corpus in the exhibition hall environment;

[0031] The corpus in the exhibition hall environment is the basic information of the exhibits in the exhibition hall;

[0032] The basic information of the exhibits in the exhibition hall includes the basic characteristics, uses, parameter information of the exhibits, and the location information of the exhibition hall;

[0033] Step 22: Preprocess the corpus in the exhibition hall environment collected in step 21 to obtain preprocessed corpus; the specific process is:

[0034] Extract key information of the corpus in the exhibition hall environment collected in step 21, generate potential problems based on the key information, and use the potential problems as preprocessed corpus;

[0035] Step 23: Based on the binary output <command:content>, preprocessed corpus, and build a dialogue database for dialogue question and answer in specific scenarios.

[0036] Preferably, in step 3, the large language model is fine-tuned for the first time based on the dialogue dataset by using all-parameter fine-tuning to obtain a large language model after one fine-tuning; the specific process is:

[0037] Step 31: Select a large language model and load the large language model into the exhibition hall environment;

[0038] Step 32:

[0039] Divide the conversation dataset into training and testing sets;

[0040] Based on the training set in the dialogue dataset, select full parameter fine-tuning to train the large language model until convergence, and obtain a preliminarily fine-tuned large language model;

[0041] The full parameters include: base model hyperparameters, optimizer and loss function, weights, and biases;

[0042] Step 33: Evaluate and tune the performance of the initially fine-tuned large language model. The specific process is as follows:

[0043] Use the test set to test the fine-tuned large language model, obtain the similarity between the output of the fine-tuned large language model and the true value, and if the similarity meets the requirements, obtain a fine-tuned large language model;

[0044] If the similarity does not meet the requirement, step 22 to step 23 are executed again until the similarity meets the requirement, and a fine-tuned large language model is obtained.

[0045] Preferably, in step 33, the performance of the preliminarily fine-tuned large language model is evaluated and tuned; the specific process is:

[0046] Use the test set to test the fine-tuned large language model, obtain the similarity between the output of the fine-tuned large language model and the true value, and if the similarity meets the requirements, obtain a fine-tuned large language model;

[0047] If the similarity does not meet the requirement, step 22 to step 23 are executed again until the similarity meets the requirement, and a fine-tuned large language model is obtained.

[0048] Preferably, the similarity is calculated by text2vec.

[0049] Preferably, in step 4, LoRA is used to perform a second fine-tuning on the large language model after the first fine-tuning based on the decision data set to obtain the large language model after the second fine-tuning; the specific process is:

[0050] Step 41: Freeze the weight W0 of the large language model after fine-tuning, W0=r×i;

[0051] r×i represents the rows and columns of weight W0;

[0052] Step 42: Initialize two low-rank matrices A and B, A = r×t, B = t×i;

[0053] r×t represents the rows and columns of matrix A respectively; t×i represents the rows and columns of matrix B respectively;

[0054] Multiply the initialized low-rank matrices A and B to replace the weight W0 of the fine-tuned large language model; obtain the fine-tuned large language model with A and B introduced;

[0055] Step 43: Input the decision data set into the fine-tuned large language model with A and B introduced once, and keep other parameters of the fine-tuned large language model except A and B unchanged, and only train A and B until convergence, so as to obtain the trained fine-tuned large language model with A and B introduced once;

[0056] The A and B in the large language model after fine-tuning after the training A and B are the trained A and B;

[0057] Multiply the trained A and B as the new weight increment ΔW;

[0058] Step 44: The weight W0 of the fine-tuned large language model frozen in step 41 and the new parameter weight ΔW obtained in step 43 are fused to obtain a fused weight matrix W, and the fused weight matrix W is used to replace the weight of the fine-tuned large language model to obtain a fine-tuned large language model.

[0059] Preferably, in step 44, the weight W0 of the fine-tuned large language model frozen in step 41 and the new parameter weight ΔW obtained in step 43 are fused to obtain a fused weight matrix W; the specific process is:

[0060] The fused weight matrix W=ΔW+W0=AB+W0.

[0061] Preferably, in step 5, the speech recognition module and the speech synthesis module are connected to the large language model after secondary fine-tuning, the user's speech problem to be tested is processed, and speech is generated to interact with the user; the specific process is:

[0062] ASR converts the user's voice question to be tested into text, which is then passed to the fine-tuned large language model. The fine-tuned large language model generates a big tuple output = <command:content>, binary output = <command:content>The large language model after the second fine-tuning generates the result which is then handed over to the text-to-speech engine, which generates speech to interact with the user.

[0063] A computer storage medium stores at least one instruction, which is loaded and executed by a processor to implement the voice interaction method based on hybrid training strategy decision and large language model.

[0064] The beneficial effects of the present invention are:

[0065] Pre-trained large models usually have extensive general knowledge, but may not perform well on specific tasks. Through fine-tuning, large models can be transformed from general to specialized, significantly improving their performance on specific tasks (such as text classification, sentiment analysis, machine translation, etc.); full parameter fine-tuning can greatly enhance the performance of large language models on specific dialogue tasks; and solve the problem that existing large language models cannot provide accurate answers due to lack of sufficient domain knowledge.

[0066] In LoRA fine-tuning, since the new weights change very little compared to the original pre-trained weights, there is almost no delay compared to the base model during inference. In addition, this approach enables the model to plan the user's input into the corresponding category of decision requirements.

[0067] Through the hybrid training strategy, the computing resource consumption when performing the same task is reduced, so that large models can be smoothly deployed on devices with limited computing power. Combining these two large model usage methods eliminates the transmission delay between large models on different machines, further improving system efficiency and solving the problem of high computing cost and long response time caused by using two large language models.

[0068] In addition, by integrating and expanding ASR (speech recognition) and TTS (text-to-speech) modules, the system's voice input and output functions can be realized, which greatly improves the user experience and makes the system more natural and easy to use. Choosing the right engine, combining streaming and cache optimization, and continuous testing and optimization will help build an efficient, responsive and user-friendly voice interaction system. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 It is a flow chart of the present invention;

[0070] Figure 2 It is a flow chart for the interaction of large language models;

[0071] Figure 3 This is a flowchart of the large language model (decision-making dialogue model) after secondary fine-tuning;

[0072] Figure 4 The flowchart of the large language model after the second fine-tuning is obtained by using LoRA to perform the second fine-tuning on the large language model after the first fine-tuning based on the decision data set. DETAILED DESCRIPTION

[0073] Specific implementation method 1: The specific process of the speech interaction method based on the decision of the hybrid training strategy and the large language model in this implementation method is as follows:

[0074] Step 1: Set a specific answer format; build a decision dataset;

[0075] The decision data set includes prompt words, user input, and expected answers;

[0076] Step 2: Build a dialogue dataset for dialogue question and answer in specific scenarios;

[0077] The dialogue dataset includes the roles played by the large language model, user input, and corresponding answers;

[0078] If the specific scene is an exhibition hall, the role is the exhibition hall introduction staff; if the specific scene is a hotel, the role is the hotel service staff;

[0079] Step 3: Fine-tune the large language model for the first time using all-parameter fine-tuning based on the dialogue dataset to obtain a fine-tuned large language model;

[0080] Step 4: Based on the decision data set, LoRA is used to perform a second fine-tuning on the large language model after the first fine-tuning to obtain a large language model after the second fine-tuning;

[0081] Step 5: Connect the speech recognition module (ASR) and speech synthesis module (TTS) to the large language model after secondary fine-tuning, process the user's voice questions to be tested, and generate voice to interact with the user.

[0082] Specific implementation method 2: This implementation method is different from specific implementation method 1 in that a specific answer format is set in step 1; a decision data set is constructed;

[0083] The decision data set includes prompt words, user input, and expected answers;

[0084] The specific process is:

[0085] Step 11: Set a specific answer format;

[0086] Set the expected answer to the tuple output = <command:content>;

[0087] Among them, the output is a binary tuple;

[0088] command is the instruction type;

[0089] Instruction types usually include categories or types related to the question, such as words like "explanation", "definition", "step", "suggestion", etc. that help clarify the intention and nature of the answer;

[0090] content is the specific content of the instruction;

[0091] The content part contains the actual answer of the large model to the instruction, which is detailed information or explanation directly related to the question;

[0092] command is the instruction type, that is, those {navigation, introduction, aborted} mentioned in step 12;

[0093] content is what needs to be done under this command. For example, <navigation: Automobile Exhibition Hall> is equivalent to telling the robot to take me to the automobile exhibition hall, and <introduction: Automobile 001> is to tell the robot to introduce automobile 001;

[0094] In this method, large language models are mainly used for navigation and introduction in the exhibition hall.

[0095] Instructions include {navigation, inrtoduction, aborted};

[0096] navigation means navigation, inrtoduction means introduction, and aborted means invalid;

[0097] Step 12: Construct a decision data set according to the set specific answer format and user input; The specific process is as follows:

[0098] Set instruction as the prompt word (According to the user input, judge whether the user's intention is navigation or introduction. If the user's input intention is navigation, output 'navigation:xxx'; if the user's input intention is introduction, output 'inrtoduction:xxx'; if the user's input has no clear meaning, output 'aborted';)

[0099] The decision dataset is constructed based on the user question input, the prompt word instruction and the bigram output. The entry format of the decision dataset is<instruction,input,output> A collection of;

[0100] To simulate the instructions and requests that users may issue in the exhibition hall, we first artificially generate some dialogues, determine the corresponding instructions, and add them to the dataset.

[0101] The other steps and parameters are the same as those in the first embodiment.

[0102] Specific implementation method three: This implementation method is different from specific implementation methods one or two in that a dialogue dataset for dialogue question and answer in a specific scenario is constructed in step 2;

[0103] The dialogue dataset includes the roles played by the large language model, user input, and corresponding answers;

[0104] The specific process is:

[0105] Step 21: Collect corpus in the exhibition hall environment;

[0106] The corpus in the exhibition hall environment is the basic information of the exhibits in the exhibition hall;

[0107] The basic information of the exhibits in the exhibition hall includes the basic characteristics, uses, parameter information of the exhibits, and the location information of the exhibition hall;

[0108] For example, in a car showroom:

[0109] Basic features refer to: advantages compared to other cars, speed, how far you can travel on a single tank of gas, etc.:

[0110] Use refers to: how many people can be carried and how many kilometers can be driven;

[0111] Parameters refer to: basic information such as vehicle length and height;

[0112] This method is temporarily applied in the exhibition hall environment. In order to meet the needs of users to understand the exhibits, it is necessary to collect basic information about the exhibits in the exhibition hall. It should include both the basic characteristics and uses of the exhibits and the parameter information of the exhibits. At the same time, in this environment, users may need a large model to guide their visit, so it is also necessary to organize the location information of the specific environment as corpus.

[0113] Step 22: preprocessing the corpus in the exhibition hall environment collected in step 21 to obtain preprocessed corpus;

[0114] The specific process is:

[0115] Extract key information of the corpus in the exhibition hall environment collected in step 21, generate potential problems based on the key information, and use the potential problems as preprocessed corpus;

[0116] For example, what questions will the user ask? You can use this information to answer them, such as the historical background of the exhibits, that is, "What is the historical background of this exhibit?" For each question, combine the exhibit information and create an answer template, such as:

[0117] Question: Instruction: What is the historical background of this exhibit? <commend:content>

[0118] Answer: This exhibit is from the XX era, represents the characteristics of XX culture, and displays XX artistic style.

[0119] Then organize these questions into question and answer pairs.

[0120] Step 23: Based on the binary output <command:content>, preprocessed corpus, and build a dialogue database for dialogue question and answer in specific scenarios.

[0121] Output the tuple = <command:content>As the input of the large language model, the preprocessed corpus is used as the output of the large language model.

[0122] This method mentions the use of <command:content>As the output of the decision model. In this step, in order to better couple the decision model, based on the binary output = <command:content>, preprocessed corpus, and construct a dialogue database for dialogue question and answer in specific scenarios. In this method, the main questions and answers about the introduction of exhibits in the exhibition hall and the navigation questions and answers about the exhibition hall exhibition area are generated.

[0123] The other steps and parameters are the same as those in the first or second embodiment.

[0124] Specific implementation method 4: This implementation method is different from any one of specific implementation methods 1 to 3 in that in step 3, the large language model is fine-tuned for the first time based on the dialogue data set by using full parameter fine-tuning to obtain a large language model after one fine-tuning;

[0125] The specific process is:

[0126] Step 31: Select a large language model and load the large language model into the exhibition hall environment;

[0127] Select a more suitable model from the currently available open source large models as the base model (such as GLM4, etc.). When selecting, you need to consider: the performance of the model, the model architecture, scalability, and the computing resources required. After selecting the model, load it into the corresponding environment to ensure that the next step of training can be carried out.

[0128] Step 32:

[0129] Divide the conversation dataset into training and testing sets;

[0130] Based on the training set in the dialogue dataset, select full parameter fine-tuning to train the large language model until convergence, and obtain a preliminarily fine-tuned large language model;

[0131] The full parameters include: base model hyperparameters, optimizer and loss function, weights, and biases;

[0132] Divide the conversation data into training set, test set and validation set, and determine the proportion of the three according to your own needs. Ensure that the data distribution is balanced on each subset. At the same time, according to the model size, data set size and own hardware resources, set the corresponding training-related hyperparameters, such as learning rate, batch size, etc., and select the appropriate optimizer and loss function so that the optimizer can effectively adjust the model parameters to minimize the loss function. Based on the pre-trained model, perform full parameter fine-tuning training, input task-specific data into the model through forward propagation, generate prediction results, and then calculate the loss value according to the task-specific loss function. The back-propagation algorithm is used to calculate the gradient of the function relative to the model parameters. Use the selected optimizer to update the model parameters according to the calculated gradient to minimize the loss function.

[0133] Step 33: Evaluate and tune the performance of the initially fine-tuned large language model. The specific process is as follows:

[0134] Use the test set to test the fine-tuned large language model to obtain the similarity between the output of the fine-tuned large language model and the true value. If the similarity meets the requirements, obtain a fine-tuned large language model (fine-tuned dialogue model).

[0135] If the similarity does not meet the requirement, re-execute step 22 to step 23 until the similarity meets the requirement, and obtain a fine-tuned large language model (fine-tuned dialogue model).

[0136] During fine-tuning, regularly use the validation set to evaluate model performance. Based on the validation set performance, adjust relevant hyperparameters in a timely manner. If you find that the model's performance on the validation set has declined, you can use methods such as early stopping or regularization to improve the model performance. After fine-tuning, use the test set to test and evaluate the model's generalization ability and performance on specific dialogue tasks. Ensure that the model performs as expected on unseen data and specific tasks.

[0137] The other steps and parameters are the same as those in the first or second embodiment.

[0138] Specific implementation method 5: This implementation method is different from the specific implementation methods 1 to 4 in that the performance of the large language model that has been preliminarily fine-tuned is evaluated and optimized in step 33; the specific process is as follows:

[0139] Use the test set to test the fine-tuned large language model to obtain the similarity between the output of the fine-tuned large language model and the true value. If the similarity meets the requirements, obtain a fine-tuned large language model (fine-tuned dialogue model).

[0140] If the similarity does not meet the requirement, re-execute step 22 to step 23 until the similarity meets the requirement, and obtain a fine-tuned large language model (fine-tuned dialogue model).

[0141] During fine-tuning, regularly use the validation set to evaluate model performance. Based on the validation set performance, adjust relevant hyperparameters in a timely manner. If you find that the model's performance on the validation set has declined, you can use methods such as early stopping or regularization to improve the model performance. After fine-tuning, use the test set to test and evaluate the model's generalization ability and performance on specific dialogue tasks. Ensure that the model performs as expected on unseen data and specific tasks.

[0142] The other steps and parameters are the same as those in Specific Embodiments 1 to 4.

[0143] Specific implementation method 6: This implementation method is different from any one of specific implementation methods 1 to 5 in that the similarity is calculated by text2vec.

[0144] The other steps and parameters are the same as those in Specific Implementation Methods 1 to 5.

[0145] Specific implementation method 7: This implementation method is different from any one of specific implementation methods 1 to 6 in that in step 4, LoRA is used to perform a second fine-tuning on the large language model after the first fine-tuning based on the decision data set to obtain a large language model after the second fine-tuning; the specific process is:

[0146] Step 41: Freeze the weight W0 of the large language model after fine-tuning, W0=r×i;

[0147] r×i represents the rows and columns of weight W0;

[0148] First, load the fine-tuned large language model and the weight W0 of the fine-tuned large language model.

[0149] When fine-tuning, the large language model W0 after fine-tuning is frozen to ensure that the general language ability of the model is not affected and to avoid a decrease in the generalization ability of the model due to fine-tuning.

[0150] Step 42: Initialize two low-rank matrices A and B, A = r×t, B = t×i;

[0151] r×t represents the rows and columns of the matrix respectively; t×i represents the rows and columns of the matrix respectively;

[0152] Then t is much smaller than r and i;

[0153] Multiply the initialized low-rank matrices A and B to replace the weight W0 of the fine-tuned large language model; obtain the fine-tuned large language model with A and B introduced;

[0154] Set the initial values ​​of A and B to random numbers close to zero so that the model can make subtle adjustments based on the pre-trained weights;

[0155] Step 43: Input the decision data set into the fine-tuned large language model with A and B introduced once, and keep other parameters of the fine-tuned large language model except A and B unchanged, and only train A and B until convergence, so as to obtain the trained fine-tuned large language model with A and B introduced once;

[0156] The A and B in the large language model after fine-tuning after the training A and B are the trained A and B;

[0157] Multiply the trained A and B as the new weight increment ΔW;

[0158] Step 44: The weight W0 of the fine-tuned large language model frozen in step 41 and the new parameter weight ΔW obtained in step 43 are merged to obtain a fused weight matrix W, and the fused weight matrix W is used to replace the weight of the fine-tuned large language model to obtain a fine-tuned large language model (decision-making dialogue model).

[0159] For a given input x, the output after adding LoRA is h = (ΔW + W0)x, ensuring that it can be adjusted for specific decision-making tasks while maintaining general capabilities.

[0160] The other steps and parameters are the same as those in Specific Embodiments 1 to 6.

[0161] Specific implementation eight: This implementation differs from any one of specific implementations one to seven in that, in step 44, the weight W0 of the large language model after fine-tuning frozen in step 41 and the new parameter weight ΔW obtained in step 43 are fused to obtain a fused weight matrix W; the specific process is as follows:

[0162] The fused weight matrix W=ΔW+W0=AB+W0.

[0163] The other steps and parameters are the same as those in Specific Embodiments 1 to 7.

[0164] Specific implementation method 9: This implementation method is different from the specific implementation methods 1 to 8 in that in step 5, the speech recognition module (ASR) and the speech synthesis module (TTS) are connected to the large language model after secondary fine-tuning, the user's speech problem to be tested is processed, and speech is generated to interact with the user; the specific process is:

[0165] ASR converts the user's voice question to be tested into text, and the text is given to the large language model (decision dialogue model) after a fine-tuning. The large language model after a fine-tuning generates a big tuple output = <command:content>, binary output = <command:content>The result is then sent to the large language model after the second fine-tuning (the fine-tuned dialogue model), and the result generated by the large language model after the second fine-tuning is sent to the text-to-speech engine, which generates speech to interact with the user.

[0166] The speech recognition module (ASR) is connected to the large language model for decision making;

[0167] Select an engine that can provide high-precision speech recognition, support the required languages ​​and dialects, and can handle speech inputs of different sound quality and noise levels. Considering that the dialogue system requires a shorter delay, the speech recognition service of iFlytek can be used in this part. Or use other speech recognition engines that support richer scenarios, such as Kaldi, FunASR and other speech recognition solutions. At the same time, deploy this part on a computing device with weaker computing power, and send the speech recognition results to the large language model deployed on the high-computing power server side using a network protocol for decision-making. In the decision-making process, it is necessary to add the prompt words mentioned in step 3 of this method at the same time.

[0168] After the large language model makes a decision, we get a <command:content>The command in the binary is analyzed. If the command is valid, the command content is used as a question and answer and then transferred to the large language model for question and answer.

[0169] Large language model question and answer external text-to-speech engine (TTS);

[0170] Choose an engine that can generate natural speech, support the required sound type and emotional expression, and adjust the speech speed and pitch. To ensure that the generated audio file is output synchronously with the user's needs and provide a smooth interactive experience. Depending on the usage scenario, you can choose streaming speech synthesis or batch generation. Commonly used speech synthesis engines include iFlytek TTS, EmotiVoice, PaddleSpeechTTS, etc.

[0171] The output of the large language model dialogue is sent to a computing device deployed remotely through a communication protocol, and streaming speech synthesis is performed, and the synthesized speech and text are played in a synchronous manner.

[0172] The other steps and parameters are the same as those in Specific Embodiments 1 to 8.

[0173] Specific implementation method ten: This implementation method is a computer storage medium, which stores at least one instruction. The at least one instruction is loaded and executed by a processor to implement the voice interaction method based on the decision-making of the hybrid training strategy and the large language model.

[0174] It should be understood that the instructions include computer program products, software or computerized methods corresponding to any method described in the present invention; the instructions can be used to program a computer system, or other electronic device. Computer storage media may include readable media on which instructions are stored, which may include but are not limited to magnetic storage media, optical storage media; magneto-optical storage media include read-only memory ROM, random access memory RAM, erasable programmable memory (e.g., EPROM and EEPROM) and flash memory layers, or other types of media suitable for storing electronic instructions.

[0175] The present invention may also have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art may make various corresponding changes and modifications based on the present invention, but these corresponding changes and modifications should all fall within the scope of protection of the claims attached to the present invention.< / command:content> < / command:content> < / command:content> < / command:content> < / command:content> < / command:content> < / command:content> < / commend:content> < / command:content> < / command:content> < / command:content> < / command:content> < / command:content>

Claims

1. A speech interaction method based on decision making of a hybrid training strategy and a large language model, characterized by: The specific process of the method is: Step 1: Set a specific answer format; build a decision dataset; The decision data set includes prompt words, user input, and expected answers; Step 2: Build a dialogue dataset for dialogue question and answer in specific scenarios; The dialogue dataset includes the roles played by the large language model, user input, and corresponding answers; Step 3: Fine-tune the large language model for the first time using all-parameter fine-tuning based on the dialogue dataset to obtain a fine-tuned large language model; Step 4: Based on the decision data set, LoRA is used to perform a second fine-tuning on the large language model after the first fine-tuning to obtain a large language model after the second fine-tuning; Step 5: Connect the speech recognition module and speech synthesis module to the large language model after secondary fine-tuning, process the user's speech questions to be tested, and generate speech to interact with the user.

2. The speech interaction method based on decision making and large language model of hybrid training strategy according to claim 1, characterized in that: In the step 1, a specific answer format is set; a decision data set is constructed; The decision data set includes prompt words, user input, and expected answers; The specific process is: Step 11: Set a specific answer format; Set the expected answer to the tuple output = <command:content> ；< / command:content> Among them, output is a two-tuple; command is the type of command; content is the specific content of the instruction; Instructions include {navigation, introduction, aborted}; Navigation means navigation, inrtoduction means introduction, and aborted means invalid; Step 12: Construct a decision dataset based on the specific answer format and user input; the specific process is as follows: Set instruction as the prompt word; The decision dataset is constructed based on the user question input, the prompt word instruction and the bigram output. The entry format of the decision dataset is<instruction,input,output> A collection of .

3. The speech interaction method based on decision making and large language model of hybrid training strategy according to claim 2, characterized in that: The said constructing a dialogue dataset for dialogue question and answering in a specific scenario; The dialogue dataset includes the roles played by the large language model, user input, and corresponding answers; The specific process is: Step 21: Collect corpus in the exhibition hall environment; The corpus in the exhibition hall environment is the basic information of the exhibits in the exhibition hall; The basic information of the exhibits in the exhibition hall includes the basic characteristics, uses, parameter information of the exhibits, and the location information of the exhibition hall; Step 22: Preprocess the corpus in the exhibition hall environment collected in step 21 to obtain preprocessed corpus; the specific process is: Extract key information of the corpus in the exhibition hall environment collected in step 21, generate potential problems based on the key information, and use the potential problems as preprocessed corpus; Step 23: Based on the binary output <command:content> , preprocessed corpus, and build a dialogue database for dialogue question and answer in specific scenarios.< / command:content> 4. The speech interaction method based on decision making and large language model of hybrid training strategy according to claim 3, characterized in that: In step 3, the large language model is fine-tuned for the first time based on the dialogue dataset by using all-parameter fine-tuning to obtain a large language model after fine-tuning. The specific process is: Step 31: Select a large language model and load the large language model into the exhibition hall environment; Step 32: Divide the conversation dataset into training and testing sets; Based on the training set in the dialogue dataset, select full parameter fine-tuning to train the large language model until convergence, and obtain a preliminarily fine-tuned large language model; The full parameters include: base model hyperparameters, optimizer and loss function, weights, and biases; Step 33: Evaluate and tune the performance of the initially fine-tuned large language model. The specific process is as follows: Use the test set to test the fine-tuned large language model, obtain the similarity between the output of the fine-tuned large language model and the true value, and if the similarity meets the requirements, obtain a fine-tuned large language model; If the similarity does not meet the requirement, step 22 to step 23 are executed again until the similarity meets the requirement, and a fine-tuned large language model is obtained.

5. The speech interaction method based on decision making and large language model of hybrid training strategy according to claim 4, characterized in that: In step 33, the performance of the preliminarily fine-tuned large language model is evaluated and tuned; the specific process is as follows: Use the test set to test the fine-tuned large language model, obtain the similarity between the output of the fine-tuned large language model and the true value, and if the similarity meets the requirements, obtain a fine-tuned large language model; If the similarity does not meet the requirement, step 22 to step 23 are executed again until the similarity meets the requirement, and a fine-tuned large language model is obtained.

6. The speech interaction method based on decision making and large language model of hybrid training strategy according to claim 5, characterized in that: The similarity is calculated by text2vec.

7. The speech interaction method based on decision making and large language model of hybrid training strategy according to claim 6, characterized in that: In step 4, LoRA is used to perform a second fine-tuning on the large language model after the first fine-tuning based on the decision data set to obtain the large language model after the second fine-tuning; the specific process is: Step 41: Freeze the weight W0 of the large language model after fine-tuning, W0=r×i; r×i represents the rows and columns of weight W0; Step 42: Initialize two low-rank matrices A and B, A = r×t, B = t×i; r×t represents the rows and columns of matrix A respectively; t×i represents the rows and columns of matrix B respectively; Multiply the initialized low-rank matrices A and B to replace the weight W0 of the fine-tuned large language model; obtain the fine-tuned large language model with A and B introduced; Step 43: Input the decision data set into the fine-tuned large language model with A and B introduced once, and keep other parameters of the fine-tuned large language model except A and B unchanged, and only train A and B until convergence, so as to obtain the trained fine-tuned large language model with A and B introduced once; The A and B in the large language model after fine-tuning after the training A and B are the trained A and B; Multiply the trained A and B as the new weight increment ΔW; Step 44: The weight W0 of the fine-tuned large language model frozen in step 41 and the new parameter weight ΔW obtained in step 43 are fused to obtain a fused weight matrix W, and the fused weight matrix W is used to replace the weight of the fine-tuned large language model to obtain a fine-tuned large language model.

8. The speech interaction method based on decision making and large language model of hybrid training strategy according to claim 7, characterized in that: In step 44, the weight W0 of the fine-tuned large language model frozen in step 41 and the new parameter weight ΔW obtained in step 43 are fused to obtain a fused weight matrix W. The specific process is: The fused weight matrix W=ΔW+W0=AB+W0.

9. The speech interaction method based on decision making and large language model of hybrid training strategy according to claim 8, characterized in that: In step 5, the speech recognition module and the speech synthesis module are connected to the large language model after secondary fine-tuning, the user's speech problem to be tested is processed, and speech is generated to interact with the user; the specific process is: ASR converts the user's voice question to be tested into text, which is then passed to the fine-tuned large language model. The fine-tuned large language model generates a big tuple output = <command:content>, binary output = <command:content>The large language model after the second fine-tuning generates the result which is then handed over to the text-to-speech engine, which generates speech to interact with the user.< / command:content> < / command:content> 10. A computer storage medium, characterized in that: The storage medium stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the voice interaction method based on the decision-making of the hybrid training strategy and the large language model as described in any one of claims 1 to 9.

Images