Speech interaction method and storage medium based on decision-making and large language model of hybrid training strategy

Abstract

The present invention relates to a speech interaction method and storage medium based on a decision-making and large language model of a hybrid training strategy. The purpose of the present invention is to solve the problem that the existing large language model cannot provide accurate answers due to the lack of sufficient domain knowledge, and the use of two large language models will bring high computational costs and long response time. The process is: setting a specific answer format; constructing a decision data set; constructing a dialogue data set for dialogue questions and answers in a specific scenario; fine-tuning the large language model for the first time using full parameter fine-tuning based on the dialogue data set to obtain a large language model after the first fine-tuning; fine-tuning the large language model after the first fine-tuning for the second time using LoRA based on the decision data set to obtain a large language model after the second fine-tuning; connecting the speech recognition module and the speech synthesis module to the large language model after the second fine-tuning, processing the user's voice questions to be tested, and generating speech to interact with the user.

Description

Technical Field

[0001] The invention relates to a voice interaction method and a storage medium. Background Art

[0002] The Large Language Model (LLM) has powerful interactive capabilities and can perform natural language dialogue tasks. Through fine-tuning technology, the dialogue performance of the large language model in specific scenarios or special task contexts is further enhanced. This technical application is based on a variety of fine-tuning strategies to enhance the application of large language models in exhibition hall robots, enabling them to respond to visitors' instructions more accurately and make corresponding decisions and dialogues. In addition, by combining existing ASR (speech recognition) and TTS (text to speech) technologies, the transition from text interaction to voice interaction is achieved, greatly improving the interactive experience between exhibition hall robots and visitors.

[0003] First, although large language models perform well across a wide range of tasks, they often fail to provide accurate answers in certain highly specialized areas due to a lack of sufficient domain knowledge. This limitation makes large language models face significant challenges when dealing with complex or specialized tasks.

[0004] Secondly, in real-world applications, speech recognition technology often encounters a critical issue: misidentification of background noise and meaningless conversations. If these misidentification results are directly fed into a large language model, they often result in ineffective or confusing conversations, impacting the user experience. To address this issue, current mainstream solutions typically employ two large language models: one to determine whether user input is valuable, and the other to respond to valuable input. However, while this approach improves conversation quality to a certain extent, it also results in significant waste of computing resources. Because it requires running two large models, this solution suffers from significant resource inefficiencies, particularly in terms of computational cost and response time. Summary of the Invention

[0005] The purpose of the present invention is to solve the problems that the existing large language model cannot provide accurate answers due to the lack of sufficient domain knowledge, and the use of two large language models will bring high computational costs and long response time, and to propose a decision-making and large model voice interaction method and storage medium based on a hybrid training strategy.

[0006] The specific process of the voice interaction method based on hybrid training strategy decision-making and large language model is as follows:

[0007] Step 1: Set a specific answer format and build a decision dataset;

[0008] The decision dataset includes prompt words, user input, and expected answers;

[0009] Step 2: Build a dialogue dataset for dialogue question answering in specific scenarios;

[0010] The dialogue dataset includes the roles played by the large language model, user input, and corresponding answers;

[0011] Step 3: Fine-tune the large language model for the first time using full-parameter fine-tuning based on the dialogue dataset to obtain a fine-tuned large language model.

[0012] Step 4: Based on the decision dataset, LoRA is used to perform a second fine-tuning on the large language model after the first fine-tuning to obtain the large language model after the second fine-tuning;

[0013] Step 5: Connect the speech recognition module and speech synthesis module to the large language model after secondary fine-tuning, process the user's speech questions to be tested, and generate speech to interact with the user.

[0014] Preferably, in step 1, a specific answer format is set; a decision data set is constructed;

[0015] The decision dataset includes prompt words, user input, and expected answers;

[0016] The specific process is:

[0017] Step 11: Set a specific answer format;

[0018] Set the expected answer to output= <command:content>;

[0019] Among them, output is a two-tuple;

[0020] command is the type of instruction;

[0021] content is the specific content of the instruction;

[0022] Instructions include {navigation, inrtoduction, aborted};

[0023] Navigation means navigation, inrtoduction means introduction, and aborted means invalid;

[0024] Step 12: Construct a decision dataset based on the specific answer format and user input. The specific process is as follows:

[0025] Set instruction as the prompt word;

[0026] The decision dataset is constructed based on the user question input, prompt word instruction and bigram output. The entry format of the decision dataset is<instruction,input,output> A collection of .

[0027] Preferably, the step of constructing a dialogue dataset for dialogue question and answer in a specific scenario;

[0028] The dialogue dataset includes the roles played by the large language model, user input, and corresponding answers;

[0029] The specific process is:

[0030] Step 21: Collect corpus data in the exhibition hall environment;

[0031] The corpus in the exhibition hall environment is the basic information of the exhibits in the exhibition hall;

[0032] The basic information of exhibits in the exhibition hall includes the basic characteristics, uses, parameter information of exhibits, and location information of the exhibition hall;

[0033] Step 22: Preprocess the corpus collected in the exhibition hall environment in step 21 to obtain preprocessed corpus. The specific process is as follows:

[0034] Extract key information of the corpus in the exhibition hall environment collected in step 21, generate potential questions based on the key information, and use the potential questions as the preprocessed corpus;

[0035] Step 23: Based on the binary output <command:content>, preprocessed corpus, and build a dialogue database for dialogue question and answer in specific scenarios.

[0036] Preferably, in step 3, the large language model is fine-tuned for the first time using all-parameter fine-tuning based on the dialogue dataset to obtain a large language model after fine-tuning. The specific process is:

[0037] Step 31: Select a large language model and load it into the exhibition hall environment;

[0038] Step 32:

[0039] Divide the conversation dataset into training and testing sets;

[0040] Based on the training set of the dialogue dataset, select full parameter fine-tuning to train the large language model until convergence, and obtain a preliminarily fine-tuned large language model;

[0041] The full parameters include: base model hyperparameters, optimizer and loss function, weights, and biases;

[0042] Step 33: Evaluate and optimize the performance of the initially fine-tuned large language model. The specific process is as follows:

[0043] Use the test set to test the fine-tuned large language model and obtain the similarity between the output of the fine-tuned large language model and the true value. If the similarity meets the requirements, the fine-tuned large language model is obtained.

[0044] If the similarity does not meet the requirements, re-execute steps 22 to 23 until the similarity meets the requirements, and obtain a fine-tuned large language model.

[0045] Preferably, in step 33, the performance of the preliminarily fine-tuned large language model is evaluated and tuned; the specific process is as follows:

[0046] Use the test set to test the fine-tuned large language model and obtain the similarity between the output of the fine-tuned large language model and the true value. If the similarity meets the requirements, the fine-tuned large language model is obtained.

[0047] If the similarity does not meet the requirement, re-execute step 22 to step 23 until the similarity meets the requirement, and obtain a fine-tuned large language model.

[0048] Preferably, the similarity is calculated by text2vec.

[0049] Preferably, in step 4, LoRA is used to perform a second fine-tuning on the large language model after the first fine-tuning based on the decision data set to obtain a large language model after the second fine-tuning; the specific process is:

[0050] Step 41: Freeze the weight W0 of the large language model after fine-tuning, W0 = r × i;

[0051] r×i represents the rows and columns of weight W0;

[0052] Step 42: Initialize two low-rank matrices A and B, A = r × t, B = t × i;

[0053] r×t represents the rows and columns of matrix A respectively; t×i represents the rows and columns of matrix B respectively;

[0054] Multiply the initialized low-rank matrices A and B to replace the weight W0 of the fine-tuned large language model; obtain the fine-tuned large language model that introduces A and B;

[0055] Step 43: Input the decision dataset into the fine-tuned large language model with A and B. All parameters except A and B in the fine-tuned large language model remain unchanged. Only A and B are trained until convergence, thereby obtaining a trained fine-tuned large language model with A and B.

[0056] The A and B in the large language model after fine-tuning after the training introduction of A and B are the trained A and B;

[0057] Multiply the trained A and B as the new weight increment ΔW;

[0058] Step 44: The weight W0 of the fine-tuned large language model frozen in step 41 and the new parameter weight ΔW obtained in step 43 are fused to obtain a fused weight matrix W. The fused weight matrix W is used to replace the weight of the fine-tuned large language model to obtain a fine-tuned large language model.

[0059] Preferably, in step 44, the weight W0 of the fine-tuned large language model frozen in step 41 and the new parameter weight ΔW obtained in step 43 are fused to obtain a fused weight matrix W. The specific process is as follows:

[0060] The fused weight matrix W = ΔW + W0 = AB + W0.

[0061] Preferably, in step 5, the speech recognition module and the speech synthesis module are connected to the large language model after secondary fine-tuning, the user's speech problem to be tested is processed, and speech is generated to interact with the user; the specific process is:

[0062] ASR converts the user's voice question into text, which is then passed to a fine-tuned large language model. The fine-tuned large language model generates a big-tuple output = <command:content>, output= <command:content>The result is then passed to the large language model after the second fine-tuning, and the result generated by the large language model after the second fine-tuning is passed to the text-to-speech engine, which generates speech to interact with the user.

[0063] A computer storage medium stores at least one instruction, which is loaded and executed by a processor to implement the voice interaction method based on hybrid training strategy decision-making and a large language model.

[0064] The beneficial effects of the present invention are:

[0065] Pre-trained large models often possess extensive general knowledge but may underperform on specific tasks. Fine-tuning can shift large models from generality to specificity, significantly improving their performance on specific tasks (such as text classification, sentiment analysis, and machine translation). Full parameter fine-tuning can significantly enhance the performance of large language models on specific conversational tasks, addressing the problem of existing large language models being unable to provide accurate responses due to a lack of sufficient domain knowledge.

[0066] In LoRA fine-tuning, since the new weights change very little compared to the original pre-trained weights, there is almost no latency during inference compared to the base model. In addition, this approach allows the model to classify user input into decision requirements for the corresponding category.

[0067] The hybrid training strategy reduces computational resource consumption when executing the same task, enabling the smooth deployment of large models on devices with limited computing power. Combining these two large model usage methods eliminates transmission delays between large models on different machines, further improving system efficiency and addressing the high computational cost and slow response time associated with using two large language models.

[0068] Furthermore, by integrating and expanding ASR (speech recognition) and TTS (text-to-speech) modules, the system's voice input and output capabilities can be implemented, significantly improving the user experience and making the system more natural and easy to use. Selecting the right engine, combined with stream processing and cache optimization, and through continuous testing and optimization, will help build an efficient, responsive, and user-friendly voice interaction system. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 Flowchart of the present invention;

[0070] Figure 2 This is a flowchart for the interaction of a large language model;

[0071] Figure 3 This is a flowchart of the large language model (decision-making dialogue model) after secondary fine-tuning;

[0072] Figure 4 This is a flowchart of how to fine-tune a large language model after the first fine-tuning using LoRA based on a decision dataset to obtain the large language model after the second fine-tuning. DETAILED DESCRIPTION

[0073] Specific implementation method 1: The specific process of the speech interaction method based on the decision-making of the hybrid training strategy and the large language model in this implementation method is as follows:

[0074] Step 1: Set a specific answer format and build a decision dataset;

[0075] The decision dataset includes prompt words, user input, and expected answers;

[0076] Step 2: Build a dialogue dataset for dialogue question answering in specific scenarios;

[0077] The dialogue dataset includes the roles played by the large language model, user input, and corresponding answers;

[0078] If the specific scene is an exhibition hall, the role is the exhibition hall introduction staff; if the specific scene is a hotel, the role is the hotel service staff;

[0079] Step 3: Fine-tune the large language model for the first time using full-parameter fine-tuning based on the dialogue dataset to obtain a fine-tuned large language model.

[0080] Step 4: Based on the decision dataset, LoRA is used to perform a second fine-tuning on the large language model after the first fine-tuning to obtain the large language model after the second fine-tuning;

[0081] Step 5: Connect the speech recognition module (ASR) and speech synthesis module (TTS) to the large language model after secondary fine-tuning, process the user's speech questions to be tested, and generate speech to interact with the user.

[0082] Specific implementation method 2: This implementation method is different from specific implementation method 1 in that a specific answer format is set in step 1; a decision data set is constructed;

[0083] The decision dataset includes prompt words, user input, and expected answers;

[0084] The specific process is:

[0085] Step 11: Set a specific answer format;

[0086] Set the expected answer to output= <command:content>;

[0087] Among them, the output is a binary tuple;

[0088] command is the instruction type;

[0089] The instruction type usually includes categories or types related to the question, such as "explanation", "definition", "step", "suggestion", etc., which are words that help clarify the intention and nature of the answer;

[0090] content is the specific content of the instruction;

[0091] The content part contains the actual answer of the large model to the instruction, which is detailed information or explanation directly related to the question;

[0092] command is the instruction type, that is, those {navigation, introduction, aborted} mentioned in step 12;

[0093] content is what needs to be done under this command. For example, <navigation: Automobile Exhibition Hall> is equivalent to telling the robot to take me to the automobile exhibition hall, and <introduction: Automobile 001> is to tell the robot to introduce automobile 001;

[0094] In this method, the large language model is mainly used for navigation and introduction in the exhibition hall.

[0095] The instructions include {navigation, inrtoduction, aborted};

[0096] navigation means navigation, inrtoduction means introduction, and aborted means invalid;

[0097] Step 12: Construct a decision data set according to the set specific answer format and user input; The specific process is as follows:

[0098] Set instruction as the prompt word (According to the user input, judge whether the user's intention is navigation or introduction. If the user input intention is navigation, output 'navigation:xxx'; if the user input intention is introduction, output 'inrtoduction:xxx'; if the user input has no clear meaning, output 'aborted';)

[0099] The decision dataset is constructed based on the user question input, prompt word instruction and bigram output. The entry format of the decision dataset is<instruction,input,output> A collection of

[0100] To simulate the instructions and requests that users may make in the exhibition hall, we first artificially generate some conversations, determine the corresponding instructions, and add them to the dataset.

[0101] Other steps and parameters are the same as those in the first embodiment.

[0102] Specific implementation method three: This implementation method differs from specific implementation methods one or two in that, in step 2, a dialogue dataset for dialogue question and answer in a specific scenario is constructed;

[0103] The dialogue dataset includes the roles played by the large language model, user input, and corresponding answers;

[0104] The specific process is:

[0105] Step 21: Collect corpus data in the exhibition hall environment;

[0106] The corpus in the exhibition hall environment is the basic information of the exhibits in the exhibition hall;

[0107] The basic information of exhibits in the exhibition hall includes the basic characteristics, uses, parameter information of exhibits, and location information of the exhibition hall;

[0108] For example, a car showroom:

[0109] Basic characteristics refer to advantages compared to other cars, speed, range on a single tank of gas, etc.

[0110] Usage refers to: how many people it can carry and how many kilometers it can travel;

[0111] Parameters refer to: basic information such as vehicle length and height;

[0112] This method is currently being applied in an exhibition hall environment. To meet users' needs for understanding exhibits, it is necessary to collect basic information about the exhibits. This should include both the basic characteristics and uses of the exhibits, as well as their parameter information. Furthermore, in this environment, users may require a large model to guide their visit, so it is also necessary to organize location information of the specific environment as a corpus.

[0113] Step 22: Preprocess the corpus in the exhibition hall environment collected in step 21 to obtain preprocessed corpus;

[0114] The specific process is:

[0115] Extract key information of the corpus in the exhibition hall environment collected in step 21, generate potential questions based on the key information, and use the potential questions as the preprocessed corpus;

[0116] For example, you can use this information to answer questions users might ask, such as questions about the historical background of an exhibit, such as "What is the historical background of this exhibit?" For each question, you can create an answer template based on the exhibit information, such as:

[0117] Question: Instruction: What is the historical background of this exhibit? <commend:content>

[0118] Answer: This exhibit is from the XX era, represents the characteristics of XX culture, and displays the XX artistic style.

[0119] Then organize these questions into question and answer pairs.

[0120] Step 23: Based on the binary output <command:content>, preprocessed corpus, and build a dialogue database for dialogue question and answer in specific scenarios.

[0121] Output= <command:content>As the input of the large language model, the preprocessed corpus is used as the output of the large language model.

[0122] This method mentions the use of <command:content>As the output of the decision model. In this step, in order to better couple the decision model, based on the binary output = <command:content>, preprocessed corpus, and constructing a dialogue database for scenario-specific dialogue question and answer. In this method, the main tasks are to generate introduction questions and answers about exhibits in the exhibition hall and navigation questions and answers about the exhibition hall exhibition area.

[0123] Other steps and parameters are the same as those in the first or second embodiment.

[0124] Specific embodiment 4: This embodiment differs from any one of specific embodiments 1 to 3 in that, in step 3, the large language model is fine-tuned for the first time based on the dialogue dataset using full parameter fine-tuning to obtain a large language model after one fine-tuning;

[0125] The specific process is:

[0126] Step 31: Select a large language model and load it into the exhibition hall environment;

[0127] Select a suitable base model from among the currently available open-source models (e.g., GLM4). Consider the model's performance, architecture, scalability, and required computing resources. Once the model is selected, load it into the appropriate environment to ensure it can be used for the next step of training.

[0128] Step 32:

[0129] Divide the conversation dataset into training and testing sets;

[0130] Based on the training set of the dialogue dataset, select full parameter fine-tuning to train the large language model until convergence, and obtain a preliminarily fine-tuned large language model;

[0131] The full parameters include: base model hyperparameters, optimizer and loss function, weights, and biases;

[0132] Divide the conversation data into training, test, and validation sets, with the ratio of the three determined based on your needs. Ensure that the data is evenly distributed across each subset. At the same time, set corresponding training-related hyperparameters, such as the learning rate and batch size, based on the model size, dataset size, and your own hardware resources. Select an appropriate optimizer and loss function so that the optimizer can effectively adjust the model parameters to minimize the loss function. Based on the pre-trained model, perform full parameter fine-tuning training. Input task-specific data into the model through forward propagation to generate prediction results. The loss value is then calculated based on the task-specific loss function. The gradient of the loss function with respect to the model parameters is calculated using the backpropagation algorithm. Use the selected optimizer to update the model parameters based on the calculated gradient to minimize the loss function.

[0133] Step 33: Evaluate and optimize the performance of the initially fine-tuned large language model. The specific process is as follows:

[0134] Use the test set to test the fine-tuned large language model and obtain the similarity between the output of the fine-tuned large language model and the true value. If the similarity meets the requirements, obtain a fine-tuned large language model (fine-tuned dialogue model).

[0135] If the similarity does not meet the requirements, re-execute steps 22 to 23 until the similarity meets the requirements, and obtain a fine-tuned large language model (fine-tuned dialogue model).

[0136] During fine-tuning, regularly evaluate model performance using the validation set. Based on the validation set performance, adjust relevant hyperparameters as appropriate. If you notice a decline in model performance on the validation set, use methods such as early stopping or regularization to improve performance. After fine-tuning, test the model on the test set to evaluate its generalization capabilities and performance on specific conversational tasks. Ensure that the model performs as expected on unseen data and specific tasks.

[0137] Other steps and parameters are the same as those in the first or second embodiment.

[0138] Specific embodiment 5: This embodiment differs from any one of specific embodiments 1 to 4 in that, in step 33, the performance of the preliminarily fine-tuned large language model is evaluated and optimized; the specific process is as follows:

[0139] Use the test set to test the fine-tuned large language model and obtain the similarity between the output of the fine-tuned large language model and the true value. If the similarity meets the requirements, obtain a fine-tuned large language model (fine-tuned dialogue model).

[0140] If the similarity does not meet the requirements, re-execute steps 22 to 23 until the similarity meets the requirements, and obtain a fine-tuned large language model (fine-tuned dialogue model).

[0141] During fine-tuning, regularly evaluate model performance using the validation set. Based on the validation set performance, adjust relevant hyperparameters as appropriate. If you notice a decline in model performance on the validation set, use methods such as early stopping or regularization to improve performance. After fine-tuning, test the model on the test set to evaluate its generalization capabilities and performance on specific conversational tasks. Ensure that the model performs as expected on unseen data and specific tasks.

[0142] The other steps and parameters are the same as those in the first to fourth embodiments.

[0143] Specific embodiment 6: This embodiment differs from any one of specific embodiments 1 to 5 in that the similarity is calculated by text2vec.

[0144] The other steps and parameters are the same as those in the first to fifth embodiments.

[0145] Specific embodiment seven: This embodiment differs from any one of specific embodiments one to six in that, in step 4, LoRA is used to perform a second fine-tuning on the large language model after the first fine-tuning based on the decision data set to obtain a large language model after the second fine-tuning; the specific process is:

[0146] Step 41: Freeze the weight W0 of the large language model after fine-tuning, W0 = r × i;

[0147] r×i represents the rows and columns of weight W0;

[0148] First, load the fine-tuned large language model and the weight W0 of the fine-tuned large language model.

[0149] When fine-tuning, the large language model W0 after fine-tuning is frozen to ensure that the general language ability of the model is not affected and to avoid a decrease in the generalization ability of the model due to fine-tuning.

[0150] Step 42: Initialize two low-rank matrices A and B, A = r × t, B = t × i;

[0151] r×t represents the rows and columns of the matrix respectively; t×i represents the rows and columns of the matrix respectively;

[0152] Then t is much smaller than r and i;

[0153] Multiply the initialized low-rank matrices A and B to replace the weight W0 of the fine-tuned large language model; obtain the fine-tuned large language model that introduces A and B;

[0154] Set the initial values ​​of A and B to random numbers close to zero so that the model can make subtle adjustments based on the pre-trained weights;

[0155] Step 43: Input the decision dataset into the fine-tuned large language model with A and B. All parameters except A and B in the fine-tuned large language model remain unchanged. Only A and B are trained until convergence, thereby obtaining a trained fine-tuned large language model with A and B.

[0156] The A and B in the large language model after fine-tuning after the training introduction of A and B are the trained A and B;

[0157] Multiply the trained A and B as the new weight increment ΔW;

[0158] Step 44: Fuse the weight W0 of the fine-tuned large language model frozen in step 41 and the new parameter weight ΔW obtained in step 43 to obtain a fused weight matrix W. Replace the weight of the fine-tuned large language model with the fused weight matrix W to obtain a fine-tuned large language model (decision-making dialogue model).

[0159] For a given input x, the output after adding LoRA is h = (ΔW + W0)x, ensuring that it can be adjusted for specific decision-making tasks while maintaining its general capabilities.

[0160] The other steps and parameters are the same as those in the first to sixth embodiments.

[0161] Specific embodiment eight: This embodiment differs from any one of specific embodiments one to seven in that, in step 44, the weight W0 of the fine-tuned large language model frozen in step 41 and the new parameter weight ΔW obtained in step 43 are fused to obtain a fused weight matrix W; the specific process is as follows:

[0162] The fused weight matrix W = ΔW + W0 = AB + W0.

[0163] The other steps and parameters are the same as those in the first to seventh embodiments.

[0164] Specific embodiment 9: This embodiment differs from any one of specific embodiments 1 to 8 in that in step 5, the speech recognition module (ASR) and the speech synthesis module (TTS) are connected to the large language model after secondary fine-tuning to process the user's speech problem to be tested and generate speech to interact with the user; the specific process is as follows:

[0165] The ASR converts the user's voice question into text, which is then passed to a fine-tuned large language model (decision-making dialogue model). The fine-tuned large language model generates a binary output. <command:content>, output= <command:content>The result is then passed to the large language model after the second fine-tuning (the fine-tuned dialogue model), and the result generated by the large language model after the second fine-tuning is passed to the text-to-speech engine, which generates speech to interact with the user.

[0166] The speech recognition module (ASR) is connected to the large language model for decision making;

[0167] Select an engine that provides high-precision speech recognition, supports the required languages ​​and dialects, and can handle voice inputs of varying sound quality and noise levels. Given the need for short latency in the dialogue system, you can use iFlytek's speech recognition service for this part. Alternatively, consider using another speech recognition engine that supports a wider range of scenarios, such as Kaldi or FunASR. Deploy this part on a less powerful computing device and use a network protocol to send the speech recognition results to a large language model deployed on a high-powered server for decision-making. During the decision-making process, include the prompt words mentioned in step 3 of this method.

[0168] After the large language model makes a decision, we get a <command:content>The command in the binary is analyzed. If the command is valid, the command content is passed as a question and answer to the large language model for question and answer.

[0169] Large language model question answering with external text-to-speech (TTS) engine;

[0170] Choose an engine that can generate natural speech, supports the desired voice type and emotional expression, and allows for adjustable speech speed and pitch. To ensure the generated audio files are synchronized with user needs and provide a smooth interactive experience, choose streaming speech synthesis or batch generation depending on the use case. Common speech synthesis engines include iFlytek TTS, EmotiVoice, and PaddleSpeechTTS.

[0171] The output of the large language model dialogue is sent to a remote computing device through a communication protocol, and streaming speech synthesis is performed, and the synthesized speech and text are played synchronously.

[0172] The other steps and parameters are the same as those in Specific Embodiments 1 to 8.

[0173] Specific embodiment ten: This embodiment is a computer storage medium, which stores at least one instruction. The at least one instruction is loaded and executed by a processor to implement the voice interaction method based on the decision-making of the hybrid training strategy and the large language model.

[0174] It should be understood that the instructions include computer program products, software, or computerized methods corresponding to any method described in the present invention; the instructions can be used to program a computer system or other electronic device. Computer storage media may include readable media on which instructions are stored, and may include but are not limited to magnetic storage media, optical storage media; magneto-optical storage media include read-only memory ROM, random access memory RAM, erasable programmable memory (e.g., EPROM and EEPROM) and flash memory layers, or other types of media suitable for storing electronic instructions.

[0175] The present invention may have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art may make various corresponding changes and modifications based on the present invention, but these corresponding changes and modifications should all fall within the scope of protection of the claims attached to the present invention.< / command:content> < / command:content> < / command:content> < / command:content> < / command:content> < / command:content> < / command:content> < / commend:content> < / command:content> < / command:content> < / command:content> < / command:content> < / command:content>

Claims

1. A speech interaction method based on a hybrid training strategy and a large language model, characterized by: The specific process of the method is: Step 1: Set a specific answer format and build a decision dataset; The decision dataset includes prompt words, user input, and expected answers; Step 2: Build a dialogue dataset for dialogue question answering in specific scenarios; The dialogue dataset includes the roles played by the large language model, user input, and corresponding answers; Step 3: Fine-tune the large language model for the first time using full-parameter fine-tuning based on the dialogue dataset to obtain a fine-tuned large language model. Step 4: Based on the decision dataset, LoRA is used to perform a second fine-tuning on the large language model after the first fine-tuning to obtain the large language model after the second fine-tuning; Step 5: Connect the speech recognition module and speech synthesis module to the large language model after secondary fine-tuning, process the user's speech questions to be tested, and generate speech to interact with the user.

2. The speech interaction method based on decision-making and large language model of hybrid training strategy according to claim 1, characterized in that: In step 1, a specific answer format is set; a decision data set is constructed; The decision dataset includes prompt words, user input, and expected answers; The specific process is: Step 11: Set a specific answer format; Set the expected answer to output= <command:content> ；< / command:content> Among them, output is a two-tuple; command is the type of instruction; content is the specific content of the instruction; Instructions include {navigation, inrtoduction, aborted}; Navigation means navigation, inrtoduction means introduction, and aborted means invalid; Step 12: Construct a decision dataset based on the specific answer format and user input. The specific process is as follows: Set instruction as the prompt word; The decision dataset is constructed based on the user question input, prompt word instruction and bigram output. The entry format of the decision dataset is<instruction,input,output> A collection of .

3. The speech interaction method based on decision-making and large language model of hybrid training strategy according to claim 2, characterized in that: The said construction of a dialogue dataset for dialogue question and answer in a specific scenario; The dialogue dataset includes the roles played by the large language model, user input, and corresponding answers; The specific process is: Step 21: Collect corpus data in the exhibition hall environment; The corpus in the exhibition hall environment is the basic information of the exhibits in the exhibition hall; The basic information of exhibits in the exhibition hall includes the basic characteristics, uses, parameter information of exhibits, and location information of the exhibition hall; Step 22: Preprocess the corpus collected in the exhibition hall environment in step 21 to obtain preprocessed corpus. The specific process is as follows: Extract key information of the corpus in the exhibition hall environment collected in step 21, generate potential questions based on the key information, and use the potential questions as the preprocessed corpus; Step 23: Based on the binary output <command:content> , preprocessed corpus, and build a dialogue database for dialogue question and answer in specific scenarios.< / command:content> 4. The speech interaction method based on decision making and a large language model using a hybrid training strategy according to claim 3, characterized in that: In step 3, the large language model is fine-tuned for the first time using full parameter fine-tuning based on the dialogue dataset to obtain a large language model after fine-tuning. The specific process is as follows: Step 31: Select a large language model and load it into the exhibition hall environment; Step 32: Divide the conversation dataset into training and testing sets; Based on the training set of the dialogue dataset, select full parameter fine-tuning to train the large language model until convergence, and obtain a preliminarily fine-tuned large language model; The full parameters include: base model hyperparameters, optimizer and loss function, weights, and biases; Step 33: Evaluate and optimize the performance of the initially fine-tuned large language model. The specific process is as follows: Use the test set to test the fine-tuned large language model and obtain the similarity between the output of the fine-tuned large language model and the true value. If the similarity meets the requirements, the fine-tuned large language model is obtained. If the similarity does not meet the requirement, re-execute step 22 to step 23 until the similarity meets the requirement, and obtain a fine-tuned large language model.

5. The speech interaction method based on decision making and a large language model using a hybrid training strategy according to claim 4, characterized in that: In step 33, the performance of the preliminarily fine-tuned large language model is evaluated and optimized. The specific process is as follows: Use the test set to test the fine-tuned large language model and obtain the similarity between the output of the fine-tuned large language model and the true value. If the similarity meets the requirements, the fine-tuned large language model is obtained. If the similarity does not meet the requirement, re-execute step 22 to step 23 until the similarity meets the requirement, and obtain a fine-tuned large language model.

6. The speech interaction method based on decision making and a large language model using a hybrid training strategy according to claim 5, characterized in that: The similarity is calculated by text2vec.

7. The speech interaction method based on decision making and a large language model using a hybrid training strategy according to claim 6, characterized in that: In step 4, LoRA is used to perform a second fine-tuning on the large language model after the first fine-tuning based on the decision data set to obtain the large language model after the second fine-tuning; the specific process is: Step 41: Freeze the weight W0 of the large language model after fine-tuning, W0 = r × i; r×i represents the rows and columns of weight W0; Step 42: Initialize two low-rank matrices A and B, A = r × t, B = t × i; r×t represents the rows and columns of matrix A respectively; t×i represents the rows and columns of matrix B respectively; Multiply the initialized low-rank matrices A and B to replace the weight W0 of the fine-tuned large language model; obtain the fine-tuned large language model that introduces A and B; Step 43: Input the decision dataset into the fine-tuned large language model with A and B. All parameters except A and B in the fine-tuned large language model remain unchanged. Only A and B are trained until convergence, thereby obtaining a trained fine-tuned large language model with A and B. The A and B in the large language model after fine-tuning after the training introduction of A and B are the trained A and B; Multiply the trained A and B as the new weight increment ΔW; Step 44: The weight W0 of the fine-tuned large language model frozen in step 41 and the new parameter weight ΔW obtained in step 43 are fused to obtain a fused weight matrix W. The fused weight matrix W is used to replace the weight of the fine-tuned large language model to obtain a fine-tuned large language model.

8. The speech interaction method based on decision making and a large language model using a hybrid training strategy according to claim 7, characterized in that: In step 44, the weight W0 of the fine-tuned large language model frozen in step 41 and the new parameter weight ΔW obtained in step 43 are fused to obtain a fused weight matrix W. The specific process is as follows: The fused weight matrix W = ΔW + W0 = AB + W0.

9. The speech interaction method based on decision making and a large language model using a hybrid training strategy according to claim 8, characterized in that: In step 5, the speech recognition module and the speech synthesis module are connected to the large language model after secondary fine-tuning, the user's speech problem to be tested is processed, and speech is generated to interact with the user; the specific process is: ASR converts the user's voice question into text, which is then passed to a fine-tuned large language model. The fine-tuned large language model generates a big-tuple output = <command:content>, output= <command:content>The result is then passed to the large language model after the second fine-tuning, and the result generated by the large language model after the second fine-tuning is passed to the text-to-speech engine, which generates speech to interact with the user.< / command:content> < / command:content> 10. A computer storage medium, characterized in that The storage medium stores at least one instruction, which is loaded and executed by the processor to implement the voice interaction method based on hybrid training strategy decision-making and large language model as described in any one of claims 1 to 9.

Images