A multi-stage large model-based voice interaction method and system

Abstract

The application discloses a speech interaction method and system based on a multi-stage large model, and belongs to the technical field of speech recognition. The application constructs a multi-stage large model framework through an intention recognition large model, a networked response large model and a local instruction large model, identifies the text category of interactive text through the intention recognition large model, selects the networked response large model or the local instruction large model for different processing of the interactive text based on the difference of the text category, and realizes speech interaction of the multi-stage large model. The intention recognition large model, the networked response large model and the local instruction large model can avoid the inaccuracy of speech interaction recognition caused by the use of a single language model in the existing speech interaction process. The real-time speech signal is directly converted into interactive text, and the interactive text is directly processed, so that the inaccuracy of keywords caused by the randomness, complexity and ambiguity of the speech information of the user is avoided, and the recognition accuracy of speech interaction is improved.

Description

Technical Field

[0001] This invention belongs to the field of speech recognition technology, and in particular relates to a speech interaction method and system based on a multi-stage large model. Background Technology

[0002] With the continuous development of speech recognition technology, more and more voice interaction devices are now widely used in daily services and industrial applications. Users wake up the voice interaction device, and the device can respond accordingly by recognizing the user's voice information. Currently, the most common voice interaction method is a question-and-answer turn-based approach.

[0003] In current voice interaction methods, a trained language processing model is typically deployed in the voice interaction device, and voice interaction is achieved by extracting keywords from the user's voice. However, due to the randomness, complexity, and ambiguity of user voice information, and the considerable complexity of the information conveyed by the user's voice, using a single language processing model to extract keywords easily leads to inaccurate voice recognition and reduces the user's interactive experience. Therefore, there is an urgent need for a voice interaction method and system based on a multi-stage large model to overcome the shortcomings of existing technologies. Summary of the Invention

[0004] The present invention aims to provide a voice interaction method and system based on a multi-stage large model to solve the above-mentioned technical problems. By identifying the text category of the interactive text and selecting different large models based on the text category, the recognition accuracy of voice interaction is improved.

[0005] To address the aforementioned technical problems, embodiments of the present invention provide a voice interaction method based on a multi-stage large model, comprising:

[0006] In response to the user's voice interaction wake-up signal, collect the user's real-time voice signal;

[0007] Based on the real-time voice signal, interactive text is generated, and the interactive text is input into a preset intent recognition model to obtain the text category corresponding to the interactive text.

[0008] If the text category is response text, then the interactive text is input into a preset online response model to generate response text; and the user is responded to based on the response text to complete the voice interaction;

[0009] If the text category is instruction text, then the interactive text is input into a preset local instruction model to obtain operation instructions; voice broadcast content is generated based on the operation instructions; and the user is responded to based on the voice broadcast content to complete the voice interaction.

[0010] It is understood that this invention constructs a multi-stage large-scale model framework through an intent recognition large-scale model, a network response large-scale model, and a local command large-scale model. The intent recognition large-scale model identifies the text category of the interactive text, and then, based on the different text categories, either the network response large-scale model or the local command large-scale model is selected to process the interactive text differently, thus realizing multi-stage large-scale model voice interaction. By using the intent recognition large-scale model, the network response large-scale model, and the local command large-scale model, the inaccuracy in voice interaction recognition caused by using only a single language model in existing voice interaction processes can be avoided. By directly converting real-time voice signals into interactive text and directly processing the interactive text, the inaccuracy of keywords caused by the randomness, complexity, and ambiguity of user voice information is avoided, thereby improving the recognition accuracy of voice interaction.

[0011] As a preferred embodiment, the step of collecting the user's real-time voice signal in response to the user's voice interaction wake-up signal includes:

[0012] Responding to the user's voice interaction wake-up signal;

[0013] The voice interaction wake-up signal is subjected to feature extraction to obtain the feature matrix of the voice interaction wake-up signal;

[0014] Based on a preset similarity algorithm and combined with the feature matrix, the global similarity score of the voice interaction wake-up signal is calculated.

[0015] If the global similarity score exceeds the preset wake word threshold, the user's real-time voice signal is collected.

[0016] This preferred solution extracts the features of the voice interaction wake-up signal and then calculates the global similarity score based on a similarity algorithm. Real-time voice signals are only collected when the wake-up word threshold is exceeded, avoiding the collection of invalid voice signals due to false wake-ups. This improves the recognition accuracy and relevance of voice interaction, ensures the effectiveness and fluency of voice interaction, provides users with a more stable and reliable voice interaction environment, and enhances the voice interaction experience.

[0017] As a preferred embodiment, the step of extracting features from the voice interaction wake-up signal to obtain a feature matrix of the voice interaction wake-up signal includes:

[0018] The voice interaction wake-up signal is divided into several frames of audio signals, and the Mel frequency cepstral coefficients of each frame of audio signal are calculated based on a preset Mel filter.

[0019] The feature matrix of the voice interaction wake-up signal is constructed based on the Mel frequency cepstral coefficients of the audio signal in each frame.

[0020] This preferred solution refines the voice interaction wake-up signal by dividing it into several frames of audio signals and calculating the Mel frequency cepstral coefficients of each frame of audio signal using a Mel filter. This improves the accuracy of the Mel frequency cepstral coefficients, thereby extracting the key features of the voice interaction wake-up signal more accurately. Consequently, it improves the accuracy of the subsequent global similarity score, avoids invalid voice interaction wake-ups, and enhances the recognition accuracy of voice interaction.

[0021] As a preferred embodiment, the step of calculating the global similarity score of the voice interaction wake-up signal based on a preset similarity algorithm and the feature matrix includes:

[0022] The feature matrix is ​​input into a preset deep learning model to obtain the mapping score of the audio signal for each frame;

[0023] The global similarity score of the voice interaction wake-up signal is obtained by summing and averaging the mapping scores of all audio signals.

[0024] This preferred solution inputs the feature matrix into a preset deep learning model to obtain the mapping score of each frame of audio signal. Then, the global similarity score is obtained by summing and averaging all the mapping scores. The deep learning model can accurately evaluate the correlation between each frame of audio signal and the wake word. The summation and averaging operation ensures that the global similarity score accurately reflects the matching degree between the voice interaction wake-up signal and the wake word, avoiding the collection of invalid voice signals due to false wake-ups and improving the recognition accuracy of voice interaction.

[0025] As a preferred embodiment, the step of generating interactive text based on the real-time speech signal and inputting the interactive text into a preset intent recognition model to obtain the text category corresponding to the interactive text includes:

[0026] The real-time voice signal is input into a preset language recognition model to obtain interactive text;

[0027] Based on the preset prompt word engineering algorithm, an information entropy measurement function is constructed.

[0028] The interactive text is input into a preset intent recognition model. With the goal of minimizing the information entropy measurement function, the intent recognition model identifies the text category of the interactive text to obtain the text category corresponding to the interactive text.

[0029] This preferred solution obtains interactive text by inputting real-time voice signals into a preset language recognition model, and then constructs an information entropy measurement function based on a preset prompt word engineering algorithm. With the goal of minimizing the information entropy measurement function, the intent recognition model identifies the text category. The information entropy measurement function quantifies the uncertainty of the text. By minimizing the information entropy, the text category of the interactive text can be determined more accurately, thus providing a basis for subsequent selection of a network response model or a local command model, and improving the accuracy of voice interaction recognition.

[0030] As a preferred embodiment, the method for obtaining the preset local instruction large model includes:

[0031] Obtain several devices capable of performing operations and an initial large language model that has been trained;

[0032] Generate device operation instructions corresponding to each of the device's executable operations, and generate broadcast text corresponding to each of the device operation instructions;

[0033] Based on the device operation instructions and broadcast text, construct an instruction matching dataset;

[0034] The instruction matching dataset is input into the initial large language model, and the initial large language model is fine-tuned using a preset LoRA fine-tuning method to obtain a local instruction large model.

[0035] This preferred solution generates device operation commands and broadcast text through device-executable operations, and then constructs a command matching dataset. Combined with a preset LoRA fine-tuning method, the initial large language model is fine-tuned to obtain a local command large model. This local command large model can more accurately understand interactive text and generate corresponding operation commands and broadcast text, thereby improving the voice interaction device's ability to process local commands and improving the recognition accuracy of voice interaction.

[0036] Accordingly, embodiments of the present invention provide a voice interaction system based on a multi-stage large model, including: a real-time voice signal acquisition module, a text category acquisition module, a response text interaction module, and a command text interaction module;

[0037] The real-time voice signal acquisition module is used to acquire the user's real-time voice signal in response to the user's voice interaction wake-up signal.

[0038] The text category acquisition module is used to generate interactive text based on the real-time voice signal, and input the interactive text into a preset intent recognition big model to obtain the text category corresponding to the interactive text.

[0039] The response text interaction module is used to input the interaction text into a preset online response model to generate response text if the text category is response text; and to respond to the user based on the response text to complete the voice interaction.

[0040] The instruction text interaction module is used to input the interaction text into a preset local instruction model to obtain an operation instruction if the text category is instruction text; generate voice broadcast content based on the operation instruction; and respond to the user based on the voice broadcast content to complete the voice interaction.

[0041] As a preferred embodiment, the real-time voice signal acquisition module includes: a real-time voice signal acquisition unit;

[0042] The real-time voice signal acquisition unit is used to respond to the user's voice interaction wake-up signal;

[0043] The voice interaction wake-up signal is subjected to feature extraction to obtain the feature matrix of the voice interaction wake-up signal;

[0044] Based on a preset similarity algorithm and combined with the feature matrix, the global similarity score of the voice interaction wake-up signal is calculated.

[0045] If the global similarity score exceeds a preset wake-up word threshold, the user's real-time voice signal is collected. As a preferred embodiment, the real-time voice signal acquisition unit includes: a feature matrix construction subunit;

[0046] The feature matrix construction subunit is used to divide the voice interaction wake-up signal into several frames of audio signals, and calculate the Mel frequency cepstral coefficients of each frame of the audio signal based on a preset Mel filter.

[0047] The feature matrix of the voice interaction wake-up signal is constructed based on the Mel frequency cepstral coefficients of the audio signal in each frame.

[0048] As a preferred embodiment, the real-time voice signal acquisition unit further includes: a similarity calculation subunit;

[0049] The similarity calculation subunit is used to input the feature matrix into a preset deep learning model to obtain the mapping score of the audio signal in each frame;

[0050] The global similarity score of the voice interaction wake-up signal is obtained by summing and averaging the mapping scores of all audio signals.

[0051] It is understood that this invention constructs a multi-stage large-scale model framework through an intent recognition large-scale model, a network response large-scale model, and a local command large-scale model. The intent recognition large-scale model identifies the text category of the interactive text, and then, based on the different text categories, either the network response large-scale model or the local command large-scale model is selected to process the interactive text differently, thus realizing multi-stage large-scale model voice interaction. By using the intent recognition large-scale model, the network response large-scale model, and the local command large-scale model, the inaccuracy in voice interaction recognition caused by using only a single language model in existing voice interaction processes can be avoided. By directly converting real-time voice signals into interactive text and directly processing the interactive text, the inaccuracy of keywords caused by the randomness, complexity, and ambiguity of user voice information is avoided, thereby improving the recognition accuracy of voice interaction. Attached Figure Description

[0052] Figure 1 A flowchart illustrating the steps of a voice interaction method based on a multi-stage large model, provided in an embodiment of the present invention;

[0053] Figure 2 This is a schematic diagram of the structure of a voice interaction system based on a multi-stage large model, provided as an embodiment of the present invention. Detailed Implementation

[0054] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] Example 1

[0056] Please refer to Figure 1 , Figure 1 The flowchart of a voice interaction method based on a multi-stage large model provided in this embodiment of the invention includes steps S101 to S104.

[0057] Step S101: In response to the user's voice interaction wake-up signal, collect the user's real-time voice signal.

[0058] In this embodiment, the step of collecting the user's real-time voice signal in response to the user's voice interaction wake-up signal includes:

[0059] Responding to the user's voice interaction wake-up signal;

[0060] The voice interaction wake-up signal is subjected to feature extraction to obtain the feature matrix of the voice interaction wake-up signal;

[0061] Based on a preset similarity algorithm and combined with the feature matrix, the global similarity score of the voice interaction wake-up signal is calculated.

[0062] If the global similarity score exceeds the preset wake word threshold, the user's real-time voice signal is collected.

[0063] This embodiment extracts the features of the voice interaction wake-up signal and then calculates the global similarity score based on the similarity algorithm. Real-time voice signals are only collected when the wake-up word threshold is exceeded, avoiding the collection of invalid voice signals due to false wake-ups. This improves the recognition accuracy and relevance of voice interaction, ensures the effectiveness and fluency of voice interaction, provides users with a more stable and reliable voice interaction environment, and enhances the voice interaction experience.

[0064] In this embodiment, the step of extracting features from the voice interaction wake-up signal to obtain the feature matrix of the voice interaction wake-up signal includes:

[0065] The voice interaction wake-up signal is divided into several frames of audio signals, and the Mel frequency cepstral coefficients of each frame of audio signal are calculated based on a preset Mel filter.

[0066] The feature matrix of the voice interaction wake-up signal is constructed based on the Mel frequency cepstral coefficients of the audio signal in each frame.

[0067] This embodiment refines the voice interaction wake-up signal by dividing it into several frames of audio signals and calculating the Mel frequency cepstral coefficients of each frame of audio signal using a Mel filter. This improves the accuracy of the Mel frequency cepstral coefficients, thereby extracting the key features of the voice interaction wake-up signal more accurately. Consequently, it improves the accuracy of the subsequent global similarity score, avoids invalid voice interaction wake-ups, and enhances the recognition accuracy of voice interaction.

[0068] In this embodiment, calculating the global similarity score of the voice interaction wake-up signal based on a preset similarity algorithm and the feature matrix includes:

[0069] The feature matrix is ​​input into a preset deep learning model to obtain the mapping score of the audio signal for each frame;

[0070] The global similarity score of the voice interaction wake-up signal is obtained by summing and averaging the mapping scores of all audio signals.

[0071] This embodiment inputs the feature matrix into a preset deep learning model to obtain the mapping score of each frame of audio signal. Then, the global similarity score is obtained by summing and averaging all the mapping scores. The deep learning model can accurately evaluate the correlation between each frame of audio signal and the wake word. The summation and averaging operation makes the global similarity score accurately reflect the matching degree between the voice interaction wake-up signal and the wake word, avoiding the collection of invalid voice signals due to false wake-ups and improving the recognition accuracy of voice interaction.

[0072] In an optional embodiment, the voice interaction wake-up signal is divided into N frames of audio signals, and each frame of audio signal is input into a preset Mel filter to calculate the Mel frequency cepstral coefficients (MFCCs) of each frame of audio signal; wherein, the formula for calculating the Mel frequency cepstral coefficients (MFCCs) is:

[0073]

[0074] Among them, E k Let K represent the energy of the Mel filter bank, where K is the number of Mel filter bank energies; i represents the i-th frame.

[0075] Then, based on the Mel-frequency cepstral coefficients of each frame of audio signal, a feature matrix X = [X1, X2, ..., X] is constructed for the voice interaction wake-up signal. N ]; where X n This represents the eigenvector of the nth frame audio signal, i.e., the Mel frequency cepstral coefficients;

[0076] The feature matrix is ​​then input into a preset deep learning model, and the model performs mapping to obtain the mapping score for each frame of the audio signal. The mapping formula is as follows:

[0077] s n =f model (X n );

[0078] Among them, s n The mapping score of the audio signal in the nth frame;

[0079] Then, the mapping scores of all audio signals are summed and averaged to obtain the global similarity score S of the voice interaction wake-up signal. The formula for calculating the global similarity score S is as follows:

[0080]

[0081] Then, the wake word threshold T is set to 1400. The global similarity score and the wake word threshold are compared. If the global similarity score exceeds the wake word threshold, the voice interaction device is woken up and the voice interaction device collects the user's real-time voice signal.

[0082]

[0083] It's important to note that the Mel Filter Bank is a key technology in speech recognition and audio analysis. The Mel Filter Bank maps a linear frequency scale to a Mel frequency scale, mimicking how the human ear perceives sound. Specifically, the human ear's frequency discrimination ability is strong in the low-frequency range, gradually decreasing as the frequency increases. Its design aims to convert sound signals into Mel frequency cepstral coefficients, which better match human auditory perception. Deep Neural Networks (DNNs) automatically learn complex feature representations of data through multiple layers of nonlinear transformations. Their structure consists of an input layer, hidden layers, and an output layer. Each layer contains multiple neurons connected by weights, and the weights are optimized using a backpropagation algorithm to minimize prediction error.

[0084] Step S102: Generate interactive text based on the real-time voice signal, and input the interactive text into a preset intent recognition model to obtain the text category corresponding to the interactive text.

[0085] In this embodiment, the step of generating interactive text based on the real-time speech signal and inputting the interactive text into a preset intent recognition model to obtain the text category corresponding to the interactive text includes:

[0086] The real-time voice signal is input into a preset language recognition model to obtain interactive text;

[0087] Based on the preset prompt word engineering algorithm, an information entropy measurement function is constructed.

[0088] The interactive text is input into a preset intent recognition model. With the goal of minimizing the information entropy measurement function, the intent recognition model identifies the text category of the interactive text to obtain the text category corresponding to the interactive text.

[0089] In one optional embodiment, the language recognition model is set as a Transformer model. Real-time speech signals are input into the Transformer model to obtain interactive text. Then, based on the prompt word engineering algorithm, an information entropy measurement function is constructed, as shown below:

[0090] H(Y|X)=-∑ y∈Y P(y|x)logP(y|x);

[0091] Where Y refers to the output distribution of the intent recognition model, and X refers to the input prompt words of the intent recognition model;

[0092] The interactive text is then used as input prompts and input into a preset intent recognition model. With the goal of minimizing the information entropy measurement function, the intent recognition model identifies the text category of the interactive text, resulting in the text category corresponding to the interactive text. The text categories include: response text, instruction text, and invalid text. Response text refers to text that requires accessing publicly available, real-time information to answer user questions; instruction text refers to text that requires matching private, personalized device instructions locally to call an interface to execute corresponding functions; invalid text refers to noise or meaningless text contained during the voice acquisition process.

[0093] It's important to note that the Intent Recognition Large Language Model (IR-LLM) is a deep learning model trained on massive amounts of data and large-scale parameters, specifically designed to understand and recognize the intent or purpose in user input text or speech. It combines the generalization capabilities of a large language model (LLM) with a dedicated model for intent recognition tasks. Through pre-training, it learns general language patterns and then adapts them to domain-specific intent classification tasks via fine-tuning or prompt engineering. This allows it to automatically extract key information from user input and determine their potential needs. The prompt engineering algorithm is a core technology in Natural Language Processing (NLP) that interacts with large language models (LLM). It's a methodology that guides the large language model to generate expected outputs by designing, optimizing, and structuring input prompts. Combining linguistics, contextual analysis, and model characteristics, it transforms user intent into instructions that the model can understand, thereby improving the accuracy, efficiency, and controllability of task execution. The Transformer model is a deep learning model architecture based on a "self-attention mechanism" for efficiently processing sequential data (such as text and speech).

[0094] This embodiment obtains interactive text by inputting real-time voice signals into a preset language recognition model, and then constructs an information entropy measurement function based on a preset prompt word engineering algorithm. With the goal of minimizing the information entropy measurement function, the intent recognition model identifies the text category. The information entropy measurement function quantifies the uncertainty of the text. By minimizing the information entropy, the text category of the interactive text can be determined more accurately, thus providing a basis for subsequent selection of a network response model or a local command model, and improving the accuracy of voice interaction recognition.

[0095] In an optional embodiment, if the text category is invalid text, no voice interaction operation is performed, and it is directly regarded as a completed voice interaction.

[0096] Step S103: If the text category is response text, then input the interaction text into the preset network response model to generate response text; and respond to the user based on the response text to complete the voice interaction.

[0097] In one optional embodiment, the Network Response Large Language Model (NR-LLM) refers to a large language model (LLM) with network retrieval capabilities; through prompt word engineering algorithms, the Network Response Large Language Model can generate response text according to a specific format;

[0098] Furthermore, if the text category is response text, the interactive text is input into the network response model, which then generates the response text through network retrieval. The response text is then displayed on the voice interaction device, enabling the voice interaction device to respond to the user based on the response text and complete the voice interaction.

[0099] Step S104: If the text category is instruction text, then input the interactive text into a preset local instruction model to obtain operation instructions; generate voice broadcast content based on the operation instructions; respond to the user based on the voice broadcast content to complete the voice interaction.

[0100] In an optional embodiment, if the text category is instruction text, the interactive text is input into a preset Local Command Large Language Model (LC-LLM) to obtain the operation instructions output by the Local Command Large Language Model; the voice interaction device completes the corresponding function operation based on the operation instructions, and generates corresponding voice broadcast content based on the content of the operation instructions; then the voice broadcast content is broadcast to the user through the audio playback device of the voice interaction device, thereby completing the voice interaction.

[0101] It should be noted that the Local Command Large Language Model (LC-LLM) is a large language model designed specifically for deployment on local devices. Its core objective is to achieve efficient, secure, and low-latency command understanding and execution capabilities in a local environment.

[0102] In this embodiment, the method for obtaining the preset local instruction large model includes:

[0103] Obtain several devices capable of performing operations and an initial large language model that has been trained;

[0104] Generate device operation instructions corresponding to each of the device's executable operations, and generate broadcast text corresponding to each of the device operation instructions;

[0105] Based on the device operation instructions and broadcast text, construct an instruction matching dataset;

[0106] The instruction matching dataset is input into the initial large language model, and the initial large language model is fine-tuned using a preset LoRA fine-tuning method to obtain a local instruction large model.

[0107] In one optional embodiment, several device-executable operations and an initial large language model that has been trained are obtained; device-executable operations refer to the functional operations that the voice interaction device can perform, including but not limited to the switching of interactive collaboration functions such as whiteboard and check-in, the adjustment of system settings such as sound and brightness, and the switching of resource management functions such as file retrieval and application market.

[0108] Next, based on the device's executable operations, a corresponding device operation command is generated for each executable operation, and a unique ID is preset for each device operation command. Then, the corresponding broadcast text for each operation command is set, thus constructing a command matching dataset. For example, the ID for the device executable operation "Open whiteboard" is "1001," and the broadcast text is "Okay, whiteboard has been opened for you." Therefore, its corresponding command matching data consists of the input prompt {"messages":[{"role":"user","content":"Open whiteboard."} and the output text {"role":"assis"}. tant","content":{"id":1001,"nb":"Okay, the whiteboard has been opened for you."}}]}; where the value of "role" in the input prompt is "user", representing the user; "content" represents the content of the input prompt; the value of "role" in the output text is "assistant", representing the model; "content" represents the content of the output text; and "nb" represents the content of the broadcast text corresponding to the device operation command; therefore, the device operation command is set as x, and the broadcast text is set as y, and the command matching dataset (x,y) is constructed;

[0109] Then, the instruction matching dataset (x, y) is used as the fine-tuning training set for the initial large language model. The LoRa fine-tuning method is used to fine-tune the initial large language model to obtain the local instruction large model. Furthermore, in the original weight matrix W0∈R of the initial large language model... d×d Introduce two low-rank matrices A∈R d×d and B∈R d×d The rank r of the low-rank matrix is ​​much smaller than the dimension d of the weight matrix, i.e., r << d; therefore, the weight matrix is ​​updated as follows:

[0110] W = W0 + ΔW = W0 + α·(B·A);

[0111] Where W is the updated weight matrix, W0 is the original weight matrix (frozen, not updated), and α is the scaling factor, typically taken as... Or r, which is used to control the influence of weights in LoRA fine-tuning;

[0112] The forward propagation calculation process of the LoRa fine-tuning method is: h=(W0+α·BA)x;

[0113] Where x is the input feature, h is the output feature, low-rank matrix A is used for initialization of random Gaussian distribution, and low-rank matrix B is used for initialization of 0 matrix, ensuring that this bypass matrix is ​​still a 0 matrix at the beginning of training;

[0114] Then, during backpropagation, the low-rank matrices A and B are updated, while the original weight matrix W0 remains unchanged. The optimization objective is:

[0115] in, Let represent the loss function, here we use cross-entropy loss, y is the true label, and the low-rank matrices A and B are updated iteratively by minimizing the loss function;

[0116] After the iteration is complete, the low-rank matrices A and B are merged into the original weights, specifically: h = W merged x = (W0 + α·BA)x;

[0117] It should be noted that LoRA (Low-Rank Adaptation) is an efficient fine-tuning technique for large pre-trained language models (LLMs). Its core idea is to introduce low-rank matrix factorization to adjust the model's capabilities for specific tasks by training only a small number of newly added parameters while keeping the original model parameters unchanged. The implementation process of LoRA generally involves forward propagation, loss calculation, backpropagation, and parameter update.

[0118] This embodiment generates device operation commands and broadcast text through device-executable operations, and then constructs a command matching dataset. Combined with the preset LoRA fine-tuning method, the initial large language model is fine-tuned to obtain a local command large model. This local command large model can more accurately understand interactive text and generate corresponding operation commands and broadcast text, thereby improving the voice interaction device's ability to process local commands and improving the recognition accuracy of voice interaction.

[0119] This embodiment constructs a multi-stage large-scale model framework by establishing a large-scale intent recognition model, a large-scale network response model, and a large-scale local command model. The intent recognition model identifies the text category of the interactive text, and then, based on the different text categories, either the large-scale network response model or the large-scale local command model is selected to process the interactive text differently, thus realizing multi-stage large-scale model-based voice interaction. By using a large-scale intent recognition model, a large-scale network response model, and a large-scale local command model, the inaccuracy in voice interaction recognition caused by using only a single language model in existing voice interaction processes can be avoided. By directly converting real-time voice signals into interactive text and directly processing the interactive text, inaccurate keywords caused by the randomness, complexity, and ambiguity of user voice information are avoided, thereby improving the recognition accuracy of voice interaction.

[0120] Example 2

[0121] Please refer to Figure 2 , Figure 2 A schematic diagram of the structure of a voice interaction system based on a multi-stage large model provided in an embodiment of the present invention includes: a real-time voice signal acquisition module 201, a text category acquisition module 202, a response text interaction module 203, and a command text interaction module 204;

[0122] The real-time voice signal acquisition module 201 is used to acquire the user's real-time voice signal in response to the user's voice interaction wake-up signal.

[0123] The text category acquisition module 202 is used to generate interactive text based on the real-time voice signal, and input the interactive text into a preset intent recognition big model to obtain the text category corresponding to the interactive text.

[0124] The response text interaction module 203 is used to input the interaction text into a preset network response model to generate response text if the text category is response text; and respond to the user based on the response text to complete the voice interaction.

[0125] The instruction text interaction module 204 is used to input the interaction text into a preset local instruction model to obtain an operation instruction if the text category is instruction text; generate voice broadcast content based on the operation instruction; and respond to the user based on the voice broadcast content to complete the voice interaction.

[0126] In this embodiment, the real-time voice signal acquisition module 201 includes: a real-time voice signal acquisition unit;

[0127] The real-time voice signal acquisition unit is used to respond to the user's voice interaction wake-up signal;

[0128] The voice interaction wake-up signal is subjected to feature extraction to obtain the feature matrix of the voice interaction wake-up signal;

[0129] Based on a preset similarity algorithm and combined with the feature matrix, the global similarity score of the voice interaction wake-up signal is calculated.

[0130] If the global similarity score exceeds a preset wake-up word threshold, the user's real-time voice signal is collected. In this embodiment, the real-time voice signal acquisition unit includes: a feature matrix construction subunit;

[0131] The feature matrix construction subunit is used to divide the voice interaction wake-up signal into several frames of audio signals, and calculate the Mel frequency cepstral coefficients of each frame of the audio signal based on a preset Mel filter.

[0132] The feature matrix of the voice interaction wake-up signal is constructed based on the Mel frequency cepstral coefficients of the audio signal in each frame.

[0133] In this embodiment, the real-time voice signal acquisition unit further includes: a similarity calculation subunit;

[0134] The similarity calculation subunit is used to input the feature matrix into a preset deep learning model to obtain the mapping score of the audio signal in each frame;

[0135] The global similarity score of the voice interaction wake-up signal is obtained by summing and averaging the mapping scores of all audio signals.

[0136] In this embodiment, the text category acquisition module 202 includes: a text category acquisition unit;

[0137] The text category acquisition unit is used to input the real-time voice signal into a preset language recognition model to obtain interactive text;

[0138] Based on the preset prompt word engineering algorithm, an information entropy measurement function is constructed.

[0139] The interactive text is input into a preset intent recognition model. With the goal of minimizing the information entropy measurement function, the intent recognition model identifies the text category of the interactive text to obtain the text category corresponding to the interactive text.

[0140] In this embodiment, the instruction text interaction module 204 includes: a preset local instruction large model acquisition unit;

[0141] The preset local instruction large model acquisition unit is used to acquire several device-executable operations and the initial large language model that has been trained.

[0142] Generate device operation instructions corresponding to each of the device's executable operations, and generate broadcast text corresponding to each of the device operation instructions;

[0143] Based on the device operation instructions and broadcast text, construct an instruction matching dataset;

[0144] The instruction matching dataset is input into the initial large language model, and the initial large language model is fine-tuned using a preset LoRA fine-tuning method to obtain a local instruction large model.

[0145] This embodiment constructs a multi-stage large-scale model framework by establishing a large-scale intent recognition model, a large-scale network response model, and a large-scale local command model. The intent recognition model identifies the text category of the interactive text, and then, based on the different text categories, either the large-scale network response model or the large-scale local command model is selected to process the interactive text differently, thus realizing multi-stage large-scale model-based voice interaction. By using a large-scale intent recognition model, a large-scale network response model, and a large-scale local command model, the inaccuracy in voice interaction recognition caused by using only a single language model in existing voice interaction processes can be avoided. By directly converting real-time voice signals into interactive text and directly processing the interactive text, inaccurate keywords caused by the randomness, complexity, and ambiguity of user voice information are avoided, thereby improving the recognition accuracy of voice interaction.

[0146] This invention constructs a multi-stage large-scale model framework by establishing an intent recognition large-scale model, a network response large-scale model, and a local command large-scale model. The intent recognition large-scale model identifies the text category of the interactive text, and then, based on the different text categories, either the network response large-scale model or the local command large-scale model is selected to process the interactive text differently, thus realizing multi-stage large-scale model-based voice interaction. By using the intent recognition large-scale model, the network response large-scale model, and the local command large-scale model, the inaccuracy in voice interaction recognition caused by using only a single language model in existing voice interaction processes can be avoided. By directly converting real-time voice signals into interactive text and directly processing the interactive text, the inaccuracy of keywords caused by the randomness, complexity, and ambiguity of user voice information is avoided, thereby improving the recognition accuracy of voice interaction.

[0147] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.

Claims

1. A voice interaction method based on a multi-stage large model, characterized in that, include: In response to the user's voice interaction wake-up signal, collect the user's real-time voice signal; Based on the real-time voice signal, interactive text is generated, and the interactive text is input into a preset intent recognition model to obtain the text category corresponding to the interactive text. If the text category is response text, then the interactive text is input into a preset online response model to generate response text; The system responds to the user based on the provided text to complete the voice interaction. If the text category is instruction text, then the interactive text is input into a preset local instruction model to obtain operation instructions; The voice broadcast content is generated based on the operation instructions; The system responds to the user based on the voice broadcast content to complete voice interaction. The method for obtaining the preset local instruction model includes: acquiring several device-executable operations and a trained initial large language model; generating device operation instructions corresponding to each device-executable operation, and generating broadcast text corresponding to each device operation instruction, and obtaining a unique ID for each device operation instruction; using the device operation instructions corresponding to the same device-executable operation as input prompts, and using the broadcast text corresponding to the device operation instructions as output text based on the unique ID, forming an instruction matching dataset; inputting the instruction matching dataset into the initial large language model, and using a preset LoRA fine-tuning method to fine-tune the initial large language model to obtain the local instruction model.

2. The voice interaction method based on a multi-stage large model as described in claim 1, characterized in that, The process of collecting the user's real-time voice signal in response to the user's voice interaction wake-up signal includes: Responding to the user's voice interaction wake-up signal; The voice interaction wake-up signal is subjected to feature extraction to obtain the feature matrix of the voice interaction wake-up signal; Based on a preset similarity algorithm and combined with the feature matrix, the global similarity score of the voice interaction wake-up signal is calculated. If the global similarity score exceeds the preset wake word threshold, the user's real-time voice signal is collected.

3. The voice interaction method based on a multi-stage large model as described in claim 2, characterized in that, The step of extracting features from the voice interaction wake-up signal to obtain the feature matrix of the voice interaction wake-up signal includes: The voice interaction wake-up signal is divided into several frames of audio signals, and the Mel frequency cepstral coefficients of each frame of audio signal are calculated based on a preset Mel filter. The feature matrix of the voice interaction wake-up signal is constructed based on the Mel frequency cepstral coefficients of the audio signal in each frame.

4. The voice interaction method based on a multi-stage large model as described in claim 3, characterized in that, The calculation of the global similarity score of the voice interaction wake-up signal based on the preset similarity algorithm and the feature matrix includes: The feature matrix is ​​input into a preset deep learning model to obtain the mapping score of the audio signal for each frame; The global similarity score of the voice interaction wake-up signal is obtained by summing and averaging the mapping scores of all audio signals.

5. The voice interaction method based on a multi-stage large model as described in claim 1, characterized in that, The process involves generating interactive text based on the real-time speech signal and inputting the interactive text into a preset intent recognition model to obtain the text category corresponding to the interactive text, including: The real-time voice signal is input into a preset language recognition model to obtain interactive text; Based on the preset prompt word engineering algorithm, an information entropy measurement function is constructed. The interactive text is input into a preset intent recognition model. With the goal of minimizing the information entropy measurement function, the intent recognition model identifies the text category of the interactive text to obtain the text category corresponding to the interactive text.

6. A voice interaction system based on a multi-stage large model, characterized in that, include: Real-time voice signal acquisition module, text category acquisition module, response text interaction module, and command text interaction module; The real-time voice signal acquisition module is used to acquire the user's real-time voice signal in response to the user's voice interaction wake-up signal. The text category acquisition module is used to generate interactive text based on the real-time voice signal, and input the interactive text into a preset intent recognition big model to obtain the text category corresponding to the interactive text. The response text interaction module is used to input the interaction text into a preset online response model to generate response text if the text category is response text; and to respond to the user based on the response text to complete the voice interaction. The instruction text interaction module is used to input the interaction text into a preset local instruction model to obtain operation instructions if the text category is instruction text; generate voice broadcast content based on the operation instructions; and respond to the user based on the voice broadcast content to complete the voice interaction. The instruction text interaction module includes: a preset local instruction model acquisition unit; the preset local instruction model acquisition unit is used to acquire several device-executable operations and a trained initial large language model; generate device operation instructions corresponding to each device-executable operation, and generate broadcast text corresponding to each device operation instruction, and obtain the unique ID of the device operation instruction; use the device operation instructions corresponding to the same device-executable operation as input prompt words, and use the broadcast text corresponding to the device operation instruction as output text based on the unique ID to form an instruction matching dataset; construct an instruction matching dataset based on the device operation instructions and broadcast text; input the instruction matching dataset into the initial large language model, and combine it with a preset LoRA fine-tuning method to fine-tune the initial large language model to obtain a local instruction model.

7. A voice interaction system based on a multi-stage large model as described in claim 6, characterized in that, The real-time voice signal acquisition module includes: a real-time voice signal acquisition unit; The real-time voice signal acquisition unit is used to respond to the user's voice interaction wake-up signal; The voice interaction wake-up signal is subjected to feature extraction to obtain the feature matrix of the voice interaction wake-up signal; Based on a preset similarity algorithm and combined with the feature matrix, the global similarity score of the voice interaction wake-up signal is calculated. If the global similarity score exceeds the preset wake word threshold, the user's real-time voice signal is collected.

8. A voice interaction system based on a multi-stage large model as described in claim 7, characterized in that, The real-time voice signal acquisition unit includes: a feature matrix construction subunit; The feature matrix construction subunit is used to divide the voice interaction wake-up signal into several frames of audio signals, and calculate the Mel frequency cepstral coefficients of each frame of the audio signal based on a preset Mel filter. The feature matrix of the voice interaction wake-up signal is constructed based on the Mel frequency cepstral coefficients of the audio signal in each frame.

9. A voice interaction system based on a multi-stage large model as described in claim 8, characterized in that, The real-time voice signal acquisition unit further includes: a similarity calculation subunit; The similarity calculation subunit is used to input the feature matrix into a preset deep learning model to obtain the mapping score of the audio signal in each frame; The global similarity score of the voice interaction wake-up signal is obtained by summing and averaging the mapping scores of all audio signals.

Images