Voice interaction method and device, electronic equipment and storage medium

Abstract

The invention provides a voice interaction method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a continuous voice representation through an encoder of a voice recognition model, and obtaining a recognition text through a decoder of the voice recognition model; mapping the continuous speech representation to a semantic space of a large language model by using an adapter module to obtain a target speech representation; semantic content information, side language information and speaker information in the questioning voice are reserved in the target voice representation; and generating a reply text by using the target voice representation and the recognition text through the large language model. By adopting a two-stage training strategy of voice text alignment and fine tuning, complete semantic content information, side language information and speaker information contained in target voice representation can be effectively understood and utilized by a large language model; and the problems of one-sided user intention understanding and lack of emotion expressive force in reply caused by loss of acoustic information in a traditional cascaded voice interaction system are avoided.

Description

Technical Field

[0001] This invention relates to the field of voice interaction technology, and in particular to a voice interaction method, device, electronic device, and storage medium. Background Technology

[0002] Voice interaction has been widely used in smart speakers, in-vehicle voice assistants, and other scenarios. To improve the interactive experience, existing technologies typically employ an improved cascaded approach, which involves extracting paralinguistic information (such as emotion) from speech using an independent model and converting it into discrete labels. These labels, along with the text results from speech recognition, are then provided to a downstream large language model for understanding and response.

[0003] However, the inherent recognition error of cascaded systems accumulates and propagates along the link, and a large amount of rich paralinguistic and speaker information in speech is lost in the speech recognition stage. This causes large language models to be unable to accurately understand user intent based on complete contextual details, resulting in responses that lack emotional expressiveness and personalized features, making it difficult to meet the needs of natural and human-like interaction and affecting user experience. Summary of the Invention

[0004] This invention provides a voice interaction method, device, electronic device, and storage medium to address the deficiencies in the prior art.

[0005] This invention provides a voice interaction method, comprising the following steps: The system acquires the user's question speech, uses the encoder of the speech recognition model to obtain the continuous speech representation of the question speech, and uses the decoder of the speech recognition model to obtain the recognized text of the question speech. Using an adapter module, the continuous speech representation is mapped to the semantic space of a large language model to obtain the target speech representation; the target speech representation retains the semantic content information, paralinguistic information, and speaker information of the question speech; Using the large language model, the target speech representation and the recognized text are applied to generate the response text; The training phase of the adapter module and the large language model includes a speech-text alignment phase and a fine-tuning phase. In the speech-text alignment phase, the parameters of the large language model and the encoder are fixed, and the adapter module is trained. In the fine-tuning phase, the parameters of the encoder and the adapter module obtained in the speech-text alignment phase are fixed, and the large language model is fine-tuned so that the large language model has the ability to jointly process the target speech representation and the recognized text.

[0006] According to a voice interaction method provided by the present invention, the training process of the voice-text alignment stage includes: The parameters of the large language model and the encoder are fixed, and the parameters of the adapter module are updated using speech recognition data; After updating the parameters of the adapter module, the parameters of the encoder are fixed, and the parameters of the large language model and the adapter module are updated using the speech recognition data, text dialogue data, and voice dialogue data.

[0007] According to a voice interaction method provided by the present invention, updating the parameters of the adapter module using voice recognition data includes: The encoder is used to encode the speech portion in the speech recognition data to obtain a continuous sample speech representation; The continuous sample speech representation is mapped through the adapter module to obtain the mapped speech representation; Using the large language model, the mapped speech representation is applied to predict the first response text. The prediction loss is determined based on the first response text and the text portion of the speech recognition data; The parameters of the adapter module are updated based on the predicted loss.

[0008] According to a voice interaction method provided by the present invention, the step of using the large language model and applying the mapped speech representation to predict and obtain a first response text includes: Using the large language model, based on the mapped speech representation and all the predicted word segments, the current word segment is predicted until all word segment predictions are completed. Then, all the predicted word segments are used as the first response text.

[0009] According to a voice interaction method provided by the present invention, the step of updating the parameters of the large language model and the adapter module using the speech recognition data, text dialogue data, and voice dialogue data includes: Acquire mixed training data, which includes the speech recognition data, the text dialogue data, and the synthesized speech dialogue data, wherein the proportion of the text dialogue data in the mixed training data is fixed at a preset value; Using the large language model and the adapter module, a second response text is generated from the mixed training data; Based on the second response text and the labeled text in the mixed training data, the generative prediction loss is calculated; The parameters of the large language model are updated based on the generative prediction loss, and the parameters of the adapter module are also updated.

[0010] According to a voice interaction method provided by the present invention, updating the parameters of the large language model based on the generative prediction loss includes: A low-rank matrix is ​​appended next to the original weight matrix of the large language model; The parameters of the low-rank matrix are updated based on the generative prediction loss.

[0011] According to a voice interaction method provided by the present invention, the training process in the fine-tuning stage includes: The fine-tuning data includes the speech recognition data, the text dialogue data, the real speech dialogue data, the dialogue error correction data, and the speech description data. The dialogue error correction data is data formed by introducing simulated speech recognition errors into the question part of the original dialogue data and pairing the question part containing the error with the response part in the original dialogue data. The speech description data is data containing the speech input and its corresponding text description annotation. Using the large language model and the adapter module, a third response text for the fine-tuned data is generated; Based on the third response text and the corresponding labeled text in the fine-tuning data, calculate the fine-tuning loss; The parameters of the large language model and the adapter module are updated based on the fine-tuning loss.

[0012] According to a voice interaction method provided by the present invention, the proportion of the voice recognition data in the fine-tuning data is less than the proportion in the mixed training data, and the proportions of the text dialogue data, the real voice dialogue data, the dialogue error correction data, and the voice description data in the fine-tuning data are equal.

[0013] According to a voice interaction method provided by the present invention, the step of generating a response text by applying the target speech representation and the recognized text through the large language model includes: Get the task prompt text for the specified current task type; The task prompt text, the target speech representation, and the recognized text are concatenated to form the input sequence of the large language model; The input sequence is fed into the large language model to generate the response text.

[0014] According to a voice interaction method provided by the present invention, the adapter module includes a linear transformation layer, a nonlinear activation function layer, and a normalization layer. The linear transformation layer is used to perform dimensional mapping on the continuous speech representation, the nonlinear activation function layer is used to perform nonlinear transformation on the mapped continuous speech representation, and the normalization layer is used to normalize the nonlinearly transformed continuous speech representation.

[0015] The present invention also provides a voice interaction device, comprising the following modules: The acquisition unit is used to acquire the user's question speech, obtain the continuous speech representation of the question speech using the encoder of the speech recognition model, and obtain the recognized text of the question speech using the decoder of the speech recognition model. The mapping unit is used to map the continuous speech representation to the semantic space of the large language model using the adapter module to obtain the target speech representation; the target speech representation retains the semantic content information, paralinguistic information and speaker information in the question speech; The generation unit is used to generate response text by applying the target speech representation and the recognized text through the large language model; The training phase of the adapter module and the large language model includes a speech-text alignment phase and a fine-tuning phase. In the speech-text alignment phase, the parameters of the large language model and the encoder are fixed, and the adapter module is trained. In the fine-tuning phase, the parameters of the encoder and the adapter module obtained in the speech-text alignment phase are fixed, and the large language model is fine-tuned so that the large language model has the ability to jointly process the target speech representation and the recognized text.

[0016] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the voice interaction methods described above.

[0017] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the voice interaction method as described above.

[0018] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements any of the voice interaction methods described above.

[0019] The voice interaction method, device, electronic device, and storage medium provided by this invention employ a two-stage training strategy of speech-text alignment and fine-tuning to train the adapter module and the large language model. This enables the large language model to effectively understand and utilize the complete semantic content information, paralinguistic information, and speaker information contained in the target speech representation. Because the above training mechanism ensures that the large language model can not only process text information but also decode non-text information in the speech signal, it effectively avoids the problems of incomplete understanding of user intent and lack of emotional expressiveness in responses caused by acoustic information loss in traditional cascaded voice interaction systems. It also reduces the risk of interaction failure caused by speech recognition errors propagating along the link. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating the voice interaction method provided by the present invention.

[0022] Figure 2 This is a flowchart illustrating the voice interaction model training method provided by the present invention.

[0023] Figure 3 This is a schematic diagram of the structure of the voice interaction device provided by the present invention.

[0024] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0026] Voice interaction, as an interactive mode in which people issue commands to machines via voice and the machines respond with text or voice, plays an important role in scenarios such as smart speakers, in-vehicle voice assistants, and customer service robots.

[0027] Currently widely used voice interaction systems typically consist of multiple cascaded modules, including an acoustic front-end, speech recognition (ASR), natural language understanding (NLU), and text-to-speech (TTS). The acoustic front-end performs preprocessing on the speech, such as noise reduction and dereverberation; ASR converts the preprocessed speech into text; NLU understands and responds to the text output by ASR; and finally, TTS converts the text response into a speech waveform.

[0028] However, in addition to semantic content information, speech signals also contain paralinguistic information (such as emotion and prosody) and speaker information (such as age and gender). ASR only focuses on semantic content extraction, which means that subsequent NLU cannot accurately understand the user's intent based on contextual details, and the speech generated by TTS also lacks emotional expressiveness. Furthermore, ASR inevitably has recognition errors, which can propagate along the link and eventually lead to interaction failure.

[0029] To match user intent and improve the human-likeness of responses, related technologies attempt to extract paralinguistic information from speech using independent models, convert it into discrete labels, and then explicitly provide it along with the recognized text to a downstream large language model for understanding and response. This is combined with a style-controllable TTS module to generate emotionally rich speech. However, the inherent recognition errors of cascaded systems accumulate and propagate along the link, and a significant amount of paralinguistic and speaker information is lost during the ASR stage. Furthermore, although discrete labels for paralinguistics are introduced, the labels lack fine-grainedness and temporal alignment with the speech signal, hindering deep information fusion. This makes it difficult for the large language model to accurately understand the user's true intent and emotional state, thus affecting the naturalness and human-likeness of the interaction.

[0030] To address this, the present invention provides a voice interaction method that aims to map the continuous speech representation obtained by the speech recognition model to the semantic space of a large language model through an adapter module. This results in a target speech representation that retains complete information (including semantic content, paralinguistics, and speaker information) in the question speech. As a result, the large language model can accurately generate the corresponding response text based on the target speech representation containing complete information, thereby avoiding the error propagation problem of cascaded systems and achieving accurate understanding of user intent and personalized interactive responses with emotional expressiveness.

[0031] in, Figure 1 This is a flowchart illustrating the voice interaction method provided by the present invention, as shown below. Figure 1 As shown, the method includes steps 110, 120 and 130.

[0032] Step 110: Obtain the user's question speech, and use the encoder of the speech recognition model to obtain the continuous speech representation of the question speech, and use the decoder of the speech recognition model to obtain the recognized text of the question speech.

[0033] Here, the user's question can be understood as the raw voice signal input by the user when interacting with the device. For example, a user can say, "Book me a flight to City A for tomorrow," through a smartphone or smart speaker. The user's question can be captured by audio acquisition devices such as a microphone.

[0034] After acquiring the user's question in speech, a pre-trained speech recognition (ASR) model is used to process the speech. As an optional embodiment, the speech recognition model can adopt an encoder-decoder architecture. The encoder can be an acoustic model based on a Conformer or similar structure, such as a model with a 16-layer Conformer structure; the decoder can be a Transformer model, such as a 6-layer Transformer model with a Monotonic Attention mechanism.

[0035] The encoder in a speech recognition model is responsible for converting the speech waveform signal of a question into an intermediate feature representation, namely, a continuous speech representation. This continuous speech representation not only contains semantic content information but also retains a wealth of paralinguistic information (such as emotion, rhythm, and speech rate) and speaker information (such as age, gender, and timbre). For example, for the same sentence "Great!", whether a user says it with an enthusiastic and excited tone or with a flat tone, the recognized text is still "Great!", but the continuous speech representation output by the encoder is significantly different in these two cases, thus preserving this difference in emotion. The decoder, on the other hand, is responsible for translating the continuous speech representation into human-readable text, resulting in the recognized text.

[0036] Step 120: Using the adapter module, the continuous speech representation is mapped to the semantic space of the large language model to obtain the target speech representation. The target speech representation retains the semantic content information, paralinguistic information, and speaker information of the question speech.

[0037] Specifically, in step 110, the feature space containing the continuous speech representation output by the encoder has a modal difference from the semantic space understood by the large language model. That is, the two have not been aligned and trained, and cannot be directly understood by the large language model.

[0038] To address the problem that heterogeneous semantic spaces prevent large language models from directly processing continuous speech representations, thus hindering the effective fusion of information from speech and text modalities, this embodiment introduces an adapter module. This adapter module can be understood as a lightweight neural network that acts as a bridge between two heterogeneous semantic spaces. It receives continuous speech representations from the encoder and projects or maps them linearly or non-linearly into a semantic space that the large language model can understand.

[0039] As an alternative embodiment, the adapter module can consist of several simple neural network layers, such as three linear projection layers, one activation function layer (e.g., GELU), and one normalization layer (e.g., LayerNorm). The input dimension of the adapter module matches the dimension of the continuous speech representation, while the output dimension matches the word embedding dimension within the large language model.

[0040] After conversion by the adapter module, the output is the target speech representation. This target speech representation is a speech feature vector that has been aligned with the semantic space of the large language model and can be directly processed by the large language model.

[0041] It is important to emphasize that, because the adapter module has undergone specialized training, the target speech representation retains the semantic content, paralinguistic information, and speaker information from the question speech. In other words, the target speech representation still carries the semantic, paralinguistic, and speaker information from the original speech, but its data format has been translated into a form that the large language model can understand.

[0042] Semantic content information refers to the core meaning expressed by the speech, corresponding to the content of the recognized text. Paralinguistic information refers to the emotions, attitudes, tones, and rhythms in speech that go beyond the literal meaning. For example, whether a user says "That's great" with an excited tone or a perfunctory tone is included in paralinguistic information. Speaker information refers to the acoustic features that distinguish different speakers, such as timbre and fundamental frequency. These features can indirectly reflect the speaker's identity, gender, age group, etc.

[0043] Step 130: Using a large language model, apply the target speech representation and text recognition to generate the response text; The training phase of the adapter module and the large language model includes a speech-text alignment phase and a fine-tuning phase. In the speech-text alignment phase, the parameters of the large language model and the encoder are fixed, and the adapter module is trained. In the fine-tuning phase, the parameters of the encoder and the adapter module obtained in the speech-text alignment phase are fixed, and the large language model is fine-tuned so that the large language model has the ability to jointly process target speech representations and recognize text.

[0044] Here, the large language model can be a pre-trained language model based on architectures such as Transformer, possessing powerful text understanding and generation capabilities, for example, a model with billions or more parameters. In this step, the large language model will simultaneously receive two forms of input: the target speech representation generated in step 120, which retains rich acoustic details, and the recognized text generated in step 110, which represents the core semantics.

[0045] As an optional implementation, the target speech representation sequence and the word embedding sequence corresponding to the recognized text can be concatenated, with specific instruction prompts added before and after the sequence to form a hybrid input sequence incorporating bimodal information. Subsequently, a large language model generates the final response text based on the hybrid input sequence using an autoregressive approach. After obtaining the response text, a speech synthesis model can be used to synthesize the response text to obtain the response speech.

[0046] To enable the adapter module to effectively convert continuous speech representations belonging to different modal spaces into a semantic space that the large language model can understand, and ultimately to allow the large language model to not only receive but also deeply understand and integrate both speech and text—two heterogeneous information sources—the training phase of the adapter module and the large language model in this embodiment includes a speech-text alignment phase and a fine-tuning phase. Specifically, when entering these two training phases, the speech recognition model can be an ASR model trained using windowing, the encoder can be the encoder of this ASR model, and the large language model can be a large language model that has been fine-tuned according to instructions.

[0047] The goal of the speech-to-text alignment stage is to train the adapter module to accurately translate the continuous speech representations output by the encoder into the semantic space of the large language model. To achieve this, the parameters of the large language model and encoder are fixed in this stage, preventing them from participating in gradient updates, and only the parameters of the adapter module are trained. The training data can consist of a large amount of speech recognition data. In this way, the entire training process focuses on optimizing the adapter module, forcing it to learn the mapping relationship between the two modal spaces.

[0048] Following the speech-text alignment stage, the adapter module effectively connects the speech and text modalities, entering the fine-tuning stage. The goal of this stage is to train the large language model to truly learn how to understand and jointly process information from both modalities. In this stage, the large language model obtained after the previous stage is fine-tuned. The training data in this stage is more complex and of higher quality; for example, it may include speech dialogue data with explicit sentiment annotations, dialogue correction data containing ASR errors, and speech description data that requires a deep understanding of speech prosody. Through fine-tuning on this data, the large language model gradually learns how to interpret the paralinguistic information (such as emotion and tone) and speaker information contained in the target speech representation, and associates, compares, and fuses this information with the semantic content of the recognized text.

[0049] For example, the model learns that when the recognized text is "I understand" but the target speech representation has impatient acoustic features, it should generate a reassuring response instead of a simple confirmation. It is through this fine-tuning phase that large language models acquire the ability to decode and utilize the semantic content information, paralinguistic information, and speaker information contained in continuous speech representations from the target speech representation.

[0050] The voice interaction method provided in this embodiment employs a two-stage training strategy of speech-text alignment and fine-tuning to train the adapter module and the large language model. This enables the large language model to effectively understand and utilize the complete semantic content, paralinguistic information, and speaker information contained in the target speech representation. Because the above training mechanism ensures that the large language model can not only process text information but also decode non-textual information (such as emotion and speaker features) in the speech signal, it effectively avoids the problems of incomplete understanding of user intent and lack of emotional expressiveness in responses caused by acoustic information loss in traditional cascaded voice interaction systems. It also reduces the risk of interaction failure caused by speech recognition errors propagating along the link.

[0051] Based on the above embodiments, the training process for the speech-text alignment stage includes: With the parameters of the large language model and encoder fixed, the parameters of the adapter module are updated using speech recognition data; After updating the parameters of the adapter module, fix the encoder parameters and update the parameters of the large language model and the adapter module using speech recognition data, text dialogue data, and voice dialogue data.

[0052] Specifically, the goal of the speech-to-text alignment stage is to teach the adapter module how to map speech representations to the semantic space of a large language model. However, directly training a randomly initialized adapter module together with a large, powerful language model with a huge number of parameters may lead to problems such as training instability and slow convergence.

[0053] Based on this, this embodiment further subdivides the speech-text alignment stage into two sub-stages, achieving cross-modal alignment and fusion more efficiently and stably through a progressive, step-by-step strategy. Specifically, this training process includes an adapter pre-alignment stage and a cross-modal joint pre-training stage.

[0054] The core objective of the adapter pre-alignment stage is to perform preliminary training on the adapter module without altering the original knowledge system of the large language model, so that it can align the continuous speech representations output by the encoder to the semantic space that the large language model can understand.

[0055] Specifically, the training process of the adapter pre-alignment stage includes: fixing the parameters of the large language model and encoder, and updating the parameters of the adapter module using speech recognition data.

[0056] Here, speech recognition data can be understood as a large amount of speech-text pairing data. For example, a piece of speech recognition data may include a speech segment (audio: "How is the weather today?") and its corresponding text annotation (text: "How is the weather today?").

[0057] Fixing the parameters of the large language model and encoder means that the weights of these two modules are frozen and do not change during backpropagation and gradient updates. During training, the speech portion of the input speech recognition data is processed by the fixed encoder to obtain a continuous speech representation. This representation is then fed into the adapter module to be trained, which outputs a pre-aligned target speech representation. Next, the target speech representation is fed into the fixed large language model, which predicts the first response text corresponding to the speech in an autoregressive manner.

[0058] The training objective of the adapter pre-alignment stage is to minimize the difference between the first response text of the large language model and the actual text annotation. Through this stage, the adapter module completes pre-training from random initialization to having preliminary cross-modal mapping capabilities.

[0059] Furthermore, considering that although the adapter module can generate representations that the LLM can understand after completing the parameter pre-alignment, the LLM itself has not yet learned how to deeply utilize this information from the speech modality. For example, the LLM has not yet learned how to combine paralinguistic information for sentiment judgment. Therefore, this embodiment introduces a cross-modal joint pre-training stage after completing the parameter update in the adapter pre-alignment stage, enabling the large language model to learn to jointly process information from both text and speech modalities.

[0060] Specifically, the training process in the cross-modal joint pre-training phase includes: fixing the encoder parameters, and using speech recognition data, text dialogue data, and speech dialogue data to update the parameters of the large language model and the adapter module.

[0061] Here, the encoder parameters are kept fixed in order to maintain the stability of speech feature extraction and avoid catastrophic forgetting during joint training, while the parameters of the large language model and the adapter module are set to a trainable state.

[0062] The cross-modal joint pre-training phase uses mixed data for training, which includes speech recognition data, text dialogue data, and speech dialogue data.

[0063] The speech recognition data can be manually annotated speech-text pairs (such as Chinese speech-text pairs, English speech-text pairs), and the data volume can be in the millions of pairs.

[0064] The text dialogue data is plain text data. Each round of dialogue consists of a question text and a response text, including single-turn and multi-turn dialogues, covering multiple fields such as home furnishing, e-commerce, navigation, education, and healthcare. The style is conversational. The data volume can reach millions of rounds. Using this type of plain text data for training can effectively maintain and enhance the original dialogue and reasoning capabilities of large language models, preventing language ability degradation during cross-modal training.

[0065] Voice dialogue data can be constructed using a hybrid approach of synthetic and real-world data. Synthetic data is built upon text-based dialogue data, synthesizing the speech corresponding to the "question text" through a model. It supports specifying prosody, emotion, and speaker, enabling the low-cost and large-scale construction of voice input dialogue data. Real-world data consists of recorded real conversations (including question and response speech), transcribed into text, covering various scenarios, rich in emotion, and of high quality, with a data volume reaching hundreds of hours. By learning from this type of data, large language models can learn to combine paralinguistic information (such as excited tone) contained in the target speech representation with the dialogue context, thereby generating more empathetic and intelligent responses.

[0066] This embodiment improves model training stability and refines cross-modal capabilities by subdividing the speech-text alignment stage into two sub-stages: adapter pre-alignment and cross-modal joint pre-training. The first sub-stage pre-trains a lightweight adapter while freezing the large language model, providing a good starting point for subsequent training. The second sub-stage uses mixed data to fine-tune the cross-modal capabilities of the large language model while preserving its strong linguistic foundation. This effectively avoids problems such as training instability, slow convergence, and damage to the original linguistic capabilities of the large language model that may result from direct end-to-end joint training, ultimately yielding a model with superior performance and more comprehensive capabilities.

[0067] To achieve efficient modality alignment, avoid the high computational cost of updating the parameters of the large language model and encoder, and prevent the large language model from forgetting the rich language knowledge it learned in the pre-training stage, this embodiment fixes the parameters of the large language model and the speech recognition model encoder, and only updates the parameters of the adapter module.

[0068] Specifically, the parameters of the adapter module are updated using speech recognition data, including: The speech component in the speech recognition data is encoded using an encoder to obtain a continuous sample speech representation; The continuous sample speech representations are mapped through the adapter module to obtain the mapped speech representations; Using a large language model, the mapped speech representation is applied to predict and obtain the first response text; The prediction loss is determined based on the text portion of the first response and the speech recognition data; Update the parameters of the adapter module based on the predicted loss.

[0069] Specifically, speech recognition data is acquired, which includes a speech component and a text component. The speech component is the audio data corresponding to specific utterances, while the text component is the text obtained by accurately transcribing the language content contained in the speech component. During the training process, the text component plays the role of the standard answer.

[0070] During training, the speech recognition model's encoder is used to encode the speech portion, converting the speech waveform signal into a continuous sample speech representation containing rich acoustic and linguistic information.

[0071] Then, the continuous sample speech representation is mapped through the adapter module to obtain the mapped speech representation. Optionally, the adapter module can perform nonlinear transformation and dimension alignment on the continuous sample speech representation to output the mapped speech representation.

[0072] Next, the large language model is used to predict the first response text by applying the mapped speech representation. Here, the large language model is also a fixed module in this stage, and its parameters remain constant. Ideally, if the adapter module's mapping is perfect, the large language model should be able to accurately generate text that is completely consistent with the original speech content based on the mapped speech representation.

[0073] Subsequently, the prediction loss is determined based on the text portion of the first response and the speech recognition data. Specifically, the prediction loss can be calculated by comparing the differences between the two.

[0074] Finally, the calculated prediction loss is used to update the parameters of the adapter module via backpropagation. Since the parameters of the encoder and the large language model are set to fixed in this stage, the update only applies to the adapter module. Ultimately, the adapter module learns how to act as an efficient translator, accurately translating the continuous speech representation corresponding to any speech into a semantic representation that the large language model can seamlessly understand, while retaining the semantic content information, paralinguistic information, and speaker information of the original speech.

[0075] This embodiment achieves efficient semantic alignment between speech and text modalities by fixing the parameters of the large language model and encoder during the speech-to-text alignment stage, training only the adapter module, and using the large language model as a supervision signal to guide the adapter in mapping speech representations to text-recognizable semantic representations. Since only the adapter module, with far fewer parameters than the large language model and encoder, needs to be trained, the computational resource consumption and time cost of model training are significantly reduced.

[0076] Based on any of the above embodiments, using a large language model, the mapped speech representation is applied for prediction to obtain the first response text, including: Using a large language model, based on the mapped speech representation and all previously predicted word segments, the current word segment is predicted until all word segment predictions are completed. Then, all predicted word segments are used as the first response text.

[0077] Specifically, word segmentation can be understood as the basic unit of natural language text. Depending on the word segmenter used by the large language model, a word segment can be a complete word (such as "weather"), a subword (such as the word "running" can be divided into "run" and "ning"), or a single Chinese character or letter, etc.

[0078] The large language model does not directly generate the complete first response text based on the mapped speech representation, but rather generates it through word segmentation and prediction. This process can be broken down into a series of prediction steps: In the first step, the large language model uses the mapped speech representation as a condition to predict the most likely first word segment.

[0079] In each subsequent step t (t>1), the large language model will simultaneously predict the probability distribution of the t-th segment based on the initial mapped speech representation and the segmentation sequence (i.e., the 1st to the (t-1st)th segment) that has been predicted and determined in all previous steps, and select one segment as the prediction result for the current step.

[0080] This iterative process will continue until the large language model predicts a specific end-of-word segmentation, or the generated segmentation sequence reaches the preset maximum length limit. Then, all the predicted segments from the start-of-word segment to the end-of-word segmentation will be concatenated and combined to obtain the final first response text.

[0081] This embodiment explicitly uses an autoregressive approach to segment and generate text word by word in a large language model, making full use of the powerful context modeling capabilities of the large language model. This ensures that the prediction of each word segment can simultaneously take into account global information from the speech modality (i.e., the mapped speech representation) and local context information from the text modality (i.e., the generated word segments), thereby ensuring the accuracy of the generated first response text.

[0082] Based on any of the above embodiments, updating the parameters of the large language model and the adapter module using speech recognition data, text dialogue data, and voice dialogue data includes: Acquire mixed training data, which includes speech recognition data, text dialogue data, and synthesized speech dialogue data. The proportion of text dialogue data in the mixed training data is fixed at a preset value. Using a large language model and adapter module, a second response text is generated from the mixed training data; Based on the second response text and the labeled text in the mixed training data, the generative prediction loss is calculated; The parameters of the large language model are updated based on the generative prediction loss, and the parameters of the adapter module are also updated.

[0083] Specifically, after completing the previous stage of training, this embodiment aims to jointly fine-tune the large language model and the adapter module through multi-task learning, so that while maintaining and enhancing its original powerful language capabilities, it can further master the ability to collaboratively process speech modal features and text modal information. This will enable it to accurately understand and respond to complex speech commands that contain rich paralinguistic information such as emotion, rhythm, and intent, ultimately achieving a more intelligent and human-like voice interaction effect.

[0084] The synthesized speech dialogue data used in this training phase can be understood as dialogue data in the form of "speech input - text output" generated through technical means. As an optional implementation, a large amount of text dialogue data can be collected first, and then text-to-speech technology can be used to convert the user's question text portion of the dialogue data into speech, thereby constructing a large amount of synthesized speech dialogue data. For example, the question text "Please check my flight tickets to Shanghai tomorrow" can be converted into a speech segment and paired with the original answer text "Checking for you...".

[0085] Considering that using only multimodal data during the fine-tuning phase might cause the large language model to forget or weaken its original powerful language capabilities acquired through pre-training on massive amounts of plain text data while learning new capabilities, this embodiment uses a mixed ratio of speech recognition data, text dialogue data, and synthesized speech dialogue data. Specifically, the proportion of text dialogue data in the mixed training data is fixed at a pre-selective value. For example, the preset value can be set to 50%. That is, in each training batch, half of the data is ensured to be plain text dialogue data, ensuring that the large language model receives a sufficient amount of text information during training, thereby stabilizing its core capabilities as the foundation of the language model and preventing performance degradation when adapting to multimodal tasks.

[0086] Specifically, for each sample in the mixed training data, appropriate processing is performed according to its data type to generate a response. For speech recognition data or synthesized speech dialogue data, the speech component is first converted into a target speech representation through an encoder and adapter module, then concatenated with the corresponding prompt text before being fed into the large language model to generate the second response text. For text dialogue data, the text input component is directly concatenated with the corresponding prompt text before being fed into the large language model to generate the second response text.

[0087] Next, based on the second response text and the labeled text in the mixed training data, the generative prediction loss is calculated. Here, the labeled text refers to the standard answer or expected output corresponding to each sample in the mixed training data. For example, for speech recognition data, the labeled text is its corresponding transcribed text; for dialogue data, the labeled text is the standard response content.

[0088] By comparing the "second response text" generated in the previous step with the corresponding "annotated text," the generative prediction loss can be calculated. As an optional implementation, this loss can also be calculated using cross-entropy loss. This loss value reflects the gap between the model's current generated result and the expected result.

[0089] Finally, the parameters of the large language model are updated based on the generative prediction loss, and the parameters of the adapter module are also updated. As an optional implementation, the gradient can be calculated using backpropagation based on the generative prediction loss to update the parameters of both the large language model and the adapter module. Unlike the speech-text alignment stage, in this stage, the gradient from backpropagation will be used to update all or some of the parameters of the large language model, as well as the parameters of the adapter module. This joint update approach enables the large language model to learn how to collaboratively utilize speech representations and text information to complete more complex instruction tasks, while also allowing the mapping capabilities of the adapter module to be further optimized and fine-tuned in richer task scenarios.

[0090] This embodiment employs multi-task training with mixed data including speech recognition data, text dialogue data, and synthesized speech dialogue data. By updating the parameters of the large language model and adapter module simultaneously based on a unified generative prediction loss, the large language model achieves efficient acquisition of complex multimodal interaction capabilities, including processing speech commands and understanding emotions and intentions in speech, while maintaining its original powerful text processing capabilities.

[0091] Based on any of the above embodiments, the parameters of the large language model are updated according to the generative prediction loss, including: A low-rank matrix is ​​appended next to the original weight matrix of the large language model; The parameters of the low-rank matrix are updated based on the generative prediction loss.

[0092] In traditional fine-tuning, directly updating all parameters of a large language model incurs significant computational and storage costs, and may cause the model to forget its existing general knowledge when learning new capabilities. To address this, this embodiment appends a low-rank matrix to the original weight matrix of the large language model and updates the parameters of the low-rank matrix based on the generative prediction loss.

[0093] Specifically, when updating the parameters of a large language model, the pre-trained original weight matrix is ​​not modified; instead, it is frozen and remains unchanged throughout the training process. Then, for specific weight matrices in the model, such as the weight matrices of the query, key, value, and output linear layers in a Transformer structure, a lower-rank matrix consisting of two smaller matrices is connected in parallel. Since the original weight matrix is ​​fixed, the only parameters that need to be learned during the entire training process are those of the lower-rank matrix. The number of parameters in the lower-rank matrix is ​​much smaller than that of the original weight matrix, typically only a few percent or even less. Therefore, updating the parameters of the lower-rank matrix significantly reduces the gradient computation and memory usage in a single training iteration, avoiding the extremely high computational resource threshold and excessively long training time caused by updating all parameters. Furthermore, by preserving the original weight matrix, the catastrophic forgetting of existing general knowledge when learning new capabilities is also avoided.

[0094] This embodiment updates the parameters of the additional low-rank matrix while freezing the original weight matrix of the large language model, which significantly reduces the number of parameters that need to be trained and stored, and achieves efficient parameter fine-tuning of the large language model, greatly reducing training costs and deployment flexibility.

[0095] Based on any of the above embodiments, the training process in the fine-tuning phase includes: The fine-tuning data includes speech recognition data, text dialogue data, real speech dialogue data, dialogue error correction data, and speech description data. Dialogue error correction data is data that is constructed by introducing simulated speech recognition errors into the question part of the original dialogue data and pairing the question part containing the error with the response part in the original dialogue data. Speech description data is data that includes speech input and its corresponding text description annotations. Using a large language model and adapter module, a third response text for fine-tuning data is generated; Calculate the fine-tuning loss based on the third response text and the corresponding labeled text in the fine-tuning data; Based on the fine-tuning loss, update the parameters of the large language model and the adapter module.

[0096] Specifically, fine-tuning data is a mixed dataset containing multiple task types, used to fine-tune instructions for large language models. Through diverse data, the model can be guided to learn different capabilities.

[0097] The speech recognition data used in this stage is similar to the data used in the speech-to-text alignment stage, namely speech-to-text pairs. Continuing to use this data in this stage aims to prevent the model from forgetting its most basic speech-to-text capabilities while learning new ones.

[0098] Text-based dialogue data refers to data that contains only text-based instruction-response pairs. Examples include general question-and-answer data, translation data, and summarization data. This type of data is introduced to maintain and enhance the powerful general language capabilities and instruction-following abilities of the large language model itself.

[0099] Real-world voice dialogue data refers to data pairs of "user voice - intelligent assistant reply text" collected from real user interaction scenarios. This type of data enables models to learn how to generate appropriate responses under realistic voice input with various accents, speaking speeds, and background noise.

[0100] Dialogue correction data is specially constructed in this embodiment to improve the model's robustness to speech recognition errors. As an optional embodiment, a speech recognition model can be used to obtain the first response text corresponding to the speech dialogue data. The distribution of entity words and error types in the first response text that are incorrectly recognized can be statistically analyzed. Based on this distribution, question text in the speech dialogue data can be randomly inserted, deleted, and replaced to construct a "wrong question - correct response" sample, which is the dialogue correction data. For example, the question voice is "Play the song 'Sunny Day'", and the corresponding response is "Okay, I'll play the song 'Sunny Day' for you right away." The question text can be changed from "Play the song 'Sunny Day'" to "Play the song 'Sweet'". Finally, "Play the song 'Sweet' - Okay, I'll play the song 'Sunny Day' for you right away" can be used as dialogue correction data. Through training with this data, the model can learn that when the received recognized text does not perfectly match the speech features, it should trust the true information contained in the speech features more, thus ignoring errors in the recognized text and providing the correct response.

[0101] Furthermore, the speech description data is specially constructed in this embodiment, containing speech input and its corresponding text description annotations. These text description annotations are not a transcription of the speech content, but rather a description of the paralinguistic features of the speech segment, such as age, gender, speech rate, emotion, intention, and transcribed text. For example, for an excited "That's great!", the corresponding speech description data could be "(user speech) → (description text: The user sounds very happy)". Using this type of data for training allows the model to be explicitly taught to understand and perceive paralinguistic information such as emotion and tone in speech.

[0102] Then, using a large language model and an adapter module, the third response text of the fine-tuned data is generated. For data containing speech input (such as speech recognition data, real speech dialogue data, etc.), the speech portion is first extracted into continuous speech representation by an encoder, and then mapped to the target speech representation by an adapter module; then, the target speech representation is fed into the large language model to obtain the third response text. For data containing only text (such as text dialogue data), the input text portion is directly fed into the large language model to obtain the third response text.

[0103] Next, the fine-tuning loss is calculated based on the third response text and the corresponding labeled text in the fine-tuning data. The labeled text can be understood as the target output text, pre-defined as the standard answer for each input sample in the training data. This labeled text is the target of the model's learning and the basis for calculating the loss. Specifically, for speech recognition data, the labeled text is the accurate textual transcription of the speech; for text dialogue data and real speech dialogue data, the labeled text is the ideal response to the question; for speech description data, the labeled text is a textual description of the paralinguistic information in the speech; particularly, for dialogue error correction data, the labeled text is the correct response to the original, correct user intent, rather than a response to the incorrectly recognized text introduced by simulation.

[0104] As an optional implementation, the third response text can be compared with the labeled text corresponding to each sample in the fine-tuning data, and the difference between the two can be calculated to obtain the fine-tuning loss. For example, considering that the amount of data of different task types in the mixed dataset may be unbalanced, and that the importance and learning difficulty of different tasks also vary, the fine-tuning losses from different types of data can be weighted and summed to obtain the total loss value. The parameters of the large language model and the adapter module can then be updated with the total loss value. This allows for a flexible balance of the learning priorities of different tasks, ensuring that the model can acquire multiple high-order capabilities such as dialogue error correction and speech description in a balanced manner, while preventing the performance degradation of basic capabilities such as speech recognition, ultimately resulting in a model with stronger comprehensive capabilities.

[0105] This embodiment constructs a hybrid fine-tuning dataset encompassing multiple tasks such as speech recognition, text-based dialogue, real-world speech dialogue, dialogue correction, and speech description. It then performs joint instruction fine-tuning on the large language model and adapter module, enabling the model to handle various complex voice interaction tasks, including speech recognition, dialogue correction, emotion understanding, and content generation, within a single framework. By incorporating targeted dialogue correction and speech description data for training, the model's robustness and intelligence in real-world interaction scenarios are significantly improved. This allows it to not only understand literal meaning but also grasp implied meanings and possesses a degree of self-correction capability when speech recognition errors occur, greatly enhancing the user experience.

[0106] Based on any of the above embodiments, the proportion of speech recognition data in the fine-tuning data is less than that in the mixed training data, and the proportions of text dialogue data, real speech dialogue data, dialogue error correction data, and speech description data in the fine-tuning data are the same.

[0107] Specifically, in the speech-to-text alignment stage, the core task is to teach the adapter module to perform cross-modal mapping, that is, to learn "what sound corresponds to what text". Therefore, speech recognition data dominates the training data in this stage, and its proportion can be relatively high, for example, it can account for more than 90% of the total data, to ensure that the model can learn robust acoustic-to-text correspondences from massive amounts of data.

[0108] However, during the fine-tuning phase, the training objective shifts from simple acoustic text alignment to teaching the large language model how to perform complex instruction following and dialogue interactions with multimodal information. If the fine-tuning data still contains a large amount of speech recognition data, the large language model may tend to perform simple transcription tasks, i.e., directly reproducing the heard speech representations as text, which would suppress its powerful language understanding and generation capabilities.

[0109] Based on this, this embodiment reduces the proportion of speech recognition data in the fine-tuning data during the fine-tuning stage. For example, the proportion of speech recognition data in the fine-tuning data can be reduced to 5% or lower, thereby avoiding the model from focusing excessively on the underlying transcription task during the fine-tuning stage, and instead guiding the model's attention to learning higher-order interaction capabilities.

[0110] Furthermore, to cultivate a comprehensive and balanced capability for the model, this embodiment allocates equal proportions of text dialogue data, real speech dialogue data, dialogue error correction data, and speech description data to the fine-tuning data. For example, if speech recognition data accounts for 5%, the remaining 95% of the data is evenly distributed among these four types, with each type accounting for 23.75%. This balanced allocation ensures that the large language model learns and strengthens its text understanding ability, multimodal emotion perception ability, dialogue robustness and error correction ability, and paralinguistic feature description ability to an equal degree during fine-tuning, avoiding any imbalance in the model's capabilities and thus developing into a powerful multimodal speech dialogue model with balanced development across all aspects.

[0111] This embodiment significantly reduces the proportion of low-value speech recognition data by optimizing the data composition during the fine-tuning stage, and introduces four types of high-value instruction data with balanced proportions. This achieves comprehensive, balanced, and targeted training of the large language model's dialogue capabilities. Because this data allocation strategy avoids the model's ability from degenerating into simple transcription tasks, and ensures that the model is equally strengthened across multiple key dimensions such as language foundation, multimodal understanding, robustness, and emotion perception, it effectively avoids the problems of single-function models and poor interactive experiences. This allows the finally trained model to better understand complex real-world speech interaction scenarios and provide more intelligent, robust, and human-like responses.

[0112] Based on any of the above embodiments, a response text is generated by applying the target speech representation and recognized text through a large language model, including: Get the task prompt text for the specified current task type; The task prompt text, target speech representation, and recognized text are concatenated to form the input sequence of the large language model; The input sequence is fed into a large language model to generate the response text.

[0113] Here, the task prompt text can be understood as a pre-defined text instruction that informs the large language model what task needs to be performed. Since the model framework constructed in this embodiment is a general speech understanding and interaction framework, it can be applied to various downstream tasks. If all inputs are sent to the large language model indiscriminately, the model may not be able to accurately determine whether the user's current intention is to chat, query information, or execute a command. Therefore, introducing task prompt text provides the large language model with clear context and task constraints.

[0114] For example, if the task to be performed is a general voice question-and-answer task, the task prompt text can be a specific identifier such as [SPEECH_Q&A]; if the task to analyze the emotions in the user's voice is to be performed, the task prompt text can be [SENTIMENT_ANALYSIS]; if the task to perform voiceprint recognition to confirm the user's identity is to be performed, the task prompt text can be [VOICEPRINT_RECOGNITION].

[0115] After obtaining the task prompt text, the task prompt text, target speech representation, and recognized text are concatenated to form the input sequence of the large language model. Here, concatenation refers to connecting multiple information segments in a predetermined order. As a preferred embodiment, the concatenation can be performed in the order of "task prompt text + target speech representation + recognized text," so that the large language model is first informed of "what to do" through the task prompt text, then detailed acoustic information of "how it was said" through the target speech representation, and finally the core semantic content of "what was said" through the recognized text.

[0116] Finally, the input sequence is fed into the large language model. Since the input sequence simultaneously and completely contains information from three dimensions—task prompt text to clarify the task objective, target speech representation carrying paralinguistic information such as user emotion and tone, and recognition text expressing literal semantics—the large language model is able to perform comprehensive deep understanding and reasoning based on multimodal information, rather than being limited to single-modal understanding of the recognition text. As a result, the large language model can generate response text that is both accurate and appropriate to the user's current emotional state.

[0117] This embodiment introduces task prompt text and concatenates it with the target speech representation and recognized text into a unified input sequence to drive the large language model, achieving flexible and explicit task guidance for the large language model. Because the task prompt text provides clear context and task constraints for the large language model, a single model architecture can support diverse speech interaction tasks by changing different prompt texts, greatly enhancing the model's reusability and scalability.

[0118] Based on any of the above embodiments, the adapter module includes a linear transformation layer, a nonlinear activation function layer, and a normalization layer. The linear transformation layer is used to perform dimensional mapping on the continuous speech representation, the nonlinear activation function layer is used to perform nonlinear transformation on the mapped continuous speech representation, and the normalization layer is used to perform normalization processing on the nonlinearly transformed continuous speech representation.

[0119] Here, the linear transformation layer is used to perform dimensionality mapping on the continuous speech representation. This linear transformation layer can be one or more fully connected layers, used to transform the dimension of the input feature vector into the target dimension. Since the dimension of the continuous speech representation output by the encoder of a speech recognition model is usually inconsistent with the word embedding dimension that a large language model can accept, this embodiment uses a linear transformation layer to align these two dimensions.

[0120] Furthermore, the adapter module also includes a nonlinear activation function layer for nonlinear transformation of the mapped continuous speech representation. If the adapter module consists only of stacked linear transformation layers, its overall effect is equivalent to a single linear transformation, which greatly limits the model's expressive power and makes it difficult to learn the complex nonlinear relationships between the speech space and semantic space. Therefore, introducing a nonlinear activation function layer is key to improving the model's learning ability. As an optional implementation, the nonlinear activation function layer can employ the Gaussian Error Linear Unit (GELU) activation function.

[0121] In addition, the adapter module includes a normalization layer for normalizing the continuous speech representation after nonlinear transformation. Here, the normalization layer adjusts and scales the output of each network layer to stabilize its distribution, helping to alleviate the internal covariate shift problem during deep neural network training. It effectively avoids gradient vanishing or exploding, thus making the entire training process more stable and accelerating model convergence. As an optional implementation, the normalization layer can use layer normalization (LayerNorm).

[0122] In an alternative embodiment, to construct a lightweight yet efficient adapter, the three layers described above can be combined and stacked. For example, the adapter module can be composed of three basic processing units stacked together, each of which sequentially includes a linear transformation layer, a GELU activation function layer, and a LayerNorm normalization layer.

[0123] Based on any of the above embodiments Figure 2 This is a flowchart illustrating the voice interaction model training method provided by the present invention, as shown below. Figure 2 As shown, the voice interaction model includes a speech recognition model, an adapter module, a large language model, and a speech synthesis model. The speech recognition model comprises an encoder and a decoder. The encoder uses a 16-layer Conformer, and the decoder uses a 6-layer Transformer with Monotonic Attention. The adapter module includes 3 linear layers, a GELU activation function, and a LayerNorm layer. The input layer dimension is the same as the input speech representation dimension, while the intermediate layer dimensions are 2 and 4 times the input speech representation dimension, respectively. The large language model uses the Spark large model with an autoregressive Transformer architecture, containing 32 Transformer layers, 7 bytes of parameters, and a vocabulary of approximately 150,000 words. The speech synthesis model uses the iFlytek hyper-humanoid synthesis model based on a discrete unit + LLM architecture.

[0124] Before training the voice interaction model, training data is required. Training data includes speech recognition data, text dialogue data, voice dialogue data (synthetic voice dialogue data and real voice dialogue data), dialogue error correction data, and voice description data.

[0125] Let voice input be represented as [Q_audio], text input as [Q_text], and text output as [answer], with angle brackets indicating special tokens, such as... <end>The prompt distinguishes between different tasks. <user>xxx <bot>In yyy, "xxx" represents the user's input, which can include text, speech representation, or a combination of both, while yyy represents the model's expected output. The specific inputs for different training data and corresponding tasks are as follows: For speech recognition data, the task input is { <user> <speech>[Q_audio] converts speech to text. <end> <bot>[answer]}, the task input instructs the model to perform a speech recognition task, that is, to directly transcribe the input speech part [Q_audio] into the corresponding text content [answer].

[0126] For text-based dialogue data, the task input is { <user> <text>[Q_text] Answer the question <end> <bot>[answer]}, the task input instructs the model to perform a pure text dialogue task, that is, to perform semantic understanding based on the input text content [Q_text] and generate the corresponding text response [answer].

[0127] For voice dialogue data, the task input is { <user> <speech>[Q_audio] answered the question. <end> <bot>[answer]}, the task input instructs the model to perform an end-to-end voice dialogue task, that is, to directly understand the input voice [Q_audio] and generate the corresponding text response [answer].

[0128] For dialogue error correction data, the task input is { <user> <text>[Q_text] <speech>[Q_audio] answered the question. <end> <bot>[answer]}, the task input instructs the model to perform a multimodal error correction task, that is, when the speech recognition result [Q_text] may contain errors, the model needs to jointly understand [Q_text] and the original speech information [Q_audio] to generate a more accurate response [answer].

[0129] For speech description data, the task input is { <user> <speech>[Q_audio] analyzes speech including age, gender, speech rate, emotion, intention, and speech content, and returns a description in a fixed format, for example: This middle-aged male speaks at a slightly faster pace and sounds somewhat dissatisfied, wanting to complain. <end> <bot>[answer]}, the task input instructs the model to perform a deep speech analysis task, that is, not only to understand the literal content of speech, but also to analyze and extract the paralinguistic information (such as age, gender, speech rate, emotion) and intent contained therein, and generate descriptive text [answer] according to the specified structured format.

[0130] Next, multi-stage training is performed using the aforementioned training data to ensure that each module of the model is fully trained and that the model's learned representations can be adapted to multiple tasks. Specifically: (1) Pre-training stage: The real-time speech recognition model is trained based on the windowing method, and the text LLM with fine-tuning of instructions is used as the large language model.

[0131] (2) Voice-to-text alignment stage: The encoder parameters of the fixed speech recognition model and the large language model parameters are used to train the adapter module using speech recognition data. The adapter module can be trained using the Adamw optimizer and a learning rate of 1e-4.

[0132] After training the adapter module as described above, the encoder parameters are fixed, and the large language model is updated using the LoRA module to update the adapter module parameters and the LoRA module parameters. A learning rate of 5e-5 can be used, and the LoRA rank can be 32. The training data includes speech recognition data, text dialogue data, and synthesized speech dialogue data, with text dialogue data accounting for 50%.

[0133] (3) Fine-tuning stage: The quality of speech recognition data, text dialogue data, and synthesized speech dialogue data is optimized, and real-world, multi-scenario, high-quality speech dialogue data, dialogue error correction data, and speech description data are added. The proportion of speech recognition data is reduced, while the amount of other types of data is kept balanced at approximately 1:1:1:1. The parameters of the large language model are updated using a learning rate of 2e-5.

[0134] In summary, the alignment training through the three stages described above endows the large language model with the functions of language recognition, dialogue, audio understanding, and error correction. The pre-training and speech-text alignment stages achieve results comparable to the cascaded effects of ASR+LLM in synthesized speech dialogue tasks. The fine-tuning stage incorporates dialogue error correction data and speech description data for training, achieving error correction accuracy and emotion recognition accuracy of over 80%. Combined with high-quality real-world speech dialogue data covering various scenarios and expressions, the large language model's responses become more reliable and human-like.

[0135] The voice interaction device provided by the present invention will be described below. The voice interaction device described below can be referred to in correspondence with the voice interaction method described above.

[0136] Based on any of the above embodiments Figure 3 This is a schematic diagram of the structure of the voice interaction device provided by the present invention, as shown below. Figure 3 As shown, the device includes: The acquisition unit 310 is used to acquire the user's question speech, and use the encoder of the speech recognition model to obtain the continuous speech representation of the question speech, and use the decoder of the speech recognition model to obtain the recognized text of the question speech; the continuous speech representation retains the semantic content information, paralinguistic information and speaker information in the question speech; The mapping unit 320 is used to map the continuous speech representation to the semantic space of the large language model using the adapter module to obtain the target speech representation. The generation unit 330 is used to generate response text by applying the target speech representation and recognized text through a large language model; The training phase of the adapter module and the large language model includes a speech-text alignment phase and a fine-tuning phase. In the speech-text alignment phase, the parameters of the large language model and the encoder are fixed, and the adapter module is trained. In the fine-tuning phase, the parameters of the encoder and the adapter module obtained in the speech-text alignment phase are fixed, and the large language model is fine-tuned so that the large language model has the ability to jointly process target speech representations and recognize text.

[0137] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 4 As shown, the electronic device may include a processor 410, a communications interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communications interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute voice interaction methods.

[0138] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0139] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the voice interaction methods provided by the above methods.

[0140] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the voice interaction methods provided by the methods described above.

[0141] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0142] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0143] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.< / bot> < / end> < / speech> < / user> < / bot> < / end> < / speech> < / text> < / user> < / bot> < / end> < / speech> < / user> < / bot> < / end> < / text> < / user> < / bot> < / end> < / speech> < / user> < / bot> < / user> < / end>

Claims

1. A voice interaction method, characterized in that, include: The system acquires the user's question speech, uses the encoder of the speech recognition model to obtain the continuous speech representation of the question speech, and uses the decoder of the speech recognition model to obtain the recognized text of the question speech. Using an adapter module, the continuous speech representation is mapped to the semantic space of a large language model to obtain the target speech representation; the target speech representation retains the semantic content information, paralinguistic information, and speaker information of the question speech; Using the large language model, the target speech representation and the recognized text are applied to generate the response text; The training phase of the adapter module and the large language model includes a speech-text alignment phase and a fine-tuning phase. In the speech-text alignment phase, the parameters of the large language model and the encoder are fixed, and the adapter module is trained. In the fine-tuning phase, the parameters of the encoder and the adapter module obtained in the speech-text alignment phase are fixed, and the large language model is fine-tuned so that the large language model has the ability to jointly process the target speech representation and the recognized text.

2. The voice interaction method according to claim 1, characterized in that, The training process for the speech-text alignment phase includes: The parameters of the large language model and the encoder are fixed, and the parameters of the adapter module are updated using speech recognition data; After updating the parameters of the adapter module, the parameters of the encoder are fixed, and the parameters of the large language model and the adapter module are updated using the speech recognition data, text dialogue data, and voice dialogue data.

3. The voice interaction method according to claim 2, characterized in that, The step of updating the parameters of the adapter module using speech recognition data includes: The encoder is used to encode the speech portion in the speech recognition data to obtain a continuous sample speech representation; The continuous sample speech representation is mapped through the adapter module to obtain the mapped speech representation; Using the large language model, the mapped speech representation is applied to predict the first response text. The prediction loss is determined based on the first response text and the text portion of the speech recognition data; The parameters of the adapter module are updated based on the predicted loss.

4. The voice interaction method according to claim 3, characterized in that, The process of using the large language model and applying the mapped speech representation to predict and obtain the first response text includes: Using the large language model, based on the mapped speech representation and all the predicted word segments, the current word segment is predicted until all word segment predictions are completed. Then, all the predicted word segments are used as the first response text.

5. The voice interaction method according to claim 2, characterized in that, The step of updating the parameters of the large language model and the adapter module using the speech recognition data, text dialogue data, and voice dialogue data includes: Acquire mixed training data, which includes the speech recognition data, the text dialogue data, and the synthesized speech dialogue data, wherein the proportion of the text dialogue data in the mixed training data is fixed at a preset value; Using the large language model and the adapter module, a second response text is generated from the mixed training data; Based on the second response text and the labeled text in the mixed training data, the generative prediction loss is calculated; The parameters of the large language model are updated based on the generative prediction loss, and the parameters of the adapter module are also updated.

6. The voice interaction method according to claim 5, characterized in that, The step of updating the parameters of the large language model based on the generative prediction loss includes: A low-rank matrix is ​​appended next to the original weight matrix of the large language model; The parameters of the low-rank matrix are updated based on the generative prediction loss.

7. The voice interaction method according to claim 5, characterized in that, The training process during the fine-tuning phase includes: The fine-tuning data includes the speech recognition data, the text dialogue data, the real speech dialogue data, the dialogue error correction data, and the speech description data. The dialogue error correction data is data formed by introducing simulated speech recognition errors into the question part of the original dialogue data and pairing the question part containing the error with the response part in the original dialogue data. The speech description data is data containing the speech input and its corresponding text description annotation. Using the large language model and the adapter module, a third response text for the fine-tuned data is generated; Based on the third response text and the corresponding labeled text in the fine-tuning data, calculate the fine-tuning loss; The parameters of the large language model and the adapter module are updated based on the fine-tuning loss.

8. The voice interaction method according to claim 7, characterized in that, The proportion of speech recognition data in the fine-tuning data is less than that in the mixed training data, and the proportions of text dialogue data, real speech dialogue data, dialogue error correction data, and speech description data in the fine-tuning data are the same.

9. The voice interaction method according to any one of claims 1 to 8, characterized in that, The step of generating a response text by applying the target speech representation and the recognized text to the large language model includes: Get the task prompt text for the specified current task type; The task prompt text, the target speech representation, and the recognized text are concatenated to form the input sequence of the large language model; The input sequence is fed into the large language model to generate the response text.

10. The voice interaction method according to any one of claims 1 to 8, characterized in that, The adapter module includes a linear transformation layer, a nonlinear activation function layer, and a normalization layer. The linear transformation layer is used to perform dimensional mapping on the continuous speech representation. The nonlinear activation function layer is used to perform nonlinear transformation on the mapped continuous speech representation. The normalization layer is used to normalize the nonlinearly transformed continuous speech representation.

11. A voice interaction device, characterized in that, include: The acquisition unit is used to acquire the user's question speech, obtain the continuous speech representation of the question speech using the encoder of the speech recognition model, and obtain the recognized text of the question speech using the decoder of the speech recognition model. The mapping unit is used to map the continuous speech representation to the semantic space of the large language model using the adapter module to obtain the target speech representation; the target speech representation retains the semantic content information, paralinguistic information and speaker information in the question speech; The generation unit is used to generate response text by applying the target speech representation and the recognized text through the large language model; The training phase of the adapter module and the large language model includes a speech-to-text alignment phase and a fine-tuning phase. In the speech-to-text alignment phase, the parameters of the large language model and the encoder are fixed, and the adapter module is trained. In the fine-tuning phase, the parameters of the encoder and the adapter module obtained in the speech-to-text alignment phase are fixed, and the large language model obtained after the speech-to-text alignment phase is fine-tuned so that the large language model has the ability to jointly process the target speech representation and the recognized text.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the voice interaction method as described in any one of claims 1 to 10.

13. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the voice interaction method as described in any one of claims 1 to 10.

Images