Voice interaction method, device and electronic equipment

Abstract

The application provides a speech interaction method and device and electronic equipment, and relates to the technical field of speech processing. The method comprises the following steps: acquiring speech information input by a user, and acquiring historical intention text of the user; inputting the speech information into a speech encoder of a spoken language understanding model to obtain acoustic coding features output by the speech encoder; inputting the historical intention text into a text encoder of the spoken language understanding model to obtain text coding features output by the text encoder; inputting the acoustic coding features and the text coding features into an intention recognition module of the spoken language understanding model to obtain an intention recognition result output by the intention recognition module, so as to be used for speech interaction. The application can improve the accuracy of the intention recognition result and accurately acquire the real intention of the user.

Description

Technical Field

[0001] This invention relates to the field of speech processing technology, and in particular to a speech interaction method, apparatus, and electronic device. Background Technology

[0002] Spoken Language Understanding (SLU) technology, as a core technology of human-computer interaction, aims to directly parse executable intents and slot parameters from user voice input. Existing SLU systems generally adopt a cascaded architecture, which first converts the speech signal into text through an Automatic Speech Recognition (ASR) module, and then extracts the intent and slots from the text through a Natural Language Understanding (NLU) module.

[0003] However, this traditional architecture has inherent limitations: First, it suffers from severe modal fragmentation. Optimizing ASR and NLU as independent modules makes efficient joint fine-tuning difficult. Furthermore, text errors generated by the ASR module are directly passed to the NLU module and amplified, leading to overall performance degradation and poor accuracy in the final intent recognition result. In addition, existing SLU systems struggle to effectively clarify incomplete, ambiguous, or ambiguous user speech input through multi-turn dialogues, further reducing intent recognition accuracy and failing to meet the needs of complex human-computer interaction scenarios.

[0004] Therefore, improving the accuracy of intent recognition results during voice interaction to accurately obtain the user's true intent is an urgent problem to be solved. Summary of the Invention

[0005] This invention provides a voice interaction method, device, and electronic device to solve the problem of low accuracy of intent recognition results during voice interaction.

[0006] This invention provides a voice interaction method, comprising:

[0007] Obtain the user's voice input and the user's historical intent text;

[0008] The speech information is input into the speech encoder of the spoken language understanding model to obtain the acoustic coding features output by the speech encoder.

[0009] The historical intent text is input into the text encoder of the spoken language understanding model to obtain the text encoding features output by the text encoder;

[0010] The acoustic coding features and the text coding features are input into the intent recognition module of the spoken language understanding model to obtain the intent recognition result output by the intent recognition module for use in voice interaction.

[0011] According to a voice interaction method provided by the present invention, the spoken language understanding model is trained in the following manner:

[0012] Obtain the sample set;

[0013] The initial spoken language understanding model is fine-tuned using the sample set to obtain the spoken language understanding model; wherein the initial spoken language understanding model includes a pre-trained speech encoder, a pre-trained text encoder, and a pre-trained intent recognition module, and the pre-trained intent recognition module is a pre-trained large speech model.

[0014] According to a voice interaction method provided by the present invention, the step of fine-tuning an initial spoken language understanding model using the sample set to obtain the spoken language understanding model includes:

[0015] Freeze the pre-trained speech encoder and the pre-trained text encoder, and freeze the feature extraction layer of the pre-trained intent recognition module;

[0016] The semantic mapping layer of the pre-trained intent recognition module is fine-tuned using the sample set to obtain the spoken language understanding model.

[0017] According to a voice interaction method provided by the present invention, the step of obtaining a sample set includes:

[0018] Obtain real voice samples;

[0019] Acquire text corpus, and synthesize initial speech samples based on the text corpus;

[0020] The initial speech sample is subjected to data augmentation processing to obtain a synthesized speech sample; wherein, the data augmentation processing includes at least one of adding background noise, adding reverberation, adjusting speech rate, and adjusting pitch.

[0021] The sample set is constructed based on the real speech samples and the synthesized speech samples.

[0022] According to a voice interaction method provided by the present invention, the step of constructing the sample set based on the real voice samples and the synthesized voice samples includes:

[0023] The real speech samples are subjected to speech recognition processing to obtain real text;

[0024] Using a large language model, the real speech samples are labeled based on the real text to obtain a real sample set;

[0025] Using the large language model, the synthesized speech samples are labeled according to the text corpus to obtain a synthesized sample set;

[0026] The sample set includes the real sample set and the synthetic sample set.

[0027] According to a voice interaction method provided by the present invention, the step of fine-tuning an initial spoken language understanding model using the sample set to obtain the spoken language understanding model further includes:

[0028] The sample set is divided into a training set, a validation set, and a test set;

[0029] The initial spoken language comprehension model is iteratively fine-tuned based on the training set. In each round of fine-tuning, the parameters of the current spoken language comprehension model are adjusted through the backpropagation algorithm.

[0030] After each round of fine-tuning, the adjusted current spoken language understanding model is validated based on the validation set, and the validation loss is calculated.

[0031] When the preset number of fine-tuning rounds is reached, or when the preset early stopping condition is met according to the verification loss, the training ends, and the spoken language understanding model corresponding to the minimum value in the verification loss is determined as the pre-trained spoken language understanding model.

[0032] The performance of the pre-trained spoken language comprehension model is evaluated based on the test set.

[0033] Upon successful evaluation, the pre-trained spoken language comprehension model is identified as the spoken language comprehension model.

[0034] According to a voice interaction method provided by the present invention, obtaining the user's historical intent text includes:

[0035] Obtain the user's historical intent recognition results;

[0036] Based on the historical intent recognition results and the preset prompt word template, the historical intent text is generated; wherein, the preset prompt word template includes the splicing relationship and position of intent, slot and interaction information.

[0037] According to a voice interaction method provided by the present invention, the intent recognition result includes interaction information. After inputting the acoustic coding features and the text coding features into the intent recognition module of the spoken language understanding model to obtain the intent recognition result output by the intent recognition module, the method further includes:

[0038] Interactive voice is generated based on the interactive information to conduct voice interaction, and the type of voice interaction is confirmation, follow-up question, or notification.

[0039] The present invention also provides a voice interaction device, comprising:

[0040] The acquisition module is used to acquire the user's voice input and the user's historical intent text.

[0041] The speech encoding module is used to input the speech information into the speech encoder of the spoken language understanding model to obtain the acoustic coding features output by the speech encoder;

[0042] The text encoding module is used to input the historical intent text into the text encoder of the spoken language understanding model to obtain the text encoding features output by the text encoder;

[0043] An intent recognition module is used to input the acoustic coding features and the text coding features into the intent recognition module of the spoken language understanding model to obtain the intent recognition result output by the intent recognition module for use in voice interaction.

[0044] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the voice interaction methods described above.

[0045] The voice interaction method, device, and electronic device provided by this invention first acquire the user's input voice information and the user's historical intent text. Then, the voice information is input to the speech encoder of the spoken language understanding model to obtain acoustic coding features, and simultaneously, the historical intent text is input to the text encoder of the spoken language understanding model to obtain text coding features. Furthermore, the acoustic coding features and text coding features are input to the intent recognition module of the spoken language understanding model to obtain the intent recognition result, which is then used for voice interaction. This invention adopts an end-to-end single-model architecture, directly establishing a mapping from speech to semantics and outputting the final intent recognition result. It abandons the traditional cascaded scheme, eliminating the problem of ASR module errors propagating and amplifying to the NLU module, thus improving the accuracy of intent recognition results and shortening the processing link from voice input to intent output, thereby improving intent recognition efficiency. Further, after acquiring the voice information, the historical intent text corresponding to the user's historical dialogues is acquired and encoded in parallel to obtain text coding features, which assist the speech coding features in determining the intent recognition result. By utilizing a multi-turn dialogue mechanism, this invention can accurately obtain the user's true intent even when the voice input is incomplete, vague, or ambiguous, further improving the accuracy of intent recognition results. In summary, this invention can significantly improve the accuracy of intent recognition results during voice interaction, quickly and accurately obtaining the user's true intent. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0047] Figure 1 This is a system architecture diagram of the voice interaction system provided by the present invention;

[0048] Figure 2 This is one of the flowcharts illustrating the voice interaction method provided by the present invention;

[0049] Figure 3 This is a schematic diagram of the structure of the spoken language comprehension model provided by the present invention;

[0050] Figure 4 This is the second flowchart of the voice interaction method provided by the present invention;

[0051] Figure 5 This is the third flowchart of the voice interaction method provided by the present invention;

[0052] Figure 6 This is a schematic diagram of the structure of the voice interaction device provided by the present invention;

[0053] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0055] This invention proposes a voice interaction method, device, and electronic device, which are described below in conjunction with... Figures 1-7 Describe it.

[0056] Figure 1 This is a system architecture diagram of the voice interaction system provided by the present invention, such as... Figure 1 As shown, the voice interaction system includes an electronic device 01 and a server 02.

[0057] The electronic device can be a smartphone, tablet, laptop, smart speaker, smartwatch, smart bracelet, extended reality (XR) device, in-vehicle terminal, robot, etc., used to execute the voice interaction method of this invention. Extended reality devices include, but are not limited to, VR (Virtual Reality) devices, AR (Augmented Reality) devices, and MR (Mixed Reality) devices. Server 02 can be a server used to train the spoken language understanding model.

[0058] After training the spoken language understanding model, server 02 deploys the trained spoken language understanding model to electronic device 01. Electronic device 01 can process the received speech information based on the spoken language understanding model and further execute corresponding interactive operations based on the intent recognition results output by the model, such as asking follow-up questions or performing control operations.

[0059] Figure 2 This is one of the flowcharts illustrating the voice interaction method provided by the present invention, such as... Figure 2 As shown, the voice interaction method includes steps S110, S120, S130 and S140.

[0060] It should be noted that the execution order of steps S120 and S130 is not important.

[0061] Step S110: Obtain the voice information input by the user and obtain the user's historical intent text.

[0062] In this embodiment of the invention, the voice interaction method is applied to an electronic device. The electronic device can be a smartphone, tablet computer, laptop computer, smart speaker, smartwatch, smart bracelet, XR device, in-vehicle terminal, robot, or other electronic device with integrated voice interaction functionality.

[0063] This voice interaction method is applicable to many scenarios such as mobile application control, vehicle control, smart home control, and robot interaction.

[0064] When a user issues a voice command (e.g., “Xiao X, please turn on the air conditioner”), the voice acquisition module (e.g., microphone) in the electronic device can acquire the voice information.

[0065] Furthermore, after acquiring the speech information, the speech information can be detected through the VAD (Voice Activity Detection) module to obtain the start and end times of the valid speech, thereby determining the valid speech information, and then inputting the valid speech information into the speech encoder of the spoken language understanding model.

[0066] Historical intent text is generated based on historical dialogues, including the intents and slots in the historical dialogues, and may also include historical interaction information, which can be inquiries, confirmations, or notifications.

[0067] In one embodiment, historical dialogues can be obtained first, and historical intent text can be generated based on the historical dialogues and preset prompt word templates. The preset prompt word templates include the splicing relationship and position of intents, slots, and interaction information.

[0068] In another embodiment, a historical dialogue database can be constructed to store the voice input of each round of dialogue, the recognized intent results (including intent and slot), interaction information, etc. The user's historical intent recognition results can be directly obtained from this historical dialogue database. Then, based on the historical intent recognition results and a preset prompt word template, historical intent text is generated; wherein, the preset prompt word template includes the splicing relationship and position of intent, slot, and interaction information.

[0069] For example, the preset prompt template can be: "Extract intent and slot from speech; if information is incomplete, prepare follow-up questions: intent = [intent type], slot = [{slot name: slot value}], follow-up questions = [interaction information]". Assume the intent recognition result of the previous round of dialogue (i.e., the historical intent recognition result) is: intent = play music, the artist slot is XX artist, the song title slot is empty, and the interaction information is "Which song do you want to listen to?". In this case, the generated historical intent text can be: "Extract intent and slot from speech; if information is incomplete, prepare follow-up questions: intent = [play music], slot = [{artist: XX artist}, {song title: empty}], follow-up questions = [Which song do you want to listen to?]". It should be noted that if no historical intent recognition result is obtained, meaning the user's current speech information is from the first round of dialogue, then the historical intent text is the preset prompt template, where the intent type, slot name, slot value, and interaction information are all empty.

[0070] Step S120: Input the speech information into the speech encoder of the spoken language understanding model to obtain the acoustic coding features output by the speech encoder.

[0071] Step S130: Input the historical intent text into the text encoder of the spoken language understanding model to obtain the text encoding features output by the text encoder.

[0072] In this embodiment of the invention, the spoken language comprehension model is pre-trained, and its structure can be referred to as follows. Figure 3 It includes a speech encoder, a text encoder, and an intent recognition module. The training process of the spoken language understanding model can be referred to the following embodiment, which will not be elaborated here.

[0073] First, the speech information is input into the speech encoder of the spoken language understanding model. The speech encoder then extracts and encodes the acoustic features of the speech information, resulting in the acoustic coding features output by the speech encoder. These acoustic coding features contain rich semantic information.

[0074] Simultaneously, the historical intent text is input into the text encoder of the spoken language understanding model, and the text encoder is used to encode the historical intent text to obtain the text encoding features output by the text encoder.

[0075] Furthermore, when inputting historical intent text into the text encoder of the spoken language understanding model, the historical intent text can be segmented into words before input.

[0076] Step S140: Input the acoustic coding features and the text coding features into the intent recognition module of the spoken language understanding model to obtain the intent recognition result output by the intent recognition module for use in voice interaction.

[0077] After obtaining the acoustic coding features and text coding features, the acoustic coding features and text coding features are input into the intent recognition module of the spoken language understanding model. The intent recognition module is used to perform intent recognition and obtain the intent recognition result.

[0078] The intent recognition result includes the intent, slot, and interaction information. The slot can be represented as a key-value pair. For example, the intent recognition result might be: intent: play music, slot_singer: artist XX, slot_song: sunny day, interaction information: OK, play "Sunny Day" by artist XX. Here, intent represents the intent, slot_singer represents the artist slot, and slot_song represents the song title.

[0079] It should be noted that in this embodiment of the invention, the output of interactive information is achieved based on the capabilities of a large speech model. Traditional end-to-end models cannot directly achieve this capability through the model.

[0080] Furthermore, interactive voice is generated based on the interaction information for voice interaction, the type of which is confirmation, follow-up questioning, or notification. The specific execution process can be found in the following embodiments, and will not be elaborated upon here.

[0081] Furthermore, if the intent and slot indicate an executable instruction, the corresponding control interface can be invoked based on the intent and slot information to execute the instruction. The specific execution process can be found in the following embodiments, which will not be elaborated upon here.

[0082] The voice interaction method provided in this invention first acquires the user's input voice information and the user's historical intent text. Then, the voice information is input to the speech encoder of the spoken language understanding model to obtain acoustic coding features, and the historical intent text is input to the text encoder of the spoken language understanding model to obtain text coding features. The acoustic coding features and text coding features are then input to the intent recognition module of the spoken language understanding model to obtain the intent recognition result, which is used for voice interaction. This invention adopts an end-to-end single-model architecture, directly establishing a mapping from speech to semantics and outputting the final intent recognition result. It abandons the traditional cascaded scheme, eliminating the problem of ASR module errors propagating and amplifying to the NLU module, thus improving the accuracy of intent recognition results and shortening the processing link from voice input to intent output, thereby improving intent recognition efficiency. Furthermore, after acquiring the voice information, the historical intent text corresponding to the user's historical dialogues is acquired and encoded in parallel to obtain text coding features, which assist the speech coding features in determining the intent recognition result. By utilizing a multi-turn dialogue mechanism, even when voice input is incomplete, vague, or ambiguous, this invention can accurately obtain the user's true intent, further improving the accuracy of intent recognition results. In summary, this invention significantly improves the accuracy of intent recognition results during voice interaction, quickly and accurately obtaining the user's true intent.

[0083] Based on any of the above embodiments, the training steps of the spoken language comprehension model include: step S10 and step S20.

[0084] Step S10: Obtain the sample set.

[0085] Each sample in the sample set includes at least speech, intent label, and slot label, and may also include multi-turn dialogue-related annotations. Among them, multi-turn dialogue-related annotations include at least follow-up questions and may also include follow-up question triggering conditions for each round of dialogue.

[0086] In one implementation, a sample set is constructed based on real speech samples. Specifically, real speech containing intent is collected as real speech samples. Then, speech recognition processing is performed on the real speech samples to obtain real text. Subsequently, a large language model is used to annotate the real speech samples based on the real text to obtain the sample set.

[0087] In another implementation, a sample set is constructed based on the synthesized speech samples. Specifically, a text corpus is acquired, initial speech samples are synthesized based on the text corpus, and data augmentation processing is performed on the initial speech samples to obtain synthesized speech samples; then, a large language model is used to annotate the synthesized speech samples based on the text corpus to obtain a sample set.

[0088] In another embodiment, a sample set is constructed based on real speech samples and synthesized speech samples. Specifically, real speech samples are acquired; simultaneously, text corpora are acquired, initial speech samples are synthesized based on the text corpora, and data augmentation processing is performed on the initial speech samples to obtain synthesized speech samples; wherein, the data augmentation processing includes at least one of adding background noise, adding reverberation, adjusting speech rate, and adjusting pitch. Furthermore, speech recognition processing is performed on the real speech samples to obtain real text, and a large language model is used to annotate the real speech samples based on the real text to obtain a real sample set; simultaneously, a large language model is used to annotate the synthesized speech samples based on the text corpora to obtain a synthesized sample set, and the aforementioned real sample set and the aforementioned synthesized sample set constitute a sample set.

[0089] Step S20: Fine-tune the initial spoken language understanding model using the sample set to obtain the spoken language understanding model. The initial spoken language understanding model includes a pre-trained speech encoder, a pre-trained text encoder, and a pre-trained intent recognition module, wherein the pre-trained intent recognition module is a pre-trained large-scale speech model.

[0090] After constructing the sample set, the initial spoken language comprehension model is fine-tuned using the sample set to obtain the spoken language comprehension model.

[0091] The initial spoken language understanding model includes a speech encoder (denoted as the initial speech encoder), a text encoder (denoted as the initial text encoder), and an intent recognition module (denoted as the initial intent recognition module).

[0092] It should be noted that traditionally, there are two independent ASR modules and NLU modules, each with its own model structure and parameters, and the two modules are trained independently. However, the embodiments of this invention adopt an end-to-end single-model architecture, which is simpler in overall architecture, reduces the difficulty of development, debugging and maintenance, and improves training efficiency and training cost.

[0093] Furthermore, considering that in the existing technology, although some end-to-end spoken language understanding models attempt to process speech input directly, they still need to re-align speech-text modalities for specific scenarios, which not only increases the complexity and cost of model training, but also makes it difficult to fully utilize the rich speech-text association knowledge already learned by the pre-trained speech large model.

[0094] Therefore, in this embodiment of the invention, the initial intent recognition module can be a pre-trained intent recognition module, which is a pre-trained large-scale speech model, such as the Vita-Audio model or the GLM-4-voice (General Language Model-4-Voice) series model. Combined with a text encoder, during subsequent fine-tuning, the model is guided to focus on the intent and slot extraction target through cue word embedding. This approach avoids re-aligning text and speech modalities, fully utilizes the speech-text association capabilities already possessed by the pre-trained end-to-end speech model, simplifies the model training process, and reduces adaptation costs.

[0095] Furthermore, the initial speech encoder can be a pre-trained speech encoder, such as Hubert (hidden unit bidirectional encoder representations from transformers), WAV2Vec (wave to vector), or Whisper (a multi-task speech recognition model); the initial text encoder can be a pre-trained text encoder, such as Word2Vec (word to vector) or FastText. Since all parts of the initial model are pre-trained, fine-tuning can be performed using a small batch of samples when training the initial spoken language understanding model, which can greatly improve training efficiency and save training costs.

[0096] In one implementation, the sample set is divided into a training set and a validation set. Based on the training set and the validation set, the initial spoken language comprehension model is fine-tuned to obtain a pre-trained spoken language comprehension model.

[0097] In another implementation, the sample set is divided into a training set, a validation set, and a test set; the initial spoken language comprehension model is fine-tuned based on the training and validation sets to obtain a pre-trained spoken language comprehension model; the performance of the pre-trained spoken language comprehension model is evaluated based on the test set; if the evaluation passes, the pre-trained spoken language comprehension model is determined to be the spoken language comprehension model. The specific execution process can be found in the following embodiments, which will not be elaborated here.

[0098] Furthermore, when fine-tuning the initial spoken language understanding model, in one embodiment, the feature extraction layer of the intent recognition module can be frozen, and the semantic mapping layer of the pre-trained speech encoder, pre-trained text encoder, and pre-trained intent recognition module can be fine-tuned. In another embodiment, the feature extraction layers of the pre-trained speech encoder, pre-trained text encoder, and pre-trained intent recognition module are frozen, and only the semantic mapping layer of the pre-trained intent recognition module is fine-tuned. In the first embodiment, if the pre-trained speech encoder and pre-trained text encoder are also fine-tuned, overfitting to downstream tasks may occur. Therefore, compared to the first embodiment, fine-tuning only the semantic mapping layer of the pre-trained intent recognition module results in a better spoken language understanding model.

[0099] The voice interaction method provided in this invention adopts an end-to-end single-model architecture, which simplifies the overall architecture. Compared with the traditional cascaded architecture, it reduces the difficulty of development, debugging, and maintenance, and improves training efficiency and cost. Furthermore, since each module of the initial spoken language understanding model is pre-trained, the capabilities of the pre-trained speech encoder, pre-trained text encoder, and pre-trained large-scale speech model can be fully utilized. Only a small sample set is needed to fine-tune the spoken language understanding model, further improving training efficiency and cost.

[0100] Based on any of the above embodiments Figure 4 This is the second flowchart illustrating the voice interaction method provided by the present invention, as shown below. Figure 4 As shown, step S20 includes steps S21 and S22.

[0101] Step S21: Freeze the pre-trained speech encoder and the pre-trained text encoder, and freeze the feature extraction layer of the pre-trained intent recognition module.

[0102] During the initial training of the spoken language understanding model, a freeze-tuning approach was adopted. Specifically, the pre-trained speech encoder and pre-trained text encoder were frozen, as well as the feature extraction layer of the pre-trained intent recognition module.

[0103] Step S22: Using the sample set, fine-tune the semantic mapping layer of the pre-trained intent recognition module to obtain the spoken language understanding model.

[0104] Then, using the sample set, the semantic mapping layer of the pre-trained intent recognition module is fine-tuned to obtain the spoken language understanding model.

[0105] Specifically, based on the multi-turn dialogue-related standards and preset prompt word templates in the sample set, sample intent text is generated. The sample speech and sample intent text are then input into the initial spoken language understanding model for fine-tuning.

[0106] Furthermore, the loss function used in the fine-tuning of the spoken language understanding model is determined based on the predicted intent recognition results and the ground truth annotations, and the cross-entropy loss function can be used. Specifically:

[0107] ;

[0108] in, The cross-entropy loss represents the predicted intent recognition result p and the ground truth annotation q, where n represents the total number of words in the sample intent text. This represents the predicted intent recognition result corresponding to the i-th word segment. This represents the actual annotation corresponding to the i-th word segment.

[0109] The voice interaction method provided in this invention employs a freeze-fine-tuning approach. It freezes the underlying feature extraction layer of the initial spoken language understanding model (i.e., the pre-trained speech encoder, pre-trained text encoder, and the feature extraction layer of the pre-trained large-scale speech model). Then, it performs lightweight fine-tuning of the semantic mapping layer of the pre-trained large-scale speech model through cue word embedding. This guides the model to focus on intent and slot extraction targets, achieving accurate extraction of intent and slots in small-sample scenarios. By utilizing the pre-trained large-scale speech model's existing speech-text association capabilities, it avoids re-aligning text-speech modalities, simplifies the model training process, and reduces adaptation costs.

[0110] Based on any of the above embodiments Figure 5 This is the third flowchart of the voice interaction method provided by the present invention, as shown below. Figure 5 As shown, step S10 includes steps S11, S12, S13 and S14.

[0111] It should be noted that the execution order of steps S11 and steps S12-S13 is not important.

[0112] Step S11: Obtain real speech samples.

[0113] The performance of a model largely depends on the quantity and quality of its training data. Training a model solely with limited real-world recordings will result in insufficient generalization ability, making it difficult to handle diverse real-world usage scenarios. However, collecting large-scale, high-quality real-world speech data covering various speakers, accents, speech rates, and acoustic environments is extremely costly and time-consuming. Therefore, in this embodiment, based on collected real-world speech data as training data, speech synthesis technology is used to generate a massive amount of basic speech data. Data augmentation techniques are then used to simulate various complex real-world acoustic environments to efficiently construct high-quality training data. This helps improve the robustness and generalization ability of the spoken language understanding model, effectively reducing the false alarm rate and false wake-up rate in complex environments.

[0114] The process of acquiring real speech samples involves collecting spoken speech data containing intent from a large number of different speakers (who can be recruited volunteers) in various acoustic environments. The number of speakers should be determined based on the specific application scenario, and speakers can cover different ages, genders, and regions to obtain various types of speech data. Acoustic environments can include, but are not limited to: quiet indoor environments, noisy office environments, outdoor environments such as streets, and open hall environments with high echoes.

[0115] Step S12: Obtain text corpus and synthesize initial speech samples based on the text corpus.

[0116] The text corpus consists of text containing intent. It can be generated using existing large language models, such as Gemini 1.5Pro, Claude 3, Llama series models, SparkDesk, DeepSeek, and Wenxin Yiyan.

[0117] Then, speech samples are synthesized based on the text corpus, and these are recorded as initial speech samples. Specifically, a speech synthesis model can be used to synthesize speech from the text corpus to obtain the initial speech samples. For example, the speech synthesis model can be a TTS (Text-to-Speech) model, which, by inputting text corpus and selecting several different virtual speaker timbres (including male, female, and child voices), generates several speech samples with standard pronunciation but different timbres, which are recorded as initial speech samples.

[0118] Step S13: Perform data augmentation processing on the initial speech sample to obtain a synthesized speech sample. The data augmentation processing includes at least one of the following: adding background noise, adding reverberation, adjusting speech rate, and adjusting pitch.

[0119] Since the synthesized speech samples are all pure human voices, further data augmentation processing is needed to address the complexity of the environment. Specifically, data augmentation processing is performed on the initial speech samples to obtain the data-augmented speech samples, denoted as the synthesized speech samples. This data augmentation processing includes at least one of the following: adding background noise, adding reverberation, adjusting speech rate, and adjusting pitch. Through this data augmentation processing, the model's tolerance to various interference factors in real-world application scenarios is improved, enabling it to maintain stable and accurate recognition performance in noisy environments and with different pronunciation habits, thus enhancing the model's applicability and reliability in complex acoustic environments.

[0120] Step S14: Construct the sample set based on the real speech samples and the synthesized speech samples.

[0121] After obtaining real speech samples and synthesized speech samples, a sample set is constructed based on the real speech samples and synthesized speech samples.

[0122] Specifically, speech recognition processing is performed on real speech samples to obtain real text. Then, a large language model is used to annotate the real speech samples based on the real text, resulting in a real sample set. Simultaneously, the large language model is used to annotate synthesized speech samples based on the text corpus, resulting in a synthesized sample set. Through this method, the constructed sample set includes both the real and synthesized sample sets. The specific execution process can be found in the following embodiment, which will not be elaborated upon here.

[0123] The voice interaction method provided in this invention collects real voice data as real voice samples, generates a massive amount of basic voice data using speech synthesis technology, and then simulates various complex real-world acoustic environments through data augmentation to generate synthesized voice data as synthesized voice samples. A sample set is then constructed based on the aforementioned real and synthesized voice samples. This approach efficiently constructs a large and highly diverse high-quality training dataset, laying the foundation for training a highly accurate and robust spoken language understanding model.

[0124] Based on any of the above embodiments, step S14 includes: step S14-1, step S14-2, and step S14-3.

[0125] It should be noted that the execution order of steps S14-1 to S14-2 and step S14-3 is not important.

[0126] Step S14-1: Perform speech recognition processing on the real speech sample to obtain the real text.

[0127] Step S14-2: Using a large language model, the real speech samples are labeled according to the real text to obtain a real sample set.

[0128] Real speech samples are processed using speech recognition to convert them into text, which is then recorded as real text. Next, a large language model is used to annotate the real speech samples based on the real text, resulting in a real sample set. The annotations include, but are not limited to: intent, slots, and multi-turn dialogue-related annotations. Multi-turn dialogue-related annotations include, but are not limited to, follow-up questions and the triggering conditions for follow-up questions in each turn of the dialogue. The large language model can be, but is not limited to: Gemini 1.5 Pro model, Claude 3 model, Llama series models, SparkDesk, DeepSeek model, and Wenxin Yiyan, etc.

[0129] Furthermore, after labeling real speech samples using a large language model, manual verification can be performed. Specifically, the labeled real text is sent to the user terminal for experts to verify through the user terminal, and the verification results are then fed back, which constitutes the real sample set.

[0130] Step S14-3: Using the large language model, the synthesized speech samples are labeled according to the text corpus to obtain a synthesized sample set.

[0131] The sample set includes the real sample set and the synthetic sample set.

[0132] Meanwhile, using a large language model, the synthesized speech samples are labeled based on the text corpus to obtain a synthesized sample set.

[0133] Furthermore, after annotating the synthesized speech samples using a large language model, manual verification can be performed. Specifically, the annotated synthesized text is sent to the user terminal for experts to verify through the user terminal, and the verification results are then fed back, which constitutes the synthesized sample set.

[0134] The real and synthetic sample sets obtained through the above methods will be used as the sample sets for training the spoken language comprehension model.

[0135] The voice interaction method provided in this invention can greatly improve annotation efficiency and reduce annotation costs by using a large language model to annotate real and synthetic voice samples to construct a sample set.

[0136] Based on any of the above embodiments, step S20 further includes: step S23, step S24, step S25, step S26, step S27 and step S28.

[0137] Step S23: Divide the sample set into a training set, a validation set, and a test set.

[0138] The sample set is divided into a training set, a validation set, and a test set. The division can be done according to a preset ratio, which can be set in advance, for example, 70%:15%:15% or 80%:10%:10%. There is no specific limitation here.

[0139] The training set is used to learn and update the initial parameters of the spoken language comprehension model, the validation set is used to monitor model performance, adjust hyperparameters, and perform early stopping during fine-tuning, and the test set is used to finally evaluate the model's performance.

[0140] Step S24: Iteratively fine-tune the initial spoken language comprehension model based on the training set. In each round of fine-tuning, the parameters of the current spoken language comprehension model are adjusted through the backpropagation algorithm.

[0141] Step S25: After each round of fine-tuning, the adjusted current spoken language understanding model is validated based on the validation set, and the validation loss is calculated.

[0142] The initial spoken language comprehension model is iteratively fine-tuned using the training set. In each round of fine-tuning, the parameters of the current spoken language comprehension model are adjusted using the backpropagation algorithm. After each round of fine-tuning, the adjusted spoken language comprehension model is validated using the validation set, and the validation loss is calculated. The initial learning rate can be set to 1e. -4 (i.e., 0.0001), which can be dynamically adjusted based on the verification loss.

[0143] Specifically, in the first round of fine-tuning, the initial spoken language comprehension model is first fine-tuned based on the training set. The parameters of the initial spoken language comprehension model are updated using the backpropagation algorithm to obtain the first trained model, denoted as the first intermediate model. Then, the first intermediate model is validated based on the validation set, and the validation loss is calculated, denoted as the first validation loss. Next, in the second round of fine-tuning, the first intermediate model is fine-tuned based on the training set, and the parameters of the first intermediate model are updated using the backpropagation algorithm to obtain the second trained model, denoted as the second intermediate model. Then, the second intermediate model is validated based on the validation set, and the validation loss is calculated, denoted as the second validation loss. This process continues for subsequent rounds of fine-tuning and validation.

[0144] Step S26: When the number of fine-tuning rounds reaches the preset number of rounds, or when the preset early stopping condition is met according to the verification loss, the training ends, and the spoken language understanding model corresponding to the minimum value in the verification loss is determined as the pre-trained spoken language understanding model.

[0145] When the preset number of fine-tuning rounds is reached, training ends, and the spoken language understanding model corresponding to the minimum value in the validation loss is identified as the pre-trained spoken language understanding model. The preset number of training rounds is a pre-set maximum number of training rounds, for example, 20.

[0146] Alternatively, training can be terminated when a preset early stopping condition is met based on the validation loss. The spoken language understanding model corresponding to the minimum value of the validation loss is then identified as the pre-trained spoken language understanding model. The preset early stopping condition is a pre-defined condition, such as the validation loss not decreasing for more than two consecutive rounds. It should be noted that if the validation loss decreases continuously, it indicates that the model is learning well and its generalization ability is improving. If the validation loss stops decreasing for two consecutive rounds, it indicates that the model may be overfitting. In this case, training can be stopped early without completing all rounds, thus significantly reducing unnecessary training time and computational resources.

[0147] For example, after the first round of fine-tuning, the first validation loss is calculated to be 0.86; after the second round, the second validation loss is calculated to be 0.66; after the third round, the third validation loss is calculated to be 0.50; after the fourth round, the fourth validation loss is calculated to be 0.52; after the fifth round, the fifth validation loss is calculated to be 0.52; and after the sixth round, the sixth validation loss is calculated to be 0.53. After the sixth round of fine-tuning, the validation loss has not decreased for three rounds, thus satisfying the aforementioned pre-set early stopping condition. Therefore, iterative fine-tuning is stopped, and the model obtained after the third round of fine-tuning is used as the initially trained spoken language understanding model.

[0148] By introducing a validation set for real-time monitoring and tuning during the training phase, the training process can be effectively guided, preventing performance degradation caused by overfitting the model to the training set. This ensures that the final spoken language understanding model has strong generalization performance. Furthermore, stopping iteration when the early stopping condition is met optimizes training efficiency and saves computational resources.

[0149] Step S27: Evaluate the performance of the pre-trained spoken language comprehension model based on the test set.

[0150] After the initial training is completed, the performance of the initially trained spoken language comprehension model is evaluated based on the test set.

[0151] When conducting performance evaluation, the performance metrics used include, but are not limited to, false wake-up rate and missed wake-up rate. False wake-up rate refers to the probability that a non-target speaker is mistakenly woken up when the speaker says the wake word, while missed wake-up rate refers to the probability that the system rejects the speaker when the target speaker says the wake word.

[0152] The evaluation is considered passed if the performance indicators meet the preset evaluation criteria; otherwise, it is considered failed. The preset evaluation criteria can be set according to actual needs and are not specifically limited here.

[0153] Step S28: When the evaluation is passed, the pre-trained spoken language comprehension model is determined to be the spoken language comprehension model.

[0154] Once the evaluation is passed, the pre-trained spoken language understanding model can be deployed to electronic devices as the final model for performing voice interaction tasks.

[0155] Furthermore, if the evaluation fails, in one embodiment, the sample set can be adjusted based on the performance evaluation results, and the initial spoken language understanding model can be fine-tuned using the adjusted sample set to obtain the spoken language understanding model. In another embodiment, the structure of the initial spoken language understanding model can be adjusted based on the performance evaluation results, for example, by increasing the depth or width of the speech encoder to improve its representational ability. Then, the adjusted initial spoken language understanding model can be fine-tuned using the sample set to obtain the spoken language understanding model.

[0156] The voice interaction method provided in this invention divides the sample set into a training set, a validation set, and a test set. While fine-tuning the model using the training set, the training process is monitored and optimized in real time using an independent validation set. This effectively prevents overfitting and ensures the generalization ability of the final model. Simultaneously, performance evaluation of the pre-trained model using an independent test set ensures the model's reliability.

[0157] Based on any of the above embodiments, the step "obtaining the user's historical intent text" includes: step S111 and step S112.

[0158] Step S111: Obtain the user's historical intent recognition results.

[0159] A historical dialogue database is pre-built to store the voice input for each round of dialogue, the recognized intent results (including intent and slot), interaction information, etc.

[0160] When obtaining historical intent text, the user's historical intent recognition results can be obtained from the historical dialogue database first.

[0161] For example, in the first round of dialogue, the user says, "Play a song by artist XX." The corresponding intent recognition result is: Intent = [Play music], Slot = [{Artist: artist XX}, {Song title: empty}]. At this point, the system will ask: Which song would you like to listen to?

[0162] Correspondingly, the intent recognition result of this round of dialogue will be stored as: intent = play music, the artist slot is XX artist, and the song title slot is empty. At the same time, the interaction information "Which song do you want to listen to?" will be stored.

[0163] Step S112: Generate the historical intent text based on the historical intent recognition result and the preset prompt word template; wherein, the preset prompt word template includes the splicing relationship and position of intent, slot and interaction information.

[0164] Then, based on the historical intent recognition results and the preset prompt word template, historical intent text is generated; wherein, the preset prompt word template includes the splicing relationship and position of intent, slot and interaction information.

[0165] For example, the preset prompt template can be: "Extract intent and slot from speech; if information is incomplete, prepare follow-up questions: intent = [intent type], slot = [{slot name: slot value}], follow-up questions = [interaction information]". Assume the intent recognition result of the previous round of dialogue (i.e., the historical intent recognition result) is: intent = play music, the artist slot is XX artist, the song title slot is empty, and the interaction information is "Which song do you want to listen to?". In this case, the generated historical intent text can be: "Extract intent and slot from speech; if information is incomplete, prepare follow-up questions: intent = [play music], slot = [{artist: XX artist}, {song title: empty}], follow-up questions = [Which song do you want to listen to?]". It should be noted that if no historical intent recognition result is obtained, meaning the user's current speech information is from the first round of dialogue, then the historical intent text is the preset prompt template, where the intent type, slot name, slot value, and interaction information are all empty.

[0166] The voice interaction method provided in this invention obtains the user's historical intent recognition results. Then, based on the historical intent recognition results and a preset prompt word template, historical intent text is generated to generate text encoding features. The text encoding features can assist the voice encoding features in determining the intent recognition results. By obtaining historical intent text and utilizing the multi-turn dialogue mechanism, even when the voice input is incomplete, ambiguous, or unclear, the user's true intent can be accurately obtained, further improving the accuracy of intent recognition results.

[0167] Based on any of the above embodiments, the intent recognition result includes interactive information. After the step "inputting the acoustic coding features and the text coding features into the intent recognition module of the spoken language understanding model to obtain the intent recognition result output by the intent recognition module", the method further includes: step S150.

[0168] Step S150: Generate interactive voice based on the interactive information to conduct voice interaction, wherein the type of voice interaction is confirmation, follow-up question, or notification.

[0169] The intent recognition result includes the intent, slot, and interaction information. Among them, the slot can be represented in the form of key-value pairs.

[0170] After obtaining the intent recognition result, interactive voice is generated based on the interaction information to conduct voice interaction. The types of voice interaction are confirmation, follow-up questioning, or notification.

[0171] Furthermore, if the intent and slot indicate an executable instruction, the corresponding control interface can be invoked based on the intent and slot information to execute the instruction.

[0172] For example, if the intent is to play music, the artist slot is "XX artist", but the song title slot is incomplete, and the interaction information is "Which song do you want to listen to?", then a follow-up question-type interactive voice message "Which song do you want to listen to?" can be generated and played to conduct voice interaction.

[0173] For example, in a multi-turn dialogue, the intent is identified as booking tickets. The departure location slot is Beijing, the arrival location slot is Shanghai, and the date slot is tomorrow and the day after tomorrow. Due to the conflict in the slot information, the interaction information is "Would you like to book tickets for tomorrow or the day after tomorrow?". At this time, a confirmation-type interactive voice message "Would you like to book tickets for tomorrow or the day after tomorrow?" can be generated and played to conduct voice interaction.

[0174] For example, if the intent is "Play music," the artist slot is "Singer XX," the song title slot is "Sunny Day," and the interaction message is "Okay, playing 'Sunny Day' by artist XX," then a notification-type interactive voice message can be generated and played to facilitate voice interaction. Furthermore, since the current intent and slot indicate an executable command, the music API (Application Programming Interface) can be invoked simultaneously to play "Sunny Day" by artist XX.

[0175] It should be noted that in this embodiment of the invention, the interaction (including follow-up questions, confirmations, or notifications) is implemented based on the capabilities of the large voice model. Traditional end-to-end models cannot directly implement this capability through the model.

[0176] The voice interaction method provided in this invention utilizes the interactive information included in the intent recognition result output by the aforementioned spoken language understanding model. Different types of interactive voices, such as follow-up questions, confirmations, or notifications, can be directly generated based on this interactive information for voice interaction. Compared to traditional spoken language understanding models, this invention provides a more efficient and intelligent voice interaction experience.

[0177] The voice interaction device provided by the present invention will be described below. The voice interaction device described below can be referred to in correspondence with the voice interaction method described above.

[0178] Figure 6 This is a structural schematic diagram of the voice interaction device provided by the present invention, as shown below. Figure 6 As shown, the device includes an acquisition module 610, a speech encoding module 620, a text encoding module 630, and an intent recognition module 640; wherein:

[0179] The acquisition module 610 is used to acquire the user's input voice information and acquire the user's historical intent text;

[0180] The speech encoding module 620 is used to input the speech information into the speech encoder of the spoken language understanding model to obtain the acoustic coding features output by the speech encoder.

[0181] The text encoding module 630 is used to input the historical intent text into the text encoder of the spoken language understanding model to obtain the text encoding features output by the text encoder;

[0182] The intent recognition module 640 is used to input the acoustic coding features and the text coding features into the intent recognition module of the spoken language understanding model to obtain the intent recognition result output by the intent recognition module for use in voice interaction.

[0183] The voice interaction device provided in this embodiment of the invention first acquires the user's input voice information and the user's historical intent text. Then, the voice information is input to the speech encoder of the spoken language understanding model to obtain acoustic coding features, and the historical intent text is input to the text encoder of the spoken language understanding model to obtain text coding features. The acoustic coding features and text coding features are then input to the intent recognition module of the spoken language understanding model to obtain the intent recognition result, which is used for voice interaction. This embodiment of the invention adopts an end-to-end single-model architecture, directly establishing a mapping from speech to semantics and outputting the final intent recognition result. It abandons the traditional cascaded scheme, eliminating the problem of ASR module errors propagating and amplifying to the NLU module, thus improving the accuracy of intent recognition results and shortening the processing link from voice input to intent output, thereby improving intent recognition efficiency. Furthermore, after acquiring the voice information, the historical intent text corresponding to the user's historical dialogue is acquired and encoded in parallel to obtain text coding features, which assist the speech coding features in determining the intent recognition result. By utilizing a multi-turn dialogue mechanism, even when voice input is incomplete, vague, or ambiguous, this invention can accurately obtain the user's true intent, further improving the accuracy of intent recognition results. In summary, this invention significantly improves the accuracy of intent recognition results during voice interaction, quickly and accurately obtaining the user's true intent.

[0184] It should be noted that the voice interaction device provided in this embodiment of the invention can implement all the method steps implemented in the above-mentioned voice interaction method embodiment and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail.

[0185] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7As shown, the electronic device may include a processor 710, a communications interface 720, a memory 730, and a communication bus 740, wherein the processor 710, communications interface 720, and memory 730 communicate with each other via the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute a voice interaction method, which includes: acquiring user-inputted voice information and acquiring the user's historical intent text; inputting the voice information into a speech encoder of a spoken language understanding model to obtain acoustic coding features output by the speech encoder; inputting the historical intent text into a text encoder of the spoken language understanding model to obtain text coding features output by the text encoder; inputting the acoustic coding features and the text coding features into an intent recognition module of the spoken language understanding model to obtain an intent recognition result output by the intent recognition module; and executing a corresponding interactive operation based on the intent recognition result.

[0186] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0187] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the voice interaction methods provided by the above methods.

[0188] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the voice interaction methods provided by the methods described above.

[0189] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0190] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0191] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A voice interaction method, characterized in that, include: The system acquires user-input voice information and obtains the user's historical intent recognition results. Based on the historical intent recognition results and a preset prompt word template, it generates historical intent text. The preset prompt word template includes the splicing relationship and position of intent, slot, and interaction information. The speech information is input into the speech encoder of the spoken language understanding model to obtain the acoustic coding features output by the speech encoder. The historical intent text is input into the text encoder of the spoken language understanding model to obtain the text encoding features output by the text encoder; the text encoder and the speech encoder are parallel. The acoustic coding features and the text coding features are input into the intent recognition module of the spoken language understanding model to obtain the intent recognition result output by the intent recognition module. The intent recognition result includes interaction information for use in voice interaction. The spoken language comprehension model was trained in the following manner: Obtain a sample set and an initial spoken language understanding model; wherein the initial spoken language understanding model includes a pre-trained speech encoder, a pre-trained text encoder, and a pre-trained intent recognition module, and the pre-trained intent recognition module is a pre-trained large speech model; Freeze the pre-trained speech encoder and the pre-trained text encoder, and freeze the feature extraction layer of the pre-trained intent recognition module; The semantic mapping layer of the pre-trained intent recognition module is fine-tuned using the sample set to obtain the spoken language understanding model.

2. The voice interaction method according to claim 1, characterized in that, The acquisition of the sample set includes: Obtain real voice samples; Acquire text corpus, and synthesize initial speech samples based on the text corpus; The initial speech sample is subjected to data augmentation processing to obtain a synthesized speech sample; wherein, the data augmentation processing includes at least one of adding background noise, adding reverberation, adjusting speech rate, and adjusting pitch. The sample set is constructed based on the real speech samples and the synthesized speech samples.

3. The voice interaction method according to claim 2, characterized in that, The step of constructing the sample set based on the real speech samples and the synthesized speech samples includes: The real speech samples are subjected to speech recognition processing to obtain real text; Using a large language model, the real speech samples are labeled based on the real text to obtain a real sample set; Using the large language model, the synthesized speech samples are labeled according to the text corpus to obtain a synthesized sample set; The sample set includes the real sample set and the synthetic sample set.

4. The voice interaction method according to claim 1, characterized in that, The step of fine-tuning the initial spoken language comprehension model using the sample set to obtain the spoken language comprehension model further includes: The sample set is divided into a training set, a validation set, and a test set; The initial spoken language comprehension model is iteratively fine-tuned based on the training set. In each round of fine-tuning, the parameters of the current spoken language comprehension model are adjusted through the backpropagation algorithm. After each round of fine-tuning, the adjusted current spoken language understanding model is validated based on the validation set, and the validation loss is calculated. When the preset number of fine-tuning rounds is reached, or when the preset early stopping condition is met according to the verification loss, the training ends, and the spoken language understanding model corresponding to the minimum value in the verification loss is determined as the pre-trained spoken language understanding model. The performance of the pre-trained spoken language comprehension model is evaluated based on the test set. Upon successful evaluation, the pre-trained spoken language comprehension model is identified as the spoken language comprehension model.

5. The voice interaction method according to any one of claims 1 to 4, characterized in that, After inputting the acoustic coding features and the text coding features into the intent recognition module of the spoken language understanding model and obtaining the intent recognition result output by the intent recognition module, the method further includes: Interactive voice is generated based on the interactive information to conduct voice interaction, and the type of voice interaction is confirmation, follow-up question, or notification.

6. A voice interaction device, characterized in that, include: The acquisition module is used to acquire the user's input voice information and the user's historical intent recognition results, and generate historical intent text based on the historical intent recognition results and the preset prompt word template; wherein, the preset prompt word template includes the splicing relationship and position of intent, slot and interaction information; The speech encoding module is used to input the speech information into the speech encoder of the spoken language understanding model to obtain the acoustic coding features output by the speech encoder; A text encoding module is used to input the historical intent text into the text encoder of the spoken language understanding model to obtain the text encoding features output by the text encoder; the text encoder and the speech encoder are parallel. An intent recognition module is used to input the acoustic coding features and the text coding features into the intent recognition module of the spoken language understanding model to obtain the intent recognition result output by the intent recognition module. The intent recognition result includes interaction information for use in voice interaction. The voice interaction device further includes a module for: Obtain a sample set and an initial spoken language understanding model; wherein the initial spoken language understanding model includes a pre-trained speech encoder, a pre-trained text encoder, and a pre-trained intent recognition module, and the pre-trained intent recognition module is a pre-trained large speech model; Freeze the pre-trained speech encoder and the pre-trained text encoder, and freeze the feature extraction layer of the pre-trained intent recognition module; The semantic mapping layer of the pre-trained intent recognition module is fine-tuned using the sample set to obtain the spoken language understanding model.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the voice interaction method as described in any one of claims 1 to 5.

Images