Voice interaction method, device, equipment, storage medium and intelligent terminal

Abstract

The present disclosure relates to a voice interaction method, device, equipment, storage medium and intelligent terminal. The method comprises: obtaining an audio segment; inputting an intent recognition prompt word and the audio segment into an intent recognition model to obtain conversation intent information output by the intent recognition model; wherein the conversation intent information represents the audio segment and a conversation intent of a voice conversation system; and processing the audio segment according to the conversation intent information. According to the embodiment of the present disclosure, the intent recognition model determines the intent of the audio segment to have a conversation with the voice conversation system, so as to realize the determination of the conversation intent information through a shorter link, and realize the natural voice interaction without the need of a wake-up word and a fixed sentence trigger, the time required for response is shorter, the perception of the general instruction is realized, and the natural and smooth conversation is realized.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a voice interaction method, apparatus, device, storage medium, and smart terminal. Background Technology

[0002] With the development of voice technology, voice interaction is becoming increasingly common. Related technologies require first performing speech recognition on the user's voice to obtain recognized text, then detecting wake words or fixed phrases in the recognized text, and finally triggering voice interaction based on the detection results. However, this interaction method has a long interaction chain, requiring a long response time and only allowing passive responses, thus disrupting the natural flow of the conversation. Summary of the Invention

[0003] To address the aforementioned technical problems, this disclosure provides a voice interaction method, apparatus, device, storage medium, and smart terminal.

[0004] In a first aspect, this disclosure provides a voice interaction method, the method comprising: Obtain audio segments; The intent recognition prompt and the audio segment are input into the intent recognition model to obtain the dialogue intent information output by the intent recognition model; wherein, the dialogue intent information represents the dialogue intent between the audio segment and the voice dialogue system; The audio segment is processed based on the dialogue intent information.

[0005] Secondly, this disclosure provides a voice interaction device, the device comprising: The acquisition module is used to acquire audio segments; The model module is used to input the intent recognition prompt and the audio segment into the intent recognition model to obtain the dialogue intent information output by the intent recognition model; wherein, the dialogue intent information represents the dialogue intent between the audio segment and the voice dialogue system; The processing module is used to process the audio segment according to the dialogue intent information.

[0006] Thirdly, embodiments of this disclosure also provide an electronic device, including: processor; Memory, used to store executable instructions; The processor is used to read executable instructions from memory and execute the executable instructions to implement the voice interaction method of the first aspect mentioned above.

[0007] Fourthly, embodiments of this disclosure also provide a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to implement the voice interaction method of the first aspect described above.

[0008] The technical solutions provided in this disclosure have the following advantages compared with the prior art: A voice interaction method, apparatus, device and storage medium according to an embodiment of this disclosure, wherein the method includes: obtaining an audio segment; inputting an intent recognition prompt word and the audio segment into an intent recognition model to obtain dialogue intent information output by the intent recognition model; wherein the dialogue intent information represents the dialogue intent between the audio segment and the voice dialogue system; and processing the audio segment according to the dialogue intent information.

[0009] By adopting the above technical solution, audio segments and intent recognition prompts are input into the intent recognition model. The intent recognition model directly obtains dialogue intent information that represents the dialogue intent between the audio segment and the voice dialogue system. Based on the dialogue intent information, the audio segment is processed for subsequent dialogue. The intent recognition model determines the intent of the audio segment to engage in dialogue with the voice dialogue system. The determination of dialogue intent information is achieved with a shorter link. Furthermore, it enables natural voice interaction without wake words or fixed sentence patterns, requires a shorter response time, and achieves the perception of generalized commands, resulting in natural and fluent dialogue. Attached Figure Description

[0010] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0011] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 A flowchart illustrating a voice interaction method provided in an embodiment of this disclosure; Figure 2 A schematic diagram of the architecture of a voice interaction method provided in an embodiment of this disclosure; Figure 3 A flowchart illustrating yet another voice interaction method provided in this disclosure embodiment; Figure 4 This is a schematic diagram of the structure of a voice interaction device provided in an embodiment of the present disclosure; Figure 5 This is a schematic diagram of the hardware circuit structure of a voice interaction device provided in an embodiment of the present disclosure. Detailed Implementation

[0013] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0014] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0015] For voice assistants in scenarios such as in-vehicle systems and embody intelligent robots, users need to say a specific wake-up word or use a limited set of wake-up-free phrases to trigger the voice assistant. This interaction method fails to recognize the user's generalized commands or whether the user has an intention to interact with the voice assistant. Each interaction requires the user to say a wake-up word or fulfill a fixed phrase, failing to generalize to the user's diverse natural expressions. This resembles walkie-talkie communication, disrupting the natural flow of the conversation. The voice assistant can only respond passively, unable to proactively care for or remind the user, lacking a sense of companionship, and struggling to engage in natural and fluent dialogue with the user.

[0016] To address the problems in the related technologies, embodiments of this disclosure provide a voice interaction method, apparatus, device, storage medium, and smart terminal.

[0017] Figure 1 This is a flowchart illustrating a voice interaction method provided in an embodiment of this disclosure. The method can be executed by a voice interaction device, which can be implemented using software and / or hardware, and is generally integrated into an electronic device. This voice interaction device and / or electronic device can be configured in a vehicle or an embodied intelligent robot to realize voice interaction in in-vehicle or embodied intelligent robot scenarios. Figure 1 As shown, the method includes: Step 101: Obtain the audio segment.

[0018] The audio segment can be the audio to be identified, and can be an audio block divided from the overall audio during a round of voice interaction. This embodiment does not limit the segmentation conditions of the audio segment; for example, the audio segment can be divided based on the duration of pauses in the speech.

[0019] In this embodiment of the disclosure, with the user's authorization, the audio acquisition device can acquire audio, which records the user's voice, and the voice interaction device can acquire the acquired audio segment.

[0020] In some embodiments of this disclosure, obtaining an audio segment includes: obtaining an audio signal; if the audio signal meets the segmentation conditions, determining an audio segment based on the portion of the audio signal that has not been processed by the intent recognition model.

[0021] The audio signal can be a signal obtained by converting ambient sound, which can be understood as accumulated speech data. The segmentation condition can be a pre-set condition for dividing the currently acquired, unprocessed audio signal into audio segments; this embodiment does not limit the segmentation condition. The intent recognition model can be a neural network model used to determine dialogue intent information; this intent recognition model can be a Multimodal Large Language Model (MLLM). This embodiment does not limit the parameters of the intent recognition model; for example, it can be 1.5B.

[0022] In this embodiment, the Voice Activity Detection (VAD) module can continuously detect multiple audio signals. If the audio signal meets the segmentation conditions, the part of the audio signal that has not been processed by the intent recognition model is extracted, and the part of the signal is preprocessed such as front-end acoustic processing to obtain an audio segment.

[0023] In some embodiments of this disclosure, the audio signal satisfies segmentation conditions, including: the unprocessed duration of the audio signal reaches a first duration threshold, or the silence duration of the audio signal reaches a second duration threshold.

[0024] The unprocessed duration can be the length of the portion of the audio signal that has not been processed by the intent recognition model. The first duration threshold can be a threshold set for the unprocessed duration; this embodiment does not limit the specific value of the first duration threshold. This first duration threshold can be associated with the broadcasting state, which indicates whether the voice dialogue system is performing voice broadcasting, i.e., whether the voice dialogue system is responding to the user via voice. The broadcasting state includes a broadcasting in progress state and a silent state. The broadcasting in progress state indicates that the voice dialogue system is currently performing voice broadcasting. The silent state indicates that the voice dialogue system is not currently performing voice broadcasting. The first duration threshold corresponding to the broadcasting in progress state can be less than the first duration threshold corresponding to the silent state, thereby allowing for a shorter interval to determine whether to interrupt the voice broadcast. For example, the first duration threshold corresponding to the broadcasting in progress state can be 0.8 seconds, and the first duration threshold corresponding to the silent state can be 4.8 seconds.

[0025] The silence duration can be the latest silence duration, or the duration during which no human voice is detected, as determined by silence detection. The second duration threshold can be a threshold set for the silence duration of the audio signal. This embodiment does not limit the second duration threshold; for example, the second duration threshold can be 100 milliseconds.

[0026] In this embodiment, to meet real-time requirements, the following strategy for determining audio segments is adopted. This real-time requirement can be an end-to-end response time of less than 300 milliseconds. In this embodiment, the voice interaction device can determine the currently accumulated audio signal as an audio segment every first duration threshold. Alternatively, it can determine the currently accumulated audio signal as an audio segment when a first silence signal is reached. This first silence signal can indicate that the silence duration of the audio signal has reached a second duration threshold. Therefore, based on the first duration threshold, the audio signal can be periodically divided into audio segments, achieving streaming processing of the audio signal. Furthermore, audio segmentation is also performed when a speech pause occurs, enabling timely processing of the audio segment before the speech pause.

[0027] Step 102: Input the intent recognition prompt and audio segment into the intent recognition model to obtain the dialogue intent information output by the intent recognition model; wherein, the dialogue intent information represents the dialogue intent between the audio segment and the voice dialogue system.

[0028] The intent recognition prompt can be a system prompt, which allows the diagram recognition model to determine the user's dialogue intent based on the input and output corresponding dialogue intent information. This intent recognition prompt can include role definitions and task instructions. Dialogue intent information can be understood as a state signal for dialogue control in full-duplex communication. Dialogue intent can be understood as a phased representation of the user's dialogue with the voice dialogue system. This dialogue intent information enables full-duplex communication between the user and the voice dialogue system. Based on this dialogue intent information, full-duplex dialogue state control is implemented, which can be a dialogue management method for simultaneous voice interaction between the user and the voice dialogue system. This dialogue intent information can be recorded in the form of tokens. This dialogue intent information can include one or more of the following: vehicle-mounted dialogue identifier, rejection identifier, waiting identifier, chat dialogue identifier, and interruption identifier. This dialogue intent identifier enables dynamic response control of the voice dialogue system.

[0029] Full-duplex can be understood as allowing users and the voice dialogue system to communicate bidirectionally simultaneously during voice interaction, supporting functions such as interruption at any time, continuous listening, and rejection. This dialogue intent information can be used to control the voice dialogue system's voice playback, audio processing, and other functions.

[0030] The "Vehicle-to-Vehicle Dialogue" indicator indicates that the user has uttered a complete vehicle-related command and is interacting with the voice dialogue system, in which case the system will respond. The "Rejection" indicator indicates that the user is not interacting with the voice dialogue system but is talking to themselves or chatting with someone else; in this case, the system will not respond. The "Waiting" indicator indicates that the user has not finished speaking. The system will continue listening and wait for the user to finish before deciding whether to respond. The "Interruption" indicator indicates that the user intentionally interrupts the voice dialogue system's prompts and steers the conversation in the wrong direction; the system should stop speaking and switch to listening mode. This interruption can be triggered by one or more of the following: denial, expression of dissatisfaction, further questioning, or changing the topic. The "Chatting" indicator indicates that the user is uttering commands other than vehicle-related commands and is interacting with the voice dialogue system. These commands can include question-and-answer commands, casual conversation commands, etc.; in this case, the system will respond.

[0031] In this embodiment, the voice interaction device can encode audio segments using an audio encoder (Whisper Encoder) to obtain speech signal features, which can be a sequence of word units in the audio. These speech signal features and intent recognition prompts are then input into an intent recognition model, which determines the dialogue intent information. This embodiment does not limit the parameter size of the audio encoder; for example, it can be 640M.

[0032] In some embodiments of this disclosure, the intent recognition prompt and audio segment are input into the intent recognition model to obtain the dialogue intent information output by the intent recognition model, including: determining the dialogue context and judging the prompt; wherein, the judging prompt is used to prompt the intent recognition model to perform the intent judgment corresponding to the broadcast state, and the broadcast state represents the state of the voice broadcast of the voice dialogue system; the intent recognition prompt, audio segment, dialogue context and judging prompt are input into the intent recognition model to obtain the dialogue intent information output by the intent recognition model.

[0033] The context can be the recognized text corresponding to multiple audio segments preceding the current audio segment. For example, the context can be the historical query text of the user in the corresponding audio region. The context may not include the response from the voice dialogue system, and can retain up to three rounds of historical query text. The user in the audio region can refer to the user's acoustic region.

[0034] In some embodiments of this disclosure, the broadcast state includes a broadcasting state and a silent state. A first judgment prompt word corresponding to the broadcasting state is used by the schematic diagram recognition model to determine whether the audio segment corresponds to an interruption intent; a second judgment prompt word corresponding to the silent state is used by the schematic diagram recognition model to determine whether the audio segment corresponds to other intents, which are dialogue intents other than the interruption intent. Other intents may include vehicle-to-everything (V2X) dialogue intents, rejection intents, waiting intents, and chat dialogue intents. The dialogue intent information corresponding to the first judgment prompt word includes at least one of an interruption flag and a rejection flag; the dialogue intent information corresponding to the second judgment prompt word includes at least one of a V2X dialogue flag, a rejection flag, a waiting flag, and a chat dialogue flag.

[0035] The judgment prompts correspond to the playback status. The playback status can be determined based on the playback identifier of the Text-to-Speech (TTS) system. For example, the first judgment prompt corresponding to the playback status can be used to determine whether the audio segment has an intention to interrupt. For example, the first judgment prompt could be: "Judging whether the user's audio content in this round has an intention to interrupt the voice playback; the audio may be incomplete." The second judgment prompt corresponding to the silence status can be used to determine the user's intention to respond; for example, the judgment prompt could be: "Judging the user's interactive intention in the audio content in this round."

[0036] In this embodiment, the voice interaction device can acquire the recognized text corresponding to the user's multiple rounds of questions preceding the audio segment, and use this recognized text as the context of the dialogue. Furthermore, based on a pre-set correspondence between the broadcast state and the judgment prompt word, it determines the judgment prompt word corresponding to the current broadcast state. Optionally, this judgment prompt word can be determined from the first audio segment of the current round of dialogue, and the judgment prompt words for subsequent audio segments in the current round are the same as the judgment prompt word corresponding to the first audio segment.

[0037] Figure 2 This is a schematic diagram of the architecture of a voice interaction method provided in an embodiment of this disclosure, such as... Figure 2 As shown in this embodiment, the voice interaction device can encode audio segments using a Whisper Encoder to obtain speech signal features. This audio encoder can be trained and output features aligned with the input requirements of the intent recognition model. Thus, the speech signal features extracted by the audio encoder and the text features required by the intent recognition model (which serves as a large language model) are mapped to the same feature dimension, achieving alignment. Furthermore, the voice interaction device can input the dialogue context into a text lexical generator to obtain the text encoding corresponding to that dialogue context. Figure 2The intent recognition prompts and judgment prompts are omitted above. Furthermore, the voice interaction device can concatenate the intent recognition prompts, text encoding, speech signal features, and judgment prompts, and input the concatenated sequence into the intent recognition model for context prefilling and decoding inference, thereby achieving the fusion of the aforementioned multimodal inputs to predict dialogue intent information.

[0038] In the above scheme, based on the dialogue context and judgment prompts, richer data is provided for generating dialogue intent information, making the final generated dialogue intent information more consistent with the user's true dialogue intent.

[0039] In one optional implementation, the voice interaction device can employ a key-value cache (KV-Cache) mechanism to reuse historical calculation results. When the length of the input intent recognition model exceeds a set length limit, a portion of the key-value cache is discarded from the starting position of the text encoding corresponding to the dialogue context to control the total length. Thus, during the streaming audio encoding and inference computation process, by combining early response for audio segments and the key-value cache reuse mechanism, a relatively fast end-to-end response latency is achieved under the constraint of edge computing power.

[0040] Step 103: Process the audio segments according to the dialogue intent information.

[0041] In this embodiment of the disclosure, the voice interaction device can perform corresponding dialogue management processing based on the predicted dialogue intent information and in conjunction with post-processing logic.

[0042] In some embodiments of this disclosure, the number of audio segments is at least one. The audio segments are processed according to the dialogue intent information, including: if the latest audio segment meets the input end condition, the latest dialogue intent information is determined as the target intent information, and at least one identified text corresponding to at least one audio segment in the target dialogue round is processed according to the target intent information; if the latest audio segment does not meet the input end condition, the next audio segment is obtained, and the next audio segment is processed until the next audio segment meets the input end condition.

[0043] The input termination condition can be referred to as a pre-set condition representing the end of the user's dialogue input in the current round. In some embodiments of this disclosure, the input termination condition is that the silence duration of the audio segment is greater than a third duration threshold. The third duration threshold can be a threshold set for the silence duration of the audio segment; this embodiment does not limit the third duration threshold. For example, the second duration threshold can be 400 milliseconds. The target intent information can be the intent information corresponding to the dialogue round in which the audio segment is located. The recognized text can be the text obtained by performing speech recognition on the audio segment.

[0044] In this embodiment, the voice interaction device can segment the audio signal according to segmentation conditions, thereby realizing streaming processing of the audio signal, and can determine whether the current round of voice input has ended by input termination conditions. For example, the voice interaction status can determine whether the silence duration of the latest audio segment is greater than a third duration threshold.

[0045] Upon the arrival of the second silence signal, the dialogue intent information corresponding to the latest audio segment is determined as the target intent information. The second silence signal can indicate that the silence duration of the audio segment has reached a third duration threshold. Further, at least one recognized text corresponding to at least one audio segment within the target dialogue round is acquired, and text processing is performed on the at least one recognized text based on the target intent information. The dialogue round can be divided based on the satisfaction of an input termination condition; that is, satisfying one input termination condition generates one dialogue round.

[0046] In some embodiments of this disclosure, text processing is performed on at least one identified text corresponding to at least one audio segment within the target dialogue round based on the target intent information, including at least one of the following: if the target intent information is a vehicle-to-everything (V2X) dialogue identifier, then at least one identified text is routed to the V2X control model; if the target intent information is a chat dialogue identifier, then at least one identified text is routed to the chat model; if the target intent information is a rejection identifier, then the processing of at least one identified text is blocked; if the target intent information is an interruption identifier, then a playback stop command corresponding to at least one identified text is sent to the voice dialogue system; if the target intent information is a waiting identifier, then at least one identified text is saved, and the third duration threshold of the next dialogue round is extended.

[0047] In this embodiment, if the target intent information is a vehicle-to-machine dialogue identifier, it indicates that the user has spoken a vehicle-to-machine command during the target dialogue. The voice interaction device then routes the at least one recognized text to the vehicle-to-machine control model, which performs vehicle-to-machine control based on the at least one recognized text. The vehicle-to-machine control model can be a neural network model used for vehicle-to-machine control.

[0048] If the target intent information is a chat dialogue identifier, it means that the user has given a command other than a vehicle-mounted system command within the target dialogue round, indicating that the user wants to chat with the voice dialogue system rather than control the vehicle-mounted system. In this case, the voice interaction device can route the at least one recognized text to the chat model, which then provides a chat response based on the recognized text. The chat model can be a neural network model used for chatting.

[0049] If the target intent information is a rejection flag, it means that the user did not interact with the voice dialogue system during the target dialogue round. In this case, the voice interaction device may not process or respond to the at least one recognized text. If the target intent information is an interruption flag, it means that the user wants to interrupt the voice dialogue system's playback during the target dialogue round. In this case, the voice interaction state can send a playback stop command to the voice dialogue system. This playback stop command can be used to instruct the voice dialogue system to stop voice playback.

[0050] If the target intent information is a waiting flag, indicating that the user has not completed a full conversation within the target dialogue turn, the voice interaction device can save at least one recognized text. This recognized text, combined with the recognized text of the next dialogue turn, can then be used for processing, extending the third duration threshold of the next dialogue turn. This next dialogue turn can be the dialogue turn following the target dialogue turn. Specifically, the voice interaction device can obtain a first predicted probability corresponding to the waiting flag and a second predicted probability corresponding to the candidate flag. Based on the first and second predicted probabilities, the third duration threshold is extended to obtain a new third duration threshold.

[0051] In related technologies, when performing speech recognition, audio is first determined through Voice Activity Detection (VAD), then converted into text through offline recognition, and finally semantic understanding is performed through template matching or Natural Language Understanding (NLU) models. The end of speech is determined by an End of Turn (EOT) model to determine if the user has finished speaking. This interaction method uses a multi-model cascaded architecture, which does not fully integrate speech and semantic information, and does not support interruption, thus failing to achieve a full-duplex interactive experience. The above solution, however, abstracts the full-time wake-up-free interaction scenario into five dialogue control states through five types of dialogue intent information. A unified classification and prediction is performed through an on-device intent recognition model, achieving integrated decision-making for processing the recognized text. This enables reasonable control of the dialogue state, supports user interruption during the voice dialogue system's playback, and allows for functions such as waiting for complete user input, achieving precise listening control adapted to user voice input.

[0052] In this embodiment, if the latest audio segment does not meet the input end condition, no additional processing is performed if the dialogue intent information is a vehicle-to-everything (V2X) dialogue identifier, a rejection identifier, an interruption identifier, or a chat dialogue identifier. If the dialogue intent information is a waiting identifier, the third duration threshold is extended, and the next audio segment is obtained. The process is then repeated based on the extended third duration threshold until the next audio segment meets the input end condition.

[0053] In some embodiments of this disclosure, if the latest audio segment does not meet the input end condition, the next audio segment is obtained, and the next audio segment is processed upon return, including: if the latest audio segment does not meet the input end condition and the dialogue intent information is a waiting identifier, a first prediction probability corresponding to the waiting identifier and a second prediction probability corresponding to the candidate identifier are obtained; a third duration threshold is extended according to the first prediction probability and the second prediction probability to obtain a new third duration threshold; the next audio segment after the latest audio segment is obtained, and the next audio segment is processed upon return based on the new third duration threshold.

[0054] The first predicted probability can be the probability that the dialogue intent information determined by the intent recognition model is a waiting identifier. Candidate identifiers can be vehicle-mounted dialogue identifiers and chat dialogue identifiers. The second predicted probability can be the probability that the dialogue intent information determined by the intent recognition model is a candidate identifier.

[0055] In this implementation, if the latest audio segment does not meet the input end condition and the dialogue intent information corresponding to the latest audio segment is a waiting identifier, it indicates that the user has not completed the current round of dialogue and the user has not completed dialogue input. The voice interaction device can then obtain a first predicted probability, which represents the probability that the dialogue intent information determined by the intent recognition model is a waiting identifier, and obtain a second predicted probability that the dialogue intent information is a candidate identifier. Based on the first and second predicted probabilities, the third duration threshold is extended to obtain an extended third duration threshold. Further, the next audio segment following the current audio segment is obtained, and the next audio segment following the current audio segment is processed based on the extended third duration threshold. Thus, in cases where the user has not fully expressed themselves but an audio segment has already been extracted, extending the third duration threshold increases the condition for determining the end of the dialogue round, allowing the user to fully express themselves within the dialogue round.

[0056] In some embodiments of this disclosure, the third duration threshold is extended based on the first prediction probability and the second prediction probability, including: normalizing the sum of the first prediction probability and the second prediction probability to obtain the adjustment probability; weighting and summing the baseline duration corresponding to the candidate identifier and the baseline duration corresponding to the intent information based on the adjustment probability to obtain the adjustment duration threshold; if the adjustment duration threshold is greater than the third duration threshold, the third duration threshold is updated based on the adjustment duration threshold.

[0057] The adjustment probability can be the probability obtained after normalizing the sum. The waiting identifier and the candidate identifier can each have their own corresponding adjustment probability. The baseline duration can be a pre-set duration for both the waiting identifier and the candidate identifier; this embodiment does not impose any restrictions on this baseline duration.

[0058] In this embodiment, the adjustment duration threshold can be calculated from the probabilities corresponding to the vehicle-mounted dialogue identifier, chat dialogue identifier, and waiting identifier. The value range of the adjustment duration threshold is 0~2000ms. A value close to 0 indicates a response, and a value close to 2000ms indicates continued listening. The voice interaction device can obtain a first predicted probability and a second predicted probability, and adjust the first predicted probability and the second predicted probability proportionally so that the sum of the probabilities is 1, thus obtaining the adjustment probability.

[0059] Adjust duration threshold The formula for calculating can be shown below: ; in, The adjusted probability of the vehicle-mounted dialogue identifier output by the intent recognition model reflects the confidence that the user has finished speaking the complete command. The corresponding baseline duration is 0, indicating that there is no need to wait for an immediate response. The adjusted probability of the chat dialogue identifier output by the intent recognition model reflects the confidence that the user is in a chat dialogue. The corresponding baseline duration is 1000ms, which means that a moderate waiting time is required to confirm the user's intent. The adjustment probability of the waiting indicator output by the intent recognition model reflects the confidence level that the user's utterance has not yet been completed. The corresponding baseline duration is 2000ms, representing the maximum waiting time required to ensure complete reception of the user's instruction. The sum of the adjustment probabilities is 1. An adaptive adjustment duration threshold is dynamically calculated through weighted summation. This adjustment duration threshold can be understood as the maximum listening waiting time. Further, it is determined whether this adjustment duration threshold is greater than a third duration threshold. If so, the value of the third duration threshold is adjusted to the adjustment duration threshold; otherwise, the third duration threshold is not adjusted.

[0060] In the above scheme, when the intent recognition model determines that the user's speech is incomplete, it uses the adjustment probabilities of each vehicle dialogue identifier, chat dialogue identifier, and waiting identifier to perform weighted calculations and dynamically determines the adjustment duration threshold for continuing to listen. This avoids premature truncation that leads to incomplete instructions and achieves adaptive duration threshold adjustment based on probability weighting.

[0061] In some embodiments of this disclosure, the first duration threshold can also be adjusted based on the adjustment duration threshold.

[0062] The voice interaction method provided in this embodiment obtains an audio segment; inputs the intent recognition prompt and the audio segment into an intent recognition model to obtain dialogue intent information output by the intent recognition model; wherein, the dialogue intent information represents the dialogue intent between the audio segment and the voice dialogue system; and processes the audio segment according to the dialogue intent information.

[0063] In this embodiment, an audio segment and intent recognition prompt are input into an intent recognition model. The intent recognition model directly obtains dialogue intent information that represents the dialogue intent between the audio segment and the voice dialogue system. Based on this dialogue intent information, the audio segment is processed for subsequent dialogue. The intent recognition model determines the intent of the audio segment to engage in dialogue with the voice dialogue system. The determination of dialogue intent information is achieved with a shorter link, and natural voice interaction without wake words or fixed sentence patterns is achieved. The response time is short, and the perception of generalized instructions is realized, resulting in a natural and fluent dialogue.

[0064] Furthermore, in scenarios such as in-vehicle cockpits, when users speak in the in-vehicle cockpit, the system can continuously listen and actively determine whether the user is interacting with the voice dialogue system. Users can interact with the voice dialogue system without saying a wake word, and the system can perceive the user's intentions to conduct natural and smooth voice interaction, thus achieving a natural dialogue interaction experience.

[0065] Figure 3 A flowchart illustrating another voice interaction method provided in this disclosure embodiment is shown below. Figure 3 As shown, in some embodiments of this disclosure, the training method for the intent recognition model includes: Step 301: Obtain sample input data and sample intent information; wherein, the sample input data includes intent recognition prompts, audio segments, dialogue context, and judgment prompts; wherein, the sample intent information corresponding to the first judgment prompt includes at least one of interruption flag and rejection flag; the sample intent information corresponding to the second judgment prompt includes at least one of vehicle-mounted dialogue flag, rejection flag, waiting flag, and chat dialogue flag.

[0066] The sample input data can be training samples used as input data. The sample intent information can be the dialogue intent identifier used as the sample, which can be understood as the truth value corresponding to the sample input data. In this embodiment, the number of sample data corresponding to the five types of dialogue intent information—interruption identifier, rejection identifier, vehicle-to-everything (V2X) dialogue identifier, waiting identifier, and chat dialogue identifier—can be selected in equal proportions.

[0067] In this embodiment, the audio encoder and intent recognition model are aligned in advance using large-scale speech recognition data, so that the speech features extracted by the audio encoder and the text features input by the large language model are mapped to the same feature dimension, achieving alignment between the two. The voice interaction device can acquire sample input data and sample intent data. Among them, the first judgment prompt word corresponding to the broadcast state corresponds to at least one of interruption flag and rejection flag, indicating that when the voice dialogue system is broadcasting, the user has two potential dialogue intentions: one is to interrupt the voice dialogue system's broadcast, and the other is not to engage in dialogue with the voice dialogue system. The second judgment prompt word corresponding to the silent state corresponds to at least one of vehicle-mounted dialogue flag, rejection flag, waiting flag, and chat dialogue flag, indicating that when the voice dialogue system is not broadcasting, the user does not have the intention to interrupt the voice broadcast.

[0068] Step 302: Train the initial model based on the sample input data and sample intent information to obtain the completed intent recognition model.

[0069] In this embodiment, the voice interaction device can concatenate the intent recognition prompts, the current context, the lexical units determined by the audio segment, and the judgment prompts in the sample input data to obtain a sample concatenation sequence. Based on this sample concatenation sequence and the sample intent information, the initial model is trained until the training completion conditions are met, resulting in a trained intent recognition model.

[0070] In this embodiment, the loss function can be the autoregressive cross-entropy loss function. As shown below: ; in, This is a sequence of spliced ​​model samples. For the first in the target sequence Each word element represents a target sequence, which can be either a sample concatenation sequence or a concatenation sequence of sample intent information. The total length of the target sequence. This represents the concatenated sequence of samples given the input. and the preceding preorder lexicon generated before the t-th term. Under the condition, the model predicts the first The model calculates the conditional probabilities of each lexical unit. By minimizing this loss function, the model can learn to accurately predict dialogue control state indicators based on multimodal inputs, i.e., accurately predict dialogue intent information.

[0071] In the above scheme, when voice broadcasting is in progress, the dialogue intent is either to interrupt the voice broadcast or not to engage in dialogue with the voice dialogue system. When voice broadcasting is not in progress, the dialogue intent does not include interrupting the voice broadcast, which aligns with the user's true dialogue intent when voice broadcasting is not in use. Therefore, an intent prediction model is trained based on this training data, improving the accuracy of the intent prediction model.

[0072] The voice interaction method provided in this disclosure, based on a multimodal large language model, achieves real-time wake-word-free interactive control in scenarios such as vehicles. It abstracts the user's interaction intent with the voice dialogue system into a classification problem of five state sequences. Through an intent recognition model on the device side, it fuses voice and text information for reasoning and decision-making, predicting the user's dialogue intent information and achieving real-time natural interaction without wake words. It supports long-duration, multi-turn free dialogue and continuous listening. Furthermore, it supports full-duplex interaction, allowing users to interrupt or respond during the voice dialogue system's playback, while also supporting pauses and waiting. This achieves natural, full-duplex interaction with good semantic generalization capabilities. Moreover, it achieves low-latency response time on the device side and does not rely on cloud network environments.

[0073] Figure 4 A schematic diagram of the structure of a voice interaction device provided in an embodiment of this disclosure is shown.

[0074] In some embodiments of this disclosure, Figure 4 The voice interaction device shown can be executed by an electronic device or a server. The electronic device can be, but is not limited to, in-vehicle infotainment systems, embody intelligent robots, and other terminals. The server can be a server cluster or a cloud server.

[0075] like Figure 4 As shown, the voice interaction device 400 may include: an acquisition module 401, a model module 402, and a processing module 403.

[0076] Module 401 is used to acquire audio segments; Model module 402 is used to input the intent recognition prompt word and the audio segment into the intent recognition model to obtain the dialogue intent information output by the intent recognition model; wherein, the dialogue intent information represents the dialogue intent between the audio segment and the voice dialogue system; The processing module 403 is used to process the audio segment according to the dialogue intent information.

[0077] Optionally, the obtaining module 401 is specifically used for: Obtain audio signals; If the audio signal meets the segmentation condition, then the audio segment is determined based on the portion of the audio signal that has not been processed by the intent recognition model.

[0078] Optionally, the audio signal satisfies segmentation conditions, including: The unprocessed duration of the audio signal reaches a first duration threshold, or the silence duration of the audio signal reaches a second duration threshold.

[0079] Optionally, the model module 402 is specifically used for: Determine the context of the dialogue and the judgment prompt; wherein, the judgment prompt is used to prompt the intent recognition model to perform intent judgment corresponding to the broadcast state, and the broadcast state represents the state of the voice broadcast of the voice dialogue system; The intent recognition prompt, the audio segment, the dialogue context, and the judgment prompt are input into the intent recognition model to obtain the dialogue intent information output by the intent recognition model.

[0080] Optionally, the broadcast status includes a broadcasting state and a silent state; The first judgment prompt word corresponding to the broadcast state is used to prompt the intent recognition model to determine whether the audio segment corresponds to the interruption intent; the second judgment prompt word corresponding to the silent state is used to prompt the intent recognition model to determine whether the audio segment corresponds to other intents, wherein the other intents are dialogue intents other than the interruption intent.

[0081] Optionally, the voice interaction device 400 further includes a training module, which is used for: Obtain sample input data and sample intent information; wherein, the sample input data includes intent recognition prompts, audio segments, dialogue context, and judgment prompts; wherein, the sample intent information corresponding to the first judgment prompt includes at least one of interruption flag and rejection flag; the sample intent information corresponding to the second judgment prompt includes at least one of vehicle-mounted dialogue flag, rejection flag, waiting flag, and chat dialogue flag; The initial model is trained based on the sample input data and the sample intent information to obtain the completed intent recognition model.

[0082] Optionally, the number of audio segments is at least one, and the processing module 403 is specifically used for: If the latest audio segment satisfies the input end condition, the latest dialogue intent information is determined as the target intent information, and text processing is performed on at least one recognized text corresponding to at least one audio segment in the target dialogue round according to the target intent information; If the latest audio segment does not meet the input end condition, the next audio segment is obtained, and the process is repeated until the next audio segment meets the input end condition.

[0083] Optionally, the text processing of at least one identified text corresponding to at least one audio segment within the target dialogue round based on the target intent information includes at least one of the following: If the target intent information is a vehicle-to-machine dialogue identifier, then the at least one identified text is routed to the vehicle-to-machine control model; If the target intent information is a chat dialogue identifier, then the at least one identified text is routed to the chat model; If the target intent information is a denial identifier, then the processing of the at least one identified text is blocked; If the target intent information is an interruption identifier, then a broadcast stop command corresponding to at least one recognized text is sent to the voice dialogue system; If the target intent information is a waiting identifier, then the at least one identified text is saved, and the third duration threshold of the next dialogue round is extended.

[0084] Optionally, the input termination condition is that the silence duration of the audio segment is greater than a third duration threshold; If the latest audio segment does not meet the input end condition, then the next audio segment is obtained, and the next audio segment is processed, including: If the latest audio segment does not meet the input end condition, and the dialogue intent information is a waiting identifier, then the first prediction probability corresponding to the waiting identifier and the second prediction probability corresponding to the candidate identifier are obtained. The third duration threshold is extended based on the first prediction probability and the second prediction probability to obtain a new third duration threshold; Obtain the next audio segment after the latest audio segment, and process the next audio segment based on the new third duration threshold.

[0085] Optionally, extending the third duration threshold based on the first predicted probability and the second predicted probability includes: The sum of the first predicted probability and the second predicted probability is normalized to obtain the adjusted probability; Based on the adjustment probability, the baseline duration corresponding to the candidate identifier and the baseline duration corresponding to the intent information are weighted and summed to obtain the adjustment duration threshold. If the adjustment duration threshold is greater than the third duration threshold, then the third duration threshold is updated according to the adjustment duration threshold.

[0086] The voice interaction device 400 provided in this disclosure can execute the voice interaction method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects of executing the method.

[0087] Figure 5 A schematic diagram of the hardware circuit structure of a voice interaction device provided in an embodiment of this disclosure is shown.

[0088] like Figure 5 As shown, the voice interaction device 500 may include a controller 501 and a memory 502 storing computer program instructions.

[0089] Specifically, the controller 501 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0090] Memory 502 may include a large-capacity storage device for information or instructions. For example, and not limitingly, memory 502 may include a hard disk drive (HDD), a floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 502 may include removable or non-removable (or fixed) media. Where appropriate, memory 502 may be internal or external to the integrated gateway device. In a particular embodiment, memory 502 is a non-volatile solid-state memory. In a particular embodiment, memory 502 includes read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (Electrically Programmable ROM, EPROM), an electrically erasable programmable PROM (EEPROM), an electrically alterable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0091] The controller 501 performs the steps of the voice interaction method provided in the embodiments of this disclosure by reading and executing computer program instructions stored in the memory 502.

[0092] In one example, the voice interaction device 500 may further include a transceiver 503 and a bus 504. Wherein, as... Figure 5 As shown, the controller 501, memory 502 and transceiver 503 are connected through bus 504 and communicate with each other.

[0093] Bus 504 may include hardware, software, or both. For example, and not limitingly, a bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industrial Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, bus 504 may include one or more buses. Although specific buses are described and illustrated in the embodiments of this application, this application considers any suitable bus or interconnection.

[0094] The following are embodiments of a computer-readable storage medium provided in this disclosure. This computer-readable storage medium and the voice interaction methods of the above embodiments belong to the same inventive concept. For details not described in detail in the embodiments of the computer-readable storage medium, please refer to the embodiments of the above voice interaction methods.

[0095] This embodiment provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform a voice interaction method.

[0096] Of course, the computer-executable instructions provided in the embodiments of this disclosure are not limited to the above-described method operations, but can also perform related operations in the voice interaction method provided in any embodiment of this disclosure.

[0097] Based on the above description of the implementation methods, those skilled in the art can clearly understand that this disclosure can be implemented using software and necessary general-purpose hardware, and of course, it can also be implemented using hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or optical disk, etc., including several instructions to cause a computer cloud platform (which may be a personal computer, a server, or a network cloud platform, etc.) to execute the voice interaction methods provided in the various embodiments of this disclosure.

[0098] This disclosure also provides a smart terminal, including at least one of the following: the aforementioned voice interaction device; the aforementioned electronic device; and the aforementioned computer-readable storage medium. The smart terminal may include a vehicle or a unibody intelligent robot.

[0099] Note that the above description is merely a preferred embodiment and the technical principles employed in this disclosure. Those skilled in the art will understand that this disclosure is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of this disclosure. Therefore, although this disclosure has been described in detail through the above embodiments, it is not limited to the above embodiments. Many other equivalent embodiments may be included without departing from the concept of this disclosure, and the scope of this disclosure is determined by the scope of the appended claims.

Claims

1. A voice interaction method, characterized in that, include: Obtain audio segments; The intent recognition prompt and the audio segment are input into the intent recognition model to obtain the dialogue intent information output by the intent recognition model; wherein, the dialogue intent information represents the dialogue intent between the audio segment and the voice dialogue system; The audio segment is processed based on the dialogue intent information.

2. The method according to claim 1, characterized in that, The obtained audio segment includes: Obtain audio signals; If the audio signal meets the segmentation condition, then the audio segment is determined based on the portion of the audio signal that has not been processed by the intent recognition model.

3. The method according to claim 2, characterized in that, The audio signal satisfies segmentation conditions, including: The unprocessed duration of the audio signal reaches a first duration threshold, or the silence duration of the audio signal reaches a second duration threshold.

4. The method according to any one of claims 1-3, characterized in that, The step of inputting the intent recognition prompt and the audio segment into the intent recognition model to obtain the dialogue intent information output by the intent recognition model includes: Determine the context of the dialogue and the judgment prompt; wherein, the judgment prompt is used to prompt the intent recognition model to perform intent judgment corresponding to the broadcast state, and the broadcast state represents the state of the voice broadcast of the voice dialogue system; The intent recognition prompt, the audio segment, the dialogue context, and the judgment prompt are input into the intent recognition model to obtain the dialogue intent information output by the intent recognition model.

5. The method according to claim 4, characterized in that, The broadcast status includes a broadcasting state and a silent state; The first judgment prompt word corresponding to the state in the broadcast is used to prompt the intent recognition model to determine whether the audio segment corresponds to the interruption intent; The second judgment prompt word corresponding to the silent state is used to prompt the intent recognition model to determine whether the audio segment corresponds to other intents, wherein the other intents are dialogue intents other than the interruption intent; The dialogue intent information corresponding to the first judgment prompt word includes at least one of interruption indicator and rejection indicator; the dialogue intent information corresponding to the second judgment prompt word includes at least one of vehicle-mounted dialogue indicator, rejection indicator, waiting indicator, and chat dialogue indicator.

6. The method according to any one of claims 1-5, characterized in that, The number of audio segments is at least one, and the processing of the audio segments according to the dialogue intent information includes: If the latest audio segment satisfies the input end condition, the latest dialogue intent information is determined as the target intent information, and text processing is performed on at least one recognized text corresponding to at least one audio segment in the target dialogue round according to the target intent information; If the latest audio segment does not meet the input end condition, the next audio segment is obtained, and the process is repeated until the next audio segment meets the input end condition.

7. The method according to claim 6, characterized in that, The text processing of at least one recognized text corresponding to at least one audio segment within the target dialogue round based on the target intent information includes at least one of the following: If the target intent information is a vehicle-to-machine dialogue identifier, then the at least one identified text is routed to the vehicle-to-machine control model; If the target intent information is a chat dialogue identifier, then the at least one identified text is routed to the chat model; If the target intent information is a denial identifier, then the processing of the at least one identified text is blocked; If the target intent information is an interruption identifier, then a broadcast stop command corresponding to at least one recognized text is sent to the voice dialogue system; If the target intent information is a waiting identifier, then the at least one identified text is saved, and the third duration threshold of the next dialogue round is extended.

8. The method according to any one of claims 6-7, characterized in that, The input termination condition is that the silence duration of the audio segment is greater than the third duration threshold. If the latest audio segment does not meet the input end condition, then the next audio segment is obtained, and the next audio segment is processed, including: If the latest audio segment does not meet the input end condition, and the dialogue intent information is a waiting identifier, then the first prediction probability corresponding to the waiting identifier and the second prediction probability corresponding to the candidate identifier are obtained. The third duration threshold is extended based on the first prediction probability and the second prediction probability to obtain a new third duration threshold; Obtain the next audio segment after the latest audio segment, and process the next audio segment based on the new third duration threshold.

9. The method according to claim 8, characterized in that, The step of extending the third duration threshold based on the first prediction probability and the second prediction probability includes: The sum of the first predicted probability and the second predicted probability is normalized to obtain the adjusted probability; Based on the adjustment probability, the baseline duration corresponding to the candidate identifier and the baseline duration corresponding to the intent information are weighted and summed to obtain the adjustment duration threshold. If the adjustment duration threshold is greater than the third duration threshold, then the third duration threshold is updated according to the adjustment duration threshold.

10. A voice interaction device, characterized in that, include: The acquisition module is used to acquire audio segments; The model module is used to input the intent recognition prompt and the audio segment into the intent recognition model to obtain the dialogue intent information output by the intent recognition model; wherein, the dialogue intent information represents the dialogue intent between the audio segment and the voice dialogue system; The processing module is used to process the audio segment according to the dialogue intent information.

11. An electronic device, characterized in that, The electronic device includes: Processor and memory; The processor executes the steps of the method as described in any one of claims 1 to 9 by invoking programs or instructions stored in the memory.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program or instructions that cause a computer to perform the steps of the method as described in any one of claims 1 to 9.

13. A smart terminal, characterized in that, Includes at least one of the following: The voice interaction device according to claim 10 above; The electronic device according to claim 11 above; The computer-readable storage medium as described in claim 12.

Images