Voice interaction method, model training method, device and electronic equipment

Abstract

This disclosure provides a voice interaction method, model training method, device, and electronic device, relating to the field of computer technology, particularly artificial intelligence, natural language processing, large-scale models, and deep learning, and applicable to scenarios such as intelligent agents, intelligent e-commerce, and digital humans. The specific implementation scheme is as follows: In response to the user's first voice signal in the current dialogue, the first voice signal is recognized to obtain first voice-text information; using a trained recognition model, based on the first voice-text information, the first voice signal, and the contextual history information of the current dialogue, dialogue intervention events are identified to obtain the recognition result; wherein, the recognition result includes a classification label and a strength label, the classification label indicating the category of the dialogue intervention event, and the strength label indicating the intensity of the intervention intent corresponding to the corresponding event type; based on the recognition result, dialogue control is performed on the voice interaction process.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to the fields of artificial intelligence, natural language processing, large models, and deep learning, and especially to a voice interaction method, a model training method, a device, and an electronic device. Background Technology

[0002] With the significant progress made in artificial intelligence in fields such as natural language processing, machine learning, and deep learning, voice interaction technology, as an important branch of artificial intelligence, relies on technologies such as speech recognition, natural language understanding, and speech synthesis to achieve natural dialogue between humans and devices.

[0003] During human-computer dialogue, users often insert various phrases, filler words, interrogative words, agreeing phrases, repetitive statements, and even unconscious vocalizations into the device or system's speech. These insertions do not necessarily indicate that the user truly wants to take control of the conversation. However, current technology generally faces a challenge: accurately judging the "intensity of the user's expressive intent" behind their vocalizations. This deficiency often leads to the robot exhibiting oversensitivity in the dialogue, frequently interrupting the user or incorrectly continuing to speak, thus negatively impacting the interactive experience and the completion of system tasks. Summary of the Invention

[0004] This disclosure provides a voice interaction method, a model training method, a device, and an electronic device.

[0005] According to one aspect of this disclosure, a voice interaction method is provided, the method comprising: responding to a first voice signal from a user in a current dialogue, recognizing the first voice signal to obtain first voice-text information; employing a trained recognition model, based on the first voice-text information, the first voice signal, and contextual history information of the current dialogue, recognizing a dialogue intervention event to obtain a current recognition result; wherein the current recognition result includes a classification label and an intensity label, the classification label indicating the category of the dialogue intervention event, and the intensity label indicating the intensity information of the intervention intent corresponding to the dialogue intervention event of the corresponding event type; and performing dialogue control on the voice interaction process based on the current recognition result.

[0006] According to another aspect of this disclosure, a model training method is provided, the method comprising: acquiring training samples; wherein the training samples include sample classification labels, sample strength labels, and sample context information for sample speech-text pairs, the sample classification labels indicating the labeled category of the dialogue intervention event to which the sample speech-text pair belongs, and the sample strength labels indicating the labeled intervention intent strength information corresponding to the dialogue intervention event matching the labeled category; employing an initial recognition model to recognize dialogue intervention events based on the sample speech-text pairs and the sample context information to obtain a recognition result; wherein the recognition result includes a predicted classification label and a predicted strength label, the predicted classification label indicating the predicted category of the dialogue intervention event to which the sample speech-text pair belongs, and the predicted strength label indicating the predicted intervention intent strength information of the dialogue intervention event matching the labeled category; and training the initial recognition model based on a first difference between the labeled category and the predicted category, and a second difference between the labeled intervention intent strength information and the predicted intervention intent strength information.

[0007] According to another aspect of this disclosure, a voice interaction device is provided, the device comprising: a first recognition module, configured to recognize a first voice signal from a user in a current dialogue to obtain first voice-text information; a second recognition module, configured to use a trained recognition model to recognize a dialogue intervention event based on the first voice-text information, the first voice signal, and the contextual history information of the current dialogue, to obtain a current recognition result; wherein the current recognition result includes a classification label and an intensity label, the classification label indicating the category of the dialogue intervention event, and the intensity label indicating the intensity information of the intervention intent corresponding to the dialogue intervention event of the corresponding event type; and a control module, configured to perform dialogue control on the voice interaction process according to the current recognition result.

[0008] According to another aspect of this disclosure, a model training apparatus is provided, the apparatus comprising: an acquisition module for acquiring training samples; wherein the training samples include sample classification labels, sample strength labels, and sample context information for sample speech-text pairs, the sample classification labels indicating the labeled category of the dialogue intervention event to which the sample speech-text pair belongs, and the sample strength labels indicating the labeled intervention intent strength information corresponding to the dialogue intervention event matching the labeled category; a recognition module for using an initial recognition model to recognize dialogue intervention events based on the sample speech-text pairs and the sample context information to obtain a recognition result; wherein the recognition result includes a predicted classification label and a predicted strength label, the predicted classification label indicating the predicted category of the dialogue intervention event to which the sample speech-text pair belongs, and the predicted strength label indicating the predicted intervention intent strength information of the dialogue intervention event matching the labeled category; and a training module for training the initial recognition model based on a first difference between the labeled category and the predicted category, and a second difference between the labeled intervention intent strength information and the predicted intervention intent strength information.

[0009] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the voice interaction method or model training method proposed above in this disclosure.

[0010] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided that stores computer instructions for causing a computer to execute the voice interaction method or model training method proposed in this disclosure above.

[0011] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the voice interaction method or model training method proposed above.

[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0013] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1 This is a flowchart illustrating a voice interaction method according to an embodiment of the present disclosure; Figure 2 This is a flowchart illustrating a voice interaction method according to another embodiment of the present disclosure; Figure 3 This is a flowchart illustrating a voice interaction method according to another embodiment of the present disclosure; Figure 4 This is a schematic flowchart of a model training method according to an embodiment of the present disclosure; Figure 5 This is a schematic diagram of the structure of a voice interaction device according to an embodiment of the present disclosure; Figure 6 This is a schematic diagram of the structure of a model training apparatus according to an embodiment of the present disclosure; Figure 7 This is a block diagram of an electronic device used to implement the voice interaction method or model training method of the embodiments of this disclosure. Detailed Implementation

[0014] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0015] In voice dialogue systems built with relevant technologies, the interaction mode strictly follows the sequential simplex communication rule. This rule means that the system and the user need to take turns controlling the speaking time. When the system is speaking, the user is assumed to be listening quietly, and there is no situation where the two speak simultaneously. However, the essence of natural human-computer dialogue is actually a dynamic full-duplex interaction mode. In actual dialogue scenarios, users often insert their voices while the system is speaking. These inserted voices take many forms, including inquiries due to questions about the system's content, interruptions due to eagerness to express their own thoughts, agreement with the system's viewpoint, feedback on the system's content, and even some unconscious speech. If the system treats all user-inserted voices as signals of intent to take over the dialogue, then the system will frequently interrupt the currently ongoing voice output, leading to a fragmented dialogue process and chaotic management of the dialogue state. This situation will seriously affect the smoothness of the interaction, making the user feel uncomfortable when talking to the system, and will also reduce the system's efficiency in performing tasks, making it unable to quickly and accurately complete the tasks assigned by the user. Conversely, if the system completely ignores the user's inserted voice, it will reduce the system's response speed, making the user feel that the system is slow to respond. It will also weaken the system's collaborative ability, causing some key user intentions to be missed and failing to meet the user's needs in a timely manner.

[0016] To address the above issues, current technical solutions can be mainly divided into two categories: The first type is the pure text-driven model. This model completely discards the paralinguistic information inherent in user speech. Paralinguistic information refers to non-textual information such as prosodic variations, timbre characteristics, and emotional tone in speech. However, this paralinguistic information plays a crucial role in accurately judging the user's immediate intention and emotional state; it is a key clue. For example, the prosodic variations in pitch, speed, and volume, as well as the timbre characteristics of the voice, can reflect the user's current emotional state and true intention. However, because the pure text-driven model lacks this important paralinguistic information, it has significant limitations in understanding and judging user intentions and cannot fully and accurately grasp the user's true thoughts.

[0017] The second type is the traditional cascaded architecture. This architecture's processing flow involves first performing speech recognition to convert speech into text; then conducting semantic understanding to analyze the meaning of the text; and finally implementing dialogue management to perform corresponding dialogue operations based on the results of semantic understanding. However, this architecture has two main problems: error propagation and modal fragmentation. When speech recognition converts speech into text, key prosodic features are lost because text can only record textual content and cannot retain prosodic information such as intonation and stress in speech. This makes it difficult for subsequent semantic understanding and dialogue management modules to perform cross-modal joint analysis, failing to integrate information from both speech and text modalities to accurately determine user intent. For example, relying solely on text recordings, we cannot distinguish between the interrogative tone of "Oh?" (rising tone, expressing doubt or inquiry) and the agreeing tone of "Oh." (falling tone, expressing agreement or understanding), and these two tones correspond to drastically different dialogue behaviors and power dynamics. If the system cannot accurately distinguish between them, incorrect responses will occur during the dialogue, affecting the quality and effectiveness of the conversation.

[0018] To address at least one of the aforementioned problems, this disclosure proposes a voice interaction method, a model training method, a device, and an electronic device.

[0019] This disclosure proposes a voice interaction method, a model training method, a device, and an electronic device.

[0020] Figure 1 This is a flowchart illustrating a voice interaction method according to an embodiment of the present disclosure. It should be noted that the voice interaction method of the present disclosure can be applied to a voice interaction device, which can be configured in an electronic device so that the electronic device can perform voice interaction functions.

[0021] Among them, electronic devices can be any device with computing capabilities, such as personal computers (PCs), mobile terminals, servers, etc. Mobile terminals can be, for example, in-vehicle devices, mobile phones, tablets, personal digital assistants, wearable devices, smart speakers, servers, server clusters, and other hardware devices with various operating systems, touch screens and / or displays.

[0022] The voice interaction device can also be software within an electronic device, such as voice interaction software. In the following embodiments, an electronic device is used as an example for illustration.

[0023] like Figure 1 As shown, the voice interaction method may include the following steps: Step 101: In response to the user's first voice signal in the current dialogue, the first voice signal is recognized to obtain the first voice text information.

[0024] The first voice information refers to the audio input data emitted by the user in the current dialogue that can be collected by the device. It is the original input carrier of voice interaction, and its physical form is a continuous sound wave vibration signal.

[0025] In this embodiment of the disclosure, after acquiring the first voice signal of the user in the current conversation, the first voice signal can be recognized to obtain the first voice-text information. As an example, ASR (Automatic Speech Recognition) technology can be used to perform speech recognition processing on the first voice signal, and the text content obtained from the speech recognition processing can be used as the first voice-text information.

[0026] In one possible implementation of this disclosure, the first speech signal may be preprocessed.

[0027] Preprocessing may include, but is not limited to: noise reduction, silence detection, filtering (such as bandpass filtering), normalization (such as signal energy normalization), etc.

[0028] Among them, silence detection can be used to distinguish between valid signal segments and "silent or noisy" signal segments in the first speech signal, making it easier to filter out meaningless signal segments.

[0029] Therefore, by preprocessing the first speech signal, the signal quality of the first speech signal is improved.

[0030] Step 102: Using the trained recognition model, based on the first speech text information, the first speech signal and the context history information of the current dialogue, the dialogue intervention event is identified to obtain the recognition result.

[0031] The identification results may include a classification label and a strength label. The classification label can be used to indicate the category of the dialogue intervention event, and the strength label can be used to indicate the strength of the intervention intent corresponding to the dialogue intervention event of the corresponding event type.

[0032] Among them, dialogue intervention events (also known as voice insertion events) refer to events in which the user intervenes in the dialogue in some way during the human-computer dialogue process, which may affect, guide or adjust the direction, content or atmosphere of the dialogue.

[0033] It should be noted that this disclosure does not restrict the categories of dialogue intervention events.

[0034] As an example, the categories can include interruptible and non-interruptive types. Interruptive events that are worth interrupting can refer to user commands or feedback issued via voice signals, which are sufficient to prompt the system (such as a chatbot, intelligent customer service, or digital human) to pause the current task, switch the conversation state, or adjust the interaction strategy. Non-interruptive events that are not worth interrupting can refer to user commands or feedback issued via voice signals, which are insufficient to prompt the system (such as a chatbot or intelligent customer service) to pause the current task, switch the conversation state, or adjust the interaction strategy.

[0035] As another example, categories can include active intervention types, passive intervention types, and indirect intervention types. Active intervention types refer to events where the user actively intervenes in the voice interaction process, such as when the user initiates a conversation. Passive intervention types refer to events where the user passively intervenes in the voice interaction process, such as when the user is forced into a conversation due to external interference. Indirect intervention types refer to events where the user indirectly intervenes in the voice interaction process, such as when the user indirectly influences the process or outcome of the human-computer dialogue through non-direct dialogue methods (such as operating the device, adjusting settings, providing feedback, etc.).

[0036] As another example, categories could include unconscious agreement type, social politeness type, content paraphrase type, preceding statement type, and so on.

[0037] It should be noted that the categories of dialogue intervention events described above are merely exemplary. In practical applications, other categories may also apply, and this disclosure does not impose any restrictions on them.

[0038] The intervention intent strength information can be used to indicate the urgency of the user's underlying need, the intensity of their desire, and the weight level of their impact on the current voice interaction process when the user initiates a dialogue intervention event. It can include, but is not limited to, intervention intent strength values ​​and intervention intent strength levels.

[0039] One approach is to use a scoring system, which is a numerical value representing the intensity of intervention intent. In other words, the intensity of intervention intent can be quantified into a specific numerical value with a certain range, such as 0 to 100 or 0 to 10.

[0040] One approach is to use a rating system, or a rating scale, to represent the intensity of intervention intent. In other words, the intensity of intervention intent can be divided into several discrete levels, such as “high”, “medium”, “low”, or numerical levels such as 1, 2, 3, etc., or letter levels such as “A+”, “A”, “A-”, “B+”, etc.

[0041] The term "trained recognition model" refers to an artificial intelligence model that has been trained using relevant sample data and possesses stable recognition capabilities; for example, it could be a multimodal large-scale speech model. It should be noted that this disclosure does not impose any restrictions on the recognition model.

[0042] As an example, contextual history information may include historical speech signals and historical speech-text information. Historical speech signals refer to the speech signals involved in the speech interaction before the current dialogue, while historical speech-text information is the text content obtained by recognizing historical speech signals.

[0043] As another example, contextual history information may include historical speech embedding vectors and historical semantic embedding vectors of historical speech signals; wherein, the historical speech embedding vector can be used to indicate the embedding vector corresponding to the historical speech signal; and the historical semantic embedding vector can be used to indicate the embedding vector corresponding to the historical speech text information.

[0044] Understandably, in practical applications of voice interaction, users frequently insert various voice signals during system conversations, such as questions, interruptions, echoing, responsive vocalizations, and unconscious speech output. If all such insertions are treated as signals of intent to take over the dialogue, the system will experience frequent interruptions, the speech flow will become fragmented, and the dialogue state machine will become disordered, severely impacting the fluency of the interaction and the efficiency of task completion. Conversely, ignoring these insertions will reduce the system's responsiveness and collaborative capabilities, leading to the omission of key intentions. Therefore, in this embodiment, a pre-trained recognition model can be used to identify dialogue intervention events based on the first speech text information, the first speech signal, and the contextual history information of the current dialogue, thereby obtaining the recognition result. As an example, the first speech text information, the first speech signal, and the contextual history information of the current dialogue can be input into the pre-trained recognition model, and the recognition result can be obtained in response to the output of the pre-trained recognition model.

[0045] Step 103: Based on the recognition results, perform dialogue control on the voice interaction process.

[0046] As an example, appropriate strategies can be adopted to control the voice interaction process based on the recognition results, such as updating the dialogue status based on the recognition results.

[0047] The voice interaction method of this disclosure embodiment recognizes a first voice signal in response to a user's first voice signal in the current dialogue to obtain first voice text information; it then uses a trained recognition model to recognize dialogue intervention events based on the first voice text information, the first voice signal, and the contextual history information of the current dialogue to obtain the recognition result; wherein the recognition result includes a classification label and a strength label, the classification label indicating the category of the dialogue intervention event, and the strength label indicating the intervention intent strength information corresponding to the dialogue intervention event of the corresponding event type; and it performs dialogue control on the voice interaction process based on the recognition result. Therefore, by combining multimodal information—namely, speech-text information, speech signal features, and contextual history information—and leveraging a pre-trained recognition model, the system can classify dialogue intervention events and quantify the intensity of intervention intentions. This not only distinguishes different categories of dialogue intervention events but also intuitively reflects the user's true intention to dominate the conversation through intensity labels. Based on the dual decision-making criteria of classification and intensity labels, the voice interaction system can implement differentiated dialogue control strategies. For example, for low-intensity, meaningless insertions, the system can maintain continuous speaking to avoid dialogue interruptions due to oversensitivity. Conversely, for high-intensity proactive takeover requests, the system can promptly terminate the current output and switch to listening mode, ensuring the full satisfaction of the user's expressed needs. This dialogue control mechanism not only improves the fluency and naturalness of human-computer voice interaction, eliminating user frustration caused by frequent interruptions or the robot "talking to itself," but also enhances the accuracy and efficiency of system task execution, improving the user experience. It provides technical support that is more aligned with users' real interaction habits for scenarios such as intelligent customer service, in-vehicle voice assistants, and smart home voice interaction.

[0048] like Figure 2 As shown, this disclosure also proposes a voice interaction method. Figure 2 This is a flowchart illustrating a voice interaction method according to another embodiment of the present disclosure, which may include the following steps: Step 201: In response to the user's first voice signal in the current dialogue, the first voice signal is recognized to obtain the first voice text information.

[0049] It should be noted that the execution process of step 201 can refer to the execution process of any embodiment of this disclosure, and will not be repeated here.

[0050] Step 202: Based on the historical speech signal and the first speech signal in the contextual historical information, the audio encoder in the recognition model is used to encode the target speech features.

[0051] The contextual history information may include historical voice signals. It should be noted that the explanation of contextual history information and historical voice signals in step 102 also applies to this embodiment, and will not be repeated here.

[0052] The recognition model may include an audio encoder. It should be noted that the explanation of the recognition model in step 102 also applies to this embodiment, and will not be repeated here.

[0053] Optionally, in some embodiments, an audio encoder (also known as an audio coding network) can be used to encode the historical speech signal to obtain historical speech features, and an audio encoder can be used to encode the first speech signal to obtain the first speech features; the target speech features are determined based on the historical speech features and the first speech features.

[0054] As an example, the historical speech signal and the first speech signal can be input into the audio encoder frame by frame to encode the speech signals and obtain the historical speech features corresponding to the historical speech signal and the first speech features corresponding to the first speech signal. Then, the target speech features can be determined based on the historical speech features and the first speech features. For example, the historical speech features and the first speech features can be weighted and fused to obtain the target speech features, or the historical speech features and the first speech features can be concatenated to obtain the target speech features.

[0055] Therefore, by encoding two types of speech signals using the same audio encoder, the consistency of feature extraction standards can be ensured, avoiding problems such as feature dimension mismatch and semantic shift caused by different encoding models, and improving the comparability and relevance of features. On the other hand, by fusing and analyzing historical speech features with first speech features to determine target speech features, for example, the temporal patterns of user speech interaction can be fully explored, such as capturing the continuous features of changes in speech rate, pause intervals, and intonation fluctuations during the conversation. This can accurately distinguish between habitual verbal tics, unconscious vocalizations, and speech signals with genuine intervention intentions, providing a high-quality acoustic feature foundation for subsequent multimodal feature fusion and intent intensity discrimination. Furthermore, it can improve the accuracy and robustness of dialogue intervention event recognition.

[0056] Optionally, in some embodiments, the audio encoder may include a first branch and a second branch.

[0057] The first branch (also known as the acoustic branch) is used to obtain speech prosody features, and the second branch (also known as the speech branch) is used to obtain acoustic intensity features. For example, either the first or second branch can employ a set of lightweight Transformers, one-dimensional convolutional layers, etc.

[0058] Optionally, in some embodiments, the first speech signal is input into a first branch of the audio encoder to obtain a first speech prosody feature, and the first speech signal is input into a second branch of the audio encoder to obtain a first acoustic intensity feature; the first speech feature is determined based on the first speech prosody feature and the first acoustic intensity feature.

[0059] Optionally, in some embodiments, the first acoustic intensity feature may include at least one of the following: short-time energy, zero-crossing rate, signal-to-noise ratio, Mel-frequency cepstral coefficient, linear prediction cepstral coefficient, background noise type, and breath sound tag value.

[0060] Background noise types can include, for example, two categories: noisy and no noise, or low noise, medium noise, strong noise, etc.

[0061] It should be noted that the above examples of background noise types are merely illustrative; in practical applications, settings can be adjusted as needed.

[0062] It should be noted that background noise type can be used for filtering in subsequent data processing, and can also be used to filter or weight feature dimensions. For example, it can be used to retain all features in low-noise scenarios, remove high-frequency interference dimensions in strong-noise scenarios, or assign higher weights to robust low-frequency features, etc.

[0063] Among them, the breathing sound label value can be, for example, 1, indicating that there is no breathing sound in the background of the voice interaction, or 0, indicating that there is breathing sound in the background of the voice interaction.

[0064] It should be noted that the above values ​​for the breathing sound label are merely illustrative and may be used in practice.

[0065] Optionally, in some embodiments of this disclosure, the first speech prosodic feature includes at least one of the following: fundamental frequency trajectory, fundamental frequency slope, fundamental frequency extreme point, fundamental frequency pause, syllable duration, phoneme duration ratio, stressed syllable, and energy change rate.

[0066] Therefore, by mining multi-dimensional acoustic intensity features and multi-dimensional speech prosody features, the accuracy and comprehensiveness of the determined first speech features can be improved. This not only achieves a three-dimensional and multi-level representation of speech signals, but also accurately distinguishes different types of dialogue intervention behaviors such as user functional speech, verbal tics, and unconscious vocalizations from the dual dimensions of prosody and intensity. This provides a highly discriminative and highly correlated acoustic foundation for subsequent multimodal fusion analysis combining historical speech features and text semantic features, and helps to improve the accuracy and robustness of dialogue intervention event recognition.

[0067] As an example, the first speech signal can be simultaneously input into the first branch and the second branch of the audio encoder to encode the first speech signal through the first branch and the second branch respectively, so as to obtain the first speech prosodic feature output by the first branch and the second speech prosodic feature output by the second branch; then, the first speech feature can be determined according to the first speech prosodic feature and the first acoustic intensity feature. For example, the first speech prosodic feature and the second acoustic intensity feature can be fused to obtain the first speech feature.

[0068] Therefore, a dual-branch parallel architecture for the audio encoder is adopted to extract multi-dimensional acoustic features from the first speech signal, avoiding the problems of single-branch feature extraction leading to a limited characterization and information loss. The first branch focuses on mining prosodic features, such as accurately capturing temporal acoustic attributes like speech rate, intonation, and pause duration, effectively distinguishing non-semantic vocalizations like echolalia and filler words from functional speech with clear intent. The second branch extracts acoustic intensity features, quantifying parameters like volume and energy of the speech signal to further aid in determining the user's initiative and willingness to intervene. Finally, the first speech feature is obtained through the fusion analysis of prosodic and acoustic intensity features, improving the adaptability and comprehensiveness of the acquired first speech signal's features. This achieves a three-dimensional and refined characterization of the speech signal, preserving its temporal dynamics while strengthening key acoustic indicators related to intervention intent. This provides high-quality data for subsequent processing, contributing to improved accuracy and robustness in dialogue intervention event recognition.

[0069] Optionally, in some embodiments of this disclosure, the historical speech signal is input into the first branch of the audio encoder to obtain the second speech prosody feature, and the historical speech signal is input into the second branch of the audio encoder to obtain the second acoustic intensity feature; the historical speech feature is determined based on the second speech prosody feature and the second acoustic intensity feature.

[0070] Step 203: Based on the historical speech and text information in the context history information and the first speech and text information, the text encoder in the recognition model is used to encode the target semantic features.

[0071] The contextual history information may also include historical voice-text information. It should be noted that the explanation of historical voice-text information in step 102 also applies to this embodiment, and will not be repeated here.

[0072] The recognition model can also include a text encoder, used to encode text content. For example, the text encoder can be the text encoding subnet in a multimodal large language model, which is a text feature extraction module based on the Transformer architecture.

[0073] Optionally, in some embodiments of this disclosure, such as Figure 3 As shown, step 203 can be achieved by following these steps: Step 2031: Use a text encoder to encode the historical speech text information to obtain historical semantic features.

[0074] For example, historical speech-text information can be input into a text encoder to encode the historical speech-text information, thereby obtaining the text features corresponding to the historical speech-text information, which are referred to as historical semantic features in this disclosure.

[0075] Step 2032: Determine the target speech text information based on the first speech text information.

[0076] As an example, the first speech text information can be identified as the target speech text information.

[0077] As another example, the first speech-text information and the reference text information can be fused to obtain the target speech-text information.

[0078] The reference text information may include, but is not limited to: the confidence level of the first speech text information, the text length of the first speech text information, the sentence category of the first speech text information, and the importance information corresponding to the context history information, etc. The reference text information can be set or added as needed.

[0079] The confidence information can be used to measure the reliability of the speech recognition result of the corresponding speech signal. For example, the confidence information of the first speech-text information can be used to measure the reliability of the speech recognition result of the first speech signal, i.e., the reliability of the first speech-text information. It should be noted that the confidence information can be descriptive, such as "the confidence of the first speech-text information is extremely high" or "the confidence of the first speech-text information is ≥0.8"; or the confidence information can be numerical, such as "the confidence information is 0.8".

[0080] The text length information can be used to indicate the number of character units (also known as words, tokens, etc.) in the first speech text information.

[0081] Among them, the statement categories may include, but are not limited to: interrogative sentences, negative sentences, confirmation statements, gratitude statements, echoing statements, retelling statements, pausing statements, meaningless sound statements, etc. Exemplarily, relevant rules can be adopted to identify the statement category of the first voice text information. For example, it can be identified whether the first voice text information belongs to a certain statement pattern (such as an interrogative sentence pattern (such as the sentence contains "ma", etc.), a negative sentence pattern (such as the sentence contains "not", "don't", "can't", etc.), etc.), and based on the recognition result, its statement category can be determined.

[0082] Among them, the importance information corresponding to the context history information can be used to measure the importance or influence degree of the context history information on the current conversation. For example, it can be determined by using relevant deep learning models based on the context history information.

[0083] Thus, by fusing the first voice text information with multi-dimensional reference text information, where the reference text information can cover key attributes such as speech recognition confidence, text length, statement category, and context history importance, the original voice text can be accurately screened and optimized from four aspects: effectiveness, integrity, functionality, and relevance. For example, based on the confidence information, low-quality text segments with fuzzy recognition can be filtered; based on the text length and statement category, meaningless short mantras and expressions with substantial content can be distinguished; based on the context history importance information, the weight of text content strongly associated with the conversation main line can be strengthened. The target voice text information obtained through fusion can not only eliminate the noise interference generated during the speech recognition process but also highlight the core semantics strongly related to the user's interaction intention, improving the accuracy and comprehensiveness of the target voice text information. And the reference semantic features generated on this basis have stronger intention representation capabilities. The target semantic features obtained after the correlation analysis with the historical semantic features can more accurately depict the true intention of the user's speech, effectively improving the accuracy of the dialogue intervention event recognition.

[0084] Step 2033, use a text encoder to encode the target voice text information to obtain reference semantic features.

[0085] Optionally, in some embodiments, the target voice text information is segmented to obtain multiple words; the word library is queried to determine the key value of each word; according to the key value, the target voice text information is encoded by using a text encoder to obtain reference semantic features.

[0086] Among them, the word library can be a pre-constructed structured database containing a large number of words and their associated attributes (such as key values, word types, semantic categories, etc.).

[0087] Key values ​​can include, for example, a word ID (Identity Document, also known as a unique code), part-of-speech tag, semantic category tag, etc. The word ID is used to uniquely identify the corresponding word; the part-of-speech tag indicates the grammatical attributes of the corresponding word, such as noun, verb, adjective, etc.; the semantic category tag indicates the semantic category to which the corresponding word belongs, etc.

[0088] As an example, a word segmenter can be used to segment the target speech text information into multiple words; then, for any word, the word database can be queried to determine its corresponding word ID; the word ID of each word is mapped to a dense vector (e.g., a 768-dimensional vector) to obtain the word embedding; based on the word embedding, the initial text representation of the target speech text information is determined; the initial text representation is input into the text encoder, and the reference semantic features are obtained in response to the output of the text encoder.

[0089] Therefore, the target speech text is first segmented into words, breaking down continuous text into independent word units, laying the foundation for subsequent semantic filtering and weight allocation. Then, by querying a word database to determine the key value of each word, the importance of different words can be accurately distinguished. For example, it can distinguish between functional core words (such as keywords expressing needs and questions) and meaningless auxiliary words (such as catchphrases, modal particles, and repetitive phrases), and different semantic weights can be assigned to each word. Subsequently, based on the weighted word features, reference semantic features are generated by a text encoder. This strengthens the representation strength of core semantics in the feature dimension and weakens the interference of meaningless words. It can both preserve the core semantic connotation of the target speech text information and effectively filter redundant information. The target semantic features obtained after correlation analysis with historical semantic features have a stronger intent discrimination ability and can accurately identify the user's speaking intent. For example, it can identify whether the user's speech is a functional expression with a real intention to take over the dialogue or meaningless insertion content. In this way, in subsequent applications, the accuracy and reliability of identifying speech intervention intent can be further improved, which can improve the human-computer interaction experience.

[0090] Step 2034: Determine the target semantic features based on historical semantic features and reference semantic features.

[0091] As an example, historical semantic features and reference semantic features can be weighted and fused to obtain target semantic features.

[0092] As another example, historical semantic features and reference semantic features can be concatenated to obtain target semantic features.

[0093] Therefore, on the one hand, historical speech text information can be encoded using a text encoder, and the extracted historical semantic features can fully preserve the contextual logic and interaction flow of the dialogue, providing an important reference for judging whether the user's current speech conforms to the continuity of the dialogue. On the other hand, reference semantic features focusing on the user's actual expression can be obtained based on the target speech text information using a text encoder. Furthermore, by generating target semantic features through correlation analysis of historical semantic features and reference semantic features, the semantic correlation between the user's current speech and historical dialogue can be captured, efficiently distinguishing different interaction intentions, such as distinguishing between "continuous agreement" and "transitional intervention." This context-related semantic feature construction method provides a highly accurate semantic foundation for subsequent multimodal feature fusion, complementing the acoustic features extracted from the audio branch, and further improving the accuracy and robustness of dialogue intervention event recognition.

[0094] Step 204: Use the multimodal fusion layer in the recognition model to fuse the target speech features and target semantic features to obtain the target features.

[0095] The recognition model may also include a multimodal fusion layer, which can be used to fuse multimodal data, such as one or more cross-modal attention networks.

[0096] As an example, target speech features and target semantic features can be input into the multimodal fusion layer in the recognition model, and the target features can be obtained in response to the output of the multimodal fusion layer.

[0097] As another example, at least one of the target speech features and target semantic features can be downsampled, and the processed features can be input into a multimodal fusion layer in the recognition model. The target features are then obtained in response to the output of the multimodal fusion layer. Downsampling reduces the sampling rate or dimensionality of the feature sequence, thereby reducing the computational burden on the model.

[0098] Step 205: Use the discriminator in the recognition model to discriminate the target features and obtain the recognition result.

[0099] The recognition model may also include a decision maker, which may include several Transformer layers, a binary classification head (or a multi-classification head, sequence labeler), etc.

[0100] As an example, the target features can be input into the discriminator in the recognition model, and the recognition result can be obtained in response to the output of the discriminator.

[0101] Step 206: Based on the recognition results, perform dialogue control on the voice interaction process.

[0102] It should be noted that the execution process of step 206 can refer to the execution process of any embodiment of this disclosure, and will not be repeated here.

[0103] Optionally, in any embodiment of this disclosure, a corresponding control decision can be determined based on a combination of the type of dialogue intervention event and the intensity of intervention intent information, and the control decision can be used to perform dialogue control on the voice interaction process.

[0104] Optionally, in some embodiments, the control decision may include at least one of the following: Action decision, buffering strategy information, and delay time parameters.

[0105] Action decisions can be used to control interactive behaviors during voice interaction. For example, interactive behaviors include stopping speaking and entering full listening mode. Buffering strategy information can be used to indicate the timing and manner of action decisions during voice interaction. For example, the buffering strategy information could be to enter full listening mode first after a short delay (e.g., 0.3s) and then stop speaking, or to stop speaking and enter full listening mode simultaneously after a long delay.

[0106] The delay time parameter can be used to indicate the timing of action decisions during voice interaction, such as 0.5s, 1s, etc.

[0107] Therefore, action decisions can clearly define the core direction of interactive behavior. For example, commands such as stopping speaking or entering full listening mode can directly respond to user dialogue intervention needs of different intensities and categories. Buffering strategy information defines the timing logic and connection method of action execution, solving the problem of abrupt breaks during dialogue state switching and ensuring a natural and smooth interaction rhythm. Delay time parameters can provide precise timing for action execution, such as dynamically matching differentiated delay thresholds for different intervention intent intensities. In this way, customized control schemes can be matched for each type of user vocal behavior, avoiding frequent interruptions due to oversensitivity and preventing interaction gaps caused by response delays. This achieves a smooth switch of control in human-computer dialogue, improves the humanization and intelligence level of the voice interaction system, and enhances the user experience.

[0108] It should be noted that control decisions can be set as needed, and this disclosure does not impose any restrictions on this.

[0109] In this embodiment, a corresponding control decision can be determined based on a combination of the category of the dialogue intervention event and the intensity of the intervention intent. As an example, a mapping relationship between the category of the dialogue intervention event, the combination of the intervention intent intensity, and the control decision can be established in advance and saved. Then, after obtaining the current identification result, the mapping relationship can be queried based on the category of the dialogue intervention event and the combination of the intervention intent intensity indicated by the current identification result to determine the corresponding control decision. For example, assuming the category of the dialogue intervention event is "not worth interrupting" and the intervention intent intensity is 2, a soft interruption mechanism can be executed; for example, the system can ask the user "What did you say?". Alternatively, assuming the category of the dialogue intervention event is "worth interrupting" and the intervention intent intensity is 5, a hard interruption mechanism can be executed; for example, the system can immediately stop speaking and switch to a listening state. It should be noted that the above examples of control decisions are merely illustrative; in practical applications, they can be configured as needed.

[0110] Therefore, by combining and analyzing the classification tags of dialogue intervention events with the tags of intervention intent intensity, differentiated control strategies can be matched for different types of users' voices, achieving smoothness and intelligence in the switching of dialogue control. This not only eliminates the rigid interactive experience of robots "talking to themselves" but also avoids the abruptness of frequently interrupting users' expressions, improving the humanized level of voice interaction and ensuring the continuity and efficiency of task execution.

[0111] Optionally, in some embodiments, the identification result may also include confidence information.

[0112] The confidence information in the identification results can be used to indicate the reliability of the identification results. It can include confidence value and confidence level. The confidence value can be expressed as a numerical value, and the confidence level can be expressed as a grade.

[0113] Optionally, in some embodiments, when the confidence information includes a confidence value, if the confidence value is less than a set threshold, it indicates that the reliability of the current identification result is low, and a confirmation inquiry strategy is adopted. The confirmation inquiry strategy refers to engaging in dialogue with the user using a confirmation inquiry approach, such as asking the user, "Sorry, I didn't hear you clearly just now, would you like to continue?".

[0114] It should be noted that, in order to prevent service interruptions caused by misjudgments by the decision-maker, the system can run minimal security rules in parallel.

[0115] Optionally, the identification results, timestamp information, etc. can be recorded and a log generated to facilitate subsequent data processing.

[0116] In one possible implementation of this disclosure, when the contextual historical information includes historical speech embedding vectors and historical semantic embedding vectors of historical speech signals, an audio encoder can be used to encode the first speech signal to obtain first speech features; target speech text information is determined based on the first speech text information; a text encoder is used to encode the target speech text information to obtain reference semantic features; a multimodal fusion layer in the recognition model is used to fuse the first speech features, reference semantic features, historical speech embedding vectors, and historical semantic embedding vectors to obtain fused features; and a discriminator in the recognition model is used to discriminate the fused features to obtain the recognition result.

[0117] As an example, a first speech signal can be input into an audio encoder, and a first speech feature can be obtained in response to the output of the audio encoder; the first speech text information and reference text information can be fused to obtain target speech text information; the target speech text information can be input into a text encoder, and a reference semantic feature can be obtained in response to the output of the text encoder; the first speech feature, reference semantic feature, historical speech embedding vector, and historical semantic embedding vector can be input into a multimodal fusion layer to obtain the fused feature output by the multimodal fusion layer; then, the fused feature can be input into the decision maker in the recognition model, and the recognition result can be obtained in response to the output of the decision maker.

[0118] Therefore, by leveraging the multimodal fusion layer of the recognition model, real-time speech features and text features, as well as historical speech embedding vectors and historical semantic embedding vectors from contextual historical information, can be deeply coupled. The resulting fused features possess real-time vocal attributes, semantic connotations, and historical interaction logic, enabling a comprehensive and three-dimensional analysis of the user's dialogue intervention intent. Furthermore, the discriminator outputs recognition results containing event categories and the intensity of intervention intent, providing a more accurate and comprehensive decision-making basis for the subsequent formulation of dialogue control strategies. This can improve the model's accuracy in perceiving intent in complex interaction scenarios, effectively reduce the probability of misjudgment caused by isolated analysis of real-time signals, and enable the voice interaction system to accurately capture the user's interaction needs, achieve smooth switching of dialogue control, and improve the naturalness and fluency of human-computer interaction.

[0119] The voice interaction method of this disclosure involves encoding target voice features using an audio encoder in a recognition model based on historical voice signals and a first voice signal from contextual historical information; encoding target semantic features using a text encoder in a recognition model based on historical voice text information and the first voice text information from contextual historical information; fusing the target voice features and target semantic features using a multimodal fusion layer in the recognition model to obtain target features; and discriminating the target features using a discriminator in the recognition model to obtain the recognition result. On the one hand, encoding the voice signal using an audio encoder allows for in-depth mining of voice features, capturing non-semantic interaction signals such as unconscious vocalizations and echolalia; on the other hand, encoding the voice text using a text encoder allows for the extraction of core semantic information, distinguishing between functional speech and meaningless verbal tics. Then, a multimodal fusion layer deeply couples speech features with semantic features to generate target features that combine the characteristics of speech signals with the semantic connotations of text, enabling a multi-dimensional and comprehensive characterization of user dialogue intervention behavior. Finally, the discriminator outputs recognition results including event category and intent intensity, providing accurate and comprehensive decision-making basis for dialogue control strategy formulation. This not only improves the accuracy and robustness of dialogue intervention intent judgment but also allows the voice interaction system to perceive the user's true intent more delicately. In practical applications, it can effectively reduce false interruptions and erroneous continuations, improve the user interaction experience, and maintain the continuity and efficiency of voice interaction system task execution.

[0120] The above explains the application process of the model. The following explains the model training process.

[0121] like Figure 4 As shown, this disclosure also proposes a model training method. Figure 4 This is a flowchart illustrating a model training method according to an embodiment of the present disclosure. The model training method may include the following steps: Step 401: Obtain training samples.

[0122] The training samples may include sample classification labels, sample strength labels, and sample context information for sample speech-text pairs. The sample classification labels can be used to indicate the annotation category of the dialogue intervention event to which the sample speech-text pair belongs, and the sample strength labels can be used to indicate the annotation intervention intent strength information corresponding to the dialogue intervention event that matches the annotation category.

[0123] The sample speech-text pair may include sample speech signals and sample speech-text information, where the sample speech-text information is obtained by performing speech recognition on the sample speech signals.

[0124] Among them, the sample context information is the context information of the dialogue environment in which the corresponding sample speech text pair is located. For example, it may include the sample historical speech signal and the sample historical speech text information. The sample historical speech text information may be obtained by performing speech recognition on the sample historical speech signal.

[0125] Optionally, in some embodiments, the sample speech-text pairs have corresponding labeled statement categories.

[0126] Step 402: Using the initial recognition model, based on the sample speech-text pairs and sample context information, the dialogue intervention event is identified to obtain the recognition result.

[0127] The recognition results may include a predicted classification label and a predicted intensity label. The predicted classification label can be used to indicate the predicted category of the dialogue intervention event to which the sample speech text belongs, and the predicted intensity label can be used to indicate the predicted intervention intent intensity information of the dialogue intervention event that matches the predicted category.

[0128] Optionally, in some embodiments, the target speech features are obtained by encoding the historical speech signal and the sample speech signal using an audio encoder in the recognition model; the target semantic features are obtained by encoding the historical speech text information and the sample speech text information using a text encoder in the recognition model; the target speech features and the target semantic features are fused using a multimodal fusion layer in the recognition model to obtain the target features; and the target features are discriminated using a discriminator in the recognition model to obtain the recognition result.

[0129] Optionally, in some embodiments, an audio encoder is used to encode the sample historical speech signal to obtain historical speech features; an audio encoder is used to encode the sample speech signal to obtain first speech features; and target speech features are determined based on the historical speech features and the first speech features.

[0130] Optionally, in some embodiments, the sample speech signal is input into a first branch of the audio encoder to obtain a first speech prosody feature; the sample speech signal is input into a second branch of the audio encoder to obtain a first acoustic intensity feature; and the first speech feature is determined based on the first speech prosody feature and the first acoustic intensity feature.

[0131] Optionally, in some embodiments, a text encoder is used to encode the historical speech text information of the samples to obtain historical semantic features; target speech text information is determined based on the sample speech text information; a text encoder is used to encode the target speech text information to obtain reference semantic features; and target semantic features are determined based on the historical semantic features and the reference semantic features.

[0132] Optionally, in some embodiments, the target speech text information is segmented to obtain multiple words; a word library is queried to determine the key value of each word; and based on the key value, a text encoder is used to encode the target speech text information to obtain reference semantic features.

[0133] Optionally, in some embodiments, the sample speech text information and the reference text information are fused to obtain the target speech text information; wherein, the reference text information includes at least one of the following: confidence information of the sample speech text information, text length information of the sample speech text information, sentence category of the sample speech text information, and importance information corresponding to the sample context history information.

[0134] It should be noted that the method for obtaining the identification results in this disclosure is similar to the method for obtaining the identification results in the above embodiments, and will not be described in detail here.

[0135] Optionally, in some embodiments, the identification result may also include the predicted statement category.

[0136] Step 403: Train the initial recognition model based on the first difference between the labeled category and the predicted category, and the second difference between the labeled intervention intent strength information and the predicted intervention intent strength information.

[0137] It is understandable that when there is a difference between the labeled category and the predicted category and / or a difference between the labeled intervention intent intensity information and the predicted intervention intent intensity information, it indicates that the accuracy of the initial identification model is not high. In order to improve the accuracy and reliability of the identification results generated by the initial identification model and improve the prediction accuracy of the model, the model parameters of the initial identification model can be adjusted. That is, in this disclosure, the model parameters in the initial identification model can be adjusted according to the first difference between the labeled category and the predicted category, and the second difference between the labeled intervention intent intensity information and the predicted intervention intent intensity information.

[0138] As one possible implementation, a first loss value can be generated based on the first difference between the labeled category and the predicted category; a second loss value can be generated based on the second difference between the labeled intervention intent strength information and the predicted intervention intent strength information; a first target loss value can be generated based on the second loss value and the first loss value; and the initial recognition model can be trained based on the first target loss value.

[0139] The first loss value and the first difference are positively correlated; that is, the smaller the first difference, the smaller the first loss value, and vice versa. The first loss value can be, for example, cross-entropy loss or focal loss.

[0140] The second loss value and the second difference are positively correlated; that is, the smaller the second difference, the smaller the second loss value, and vice versa. The second loss value can be, for example, MSE (Mean Squared Error) loss, Huber loss, etc.

[0141] In this embodiment of the disclosure, a first target loss value can be generated based on a second loss value and a first loss value. For example, the first target loss value can be obtained by weighted summation of the second loss value and the first loss value.

[0142] In this embodiment of the disclosure, the model parameters in the initial recognition model can be adjusted according to the first target loss value so as to minimize the value of the first target loss value.

[0143] It should be noted that the above example only uses minimizing the value of the first target loss as the termination condition for the initial recognition model training. In actual applications, other termination conditions can also be set. For example, the termination condition can be that the number of training iterations reaches a set number, or the termination condition can be that the training duration reaches a set duration, etc. This disclosure does not impose any restrictions on this.

[0144] Therefore, by fusing the two types of loss values ​​to generate the first target loss value and using it to train the initial recognition model, dual supervision of the model's classification and intensity regression capabilities is achieved. This enables the trained recognition model to accurately classify intervention event categories and quantify the differences in intent intensity under different categories. This dual-dimensional loss training mechanism can improve the model's perception accuracy of complex user interaction intents and effectively reduce the probability of problems such as category misjudgment and intensity estimation bias.

[0145] As one possible implementation, when the sample speech-text pair has corresponding labeled sentence categories and the recognition result includes the predicted sentence category, a third loss value can be generated based on the third difference between the labeled sentence category and the predicted sentence category; a second target loss value can be determined based on the first target loss value and the third loss value; and the initial recognition model can be trained based on the second target loss value.

[0146] Among them, the third loss value and the third difference are positively correlated. That is, the smaller the third difference, the smaller the third loss value, and vice versa.

[0147] In this embodiment of the disclosure, a second target loss value can be determined based on a first target loss value and three loss values. For example, the second target loss value can be obtained by weighted summation of the first target loss value and three loss values.

[0148] In this embodiment of the disclosure, the initial recognition model can be trained based on the second target loss value to minimize the value of the second target loss value.

[0149] It should be noted that the above example only uses the minimization of the second target loss value as the termination condition for the initial recognition model training. In actual applications, other termination conditions can also be set. For example, the termination condition can be that the number of training iterations reaches a set number, or the termination condition can be that the training duration reaches a set duration, etc. This disclosure does not impose any restrictions on this.

[0150] Therefore, based on the first target loss value formed by fusing the original category loss and intensity loss, a third loss value targeting sentence categories is introduced. By calculating the third difference between the labeled sentence category of the sample speech-text pair and the sentence category predicted by the model, the model's ability to recognize the user's spoken sentence category is strengthened. For example, it can accurately distinguish between different sentence categories such as interrogative, agreeable, and meaningless sentences, thereby compensating for the deficiency that it is difficult to define the speech by relying solely on the category of the dialogue intervention event and the intensity of the access intention. The first target loss value and the third loss value are further fused to generate a second target loss value, which is used to train the initial recognition model. This achieves triple supervision of the model's event classification ability, intensity regression ability, and sentence category recognition ability. The trained recognition model can comprehensively and three-dimensionally analyze the user's interactive intention from three levels: event attributes, intention degree, and sentence category. This effectively reduces the intention recognition bias caused by misjudgment of sentence category and improves the model's adaptability and robustness to complex interaction scenarios.

[0151] The model training method of this disclosure involves acquiring training samples, which include sample classification labels, sample strength labels, and sample context information for sample speech-text pairs. The sample classification labels indicate the labeled category of the dialogue intervention event to which the sample speech-text pair belongs, and the sample strength labels indicate the labeled intervention intent strength information corresponding to the dialogue intervention event matching the labeled category. An initial recognition model is used to recognize dialogue intervention events based on the sample speech-text pairs and sample context information to obtain recognition results. The recognition results include predicted classification labels and predicted strength labels. The predicted classification labels indicate the predicted category of the dialogue intervention event to which the sample speech-text pair belongs, and the predicted strength labels indicate the predicted intervention intent strength information of the dialogue intervention event matching the labeled category. The initial recognition model is trained based on a first difference between the labeled category and the predicted category, and a second difference between the labeled intervention intent strength information and the predicted intervention intent strength information. Therefore, by supervising the training of the initial recognition model, the accuracy and reliability of the model's recognition results can be improved.

[0152] To implement the above embodiments, this disclosure also provides a voice interaction device. For example... Figure 5 As shown, Figure 5 This is a schematic diagram of a voice interaction device according to an embodiment of the present disclosure. The voice interaction device 500 may include: a first recognition module 501, a second recognition module 502, and a control module 503.

[0153] The first recognition module 501 is used to recognize the first voice signal in response to the user's first voice signal in the current dialogue and obtain the first voice text information.

[0154] The second recognition module 502 is used to use a trained recognition model to recognize dialogue intervention events based on the first speech text information, the first speech signal and the context history information of the current dialogue, so as to obtain the recognition result. The recognition result includes a classification label and an intensity label. The classification label is used to indicate the category of the dialogue intervention event, and the intensity label is used to indicate the intensity information of the intervention intention corresponding to the dialogue intervention event of the corresponding event type.

[0155] The control module 503 is used to control the voice interaction process based on the recognition results.

[0156] In one possible implementation of this disclosure, the contextual history information includes historical speech signals and historical speech-text information, wherein the historical speech-text information is obtained by recognizing the historical speech signals; the second recognition module 502 is configured to: encode the target speech features based on the historical speech signals and the first speech signals using an audio encoder in the recognition model; encode the target semantic features based on the historical speech-text information and the first speech-text information using a text encoder in the recognition model; fuse the target speech features and the target semantic features using a multimodal fusion layer in the recognition model to obtain target features; and discriminate the target features using a discriminator in the recognition model to obtain the recognition result.

[0157] In one possible implementation of this disclosure, the second recognition module 502 is configured to: encode a historical speech signal using an audio encoder to obtain historical speech features; encode a first speech signal using an audio encoder to obtain a first speech feature; and determine a target speech feature based on the historical speech feature and the first speech feature.

[0158] In one possible implementation of this disclosure, the second recognition module 502 is configured to: input the first speech signal into a first branch of an audio encoder to obtain a first speech prosody feature; input the first speech signal into a second branch of an audio encoder to obtain a first acoustic intensity feature; and determine the first speech feature based on the first speech prosody feature and the first acoustic intensity feature.

[0159] In one possible implementation of the embodiments of this disclosure, the first acoustic intensity feature includes at least one of the following: short-time energy, zero-crossing rate, signal-to-noise ratio, Mel-frequency cepstral coefficient, linear prediction cepstral coefficient, background noise type, and breath sound tag value; the first speech prosodic feature includes at least one of the following: fundamental frequency trajectory, fundamental frequency slope, fundamental frequency extreme point, fundamental frequency pause, syllable duration, phoneme duration ratio, severe syllable, and energy change rate.

[0160] In one possible implementation of this disclosure, the second recognition module 502 is configured to: encode historical speech text information using a text encoder to obtain historical semantic features; determine target speech text information based on the first speech text information; encode the target speech text information using a text encoder to obtain reference semantic features; and determine target semantic features based on the historical semantic features and the reference semantic features.

[0161] In one possible implementation of this disclosure, the second recognition module 502 is used to: perform word segmentation on the target speech text information to obtain multiple words; query a word library to determine the key value of each word; and encode the target speech text information using a text encoder based on the key value to obtain reference semantic features.

[0162] In one possible implementation of this disclosure, the second recognition module 502 is used to: fuse the first speech text information and the reference text information to obtain the target speech text information; wherein the reference text information includes at least one of the following: confidence information of the first speech text information, text length information of the first speech text information, sentence category of the first speech text information, and importance information corresponding to context history information.

[0163] In one possible implementation of this disclosure, the control module 503 is configured to: determine a corresponding control decision based on a combination of the type of dialogue intervention event and the intensity of intervention intent information; and use the control decision to perform dialogue control on the voice interaction process.

[0164] In one possible implementation of this disclosure, the control decision includes at least one of the following: action decision, buffering strategy information, and delay time parameter; the action decision is used to control the interactive behavior during the voice interaction process; the buffering strategy information is used to indicate the timing and method of executing the action decision during the voice interaction process; and the delay time parameter is used to indicate the timing of executing the action decision during the voice interaction process.

[0165] It should be noted that the voice interaction device provided in this disclosure can implement all the method steps implemented in any of the above-described voice interaction method embodiments and can achieve the same technical effect. Therefore, the parts and beneficial effects that are the same as those in the method embodiments will not be described in detail here.

[0166] The voice interaction device of this embodiment responds to a user's first voice signal in the current dialogue, identifies the first voice signal to obtain first voice text information; uses a trained recognition model to identify dialogue intervention events based on the first voice text information, the first voice signal, and the contextual history information of the current dialogue to obtain the current recognition result; wherein, the current recognition result includes a classification label and a strength label, the classification label is used to indicate the category of the dialogue intervention event, and the strength label is used to indicate the intervention intent strength information corresponding to the dialogue intervention event of the corresponding event type; and performs dialogue control on the voice interaction process according to the current recognition result. Therefore, by combining multimodal information—namely, speech-text information, speech signal features, and contextual history information—and leveraging a pre-trained recognition model, the system can classify dialogue intervention events and quantify the intensity of intervention intentions. This not only distinguishes different categories of dialogue intervention events but also intuitively reflects the user's true intention to dominate the conversation through intensity labels. Based on the dual decision-making criteria of classification and intensity labels, differentiated dialogue control strategies can be implemented. For example, for low-intensity, meaningless insertions, the system can maintain continuous speaking to avoid dialogue interruptions due to oversensitivity. Conversely, for high-intensity proactive takeover requests, the system can promptly terminate the current output and switch to listening mode, ensuring the full satisfaction of the user's expressed needs. This dialogue control mechanism not only improves the fluency and naturalness of human-computer voice interaction, eliminating user frustration caused by frequent interruptions or the robot "talking to itself," but also enhances the accuracy and efficiency of system task execution, improving the user experience. It provides technical support that is more aligned with users' real interaction habits for scenarios such as intelligent customer service, in-vehicle voice assistants, and smart home voice interaction.

[0167] To implement the above embodiments, this disclosure also provides a model training apparatus. For example... Figure 6 As shown, Figure 6This is a schematic diagram of a model training apparatus according to an embodiment of the present disclosure. The model training apparatus 600 may include: a selection module 601, a recognition module 602, and a training module 603.

[0168] The acquisition module 601 is used to acquire training samples. The training samples include sample classification labels, sample strength labels, and sample context information for sample speech-text pairs. The sample classification labels are used to indicate the annotation category of the dialogue intervention event to which the sample speech-text pair belongs, and the sample strength labels are used to indicate the annotation intervention intent strength information corresponding to the dialogue intervention event that matches the annotation category.

[0169] The recognition module 602 is used to identify dialogue intervention events based on sample speech-text pairs and sample context information using an initial recognition model, so as to obtain recognition results. The recognition results include predicted classification labels and predicted intensity labels. The predicted classification labels are used to indicate the predicted category of the dialogue intervention event to which the sample speech-text pair belongs, and the predicted intensity labels are used to indicate the predicted intervention intent intensity information of the dialogue intervention event that matches the labeled category.

[0170] Training module 603 is used to train an initial recognition model based on a first difference between the labeled category and the predicted category, and a second difference between the labeled intervention intent intensity information and the predicted intervention intent intensity information.

[0171] In one possible implementation of this disclosure, the training module 603 is configured to: generate a first loss value based on a first difference between the labeled category and the predicted category; generate a second loss value based on a second difference between the labeled intervention intent strength information and the predicted intervention intent strength information; generate a first target loss value based on the second loss value and the first loss value; and train an initial recognition model based on the first target loss value.

[0172] In one possible implementation of this disclosure, the sample speech-text pair has a corresponding labeled sentence category, and the corresponding recognition result includes the predicted sentence category; the training module 603 is used to: generate a third loss value based on the third difference between the labeled sentence category and the predicted sentence category; determine a second target loss value based on the first target loss value and the third loss value; and train the initial recognition model based on the second target loss value.

[0173] The model training apparatus of this embodiment acquires training samples, which include sample classification labels, sample strength labels, and sample context information for sample speech-text pairs. The sample classification labels indicate the labeled category of the dialogue intervention event to which the sample speech-text pair belongs, and the sample strength labels indicate the labeled intervention intent strength information corresponding to the dialogue intervention event matching the labeled category. An initial recognition model is used to recognize dialogue intervention events based on the sample speech-text pairs and sample context information to obtain recognition results. The recognition results include predicted classification labels and predicted strength labels. The predicted classification labels indicate the predicted category of the dialogue intervention event to which the sample speech-text pair belongs, and the predicted strength labels indicate the predicted intervention intent strength information of the dialogue intervention event matching the labeled category. The initial recognition model is trained based on a first difference between the labeled category and the predicted category, and a second difference between the labeled intervention intent strength information and the predicted intervention intent strength information. Therefore, by supervising the initial recognition model, the accuracy and reliability of the model's recognition results can be improved.

[0174] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision, and disclosure of any type of information, such as user personal information, are all carried out with the user's consent and comply with relevant laws and regulations, and do not violate public order and good morals.

[0175] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0176] Figure 7 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0177] like Figure 7As shown, device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 802 or a computer program loaded from storage unit 808 into random access memory (RAM) 803. RAM 803 may also store various programs and data required for the operation of device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804.

[0178] Multiple components in device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of monitors, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0179] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as voice interaction methods or model training methods. For example, in some embodiments, the voice interaction method or model training method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the voice interaction method or model training method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform a voice interaction method or a model training method by any other suitable means (e.g., by means of firmware).

[0180] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0181] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0182] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0183] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0184] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0185] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0186] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0187] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A voice interaction method, wherein, The method includes: In response to a user’s first voice signal in the current conversation, the first voice signal is recognized to obtain first voice text information; Using a trained recognition model, based on the first speech text information, the first speech signal, and the contextual history information of the current dialogue, the recognition of dialogue intervention events is performed to obtain the recognition result; wherein, the recognition result includes a classification label and an intensity label, the classification label is used to indicate the category of the dialogue intervention event, and the intensity label is used to indicate the intensity information of the intervention intention corresponding to the dialogue intervention event of the corresponding event type; Based on the recognition results, dialogue control is implemented during the voice interaction process.

2. The method according to claim 1, wherein, The contextual history information includes historical speech signals and historical speech-text information, wherein the historical speech-text information is obtained by recognizing the historical speech signals; the step of using a trained recognition model to identify dialogue intervention events based on the first speech-text information, the first speech signal, and the contextual history information of the current dialogue, in order to obtain the recognition result, includes: Based on the historical speech signal and the first speech signal, the target speech features are obtained by encoding using the audio encoder in the recognition model. Based on the historical speech-text information and the first speech-text information, the text encoder in the recognition model is used to encode the target semantic features. The multimodal fusion layer in the recognition model is used to fuse the target speech features and the target semantic features to obtain the target features; The discriminator in the recognition model is used to identify the target features, and the recognition result is obtained.

3. The method according to claim 2, wherein, The step of encoding the target speech features based on the historical speech signal and the first speech signal using the audio encoder in the recognition model to obtain the target speech features includes: The audio encoder is used to encode the historical speech signal to obtain historical speech features; The first speech signal is encoded using the audio encoder to obtain the first speech feature; The target speech features are determined based on the historical speech features and the first speech features.

4. The method according to claim 3, wherein, The step of encoding the first speech signal using the audio encoder to obtain the first speech feature includes: The first speech signal is input into the first branch of the audio encoder to obtain the first speech prosodic feature; The first speech signal is input into the second branch of the audio encoder to obtain the first acoustic intensity feature; The first speech feature is determined based on the first speech prosody feature and the first acoustic intensity feature.

5. The method according to claim 4, wherein, The first acoustic intensity feature includes at least one of the following: short-time energy, zero-crossing rate, signal-to-noise ratio, Mel-frequency cepstral coefficient, linearly predicted cepstral coefficient, background noise type, and breath sound tag value; The first speech prosodic feature includes at least one of the following: fundamental frequency trajectory, fundamental frequency slope, fundamental frequency extreme point, fundamental frequency pause, syllable duration, phoneme duration ratio, stressed syllable, and energy change rate.

6. The method according to claim 2, wherein, The step of encoding the target semantic features based on the historical speech-text information and the first speech-text information using the text encoder in the recognition model to obtain the target semantic features includes: The historical speech text information is encoded using the text encoder to obtain historical semantic features; Based on the first speech text information, determine the target speech text information; The target speech text information is encoded using the text encoder to obtain reference semantic features; The target semantic features are determined based on the historical semantic features and the reference semantic features.

7. The method according to claim 6, wherein, The step of encoding the target speech text information using the text encoder to obtain reference semantic features includes: The target speech text information is segmented into multiple words. The terminology database is queried to determine the key value for each of the stated terms; Based on the key values, the target speech text information is encoded using the text encoder to obtain the reference semantic features.

8. The method according to claim 6, wherein, The step of determining the target speech text information based on the first speech text information includes: The first speech-text information and the reference text information are fused to obtain the target speech-text information; The reference text information includes at least one of the following: The confidence information of the first speech text information, the text length information of the first speech text information, the sentence category of the first speech text information, and the importance information corresponding to the context history information.

9. The method according to any one of claims 1-8, wherein, The step of controlling the voice interaction process based on the recognition result includes: Based on the combination of the category of the dialogue intervention event and the intensity of the intervention intention information, the corresponding control decision is determined; The control decision is used to perform dialogue control on the voice interaction process.

10. The method according to claim 9, wherein, The control decision includes at least one of the following: Action decision, buffer strategy information, delay time parameters; The action decision is used to control the interactive behavior during the voice interaction process; The buffering strategy information is used to indicate the timing and method of executing the action decision during voice interaction; The delay time parameter is used to indicate the timing of the action decision during voice interaction.

11. A model training method, wherein, The method includes: Obtain training samples; wherein, the training samples include sample classification labels, sample strength labels and sample context information for sample speech-text pairs, the sample classification labels are used to indicate the annotation category of the dialogue intervention event to which the sample speech-text pair belongs, and the sample strength labels are used to indicate the annotation intervention intent strength information corresponding to the dialogue intervention event that matches the annotation category; An initial recognition model is used to identify dialogue intervention events based on the sample speech-text pairs and the sample context information to obtain recognition results. The recognition results include a predicted classification label and a predicted intensity label. The predicted classification label is used to indicate the predicted category of the dialogue intervention event to which the sample speech-text pair belongs, and the predicted intensity label is used to indicate the predicted intervention intent intensity information of the dialogue intervention event that matches the predicted category. The initial recognition model is trained based on a first difference between the labeled category and the predicted category, and a second difference between the labeled intervention intent strength information and the predicted intervention intent strength information.

12. The method according to claim 11, wherein, The step of training the initial recognition model based on a first difference between the labeled category and the predicted category, and a second difference between the labeled intervention intent strength information and the predicted intervention intent strength information, includes: A first loss value is generated based on the first difference between the labeled category and the predicted category; A second loss value is generated based on the second difference between the labeled intervention intent strength information and the predicted intervention intent strength information; A first target loss value is generated based on the second loss value and the first loss value; The initial recognition model is trained based on the first target loss value.

13. The method according to claim 12, wherein, The sample speech-text pairs have corresponding labeled sentence categories, and correspondingly, the recognition result includes the predicted sentence category; training the initial recognition model based on the first target loss value includes: A third loss value is generated based on the third difference between the labeled statement category and the predicted statement category; Based on the first target loss value and the third loss value, determine the second target loss value; The initial recognition model is trained based on the second target loss value.

14. A voice interaction device, wherein, The device includes: The first recognition module is used to recognize the first voice signal of the user in the current dialogue and obtain the first voice text information. The second recognition module is used to use a trained recognition model to recognize dialogue intervention events based on the first speech text information, the first speech signal, and the context history information of the current dialogue, so as to obtain the recognition result; wherein, the recognition result includes a classification label and an intensity label, the classification label is used to indicate the category of the dialogue intervention event, and the intensity label is used to indicate the intensity information of the intervention intention corresponding to the dialogue intervention event of the corresponding event type. The control module is used to control the voice interaction process based on the recognition results.

15. A model training device, wherein, The device includes: The acquisition module is used to acquire training samples; wherein, the training samples include sample classification labels, sample strength labels and sample context information of sample speech-text pairs, the sample classification labels are used to indicate the annotation category of the dialogue intervention event to which the sample speech-text pair belongs, and the sample strength labels are used to indicate the annotation intervention intent strength information corresponding to the dialogue intervention event that matches the annotation category; The recognition module is used to identify dialogue intervention events based on the sample speech-text pairs and the sample context information using an initial recognition model, so as to obtain recognition results; wherein, the recognition results include a predicted classification label and a predicted intensity label, the predicted classification label is used to indicate the predicted category of the dialogue intervention event to which the sample speech-text pair belongs, and the predicted intensity label is used to indicate the predicted intervention intent intensity information of the dialogue intervention event that matches the labeled category; The training module is used to train the initial recognition model based on a first difference between the labeled category and the predicted category, and a second difference between the labeled intervention intent strength information and the predicted intervention intent strength information.

16. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 10, or the method of any one of claims 11 to 13.

17. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 10, or the method according to any one of claims 11 to 13.

18. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 10, or the method according to any one of claims 11 to 13.

Images