A voice interaction method and apparatus

Abstract

The application provides a voice interaction method and device, wherein the method comprises: judging whether the streaming audio data belongs to the interactive voice according to the obtained streaming audio data and the interaction context; determining the first end prediction result of the interactive voice and the waiting time length in response to the streaming audio data belonging to the interactive voice; continuously collecting incremental audio data within the waiting time length, and if the incremental audio data belongs to the interactive voice, splicing the incremental audio data and the streaming audio data, and determining the second end prediction result of the target audio data obtained by splicing and a new waiting time length until the new incremental audio data collected within the new waiting time length does not belong to the interactive voice; and inputting the target audio data to a downstream interaction processing unit to generate response information corresponding to the target audio data. The embodiment can dynamically determine the user turn boundary, avoid premature response or false triggering of the interaction, and thus improve the accuracy and naturalness of the streaming full-duplex voice interaction.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more specifically, to a voice interaction method and apparatus. Background Technology

[0002] With the development of technologies such as large language models, speech recognition, and speech synthesis, voice interaction systems are gradually evolving from traditional half-duplex interaction to streaming full-duplex interaction. In streaming full-duplex voice interaction, the system can continuously receive the user's voice input and determine the response timing based on the user's turn-taking status. Simultaneously, it continues to listen to the user's voice while outputting its own, supporting user interruptions and continuous interaction, thereby improving the real-time nature and naturalness of voice interaction.

[0003] However, in existing technologies, the ability to decide dialogue turns is usually deeply coupled with the specific voice interaction model or main interaction link, making it difficult to optimize related capabilities and adapt to different voice interaction architectures. Furthermore, in real-world applications, the input audio may contain background voices, conversations with others, and other non-target speech. Existing solutions have limited ability to determine whether the input speech is directed to the voice interaction system, easily misinterpreting non-interactive speech as valid interactive input. This can lead to false interruptions, incorrect responses, or inaccurate turn boundary judgments, affecting the accuracy and user experience of streaming full-duplex voice interaction. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide at least one voice interaction method, device, electronic device, and storage medium, which can dynamically determine the user's turn boundaries, avoid premature response or accidental triggering of interaction, thereby improving the accuracy and naturalness of streaming full-duplex voice interaction.

[0005] In a first aspect, embodiments of the present invention provide a voice interaction method, including: Based on the acquired streaming audio data and the interaction context, determine whether the streaming audio data belongs to interactive speech; In response to the fact that the streaming audio data belongs to interactive speech, the first end prediction result of the interactive speech is determined, and the waiting time is determined based on the first end prediction result; During the waiting period, incremental audio data of streaming audio data is continuously collected. If the incremental audio data belongs to interactive speech, the incremental audio data is concatenated with the streaming audio data, and the second end prediction result of the concatenated target audio data is determined. A new waiting period is determined based on the second end prediction result until the new incremental audio data collected during the new waiting period does not belong to interactive speech. The target audio data is input to the downstream interactive processing unit so that the interactive processing unit generates response information corresponding to the target audio data.

[0006] Optionally, based on the acquired streaming audio data and the interaction context, it is determined whether the streaming audio data belongs to interactive speech, including: Speech activity detection is performed on streaming audio data, and the streaming audio data is segmented based on the speech activity detection results to obtain at least one valid speech segment corresponding to the streaming audio data. Valid speech segments are input into a pre-trained interaction decision model, so that the interaction decision model outputs the interaction decision result of the streaming audio data based on the valid speech segments and the interaction context.

[0007] Optionally, the interaction decision model is trained based on the following steps: Obtain manually labeled data; manually labeled data includes interactive tags used to indicate whether conversational audio data belongs to interactive speech; Obtain pseudo-labeled data; the pseudo-labeled data is obtained based on the following steps: perform speech activity detection on the conversation audio data, segment the conversation audio data based on the speech activity detection results to obtain at least one speech segment; perform speech recognition on the speech segment to obtain the text information corresponding to the speech segment; concatenate the text information into dialogue data according to the time sequence of the speech segments, and determine the interaction label of each speech segment based on the dialogue data; Obtain scene annotation data; the scene annotation data is obtained based on the following steps: construct different interaction contexts for each session audio data according to scene requirements information; input the interaction context and session audio data into the annotation model so that the annotation model outputs the interaction labels of the session audio data; A training dataset is constructed based on manually labeled data, pseudo-labeled data, and scene labeled data, and an initial judgment model is trained based on the interaction labels in the training dataset. The initial decision model was fine-tuned using manually labeled data from the training dataset to obtain an interactive decision model.

[0008] Optionally, an initial decision model is trained based on the interaction labels in the training dataset, including: For each session audio data in the training dataset, complete audio data and local audio data sampled from the complete audio data are obtained respectively; Use complete audio data and partial audio data as training samples to input into the pre-trained speech model; For each training sample, a corresponding interaction label and an end prediction label are determined. The interaction label indicates whether the training sample belongs to interactive speech, and the end prediction label indicates whether the training sample corresponds to the end state of a user's turn. The pre-trained speech model is trained based on the interaction label and the end prediction label to obtain the initial judgment model.

[0009] Optionally, the pre-trained speech model is trained based on the interaction label and the end prediction label, including: Obtain the speech recognition text corresponding to each complete audio data and partial audio data in the training samples; For complete audio data and partial audio data, construct a first sample containing speech recognition text and a second sample not containing speech recognition text, respectively; The first and second samples are input into the pre-trained speech model, so that the pre-trained speech model outputs the interactive prediction result and the end prediction result of the training samples. Based on the interaction prediction results and interaction labels, determine the first loss of the training samples, and based on the termination prediction results and termination prediction labels, determine the second loss of the training samples. The pre-trained speech model is trained based on the first loss and the second loss to obtain the initial judgment model.

[0010] Optionally, the waiting time is determined based on the first termination prediction result, including: Obtain the predicted probability of the user's turn ending corresponding to the interactive speech in the first ending prediction result; The waiting time is determined based on a preset duration and a predicted probability; the waiting time is negatively correlated with the predicted probability.

[0011] Secondly, embodiments of the present invention provide a voice interaction device, including: The first judgment module is used to determine whether the streaming audio data belongs to interactive speech based on the acquired streaming audio data and the interaction context. The determination module is used to determine the first end prediction result of the interactive speech in response to the fact that the streaming audio data belongs to interactive speech, and to determine the waiting time based on the first end prediction result; The second judgment module is used to continuously collect incremental audio data of streaming audio data within the first waiting time. If the incremental audio data belongs to interactive speech, the incremental audio data is spliced ​​with the streaming audio data, and the second end prediction result of the spliced ​​target audio data is determined. A new waiting time is determined based on the second end prediction result until the new incremental audio data collected within the new waiting time does not belong to interactive speech. The transmission module is used to input the target audio data to the downstream interactive processing unit, so that the interactive processing unit can generate response information corresponding to the target audio data.

[0012] Thirdly, embodiments of the present invention also provide an electronic device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, the steps of the first aspect or any optional implementation of the first aspect are performed.

[0013] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the first aspect or any optional implementation thereof.

[0014] Fifthly, embodiments of the present invention also provide a computer program product, including a computer program that, when executed by a processor, implements the method of any of the above embodiments.

[0015] In any of the above aspects or any implementation of any aspect, based on the acquired streaming audio data and the interaction context, interactive speech judgment is performed on the current streaming audio data to determine whether the current audio is valid speech issued by the user to the voice interaction system. When the judgment result indicates that the current streaming audio data does not belong to interactive speech, subsequent interactive processing is not triggered, thereby avoiding background human voices, conversations with others, or other non-target speech from mistakenly triggering the system response. When it is determined that the current streaming audio data belongs to interactive speech, the first end prediction result of the interactive speech is further determined, and the waiting time is dynamically determined based on the first end prediction result. Since different users have different speaking habits and pauses in different semantic expression processes, the system does not need to use a fixed silence duration as the turn-ending condition, but can adaptively determine the waiting time based on the end prediction result of the current interactive speech. When the user continues to input incremental audio data within the waiting time, and the incremental audio data still belongs to interactive speech, it indicates that the user may have only paused briefly and has not yet completed the current turn. At this time, the incremental audio data is concatenated with the existing streaming audio data, and the end prediction is re-performed based on the concatenated target audio data to further determine a new waiting time. By continuously collecting data, making interactive judgments, splicing audio, and dynamically updating the waiting time, the system can gradually confirm the true turn boundary as the user continues to input voice. Only when no new interactive voice is detected within the corresponding waiting time is the currently determined target audio data input to the downstream interactive processing unit to generate response information. Therefore, this solution can avoid prematurely ending the current turn and triggering a response due to a short pause by the user. At the same time, by judging whether the incremental audio data belongs to interactive voice, it reduces false interruptions or false responses caused by non-target voice, thereby achieving dynamic determination of the user's turn boundary and improving the accuracy, real-time performance, and naturalness of streaming full-duplex voice interaction.

[0016] The beneficial effects of the aforementioned voice interaction device, electronic device, and storage medium are described in the description of the aforementioned voice interaction method, and will not be repeated here. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below. These drawings are incorporated in and constitute a part of this specification. They illustrate embodiments conforming to the present invention and, together with the specification, serve to explain the technical solutions of the present invention. It should be understood that the following drawings only show some embodiments of the present invention and should not be considered as a limitation of the scope. For those skilled in the art, other related drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 A flowchart of a voice interaction method provided by an embodiment of the present invention is shown; Figure 2 A schematic diagram of a voice interaction device provided in an embodiment of the present invention is shown; Figure 3 An exemplary system architecture in which embodiments of the present invention can be applied is shown; Figure 4 A schematic diagram of the structure of a computer system used to implement embodiments of the present invention is shown. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0020] It should be noted that the collection, use, storage, sharing and transfer of user personal information involved in the technical solution of the present invention all comply with the provisions of relevant laws and regulations, and require notification to users and obtaining their consent or authorization. When applicable, user personal information is subjected to de-identification and / or anonymization and / or encryption technical processing.

[0021] The above problems and solutions are the result of the inventor's practice and careful research. The discovery process of the above problems and the solutions proposed for the above problems should be considered as the inventor's contribution to the invention.

[0022] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0023] To facilitate understanding of this embodiment, a detailed description of the voice interaction method disclosed in this invention will be provided first. The execution entity of the voice interaction method provided in this invention is generally a computer device with certain computing capabilities. This computer device may include, for example, a terminal device, a server, or other processing devices. The terminal device may be a user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. In some possible implementations, this voice interaction method can be implemented by a processor calling computer-readable instructions stored in memory.

[0024] See Figure 1 The diagram shows a flowchart of a voice interaction method provided in an embodiment of the present invention. The method includes steps S101 to S104, wherein: S101: Based on the acquired streaming audio data and the interaction context, determine whether the streaming audio data belongs to interactive speech.

[0025] In this embodiment of the invention, streaming audio data refers to audio data continuously acquired by the voice acquisition device during voice interaction and input into the voice interaction system in batches or segments according to time sequence. Unlike non-streaming audio data, which is input and processed all at once after a complete voice segment is acquired, streaming audio data can be continuously generated and acquired by the system while the user is still speaking.

[0026] Interaction context refers to contextual information used to assist in determining whether the current streaming audio data belongs to interactive speech and to determine the user's turn status. In one feasible implementation, interaction context may include interaction task instruction information, voice interaction system information, historical memory information, and historical multimodal interaction context information. Interaction task instruction information is used to indicate the interaction task that the current model needs to perform; for example, it can be used to instruct the model to perform an interaction initiation task or an interaction termination task. Voice interaction system information may include the name, nickname, persona, application scenario, or other information used to characterize the voice interaction system, enabling the model to determine the user's intended audience based on the specific identity of the current voice interaction system. For example, when the name of the voice interaction system is "Xiaoming," for the voice content "Xiaoming, help me check the weather," the voice interaction system information can be used to determine whether the voice is directed at the current voice interaction system. Historical memory information can be key information extracted from dialogue content generated during historical interactions, used to characterize the long-term or phased contextual state of the current voice interaction process. For example, user characteristics, user preferences, and recently mentioned named entity nouns can be extracted from historical dialogue content using a language model, and this extracted information can be used as historical memory information. Compared to relatively fixed information in a voice interaction system, historical memory information can be dynamically updated as the interaction progresses, enabling the model to understand the user's subsequent speech in conjunction with the current session state. Historical multimodal interaction context information can include at least one round of historical interaction data that has occurred in the current session, such as historical user speech, text information corresponding to historical user speech, and response information generated by the voice interaction system in response to historical user speech.

[0027] For example, for the interaction initiation task, which determines whether the current audio data is an initiation of a dialogue with a voice interaction system, the interaction context could be: "You are a human-computer interaction analysis expert, skilled at identifying the speaker's dialogue partner through voice signals and dialogue context. Your analysis scenario is: a dialogue between a voice interaction device and a user. The device's microphone continuously picks up audio, and the automatic speech recognition unit converts all captured speech into text (not necessarily accurate), but not all recorded speech is the user speaking to the device." For the interaction termination task, which determines whether the user's speech has ended, the interaction context could be: "You are a human-computer interaction analysis expert, skilled at determining whether the user has finished speaking through voice signals and dialogue context. Your analysis scenario is: a dialogue between a voice interaction device and a user. The device's microphone continuously picks up audio, and the automatic speech recognition unit converts all captured speech into text (not necessarily accurate). The device needs to determine whether the user has fully expressed themselves in this round, thus deciding whether to immediately take over and respond, or continue to wait for the user to finish speaking." This allows the model to learn and distinguish between voices that have triggered interactions and other voices that have not, and to make judgments on newly input streaming audio data in conjunction with the current session state.

[0028] In this embodiment of the invention, interactive speech refers to valid speech input by the speaker into the voice interaction process with the voice interaction system as the communication target. In other words, interactive speech does not refer only to any detected human voice, but to speech with the intention to communicate with the current voice interaction system. For example, when a user directly asks a question, issues a command, or supplements the response of the voice interaction system, the corresponding speech can be identified as interactive speech; while background voices, conversations between others, self-talk unrelated to the current voice interaction, echoing speech, or other speech not directed at the current voice interaction system can be identified as non-interactive speech. By combining streaming audio data and the above-mentioned interactive context for judgment, it is possible to identify whether the current speech is truly directed at the voice interaction system during the user's continuous speaking, thereby performing subsequent turn-ending prediction and interactive processing only for interactive speech, reducing the probability of false interruptions or false responses caused by non-target speech.

[0029] In this embodiment of the invention, determining whether the streaming audio data belongs to interactive speech based on the acquired streaming audio data and the interaction context includes: performing speech activity detection on the streaming audio data; segmenting the streaming audio data based on the speech activity detection results to obtain at least one valid speech segment corresponding to the streaming audio data; and inputting the valid speech segment into a pre-trained interaction determination model so that the interaction determination model outputs the interaction determination result of the streaming audio data based on the valid speech segment and the interaction context.

[0030] In practice, streaming audio data can be continuously acquired by a voice acquisition device and input into the voice interaction system in sequential audio data blocks. Since the actual acquired streaming audio data may contain silence, environmental noise, and speech from different speakers, continuously inputting all streaming audio data into the interaction decision model would not only increase the computational burden of the model but also potentially allow a large amount of silent data without practical interactive significance to participate in subsequent judgments. Therefore, speech activity detection can be performed on the streaming audio data first to identify time regions where speech activity exists. Speech activity detection can be implemented based on a pre-trained speech activity detection model. By analyzing the acoustic features in the streaming audio data, it can determine whether there is valid human voice activity at different time locations and generate corresponding speech activity detection results. The speech activity detection results can be used to characterize the speech state corresponding to each time period in the streaming audio data. For example, the time period in which human voice activity was detected can be defined as the speech activity region, and the time period in which no human voice activity was detected can be defined as the silence region or invalid audio region. In some implementations, voice activity detection can also employ real-time processing, continuously outputting corresponding voice activity detection results as streaming audio data continues to be input, thus eliminating the need to wait for the complete acquisition of conversational audio data.

[0031] Based on the voice activity detection results, streaming audio data can be segmented to extract the audio segments containing voice activity, obtaining at least one valid voice segment corresponding to the streaming audio data. Specifically, when continuous streaming audio data is detected to contain voice activity, the continuous voice activity region can be identified as a valid voice segment; when there is a pause between adjacent voice activity regions that meets preset conditions, the streaming audio data can be segmented according to the pause position to obtain different valid voice segments. Through voice activity detection and segmentation processing, the number of silent or other invalid audio data entering the interaction judgment model can be reduced, allowing the interaction judgment model to focus on processing valid voice segments containing actual voice content. It should be noted that a valid voice segment only indicates an audio segment that is identified as containing voice activity through voice activity detection, and does not necessarily mean that a valid voice segment belongs to interactive voice. For example, in real-world usage scenarios, the voice activity detected in the streaming audio data may come from a user communicating with the voice interaction system, or it may come from background voices, conversations between others, the user's echoing voice, self-talk, or other voices not directed at the voice interaction system. Therefore, after obtaining valid speech segments, it is necessary to further determine whether the valid speech segments belong to interactive speech through an interactive judgment model.

[0032] After obtaining at least one valid speech segment, the valid speech segment can be input into a pre-trained interaction judgment model. Simultaneously, the interaction context is provided to the interaction judgment model, allowing it to comprehensively judge based on the speech content of the valid speech segment and the interaction context, and output the interaction judgment result for the streaming audio data. For example, the speech content corresponding to the current valid speech segment is "Xiaoming, help me check the weather," and the interaction context contains the identity information of the voice interaction system. When the identity information indicates that the name or nickname of the current voice interaction system is "Xiaoming," the interaction judgment model can determine that the speech belongs to interactive speech based on the valid speech segment and the interaction context. When the interaction context indicates that the current voice interaction system is not "Xiaoming," the interaction judgment model can determine that the current speech may be addressed to someone else and is not interactive speech. Furthermore, when the speech content of the valid speech segment itself is difficult to directly identify the speaker, the interaction judgment model can also combine historical interaction information and historical memory information to determine whether the current speech has a semantic connection with the previous interaction process, thereby improving the accuracy of the interaction judgment. When the interaction judgment result indicates that the streaming audio data belongs to interactive speech, subsequent end prediction processing can continue to be performed on the current streaming audio data to determine whether the user has completed the current turn. When the interaction judgment result indicates that the streaming audio data does not belong to interactive speech, subsequent interaction processing can be avoided, and new streaming audio data can continue to be received. This avoids directly using silence, environmental noise, background voices, or other speech not designed for voice interaction systems as user interaction input, thereby reducing the probability of false triggering of interactions and reducing unnecessary model inference overhead while ensuring the accuracy of interaction judgment.

[0033] In this embodiment of the invention, the interaction determination model can construct a training dataset using manually labeled data, pseudo-labeled data, and scene labeled data, and train it using a combination of pre-training and fine-tuning, thereby enabling the interaction determination model to determine whether conversational audio data belongs to interactive speech. Specifically, the interaction determination model is trained based on the following steps: acquiring manually labeled data; the manually labeled data includes interaction labels used to indicate whether the conversation audio data belongs to interactive speech; acquiring pseudo-labeled data; the pseudo-labeled data is obtained based on the following steps: performing speech activity detection on the conversation audio data, segmenting the conversation audio data based on the speech activity detection results to obtain at least one speech segment; performing speech recognition on the speech segment to obtain text information corresponding to the speech segment; concatenating the text information into dialogue data according to the time sequence of the speech segments, and determining the interaction labels of each speech segment based on the dialogue data; acquiring scene labeling data; the scene labeling data is obtained based on the following steps: constructing different interaction contexts for each conversation audio data according to scene requirement information; inputting the interaction context and conversation audio data into the labeling model so that the labeling model outputs the interaction labels of the conversation audio data; constructing a training dataset based on the manually labeled data, pseudo-labeled data, and scene labeling data, and training an initial determination model based on the interaction labels in the training dataset; fine-tuning the initial determination model using the manually labeled data in the training dataset to obtain the interaction determination model.

[0034] In practice, manually labeled data can be obtained by manually annotating conversational audio data. Specifically, conversational audio data can be analyzed according to conversation level, and corresponding interaction labels can be determined based on the actual user interaction in the audio. Interaction labels are used to indicate whether the conversational audio data belongs to interactive speech, that is, to distinguish between valid speech issued by the user to the voice interaction system and background voices, conversations with others, user echoing, or other audio not issued to the voice interaction system. Manually labeled data has high annotation accuracy, enabling the training results of the interaction judgment model to maintain good consistency with the human judgment results. However, because manual annotation requires a high investment of time and manpower, it is usually difficult to rely solely on manually labeled data to meet the needs of large-scale model training.

[0035] To expand the training data, pseudo-labeled data can be further acquired. In constructing pseudo-labeled data, speech activity detection can be performed on another batch of conversation audio data. Based on the speech activity detection results, time regions containing speech activity in the conversation audio data can be identified, and the conversation audio data can be segmented based on the speech activity detection results to obtain at least one speech segment. Then, speech recognition can be performed on each speech segment to obtain the corresponding text information. The text information is then concatenated according to the chronological order of the speech segments to obtain dialogue data that reflects the conversation process. Based on the dialogue data, the chronological order, semantic relationships, and dialogue content between speech segments can be combined to determine whether each speech segment belongs to the speech directed to the voice interaction system, and corresponding interaction labels can be assigned to each speech segment. In this way, large-scale training data with interaction labels can be automatically constructed using existing conversation audio data, thereby reducing manual annotation costs and expanding the scale of training data.

[0036] Furthermore, for the same conversational audio data, the determination of whether it belongs to interactive speech may yield different results under different interactive contexts. Therefore, different interactive contexts can be constructed for each conversational audio data based on scenario requirements to improve the adaptability of the interaction determination model to different interactive scenarios. Scenario requirements can reflect information such as the identity, interactive task, or dialogue state of the voice interaction system in different application scenarios. For example, for conversational audio data containing the content "Xiaoming, help me check the weather," if the interactive context includes the nickname of the current voice interaction system as "Xiaoming," the conversational audio data can be determined to belong to interactive speech; if the interactive context indicates that the nickname of the current voice interaction system is not "Xiaoming," the conversational audio data can be determined not to belong to interactive speech. In the process of constructing scenario annotation data, the interactive context and conversational audio data can be input into the annotation model, so that the annotation model can combine the speech content in the conversational audio data and the interactive context to analyze the current conversational audio data and output the corresponding interactive labels. It should be noted that the above methods for obtaining various types of labeled data are only illustrative examples of feasible implementation methods in the embodiments of the present invention, and do not constitute an improper limitation of the present invention. In practical applications, specific methods for obtaining various types of labeled data can be set according to actual conditions and needs, and the embodiments of the present invention do not make specific limitations in this regard.

[0037] After obtaining manually labeled data, pseudo-labeled data, and scene-labeled data, the three types of labeled data can be mixed according to a pre-set ratio to obtain a training dataset. This ratio can be set according to actual needs; for example, increasing the proportion of manually labeled data can improve the accuracy of the initial judgment model. This embodiment of the invention does not specifically limit this. Each training sample in the training dataset can include conversation audio data, the corresponding interaction context, and interaction labels. Then, the interaction labels in the training dataset can be used to train the pre-trained speech model, enabling it to learn the association between conversation audio data and interaction context, and output a judgment result indicating whether the conversation audio data belongs to interactive speech, thus obtaining the initial judgment model. Since manually labeled data has relatively high labeling accuracy, after the initial judgment model completes training based on multiple training datasets, it can be further fine-tuned using the manually labeled data in the training dataset, enabling the initial judgment model to further learn the interactive speech judgment ability consistent with the human judgment result, ultimately obtaining the interactive judgment model. This balances the scale and labeling accuracy of the training data, improving the recognition accuracy of the interactive judgment model for interactive speech in different conversational scenarios.

[0038] In this embodiment of the invention, training an initial judgment model based on interaction labels in the training dataset includes: acquiring complete audio data and partial audio data sampled from the complete audio data for each session audio data in the training dataset; inputting the complete audio data and partial audio data as training samples into a pre-trained speech model; determining corresponding interaction labels and end prediction labels for the training samples; wherein, the interaction label is used to indicate whether the training sample belongs to interactive speech; the end prediction label is used to indicate whether the training sample corresponds to the end state of a user's turn; and training the pre-trained speech model based on the interaction label and the end prediction label to obtain the initial judgment model.

[0039] In practical implementation, to enable the pre-trained speech model to process both complete and streaming audio, complete audio data and partial audio data sampled from the complete audio data can be obtained for each session audio data in the training dataset. The complete audio data can be all the audio data corresponding to a complete user turn, while the partial audio data can be a portion of the audio data extracted from the complete audio data. By using both complete audio data and partial audio data of different lengths as training samples, different states during continuous user voice input can be simulated, allowing the pre-trained speech model to learn the processing capabilities of both complete and incremental input audio. Both complete and partial audio data can be used as training samples input into the pre-trained speech model, and corresponding interaction labels and end prediction labels are determined for different training samples. The end prediction label indicates whether the training sample corresponds to the end state of the user turn. For partial audio data where the user turn has not yet ended, the corresponding end prediction label can be determined as "not ended"; for complete audio data or partial audio data within a preset end audio segment range, the corresponding end prediction label can be determined as "ended state".

[0040] In one feasible implementation, to avoid insufficient training samples for the end state, targeted sampling can be performed on the end portion of the complete audio data. Local audio data within a preset length of the end audio segment can be determined as training samples for the corresponding user turn end state, thereby increasing the proportion of end state training samples and enabling the pre-trained speech model to learn the speech features before and after the user turn ends.

[0041] After obtaining the interaction labels and end-of-turn prediction labels corresponding to the complete audio data and the partial audio data, the pre-trained speech model can be trained based on these labels. This allows the pre-trained speech model to simultaneously learn interactive speech judgment capabilities and user turn-ending state prediction capabilities, resulting in an initial judgment model. Since the training process includes both complete and partial audio data, the initial judgment model can adapt to different input formats, including complete audio and streaming audio, thus providing a model foundation for subsequent real-time interactive judgment based on streaming audio data and dynamic determination of user turn boundaries.

[0042] In this embodiment of the invention, during the training of the pre-trained speech model based on interaction tags and end prediction tags, a hybrid training method including and without speech recognition text can be further adopted. This enables the pre-trained speech model to utilize the semantic information provided by the speech recognition text while also possessing the ability to directly judge audio without relying on the speech recognition text. Specifically, training the pre-trained speech model based on interaction tags and end prediction tags includes: obtaining the speech recognition text corresponding to each complete audio data and partial audio data in the training samples; constructing a first sample containing speech recognition text and a second sample without speech recognition text for the complete audio data and partial audio data, respectively; inputting the first sample and the second sample into the pre-trained speech model, so that the pre-trained speech model outputs the interaction prediction result and end prediction result of the training samples; determining the first loss of the training samples based on the interaction prediction result and interaction tags, and determining the second loss of the training samples based on the end prediction result and end prediction tag; and training the pre-trained speech model based on the first loss and the second loss to obtain an initial judgment model.

[0043] In practical implementation, a first sample and a second sample can be constructed based on complete audio data, partial audio data, and the corresponding speech-recognition text. The first sample includes either complete or partial audio data, along with the corresponding speech-recognition text. The second sample includes either complete or partial audio data, but does not include the corresponding speech-recognition text. The first and second samples can correspond to the same interaction label and end prediction label, enabling the pre-trained speech model to learn prediction capabilities with and without speech-recognition text for the same audio content.

[0044] After inputting the first sample into the pre-trained speech model, the model outputs interaction prediction results and termination prediction results based on the audio data and speech recognition text in the first sample. It determines a first interaction loss based on the difference between the interaction prediction results and the interaction labels, and a first termination loss based on the difference between the termination prediction results and the termination prediction labels. The first interaction loss and the first termination loss can be weighted and combined to obtain the first loss corresponding to the first sample. Similarly, after inputting the second sample into the pre-trained speech model, the model outputs interaction prediction results and termination prediction results based on the audio data in the second sample. It determines a second interaction loss based on the difference between the interaction prediction results and the interaction labels, and a second termination loss based on the difference between the termination prediction results and the termination prediction labels. The second interaction loss and the second termination loss can be weighted and combined to obtain the second training loss corresponding to the second sample.

[0045] Furthermore, the first and second losses can be weighted and combined to obtain the total training loss of the pre-trained speech model, and the parameters of the pre-trained speech model can be updated based on the total training loss. The first loss corresponding to the first sample can be used to optimize the interaction prediction ability and termination prediction ability of the pre-trained speech model under the condition of speech recognition text participation, while the second loss corresponding to the second sample can be used to optimize the interaction prediction ability and termination prediction ability of the pre-trained speech model under the condition of no speech recognition text. By simultaneously using the training losses corresponding to the two types of samples to train the pre-trained speech model, the pre-trained speech model can be adapted to input formats with and without speech recognition text, and finally an initial judgment model is obtained.

[0046] S102: In response to the fact that the streaming audio data belongs to interactive speech, determine the first end prediction result of the interactive speech, and determine the waiting time based on the first end prediction result.

[0047] In this embodiment of the invention, after determining that the streaming audio data belongs to interactive speech, it indicates that the current streaming audio data is a valid interactive input sent to the voice interaction system. At this time, the first end prediction result of the interactive speech can be further determined to determine whether the user has completed the current turn of expression. Specifically, the interactive speech used for judgment can be input into the interaction judgment model and analyzed in conjunction with the interaction context, so that the interaction judgment model outputs the first end prediction result. The first end prediction result can be used to characterize the user's turn-ending state corresponding to the interactive speech, for example, it can include the predicted probability that the user's turn has ended. The higher the prediction probability, the greater the possibility that the current interactive speech has ended; the lower the prediction probability, the more likely the user may continue to express themselves.

[0048] Unlike traditional voice interaction methods that directly determine the end of a user's turn after detecting a fixed duration of silence, this invention does not directly determine whether to immediately trigger downstream interaction based on a single prediction result. Instead, it dynamically determines the waiting time based on a first end prediction result. Specifically, determining the waiting time based on the first end prediction result includes: obtaining the predicted probability of the user's turn ending corresponding to the interactive voice in the first end prediction result; and determining the waiting time based on a preset duration and the predicted probability; wherein the waiting time is negatively correlated with the predicted probability.

[0049] In practice, the waiting time can be determined based on the predicted probability of the user's turn ending in the first ending prediction result and a preset time parameter. For example, when the first ending prediction result indicates a high probability of the user's turn ending, a shorter waiting time can be determined; when the prediction probability is low, a longer waiting time can be determined. Thus, the interaction judgment model can dynamically adjust the subsequent waiting time according to the degree of completion of the current interactive speech, rather than using a fixed silence waiting threshold for different users, different speech rates, or different expression habits.

[0050] Assuming the probability of ending the dialogue is P and the preset time parameter is t, the waiting time can be determined by (1-P)×t. When P is close to 1, it indicates a higher probability that the current interactive speech has ended, resulting in a shorter waiting time and allowing the voice interaction system to proceed to subsequent processing more quickly. When P is close to 0, it indicates a higher probability that the user will continue speaking, resulting in a longer waiting time and allowing more time for subsequent incremental audio data. For example, if a user inputs "Check today's weather for me," and the interaction judgment model determines that the user's turn has essentially ended based on the current streaming audio data, it can obtain a higher probability of ending the dialogue and determine a shorter waiting time based on this probability, thus reducing system response latency. Conversely, if a user inputs "Check today's weather for me..." followed by a short pause, and the interaction judgment model determines that the user may continue speaking based on the current interactive speech, it can obtain a lower probability of ending the dialogue and determine a longer waiting time. In this case, the voice interaction system will not immediately trigger downstream interaction processing after the short pause but will continue to receive subsequent incremental audio data, thus avoiding misjudging an incomplete user expression as a complete turn.

[0051] By dynamically determining the waiting time based on the first end prediction result, the user turn-ending judgment can be transformed from a fixed-duration silence detection method to a dynamic prediction method that combines the current interactive voice content and context. This allows the waiting time to be adjusted according to the user's current expression state. On the one hand, when the user has finished expressing themselves, the waiting time can be shortened and a timely response can be given; on the other hand, when the user only pauses naturally but is still preparing to continue expressing themselves, the waiting time can be extended and subsequent voice can continue to be received, thereby reducing the probability of premature response and improving the continuity and naturalness of streaming full-duplex voice interaction.

[0052] S103: During the waiting period, continuously collect incremental audio data from the streaming audio data. If the incremental audio data belongs to interactive speech, concatenate the incremental audio data with the streaming audio data and determine the second end prediction result of the concatenated target audio data. Determine a new waiting period based on the second end prediction result until the new incremental audio data collected during the new waiting period does not belong to interactive speech.

[0053] In this embodiment of the invention, after determining the waiting time based on the first end prediction result, the voice interaction system can continue to collect incremental audio data from the streaming audio data within the waiting time and perform interactive voice judgment on the incremental audio data. Since users may pause briefly during natural dialogue before completing a sentence, sending the current streaming audio data to the downstream interactive processing unit immediately after determining that the user may have finished speaking could easily cause the system to respond prematurely. Therefore, within the waiting time, it can continue to listen to and collect subsequently arriving incremental audio data to further confirm whether the user's current turn has truly ended.

[0054] After collecting incremental audio data, it can be determined whether the incremental audio data belongs to interactive speech using the aforementioned interaction judgment method. When the incremental audio data belongs to interactive speech, it indicates that the user continued to express themselves to the voice interaction system during the waiting period, and the current user turn has not yet ended. To avoid splitting the continuous expression into multiple independent turns, the incremental audio data can be concatenated with the streaming audio data to obtain the target audio data. The target audio data can include the streaming audio data already entered by the user in the current turn and the incremental audio data entered during the waiting period, thus more completely reflecting the content of the user's current turn.

[0055] After obtaining the target audio data, a second end prediction result can be determined, and a new waiting duration can be re-determined based on this result. The second end prediction result reflects whether the user's current turn has ended after adding new voice content. Compared to the first end prediction result, the second end prediction result is based on the updated target audio data, thus allowing for a re-evaluation of the current turn's end status in conjunction with the user's added content. Within the new waiting duration, new incremental audio data continues to be collected, and it is determined whether the new incremental audio data belongs to interactive speech. If the new incremental audio data still belongs to interactive speech, it can be concatenated with the current target audio data, and end prediction and waiting duration determination can be performed again based on the updated target audio data.

[0056] The user can be considered to have ceased expressing themselves to the voice interaction system during the current waiting period if no new incremental audio data belonging to interactive speech is collected within the new waiting period, or if no new incremental audio data belonging to interactive speech is collected. At this point, the user's turn corresponding to the current target audio data can be considered complete. The target audio data can then be used as complete user input for subsequent interactive processing.

[0057] For example, if a user first enters "Help me check," the interaction judgment model determines that the user may have finished speaking based on the current streaming audio data and determines the corresponding waiting time. However, if the user continues to enter "Today's weather in Beijing" within the waiting time, this incremental audio data belongs to interactive speech. Therefore, the incremental audio data corresponding to "Today's weather in Beijing" is concatenated with the previous streaming audio data to form complete target audio data. Based on the updated audio content, the second end prediction result and a new waiting time are re-determined. If the user does not continue to enter new incremental audio data belonging to interactive speech within the new waiting time, it can be determined that the user's current turn has ended, and the target audio data containing the complete expression content is used for subsequent processing. This allows for dynamic updates to the audio boundaries of the current turn based on subsequent user input, avoiding premature termination of interaction due to natural pauses, and enabling timely entry into subsequent processing after confirming that the user will no longer speak, thereby improving the accuracy and naturalness of streaming full-duplex voice interaction.

[0058] S104: Input the target audio data to the downstream interactive processing unit so that the interactive processing unit generates response information corresponding to the target audio data.

[0059] In this embodiment of the invention, when it is determined that the user's current turn has ended based on the dynamic waiting process, the target audio data corresponding to the current user turn can be input to the downstream interaction processing unit to generate response information corresponding to the target audio data. Since the target audio data has already undergone interactive voice judgment and dynamic determination of the user turn boundary, when the downstream interaction processing unit receives the target audio data, it can process the target audio data as a relatively complete user interaction input, thereby avoiding splitting the same turn into multiple independent requests due to natural pauses in the user's expression process.

[0060] It should be noted that the interaction determination model in this embodiment of the invention is mainly used to perform interactive speech judgment and predict the end state of user turns during streaming voice interaction, while the downstream interaction processing unit is mainly used to further understand and generate the interactive content corresponding to the target audio data after determining that the user's current turn has ended. By decoupling the interaction determination model from the downstream interaction processing unit, the interaction determination model can be integrated into the existing voice interaction system as an independent dialogue turn decision unit without changing the original processing logic of the downstream interaction processing unit. Therefore, regardless of whether the downstream interaction processing unit adopts a processing architecture combining speech recognition, large language models, and speech synthesis, or other voice interaction architectures, interactive speech recognition and user turn boundary determination can be completed at the front end through the interaction determination model, and then the confirmed target audio data can be input into the downstream interaction processing unit for subsequent processing.

[0061] According to a second aspect of the embodiments of the present invention, such as Figure 2 As shown, a voice interaction device 200 is provided, including: The first judgment module 201 is used to determine whether the streaming audio data belongs to interactive speech based on the acquired streaming audio data and the interaction context. The determination module 202 is used to determine the first end prediction result of the interactive speech in response to the fact that the streaming audio data belongs to interactive speech, and to determine the waiting time based on the first end prediction result; The second judgment module 203 is used to continuously collect incremental audio data of streaming audio data within the first waiting time. If the incremental audio data belongs to interactive speech, the incremental audio data is spliced ​​with the streaming audio data, and the second end prediction result of the spliced ​​target audio data is determined. A new waiting time is determined based on the second end prediction result until the new incremental audio data collected within the new waiting time does not belong to interactive speech. The transmission module 204 is used to input the target audio data to the downstream interactive processing unit so that the interactive processing unit generates response information corresponding to the target audio data.

[0062] Optionally, the first judgment module 201 is specifically used for: Speech activity detection is performed on streaming audio data, and the streaming audio data is segmented based on the speech activity detection results to obtain at least one valid speech segment corresponding to the streaming audio data. Valid speech segments are input into a pre-trained interaction decision model, so that the interaction decision model outputs the interaction decision result of the streaming audio data based on the valid speech segments and the interaction context.

[0063] Optionally, the device further includes a training module 205; the training module 205 is specifically used for: Obtain manually labeled data; manually labeled data includes interactive tags used to indicate whether conversational audio data belongs to interactive speech; Obtain pseudo-labeled data; the pseudo-labeled data is obtained based on the following steps: perform speech activity detection on the conversation audio data, segment the conversation audio data based on the speech activity detection results to obtain at least one speech segment; perform speech recognition on the speech segment to obtain the text information corresponding to the speech segment; concatenate the text information into dialogue data according to the time sequence of the speech segments, and determine the interaction label of each speech segment based on the dialogue data; Obtain scene annotation data; the model annotation data is obtained based on the following steps: construct different interaction contexts for each session audio data according to scene requirements information; input the interaction context and session audio data into the annotation model so that the annotation model outputs the interaction labels of the session audio data; A training dataset is constructed based on manually labeled data, pseudo-labeled data, and scene labeled data, and an initial judgment model is trained based on the interaction labels in the training dataset. The initial decision model was fine-tuned using manually labeled data from the training dataset to obtain an interactive decision model.

[0064] Optionally, the training module 205 is specifically used for: For each session audio data in the training dataset, complete audio data and local audio data sampled from the complete audio data are obtained respectively; Use complete audio data and partial audio data as training samples to input into the pre-trained speech model; For each training sample, a corresponding interaction label and an end prediction label are determined. The interaction label indicates whether the training sample belongs to interactive speech, and the end prediction label indicates whether the training sample corresponds to the end state of a user's turn. The pre-trained speech model is trained based on the interaction label and the end prediction label to obtain the initial judgment model.

[0065] Optionally, the training module 205 is specifically used for: Obtain the speech recognition text corresponding to each complete audio data and partial audio data in the training samples; For complete audio data and partial audio data, construct a first sample containing speech recognition text and a second sample not containing speech recognition text, respectively; The first and second samples are input into the pre-trained speech model, so that the pre-trained speech model outputs the interactive prediction result and the end prediction result of the training samples. Based on the interaction prediction results and interaction labels, determine the first loss of the training samples, and based on the termination prediction results and termination prediction labels, determine the second loss of the training samples. The pre-trained speech model is trained based on the first loss and the second loss to obtain the initial judgment model.

[0066] Optionally, the determination module 202 is specifically used for: Obtain the predicted probability of the user's turn ending corresponding to the interactive speech in the first ending prediction result; The waiting time is determined based on a preset duration and a predicted probability; the waiting time is negatively correlated with the predicted probability.

[0067] According to a third aspect of the present invention, a voice-interactive electronic device is provided, comprising: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the first aspect of the present invention.

[0068] According to a fourth aspect of the present invention, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method provided in the first aspect of the present invention.

[0069] According to a fifth aspect of the present invention, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method of any of the above embodiments.

[0070] Figure 3 An exemplary system architecture 300 is shown that can be applied to the voice interaction method or voice interaction device implemented in this invention.

[0071] like Figure 3 As shown, system architecture 300 may include terminal devices 301, 302, and 303, a network 304, and a server 305. Network 304 serves as the medium for providing communication links between terminal devices 301, 302, and 303 and server 305. Network 304 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0072] Users can use terminal devices 301, 302, and 303 to interact with server 305 via network 304 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 301, 302, and 303, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0073] Terminal devices 301, 302, and 303 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0074] Server 305 can be a server that provides various services, such as a backend management server (for example only) that supports shopping websites browsed by users using terminal devices 301, 302, and 303. The backend management server can process received voice interaction requests and send the processing results (for example only) back to the terminal devices.

[0075] It should be noted that the voice interaction method provided in this embodiment of the invention is generally executed by server 305, and correspondingly, the voice interaction device is generally located in server 305. The voice interaction method provided in this embodiment of the invention can also be executed by terminal devices 301, 302, and 303, and correspondingly, the voice interaction device can be located in terminal devices 301, 302, and 303.

[0076] It should be understood that Figure 3The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0077] The following is for reference. Figure 4 It shows a schematic diagram of the structure of a computer system 400 suitable for implementing a terminal device of the present invention. Figure 4 The terminal device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0078] like Figure 4 As shown, the computer system 400 includes a central processing unit (CPU) 401, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 402 or programs loaded from storage section 408 into random access memory (RAM) 403. The RAM 403 also stores various programs and data required for the operation of the system 400. The CPU 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0079] The following components are connected to I / O interface 405: an input section 406 including a keyboard, mouse, etc.; an output section 407 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN card, modem, etc. The communication section 409 performs communication processing via a network such as the Internet. Drive 410 is also connected to I / O interface 405 as needed. Removable media 411, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 410 as needed so that computer programs read from them can be installed into storage section 408 as needed.

[0080] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 409, and / or installed from removable medium 411. When the computer program is executed by central processing unit (CPU) 401, it performs the functions defined above in the system of this invention.

[0081] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0082] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0083] The modules described in the embodiments of the present invention can be implemented in software or hardware. The described modules can also be housed in a processor. For example, a processor may include a first judgment module, a determination module, a second judgment module, and a transmission module. The names of these modules do not necessarily limit the module itself. For instance, the first judgment module may be described as "determining whether the streaming audio data belongs to the interactive voice module based on the acquired streaming audio data and the interaction context."

[0084] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs, and when the one or more programs are executed by the device, the device implements the following method: determining whether the acquired streaming audio data belongs to interactive speech based on the acquired streaming audio data and the interaction context; in response to the streaming audio data belonging to interactive speech, determining a first end prediction result for the interactive speech, and determining a waiting time based on the first end prediction result; continuously collecting incremental audio data of the streaming audio data within the waiting time; if the incremental audio data belongs to interactive speech, concatenating the incremental audio data with the streaming audio data, and determining a second end prediction result for the concatenated target audio data; determining a new waiting time based on the second end prediction result, until new incremental audio data collected within the new waiting time does not belong to interactive speech; inputting the target audio data to a downstream interactive processing unit so that the interactive processing unit generates response information corresponding to the target audio data.

[0085] Finally, it should be noted that the above embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A voice interaction method, characterized in that, include: Based on the acquired streaming audio data and the interaction context, determine whether the streaming audio data belongs to interactive voice; In response to the fact that the streaming audio data belongs to interactive speech, a first end prediction result of the interactive speech is determined, and a waiting time is determined based on the first end prediction result; During the waiting period, incremental audio data of the streaming audio data is continuously collected. If the incremental audio data belongs to interactive speech, the incremental audio data is concatenated with the streaming audio data, and the second end prediction result of the concatenated target audio data is determined. A new waiting period is determined based on the second end prediction result until the new incremental audio data collected during the new waiting period does not belong to interactive speech. The target audio data is input to the downstream interactive processing unit so that the interactive processing unit generates response information corresponding to the target audio data.

2. The method according to claim 1, characterized in that, Based on the acquired streaming audio data and the interaction context, determine whether the streaming audio data belongs to interactive speech, including: Speech activity detection is performed on the streaming audio data, and the streaming audio data is segmented based on the speech activity detection results to obtain at least one valid speech segment corresponding to the streaming audio data. The effective speech segments are input into a pre-trained interaction determination model, so that the interaction determination model outputs the interaction determination result of the streaming audio data based on the effective speech segments and the interaction context.

3. The method according to claim 2, characterized in that, The interaction determination model is trained based on the following steps: Obtain manually labeled data; the manually labeled data includes interactive tags used to indicate whether the conversational audio data belongs to interactive speech; Obtain pseudo-labeled data; the pseudo-labeled data is obtained based on the following steps: performing speech activity detection on the conversation audio data, segmenting the conversation audio data based on the speech activity detection results to obtain at least one speech segment; performing speech recognition on the speech segment to obtain text information corresponding to the speech segment; concatenating the text information into dialogue data according to the time sequence of the speech segments, and determining the interaction tags of each speech segment based on the dialogue data; Obtain scene annotation data; the model annotation data is obtained based on the following steps: construct different interaction contexts for each session audio data according to scene requirement information; input the interaction context and session audio data into the annotation model so that the annotation model outputs the interaction labels of the session audio data; A training dataset is constructed based on the manually labeled data, pseudo-labeled data, and scene labeled data, and an initial judgment model is trained based on the interaction labels in the training dataset. The initial judgment model is fine-tuned and trained using manually labeled data from the training dataset to obtain the interactive judgment model.

4. The method according to claim 3, characterized in that, An initial decision model is trained based on the interaction labels in the training dataset, including: For each session audio data in the training dataset, complete audio data and local audio data sampled from the complete audio data are obtained respectively. The complete audio data and the partial audio data are used as training samples and input into the pre-trained speech model. For the training sample, a corresponding interaction label and an end prediction label are determined; wherein, the interaction label is used to indicate whether the training sample belongs to interactive speech; the end prediction label is used to indicate whether the training sample corresponds to the end state of a user's turn; Based on the interaction label and the end prediction label, the pre-trained speech model is trained to obtain the initial judgment model.

5. The method according to claim 4, characterized in that, The pre-trained speech model is trained based on the interaction label and the end prediction label, including: Obtain the speech recognition text corresponding to each complete audio data and partial audio data in the training samples; For the complete audio data and the partial audio data, respectively construct a first sample containing the speech recognition text and a second sample not containing the speech recognition text; The first sample and the second sample are respectively input into the pre-trained speech model so that the pre-trained speech model outputs the interaction prediction result and the end prediction result of the training sample; Based on the interaction prediction results and interaction labels, the first loss of the training sample is determined, and based on the termination prediction results and termination prediction labels, the second loss of the training sample is determined. Based on the first loss and the second loss, the pre-trained speech model is trained to obtain the initial judgment model.

6. The method according to any one of claims 1-5, characterized in that, The waiting time is determined based on the first termination prediction result, including: Obtain the predicted probability of the user turn ending corresponding to the interactive voice from the first ending prediction result; The waiting time is determined based on a preset duration and the predicted probability; wherein the waiting time is negatively correlated with the predicted probability.

7. A voice interaction device, characterized in that, include: The first judgment module is used to determine whether the acquired streaming audio data belongs to interactive voice based on the acquired streaming audio data and the interaction context. The determination module is used to determine a first end prediction result of the interactive speech in response to the fact that the streaming audio data belongs to interactive speech, and to determine a waiting time based on the first end prediction result; The second judgment module is used to continuously collect incremental audio data of the streaming audio data within the first waiting time. If the incremental audio data belongs to interactive speech, the incremental audio data is spliced ​​with the streaming audio data, and the second end prediction result of the spliced ​​target audio data is determined. A new waiting time is determined based on the second end prediction result until the new incremental audio data collected within the new waiting time does not belong to interactive speech. The transmission module is used to input the target audio data to the downstream interactive processing unit, so that the interactive processing unit generates response information corresponding to the target audio data.

8. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.

9. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-6.

10. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-6.

Images