Voice interaction method, related device, equipment, system and storage medium

Abstract

The invention discloses a voice interaction method, a related device, equipment, a system and a storage medium. The method comprises the following steps: performing voice activity detection based on streaming audio; in response to the detected voice start endpoint, performing semantic end detection on the streaming audio from the voice start endpoint to detect a semantic end endpoint after the voice start endpoint, and continuing voice activity detection on the streaming audio from the voice start endpoint to detect a voice end endpoint after the voice start endpoint; determining an audio ending endpoint based on at least one of the voice ending endpoint and the semantic ending endpoint; and generating target content for responding to the target audio based on the target audio from the voice start endpoint to the audio end endpoint. According to the scheme, the voice recognition and interaction quality can be improved.

Description

Technical Field

[0001] The present application relates to the field of human-computer interaction technology, and in particular to a voice interaction method and related devices, equipment, systems and storage media. Background Art

[0002] With the development of artificial intelligence technology, intelligent human-computer interaction functions can be realized based on technologies such as speech recognition, speech synthesis, and natural language understanding.

[0003] In the existing technology, the end position of voice activity in the audio signal is usually detected based on voice activity detection technology (VAD), and then a reply is generated based on the separated voice segment. However, voice detection technology needs to detect whether there is a preset length of silence after the audio activity to determine whether the voice activity has ended, resulting in low accuracy of voice recognition in complex interactive environments, and thus resulting in the quality of interaction not being able to meet user needs. In view of this, how to improve the quality of voice recognition and interaction has become an urgent problem to be solved. Summary of the invention

[0004] The main technical problem solved by the present application is to provide a voice interaction method and related devices, equipment, system and storage medium, which can improve the quality of voice recognition and interaction.

[0005] In order to solve the above technical problems, the first aspect of the present application provides a voice interaction method, including performing voice activity detection based on streaming audio; in response to detecting a voice start endpoint, performing semantic end detection on the streaming audio from the voice start endpoint to detect the semantic end endpoint after the voice start endpoint, and continuing to perform voice activity detection on the streaming audio from the voice start endpoint to detect the voice end endpoint after the voice start endpoint; determining the audio end endpoint based on at least one of the voice end endpoint and the semantic end endpoint; and generating target content for responding to the target audio based on the target audio from the voice start endpoint to the audio end endpoint.

[0006] In order to solve the above technical problems, the second aspect of the present application provides a voice interaction device, including a first detection module, a second detection module, an endpoint determination module and a content generation module, the first detection module is used to perform voice activity detection based on streaming audio; the second detection module is used to respond to the detection of a voice start endpoint, perform semantic end detection on the streaming audio from the voice start endpoint to detect the semantic end endpoint after the voice start endpoint, and continue to perform voice activity detection on the streaming audio from the voice start endpoint to detect the voice end endpoint after the voice start endpoint; the endpoint determination module is used to determine the audio end endpoint based on at least one of the voice end endpoint and the semantic end endpoint; the content generation module is used to generate target content for responding to the target audio based on the target audio from the voice start endpoint to the audio end endpoint.

[0007] In order to solve the above technical problems, the third aspect of the present application provides an electronic device, including a memory and a processor coupled to each other, the memory storing program instructions, and the processor being used to execute the program instructions to implement the voice interaction method of the first aspect.

[0008] In order to solve the above technical problems, the fourth aspect of the present application provides a voice interaction system, including an audio acquisition device, a data processing device and a data output device, the audio acquisition device is used to acquire streaming audio, the data output device is used to output the target content generated by the data processing device, and the data processing device is the electronic device described in the above third aspect.

[0009] In order to solve the above technical problems, the fifth aspect of the present application provides a computer-readable storage medium, which stores program instructions that can be executed by a processor, and the program instructions are used to implement the voice interaction method of the first aspect above.

[0010] The above scheme performs voice activity detection based on streaming audio. In response to detecting a voice start endpoint, semantic end detection is performed on the streaming audio from the voice start endpoint to detect the semantic end endpoint after the voice start endpoint, and voice activity detection is continued on the streaming audio from the voice start endpoint to detect the voice end endpoint after the voice start endpoint. The audio end endpoint is determined based on at least one of the voice end endpoint and the semantic end endpoint, and the target content for responding to the target audio is generated based on the target audio from the voice start endpoint to the audio end endpoint. Since the determination of the semantic end endpoint does not require detection of whether there is silence of a preset length, the detection of the voice end endpoint does not need to rely on the integrity of the semantics of the voice activity, and both the voice end endpoint and the semantic end endpoint represent the end of the voice activity, semantic end detection and voice end detection are performed on the streaming audio respectively. In different scenarios, the audio end endpoint determined based on at least one of the voice end endpoint and the semantic end endpoint can minimize the impact of various factors on the target audio detection during the interaction process, and is helpful to determine the audio end endpoint based on the end endpoint obtained by the better detection method in the current scenario, so as to minimize the delay in the generation of the target content. Therefore, the quality of speech recognition and interaction can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is a flowchart of an embodiment of the voice interaction method of the present application; Figure 2 It is a schematic diagram of an embodiment of endpoint determination of the voice interaction method of the present application; Figure 3 It is a schematic diagram of the framework of an embodiment of the voice interaction device of the present application; Figure 4 It is a schematic diagram of the framework of an embodiment of the electronic device of the present application; Figure 5 It is a schematic diagram of the framework of an embodiment of the voice interaction system of the present application; Figure 6 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0012] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0013] The terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, "many" in this article means two or more than two.

[0014] See also Figure 1 , Figure 1 This is a flow chart of an embodiment of the voice interaction method of the present application. Specifically, it may include the following steps: Step S10: Perform voice activity detection based on the streaming audio.

[0015] In the disclosed embodiment, streaming audio has an audio format that divides audio data into small blocks for transmission and processing, allowing audio data to be gradually loaded and played in the form of a stream during transmission. For the specific data properties of streaming audio, please refer to more technical details of streaming audio, which will not be repeated here.

[0016] In one implementation scenario, voice activity detection for streaming audio can be implemented based on a VAD module (Voice Activity Detection). The VAD module identifies valid speech parts from continuous streaming audio and separates them from non-speech parts or noise. Specifically, the VAD module can determine whether the frame belongs to speech activity by calculating the zero-crossing rate of each frame of audio in the streaming audio and comparing it with a set threshold, or determine the presence of speech by detecting the pitch period of the speech signal. It should be noted that the specific method of the VAD module for voice activity detection is not limited in this application.

[0017] In another implementation scenario, voice activity detection may be performed on streaming audio based on a pre-trained acoustic detection model, a spectrum filter, etc. For the sake of brevity, the specific implementation of the voice activity detection is not described here in detail.

[0018] In a specific implementation scenario, as a possible implementation method, an acoustic detection model can be pre-trained, and the acoustic detection model can include but is not limited to a network model with a convolutional neural network, a recurrent neural network architecture, etc., and the streaming audio is input into the acoustic detection model, and the output result of the acoustic detection model is used as the result of acoustic activity detection. In order to ensure the prediction accuracy of the acoustic detection model as much as possible, sample audio can be collected, and the real start endpoint and the real end endpoint can be marked on the sample audio. On this basis, the sample audio can be processed based on the acoustic detection model to obtain the sample predicted start endpoint and predicted end endpoint of the sample audio, so that the network parameters of the acoustic detection model can be adjusted based on the difference between the real start endpoint and the predicted start endpoint, and the difference between the real end endpoint and the predicted end endpoint, until the acoustic detection model training converges, and the streaming audio collected during the interaction process can be processed based on the acoustic detection model with converged training to obtain the result of voice activity detection. It should be noted that the specific processing process of the acoustic detection model can refer to the technical details such as the network model of the neural network architecture, which will not be repeated here. The above scheme can process the streaming audio in the interaction process based on the pre-trained acoustic detection model to improve the accuracy of audio activity determination.

[0019] Step S20: In response to detecting a speech start endpoint, performing semantic end detection on the streaming audio from the speech start endpoint to detect a semantic end endpoint after the speech start endpoint, and continuing to perform voice activity detection on the streaming audio from the speech start endpoint to detect a speech end endpoint after the speech start endpoint.

[0020] In one implementation scenario, when a speech start endpoint is detected, that is, there is speech activity in the streaming audio, semantic end detection is performed on the streaming audio from the speech start endpoint to detect the semantic end endpoint after the speech start endpoint, and speech activity detection is continued on the streaming audio from the speech start endpoint to detect the speech end endpoint after the speech start endpoint. Since the detection of the speech end endpoint is related to the duration of the audio feature in the speech activity, and the detection of the semantic end endpoint is related to the semantics represented by the audio feature in the speech activity, when different categories of endpoint detection are performed on the same streaming audio after the speech start endpoint, the moment when the speech end endpoint is detected is different from the moment when the semantic end endpoint is detected. The moment of the end endpoint has different order relationships in different scenarios. For example, without considering hardware resources, it can be understood that since semantic detection is predictive, that is, based on the semantic information represented by the input streaming audio, it can predict the semantic information represented by the not-input streaming audio. Therefore, the moment of detecting the semantic end endpoint is earlier than the moment of detecting the voice end endpoint. For another example, when the semantics of the voice activity output by the target object are incomplete, since the detection of the voice end endpoint is only related to the duration of the audio features in the voice activity, the moment of detecting the voice end endpoint is earlier than the moment of detecting the semantic end endpoint.

[0021] See also Figure 2 , Figure 2 Schematic diagram of an embodiment of the voice interaction method of the present application for determining an endpoint. Figure 2 As shown, in a specific implementation scenario, voice activity detection is implemented based on a voice detection module, and semantic end detection is implemented based on a semantic detection module. With the timeline as a reference, streaming audio is continuously input. It can be understood that although the moment when the voice end endpoint is detected is inconsistent with the moment when the semantic end endpoint is detected due to the different processing methods of the voice detection module and the semantic detection module, the semantic end endpoint and the voice end endpoint are both used to represent the end of the voice activity, that is, the audio end endpoint. Therefore, the semantic end endpoint and the voice end endpoint are represented as the same moment on the timeline, or due to the different detection accuracy of the voice detection module and the semantic detection module, there is a slight difference between the semantic end endpoint and the voice end endpoint on the timeline.

[0022] In one implementation scenario, semantic end detection is performed based on a large language model, which can understand human multimodal natural language input, and obtain keyword information about the input content and the logical relationship between each element based on the input content, and generate semantically relevant output. For specific principles, please refer to more technical details of the large language model, which will not be repeated here. It should be noted that the large language model may include but is not limited to: LLAMA, Bloom and other open source large models. The network architecture of the large language model is not limited here. The large language model in this application can be an open source large model or a finished large model after parameter adjustment. The specific application is not limited.

[0023] In a specific implementation scenario, based on the speech start endpoint, a prompt instruction is constructed to instruct the large language model to understand the semantic information of the streaming audio from the speech start endpoint to determine the semantic end endpoint of the streaming audio. For example, the constructed prompt instruction is "Please understand the semantic information represented by the speech activity of the continuously input streaming audio after (the corresponding moment of the speech start endpoint), and predict the semantic end endpoint of this speech activity based on the semantic information", and input the prompt instruction to the large language model to obtain the output endpoint of the large language model as the semantic end endpoint.

[0024] Step S30: Determine the audio end endpoint based on at least one of the speech end endpoint and the semantic end endpoint.

[0025] In one implementation scenario, in response to the voice end endpoint being detected before the semantic end endpoint, the voice end endpoint is selected as the audio end endpoint, and in response to the semantic end endpoint being detected before the voice end endpoint, the semantic end endpoint is selected as the audio end endpoint.

[0026] In a specific implementation scenario, when the voice end endpoint is detected before the semantic end endpoint, that is, when the voice end endpoint is used as the audio end endpoint, for example, the voice activity semantics are incomplete, the resource consumption required for voice activity detection is much lower than the resource consumption required for semantic detection, etc., the currently executing semantic end detection is forcibly terminated.

[0027] In a specific implementation scenario, when a speech end endpoint is detected, voice activity detection is started to detect the latest speech start endpoint.

[0028] In a specific implementation scenario, when the semantic end endpoint is detected before the speech end endpoint, the voice activity detection operation on the speech end endpoint is forcibly ended, and while generating the target content, the voice activity detection operation on the latest speech start endpoint is started.

[0029] Step S40: Based on the target audio from the speech start endpoint to the audio end endpoint, generate target content for responding to the target audio.

[0030] In one implementation scenario, since the semantic end endpoint does not need to be determined by detecting whether there is silence of a preset duration, the detection of the voice end endpoint does not need to rely on the integrity of the semantics of the voice activity, and both the voice end endpoint and the semantic end endpoint represent the end of the voice activity, semantic end detection and voice end detection are performed on the streaming audio respectively. In different scenarios, the audio end endpoint determined based on at least one of the voice end endpoint and the semantic end endpoint can minimize the impact of various factors on the target audio detection during the interaction process, and is helpful to determine the audio end endpoint based on the end endpoint obtained by the better detection method in the current scenario, so as to minimize the delay in the generation of the target content. Therefore, the quality of voice recognition and interaction can be improved.

[0031] In one implementation scenario, after responding to the detection of the speech start endpoint of the target audio, and before generating the target content for responding to the target audio based on the target audio from the speech start endpoint to the audio end endpoint, the previous audio end endpoint is selected as the reference end endpoint, and the comparison result of the interval between the reference end endpoint and the current speech start endpoint and the preset time length is obtained. It should be noted that the specific setting method of the preset time length is not limited in this application, for example, a fixed time length is set according to the generation efficiency of the target content, etc. Based on the data processing method that matches the comparison result, the target content for responding to the target audio is generated. The above scheme uses the previous audio end endpoint as the reference end endpoint, and determines the generation method of the target content based on the interval between the reference end endpoint and the current speech start endpoint. When the target object continuously inputs multiple voices to be interacted, the data processing pressure can be reduced as much as possible, and the interaction efficiency can be improved.

[0032] In a specific implementation scenario, when the comparison result indicates that the interval duration is less than a preset duration, the target audio ending at the reference end endpoint is taken as historical audio, and the generation of target content for responding to the historical audio is stopped, and based on the historical audio and the current target audio, target content for responding to the historical audio and the current target audio is generated.

[0033] In another specific implementation scenario, when the comparison result indicates that the interval duration is not less than a preset duration, target content for responding to the current target audio is generated based on the current target audio.

[0034] In a specific implementation scenario, when network resources are relatively abundant, both voice activity detection and semantic end detection have a faster processing speed. Therefore, when the voice end endpoint serves as the audio end endpoint, the target audio from the voice start endpoint to the audio end endpoint has the defect of unclear semantics. Therefore, when the voice end endpoint serves as the audio end endpoint, the target content used to respond to the target audio at least includes content used to guide the target to supplement the relevant interaction, such as "Please supplement your interaction needs", "I don't quite understand what you mean, can you say it again", etc.

[0035] In one implementation scenario, after generating target content for responding to the target audio based on the target audio from the voice start endpoint to the audio end endpoint, the target content is output, and the step of performing voice activity detection based on the streaming audio is returned to start a new round of voice interaction. Specifically, the target content can be played based on an audio playback device, the corresponding text can be displayed based on a display device, etc. The output method of the target content is not limited in this application.

[0036] In a specific implementation scenario, speech synthesis is performed based on the target content to obtain playback audio for responding to the target audio. When the target content is played based on an audio playback device, in order to improve the accuracy of voice interaction and minimize the impact of the playback content on voice activity detection, echo cancellation is performed on the playback content based on the target content, and when the step of performing voice activity detection based on streaming audio is performed, the streaming audio does not contain the playback audio. For the specific implementation of speech synthesis and echo cancellation, please refer to the relevant technical details of speech synthesis, echo cancellation, etc., which will not be repeated here for the sake of brevity.

[0037] In a specific implementation scenario, when the target content of the previous target audio is output at the detection moment of the latest voice start endpoint, for example, when the audio about the interactive content is being played and interrupted by the target object, the target content of the previous target audio is stopped from being output, and the target content of the previous target audio is used as reference data for assisting in generating the target content for responding to the latest target audio in the latest round of voice interaction. The above scheme can realize the rapid interruption of the interaction process and improve the interaction efficiency, and use the target content that has been processed and generated as reference data for generating the target content for responding to the latest target audio, which can reduce the data processing pressure as much as possible and improve the interaction efficiency while improving the integrity of the target content generation.

[0038] The above scheme performs voice activity detection based on streaming audio. In response to detecting a voice start endpoint, semantic end detection is performed on the streaming audio from the voice start endpoint to detect the semantic end endpoint after the voice start endpoint, and voice activity detection is continued on the streaming audio from the voice start endpoint to detect the voice end endpoint after the voice start endpoint. The audio end endpoint is determined based on at least one of the voice end endpoint and the semantic end endpoint, and the target content for responding to the target audio is generated based on the target audio from the voice start endpoint to the audio end endpoint. Since the determination of the semantic end endpoint does not require detection of whether there is silence of a preset length, the detection of the voice end endpoint does not need to rely on the integrity of the semantics of the voice activity, and both the voice end endpoint and the semantic end endpoint represent the end of the voice activity, semantic end detection and voice end detection are performed on the streaming audio respectively. In different scenarios, the audio end endpoint determined based on at least one of the voice end endpoint and the semantic end endpoint can minimize the impact of various factors on the target audio detection during the interaction process, and is helpful to determine the audio end endpoint based on the end endpoint obtained by the better detection method in the current scenario, so as to minimize the delay in the generation of the target content. Therefore, the quality of speech recognition and interaction can be improved.

[0039] See also Figure 3 , Figure 3 1 is a flow chart of an embodiment of the voice interaction device 30 of the present application. Figure 3 As shown, the voice interaction device 30 includes a first detection module 31, a second detection module 32, an endpoint determination module 33 and a content generation module 34. The first detection module 31 is used to perform voice activity detection based on streaming audio; the second detection module 32 is used to perform semantic end detection on the streaming audio from the speech start endpoint in response to detecting the speech start endpoint, so as to detect the semantic end endpoint after the speech start endpoint, and continue to perform voice activity detection on the streaming audio from the speech start endpoint to detect the speech end endpoint after the speech start endpoint; the endpoint determination module 33 is used to determine the audio end endpoint based on at least one of the speech end endpoint and the semantic end endpoint; the content generation module 34 is used to generate target content for responding to the target audio based on the target audio from the speech start endpoint to the audio end endpoint.

[0040] Therefore, the voice interaction device 30 performs voice activity detection based on the streaming audio, and in response to detecting the voice start endpoint, performs semantic end detection on the streaming audio from the voice start endpoint to detect the semantic end endpoint after the voice start endpoint, and continues to perform voice activity detection on the streaming audio from the voice start endpoint to detect the voice end endpoint after the voice start endpoint, and determines the audio end endpoint based on at least one of the voice end endpoint and the semantic end endpoint, and generates target content for responding to the target audio based on the target audio from the voice start endpoint to the audio end endpoint. Since the determination of the semantic end endpoint does not need to be determined by detecting whether there is silence of a preset length, the detection of the voice end endpoint does not need to rely on the integrity of the semantics of the voice activity, and the voice end endpoint and the semantic end endpoint both represent the end of the voice activity, the semantic end detection and the voice end detection are performed on the streaming audio respectively, and in different scenarios, the audio end endpoint determined based on at least one of the voice end endpoint and the semantic end endpoint can minimize the impact of various factors on the target audio detection during the interaction process, and is helpful to determine the audio end endpoint based on the end endpoint obtained by the better detection method in the current scenario, so as to minimize the delay in the generation of the target content. Therefore, the quality of voice recognition and interaction can be improved.

[0041] In some disclosed embodiments, after responding to detecting the speech start endpoint of the target audio, and before generating target content for responding to the target audio based on the target audio from the speech start endpoint to the audio end endpoint, the voice interaction device 30 also includes a reference end module (not shown) for selecting the previous audio end endpoint as the reference end endpoint; the voice interaction device 30 also includes a comparison acquisition module (not shown) for acquiring a comparison result of the interval duration between the reference end endpoint and the current speech start endpoint and a preset duration; the content generation module 34 also includes a selection processing module (not shown) for generating target content based on a data processing method that matches the comparison result.

[0042] In some disclosed embodiments, when the comparison result represents an interval duration that is less than a preset duration, the voice interaction device 30 also includes a generation stop module (not shown) for stopping the generation of target content for responding to historical audio; wherein the historical audio is the target audio that ends at a reference end endpoint; the selection processing module (not shown) also includes a first generation module (not shown) for generating target content for responding to the historical audio and the current target audio based on the historical audio and the current target audio.

[0043] In some disclosed embodiments, when the comparison result indicates that the interval duration is not less than a preset duration, the selection processing module (not shown) further includes a second generation module (not shown) for generating target content in response to the current target audio based on the current target audio.

[0044] In some disclosed embodiments, the endpoint determination module 33 also includes a first selection module (not shown) for selecting a speech end endpoint as an audio end endpoint in response to the speech end endpoint being detected before the semantic end endpoint; the endpoint determination module 33 also includes a second selection module (not shown) for selecting a semantic end endpoint as an audio end endpoint in response to the semantic end endpoint being detected before the speech end endpoint.

[0045] In some disclosed embodiments, when the speech end endpoint is detected before the semantic end endpoint, the voice interaction device 30 also includes a first forcing module (not shown) for forcing the termination of the currently executed semantic end detection; and / or, when the semantic end endpoint is detected before the speech end endpoint, the voice interaction device 30 also includes a second forcing module (not shown) for forcing the termination of the voice activity detection operation on the speech end endpoint, and starting the voice activity detection operation on the latest speech start endpoint while generating the target content.

[0046] In some disclosed embodiments, the second detection module 32 also includes an instruction construction module (not shown) for constructing a prompt instruction based on the speech start endpoint; wherein the prompt instruction is used to instruct the large language model to understand the semantic information of the streaming audio from the speech start endpoint to determine the semantic end endpoint of the streaming audio; the second detection module 32 also includes a large model module (not shown) for inputting the prompt instruction to the large language model to obtain the output endpoint of the large language model as the semantic end endpoint.

[0047] In some disclosed embodiments, after generating target content for responding to the target audio based on the target audio from the voice start endpoint to the audio end endpoint, the voice interaction device 30 also includes a content output module (not shown) for outputting the target content and returning to execute the step of performing voice activity detection based on streaming audio to start a new round of voice interaction.

[0048] In some disclosed embodiments, when there is a target content output of the previous target audio at the detection moment of the latest voice start endpoint, the voice interaction device 30 also includes a data processing module (not shown) for stopping the output of the target content of the previous target audio and using the target content of the previous target audio as reference data; wherein the reference data is used to assist in generating target content for responding to the latest target audio in the latest round of voice interaction.

[0049] See also Figure 4 , Figure 4 1 is a schematic diagram of a framework of an embodiment of an electronic device 40 of the present application. Figure 4As shown, the electronic device 40 includes a memory 41 and a processor 42 coupled to each other, the memory 41 stores program instructions, and the processor 42 is used to execute the program instructions to implement the steps in any of the above-mentioned voice interaction method embodiments. Specifically, the electronic device 40 may include but is not limited to: a server, a desktop computer, a laptop computer, a tablet computer, a smart phone, etc., which are not limited here. Specifically, the processor 42 is used to control itself and the memory 41 to implement the steps in any of the above-mentioned voice interaction method embodiments. The processor 42 can also be called a CPU (Central Processing Unit). The processor 42 may be an integrated circuit chip with signal processing capabilities. The processor 42 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 42 can be implemented by an integrated circuit chip.

[0050] Therefore, the electronic device 40 performs voice activity detection based on the streaming audio, and in response to detecting the voice start endpoint, performs semantic end detection on the streaming audio from the voice start endpoint to detect the semantic end endpoint after the voice start endpoint, and continues to perform voice activity detection on the streaming audio from the voice start endpoint to detect the voice end endpoint after the voice start endpoint, and determines the audio end endpoint based on at least one of the voice end endpoint and the semantic end endpoint, and generates target content for responding to the target audio based on the target audio from the voice start endpoint to the audio end endpoint. Since the determination of the semantic end endpoint does not require detection of whether there is silence of a preset duration, the detection of the voice end endpoint does not need to rely on the integrity of the semantics of the voice activity, and the voice end endpoint and the semantic end endpoint both represent the end of the voice activity, the semantic end detection and the voice end detection are performed on the streaming audio respectively, and in different scenarios, the audio end endpoint determined based on at least one of the voice end endpoint and the semantic end endpoint can minimize the impact of various factors on the target audio detection during the interaction process, and is helpful to determine the audio end endpoint based on the end endpoint obtained by the better detection method in the current scenario, so as to minimize the delay in the generation of the target content. Therefore, the quality of voice recognition and interaction can be improved.

[0051] See also Figure 5 , Figure 5Schematic diagram of the framework of an embodiment of the voice interaction system 50 of the present application. Figure 5 As shown, the voice interaction system 50 includes an audio acquisition device 51, a data processing device 52 and a data output device 53. The audio acquisition device 51 is used to acquire streaming audio, and the data output device 53 is used to output the target content generated by the data processing device 52. The data processing device 52 is the electronic device 40 described in any of the above embodiments.

[0052] In one implementation scenario, the data processing device 52 runs at least a speech detection model and a semantic detection model, wherein the speech detection model is at least used to perform speech activity detection on the streaming audio, and the semantic detection model is at least used to perform semantic end detection on the streaming audio.

[0053] It should be noted that the specific network structure of the speech detection model and the semantic detection model is not limited in this application. For details, please refer to the detailed explanation of the voice activity detection and semantic end detection technical solutions in the aforementioned embodiments. For the sake of brevity, they will not be repeated here.

[0054] In one implementation scenario, the data processing device 52 also runs a response generation model for generating target content for responding to the target audio based on the target audio from the speech start endpoint to the audio end endpoint. For details, please refer to the detailed description of the technical solution in the aforementioned embodiment. For the sake of brevity, it will not be repeated here.

[0055] In one implementation scenario, the data output device 53 runs a content synthesis model, which is used to synthesize the target content in response to the target audio into data in a preset form and output it, for example, synthesizing the target content into audio, text, etc. For details, please refer to the detailed description of the technical solution in the aforementioned embodiment. For the sake of brevity, it will not be repeated here.

[0056] In the above scheme, the voice interaction system 50 performs voice activity detection based on streaming audio, and in response to detecting the voice start endpoint, performs semantic end detection on the streaming audio from the voice start endpoint to detect the semantic end endpoint after the voice start endpoint, and continues to perform voice activity detection on the streaming audio from the voice start endpoint to detect the voice end endpoint after the voice start endpoint, and determines the audio end endpoint based on at least one of the voice end endpoint and the semantic end endpoint, and generates target content for responding to the target audio based on the target audio from the voice start endpoint to the audio end endpoint. Since the determination of the semantic end endpoint does not require detection of whether there is silence of a preset duration, the detection of the voice end endpoint does not need to rely on the integrity of the semantics of the voice activity, and the voice end endpoint and the semantic end endpoint both represent the end of the voice activity, the semantic end detection and the voice end detection are performed on the streaming audio respectively, and in different scenarios, the audio end endpoint determined based on at least one of the voice end endpoint and the semantic end endpoint can minimize the impact of various factors on the target audio detection during the interaction process, and is helpful to determine the audio end endpoint based on the end endpoint obtained by the better detection method in the current scenario, so as to minimize the delay in the generation of the target content. Therefore, the quality of voice recognition and interaction can be improved.

[0057] See also Figure 6 , Figure 6 Schematic diagram of a framework of an embodiment of a computer-readable storage medium 60 of the present application. The computer-readable storage medium 60 includes program instructions 61 that can be executed by a processor, and the program instructions 61 are used to implement the steps in any of the above-mentioned voice interaction method embodiments.

[0058] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0059] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0060] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0061] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, and may be located in one place or distributed over multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0062] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0063] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to execute all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0064] If the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to the collection of his or her personal information; or on the device that processes personal information, the personal information processing rules are notified by obvious signs / information, and the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

Claims

1. A voice interaction method, characterized in that: include: Voice activity detection based on streaming audio; In response to detecting a speech start endpoint, performing semantic end detection on the streaming audio from the speech start endpoint to detect a semantic end endpoint after the speech start endpoint, and continuing to perform the voice activity detection on the streaming audio from the speech start endpoint to detect a speech end endpoint after the speech start endpoint; Determine an audio end endpoint based on at least one of the voice end endpoint and the semantic end endpoint; Based on the target audio from the speech start endpoint to the audio end endpoint, a target content for responding to the target audio is generated.

2. The method according to claim 1, characterized in that After the response to detecting the speech start endpoint of the target audio and before the target content for responding to the target audio is generated based on the target audio from the speech start endpoint to the audio end endpoint, the method further includes: Select the audio end endpoint mentioned above as the reference end endpoint; Obtain a comparison result of the interval time between the reference end endpoint and the current speech start endpoint and a preset time; The step of generating target content for responding to the target audio based on the target audio from the speech start endpoint to the audio end endpoint includes: The target content is generated based on a data processing method that matches the comparison result.

3. The method according to claim 2, characterized in that When the comparison result indicates that the interval duration is less than the preset duration, the method further includes: Stop generating target content for responding to historical audio; wherein the historical audio is the target audio ending at the reference end endpoint; The generating the target content based on the data processing method matching the comparison result includes: Based on the historical audio and the current target audio, the target content for responding to the historical audio and the current target audio is generated.

4. The method according to claim 2, characterized in that: In the case where the comparison result indicates that the interval duration is not less than the preset duration, generating the target content based on a data processing method matching the comparison result includes: Based on the current target audio, the target content for responding to the current target audio is generated.

5. The method according to claim 1, characterized in that The step of determining the audio end endpoint based on at least one of the voice end endpoint and the semantic end endpoint comprises: In response to the voice end endpoint being detected before the semantic end endpoint, selecting the voice end endpoint as the audio end endpoint; In response to the semantic end endpoint being detected before the voice end endpoint, the semantic end endpoint is selected as the audio end endpoint.

6. The method according to claim 5, characterized in that In the case where the speech end endpoint is detected before the semantic end endpoint, the method further comprises: forcibly ending the semantic end detection currently being executed; And / or, in the case where the semantic end endpoint is detected before the speech end endpoint, the method further includes: forcibly terminating the voice activity detection operation on the speech end endpoint, and starting the voice activity detection operation on the latest speech start endpoint while generating the target content.

7. The method according to claim 1, characterized in that The performing semantic end detection on the streaming audio from the voice start endpoint includes: Based on the speech start endpoint, construct a prompt instruction; wherein the prompt instruction is used to instruct the large language model to understand the semantic information of the streaming audio from the speech start endpoint to determine the semantic end endpoint of the streaming audio; The prompt instruction is input to the large language model to obtain an output endpoint of the large language model as the semantic end endpoint.

8. The method according to any one of claims 1 to 7, characterized in that: After generating target content for responding to the target audio based on the target audio from the speech start endpoint to the audio end endpoint, the method further includes: The target content is output, and the step of performing voice activity detection based on streaming audio is returned to start a new round of voice interaction.

9. The method according to claim 8, characterized in that In the case where there is a target content output of the previous target audio at the detection time of the latest voice start endpoint, the method further includes: Stop outputting the target content of the last target audio, and use the target content of the last target audio as reference data; wherein the reference data is used to assist in generating the target content for responding to the latest target audio in the latest round of voice interaction.

10. A voice interaction device, characterized in that: include: A first detection module, configured to perform voice activity detection based on streaming audio; A second detection module is used for, in response to detecting a speech start endpoint, performing semantic end detection on the streaming audio from the speech start endpoint to detect a semantic end endpoint after the speech start endpoint, and continuing to perform the voice activity detection on the streaming audio from the speech start endpoint to detect a speech end endpoint after the speech start endpoint; An endpoint determination module, used for determining an audio end endpoint based on at least one of the voice end endpoint and the semantic end endpoint; The content generation module is used to generate target content for responding to the target audio based on the target audio from the voice start endpoint to the audio end endpoint.

11. An electronic device, characterized in that: It comprises a memory and a processor coupled to each other, wherein the memory stores program instructions, and the processor is used to execute the program instructions to implement the voice interaction method according to any one of claims 1 to 9.

12. A voice interaction system, characterized in that: It comprises an audio acquisition device, a data processing device and a data output device, wherein the audio acquisition device is used to acquire streaming audio, the data output device is used to output target content generated by the data processing device, and the data processing device is the electronic device described in claim 11.

13. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the voice interaction method according to any one of claims 1 to 9 is implemented.

Images