Streaming voice interaction method, related device, equipment and storage medium

Abstract

The invention discloses a streaming voice interaction method, a related device, equipment and a storage medium, and the method comprises the steps: carrying out the first classification based on a currently collected first streaming voice, and obtaining a first prediction type of the first streaming voice; in response to the fact that the first prediction category is human voice, performing second classification at least based on the first streaming voice to obtain a second prediction category of the first streaming voice; in response to the fact that the second prediction category is real human voice, performing third classification at least based on the first streaming voice to obtain a third prediction category of the first streaming voice; and based on the third prediction category, executing the target interaction operation on the currently output machine conversation content. According to the scheme, false triggering of streaming voice can be reduced as much as possible, and the accuracy of streaming voice interaction is improved.

Description

Technical Field

[0001] The present application relates to the field of voice interaction technology, and in particular to a streaming voice interaction method and related devices, equipment and storage media. Background Art

[0002] With the rapid development of natural language processing and other related technologies, streaming voice interaction has been widely used in many scenarios such as intelligent assistants and customer service. Through streaming voice interaction technology, human-computer dialogue can be interrupted at any time during the dialogue like human-to-human dialogue.

[0003] However, the existing streaming voice interaction technology still has the technical defect of false triggering, which leads to problems such as responding when it should not respond or not responding when it should respond, thus affecting the interactive experience. In view of this, how to reduce the false triggering of streaming voice as much as possible and improve the accuracy of streaming voice interaction has become an urgent problem to be solved. Summary of the invention

[0004] The main technical problem solved by the present application is to provide a streaming voice interaction method and related devices, equipment and storage media, which can reduce the false triggering of streaming voice as much as possible and improve the accuracy of streaming voice interaction.

[0005] In order to solve the above technical problems, the first aspect of the present application provides a streaming voice interaction method, including: performing a first classification based on the currently collected first streaming voice to obtain a first predicted category of the first streaming voice; wherein the first predicted category is human voice, noise, or silence; in response to the first predicted category being human voice, performing a second classification at least based on the first streaming voice to obtain a second predicted category of the first streaming voice; wherein the second predicted category is real human voice or background human voice; in response to the second predicted category being real human voice, performing a third classification at least based on the first streaming voice to obtain a third predicted category of the first streaming voice; wherein the third predicted category involves whether the semantics are complete and whether it is related to the context content; based on the third predicted category, performing a target interaction operation on the machine dialogue content currently being output; wherein the target interaction operation is any one of several preset interaction operations, and the several preset interaction operations include maintaining the current topic, switching to a new topic, and continuing the original topic from the interrupted conversation.

[0006] In order to solve the above technical problems, the second aspect of the present application provides a streaming voice interaction device, including: a first classification module, a second classification module, a third classification module and an interactive operation module, the first classification module is used to perform a first classification based on the currently collected first streaming voice to obtain a first predicted category of the first streaming voice; wherein the first predicted category is human voice, noise, or silence; the second classification module is used to respond to the first predicted category being human voice, perform a second classification based at least on the first streaming voice, and obtain a second predicted category of the first streaming voice; wherein the second predicted category is real human voice or background human voice; the third classification module is used to respond to the second predicted category being real human voice, perform a third classification based at least on the first streaming voice, and obtain a third predicted category of the first streaming voice; wherein the third predicted category involves whether the semantics are complete and whether it is related to the context content; the interactive operation module is used to perform a target interactive operation on the machine dialogue content currently being output based on the third predicted category; wherein the target interactive operation is any one of several preset interactive operations, and the several preset interactive operations include maintaining the current topic, switching to a new topic, and continuing the original topic from the interrupted conversation.

[0007] In order to solve the above technical problems, the third aspect of the present application provides an electronic device, which at least includes a memory and a processor coupled to each other, the memory at least stores program instructions, and the processor is used to execute the program instructions to implement the streaming voice interaction method in the above first aspect.

[0008] In order to solve the above technical problems, the fourth aspect of the present application provides a computer-readable storage medium, which stores program instructions that can be executed by a processor, and the program instructions are used to implement the streaming voice interaction method of the above first aspect.

[0009] In the above scheme, a first classification is performed based on the first stream of speech currently collected to obtain a first predicted category of the first stream of speech, and the first predicted category is human, noise, or silence. In response to the first predicted category being human voice, a second classification is performed at least based on the first stream of speech to obtain a second predicted category of the first stream of speech, and the second predicted category is real human voice or background human voice. In response to the second predicted category being real human voice, a third classification is performed at least based on the first stream of speech to obtain a third predicted category of the first stream of speech, and the third predicted category involves whether the semantics are complete and whether it is related to the context content. Based on the third predicted category, the machine speech currently being output is classified. The target interactive operation is performed based on the content of the conversation. The target interactive operation is any one of several preset interactive operations, and several preset interactive operations include maintaining the current topic, switching to a new topic, and continuing the original topic from the interrupted conversation. Therefore, the first stream voice currently collected is hierarchically classified in the first classification, the second classification, and the third classification in sequence. In the first classification process, human voice, noise or silence are identified. In the second classification process, the human voice is further identified as a real human voice or a background human voice. In the third classification process, it is further identified whether the semantics of the real human voice are complete and whether the content is relevant, so as to minimize the false triggering of the stream voice through hierarchical and step-by-step progress. Therefore, the false triggering of the stream voice can be minimized as much as possible, and the accuracy of the stream voice interaction can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 This is a flow chart of an embodiment of the streaming voice interaction method of the present application; Figure 2 This is a process diagram of an embodiment of the streaming voice interaction method of the present application; Figure 3 It is a schematic diagram of the framework of an embodiment of the streaming voice interaction device of the present application; Figure 4 It is a schematic diagram of the framework of an embodiment of the electronic device of the present application; Figure 5 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0011] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0012] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0013] The terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the fragment " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, "many" in this article means two or more than two.

[0014] See also Figure 1 , Figure 1 This is a flow chart of an embodiment of the streaming voice interaction method of the present application. Specifically, it may include the following steps: Step S11: performing a first classification based on the currently collected first stream speech to obtain a first predicted category of the first stream speech.

[0015] In the disclosed embodiment, streaming voice is voice data in the form of a stream. Different from non-streaming voice, which usually uses voice interaction as the voice starting point (such as clicking and holding the voice button) and closes the voice interaction (such as releasing the voice button) as the voice end point for the whole voice data, streaming voice interaction can imitate the voice interaction mode of human interaction to perform human-computer interaction in a timely and real-time manner during the voice collection process. In addition, the first prediction category can specifically be human voice, noise, or silence. Exemplarily, the first classification of the first streaming voice can be enabled at t=0s, when t=1 second, since no one has spoken yet, the first prediction category is silence (or when there is background noise, the first prediction category can also be noise), when t=2 seconds, someone speaks, and the first prediction category is human voice. Of course, the above example is only a possible example in the actual application process, and does not limit other situations that may exist in the actual application process, and no examples are given here one by one.

[0016] In one implementation scenario, as a possible example, the first prediction category can be predicted by a first classification network, the first classification network can be trained based on the first sample speech, and the first sample speech can be marked with a label category, such as human voice, noise, or silence. Specifically, the first sample speech can be first classified based on the first classification network to obtain the predicted category of the first sample speech, and then the network parameters of the first classification network can be adjusted based on the difference between the predicted category and the labeled category of the first sample speech. It should be noted that the first classification network processes the first sample speech, and specifically obtains the predicted probability values ​​of the first sample speech belonging to human voice, noise, and silence, respectively. Based on this, the predicted probability values ​​of the first sample speech belonging to human voice, noise, and silence, respectively, and the labeled category of the first sample speech can be processed by loss functions such as cross entropy to obtain the training loss of the first classification network, so as to adjust the network parameters of the first classification network based on the training loss of the first classification network. In addition, the first classification network can specifically include but is not limited to: convolutional neural network, recurrent neural network, long short-term memory network, Transformer, etc., and the network structure of the first classification network is not limited here.

[0017] In another implementation scenario, different from the aforementioned implementation, as another possible example, in the process of training a first classification network based on a first sample speech, the first sample speech can also be first classified based on the first classification network to obtain a predicted category of the first sample speech, and the sample original speech can be classified and predicted based on the speech endpoint detection network to obtain a speech classification of each sample speech segment in the sample original speech, and the sample original speech is the complete speech in which the first sample speech is located, and the first sample speech is one of the sample speech segments, and the speech classification is human voice, noise, or silence, and then based on the difference between the predicted category and the labeled category of the first sample speech, and the difference between the predicted category and the speech classification of the first sample speech, the network parameters of the first classification network are adjusted. The above method, during the training process of the first classification network, by combining the difference between the predicted category and the labeled category of the first sample speech, as well as the difference between the predicted category and the speech classification of the first sample speech, can force the first classification network to learn the true labeling of the sample speech on the one hand, so that the output result of the first classification network is as close as possible to the true classification of the input speech; on the other hand, it can force the first classification network to learn the speech endpoint detection model to realize knowledge distillation based on the speech endpoint detection model, and then the knowledge / performance of the speech endpoint detection model can be transferred to the first classification network as much as possible. The combination of the above two aspects helps to improve the accuracy of the first classification network in performing the first classification.

[0018] In a specific implementation scenario, the voice endpoint detection network is used to implement voice activity detection (VAD). For specific implementation, please refer to the technical details of VAD, which will not be repeated here.

[0019] In a specific implementation scenario, the first classification network processes the first sample speech, and specifically obtains the predicted probability values ​​of the first sample speech belonging to human voice, noise, and silence, respectively. Based on this, the predicted probability values ​​of the first sample speech belonging to human voice, noise, and silence, respectively, and the labeled category of the first sample speech can be processed through loss functions such as cross entropy to obtain the first loss of the first classification network.

[0020] In a specific implementation scenario, for the sake of ease of description, the predicted probability values ​​of the first sample speech obtained by the first classification network processing the first sample speech belonging to human voice, noise, and silence respectively can be called the first probability distribution, and the predicted probability values ​​of the sample speech segments obtained by processing the sample original speech by the speech endpoint detection network belonging to human voice, noise, and silence respectively can be called the second probability distribution. The first probability distribution and the second probability distribution of the first sample speech can be measured by loss functions such as KL divergence, so as to measure the difference between the predicted category of the first sample speech and the speech classification, and obtain the second loss of the first classification network.

[0021] In a specific implementation scenario, the training loss of the first classification network can be obtained based on the first loss and the second loss, and then the network parameters of the first classification network can be adjusted based on the classification loss of the first classification network. Exemplarily, the training loss of the first classification network can be obtained by weighting, averaging, summing, etc. based on the first loss and the second loss.

[0022] Step S12: In response to the first predicted category being human voice, a second classification is performed based at least on the first stream speech to obtain a second predicted category of the first stream speech.

[0023] In the disclosed embodiment, the second prediction category is real human voice or background human voice. It should be noted that real human voice refers to human voice with real human-computer interaction intention, while background human voice can include player human voice (such as TV human voice, etc.), other speakers’ human voice (such as when a speaker who is interacting with the human-computer is talking to others, the human voice emitted by others), etc. The possible situations of background human voice will not be given one by one here.

[0024] In one implementation scenario, as a possible implementation, the first stream speech can be directly subjected to a second classification to obtain a second predicted category of the first stream speech. Exemplarily, feature extraction can be performed based on the first stream speech to obtain speech features of the first stream speech, and classification prediction can be performed based on the speech features of the first stream speech to obtain the second predicted category of the first stream speech.

[0025] In another implementation scenario, different from the aforementioned implementation, as another possible implementation, before performing the second classification, the voice collected when the voice interaction is turned on can be obtained as a reference voice. Exemplarily, when a click operation on the voice button is detected, voice data of a preset duration (such as 5 seconds, 10 seconds, etc.) can be collected as a reference voice. On this basis, a second classification can be performed based on the first streaming voice and the reference voice to obtain a second predicted category of the first streaming voice. Exemplarily, feature extraction can be performed based on the first streaming voice to obtain voice features of the first streaming voice, and feature extraction can be performed based on the reference voice to obtain voice features of the reference voice, and then the voice features of the first streaming voice and the voice features of the reference voice are fused to obtain fused features, and classification prediction is performed based on the fused features to obtain the second predicted category of the first streaming voice. In the above method, the voice collected during the voice interaction is first obtained as a reference voice, and then the second classification is performed based on the first streaming voice and the reference voice to obtain the second predicted category of the first streaming voice. The second classification of the first streaming voice can be assisted by the reference voice, which helps to improve the accuracy of the second classification.

[0026] In an implementation scenario, similar to the aforementioned first classification, the second prediction category can be predicted by the second classification network, and the second classification network can be trained based on the second sample voice, and the annotated category of the second sample voice is real human voice or background human voice. For the specific meanings of real human voice and background human voice, please refer to the aforementioned related description, which will not be repeated here. For the specific training process, please refer to the aforementioned related description of training the first classification network, which will not be repeated here. In addition, the second classification network may include but is not limited to: convolutional neural network, recurrent neural network, Transformer, etc., and the network structure of the second classification network is not limited here. For the second sample voice with the annotated category of real human voice, it can be obtained by such methods as real recording in a recording studio and voice collection of real human voice in a real interaction scene, which is not limited here. For the second sample voice with the annotated category of background human voice, as a possible example, the sample source can be collected at the target distance to the location of the voice interaction to obtain the second sample voice with the annotated category of background human voice, and the sample source is a speaker or a player. For example, during voice interaction, the speaker is generally about 50 centimeters away from the machine equipment. Then, a sample signal source can be set at a target distance such as 1 meter or 2 meters away from the speaker from the machine equipment, so that the machine equipment can collect voice from the sample signal source as a second sample voice with the labeled category of background human voice; or, as another possible example, the volume of the second sample voice with the labeled category of real human voice can be at least attenuated to obtain a second sample voice with the labeled category of background human voice. Of course, the above examples are only several possible ways to obtain the second sample voice labeled as background human voice, and they are not listed one by one here. For example, the background human voice can also be collected in a real interaction scene to obtain the second sample voice. In the above method, by collecting voice from the sample signal source and attenuating the volume of the real spoken voice to obtain the second sample voice labeled as background human voice, sample amplification can be achieved as much as possible for training the second classification network.

[0027] Step S13: In response to the second predicted category being a real human voice, a third classification is performed based at least on the first stream speech to obtain a third predicted category of the first stream speech.

[0028] In the disclosed embodiment, the third prediction category involves whether the semantics are complete and whether it is related to the context content. It should be noted that whether the semantics of the first stream speech are complete is used to measure whether the first stream speech contains relatively complete semantic information, such as "today...", "today's weather...", "today's weather...", etc., which are incomplete in semantics. Among the three, the semantics of the first two are relatively short (or close to none), and the semantics of the last one are relatively long (i.e. close to complete), while the semantics of "today's weather is very good", "how is the weather today", "Is the weather today suitable for outdoor sports" and other expressions are complete. In addition, whether the first stream speech is related to the context content is used to measure whether the first stream speech is related to the context (which can be the previous text / previous text in the stream speech) at the semantic level. For example, taking the previous text "how is the weather today" as an example, when the first stream speech is "Is it suitable for outdoor sports today", the first stream speech is related to the context content, and when the first stream speech is "Is C Mountain in City B suitable for climbing for the elderly", the first stream speech is irrelevant to the context content. Of course, the above examples are only several possible examples of the third prediction category in the actual application process, and other possible situations are not given examples one by one here.

[0029] In one implementation scenario, as a possible implementation, the third classification can be directly performed based on the first stream speech to obtain the third predicted category of the first stream speech. Exemplarily, feature extraction can be performed based on the first stream speech to obtain the speech features of the first stream speech, and then classification prediction can be performed based on the speech features of the first stream speech to obtain the third predicted category of the first stream speech.

[0030] In another implementation scenario, different from the aforementioned implementation, as another possible implementation, before executing the third classification, the recognition text of the second stream voice can be obtained as the historical text, and the second stream voice is the real human voice that performed voice interaction before the first stream voice. Exemplarily, still taking the first stream voice as "Is it suitable for outdoor sports today?" as an example, the voice data "What's the weather like today" that performed voice interaction before it can be used as the second stream voice. Of course, the above example is only a possible example in the actual application process, and the specific content of the first stream voice and the second stream voice is not limited here. On this basis, the third classification can be performed based on the first stream voice and the historical text to obtain the third predicted category of the first stream voice. Exemplarily, feature extraction can be performed based on the first stream voice to obtain the voice features of the first stream voice, and feature extraction can be performed based on the historical text to obtain the text features of the historical text, and then the voice features of the first stream voice and the text features of the historical text are fused to obtain the fusion features, and the classification prediction is performed based on the fusion features to obtain the third predicted category of the first stream voice. In the above method, the recognition text of the second stream speech is first obtained as the historical text, and then the third classification is performed based on the first stream speech and the historical text to obtain the third predicted category of the first stream speech. The third classification of the first stream speech can be assisted by the historical text, which helps to improve the accuracy of the third classification.

[0031] In one implementation scenario, similar to the related implementations of the aforementioned first and second classifications, the third classification can be performed by a third classification network. It should be noted that the third classification network may include but is not limited to: convolutional neural network, recurrent neural network, long short-term memory network, Transformer, etc., and the network structure of the third classification network is not limited here. In addition, the third classification network can be trained based on the third sample speech, and the annotation category of the third sample speech can involve whether it is semantically complete and whether it is related to the context content. The third sample speech can also be annotated with the text corresponding to the speech data of the previous speech interaction (for the convenience of description, it can be called sample historical text) and the text corresponding to the third sample speech itself (for the convenience of description, it can be called sample real text). On this basis, the third sample speech and the sample historical text can be processed based on the third classification network to obtain the predicted category of the third sample speech and the sample recognition text, so that the first loss can be obtained based on the difference between the predicted category and the labeled category of the third sample speech, and the second loss can be obtained based on the difference between the sample real text and the sample recognition text of the third sample speech. Then, the first loss and the second loss can be fused (such as weighted, averaged, summed, etc.) to obtain the training loss of the third classification network, so as to adjust the network parameters of the third classification network based on the training loss of the third classification network.

[0032] Step S14: Based on the third prediction category, a target interactive operation is performed on the machine dialogue content currently being output.

[0033] In the disclosed embodiment, the target interactive operation is any one of several preset interactive operations, and the several preset interactive operations include maintaining the current topic, switching to a new topic, and continuing the original topic from the interrupted conversation. Of course, the above examples are only several possible examples of several preset interactive operations, and other possible interactive operations are not given examples one by one here.

[0034] In one implementation scenario, in response to the third prediction category being semantically complete, the machine dialogue content currently being output can be interrupted and switched to a new topic. It should be noted that the new topic is the topic to be interacted by the first stream voice. Taking the first stream voice "How is the weather today" as an example, if the machine dialogue content currently being output is the machine dialogue content of the previous voice data "Please help me plan the itinerary to City B" (i.e., the dialogue content generated and output in response to the voice data), since the third prediction category of the first stream voice represents semantic completeness, the machine dialogue content currently being output can be interrupted. For example, the current dialogue content is generated and output to "... Mountain C located in the north of City B is one of the famous attractions in the country", and it can be interrupted here and switched to the new topic of "How is the weather today", at which time a new machine dialogue content can be generated and output, such as "It is sunny all day today, the temperature is XX degrees, the wind force is XX level, and the humidity is XX%..." In addition, when the third prediction category represents semantic completeness, as a possible example, regardless of whether the third prediction category represents related to the context content, the machine dialogue content currently being output can be interrupted and switched to a new topic.

[0035] In one implementation scenario, in response to the third prediction category being semantically incomplete and unable to determine whether it is related to the context content, the original topic of the currently output machine dialogue content can be continued from the interruption point. It should be noted that the original topic is the topic to which the currently output machine dialogue content belongs. In addition, in the case where it is impossible to determine whether it is related to the context content, it can be considered that the first-stream speech has a high degree of semantic incompleteness, so that it is impossible to determine whether it is related to the context content, such as short semantics (such as "Today..."), close to nothing (such as "Today..."), or even no semantics (such as the first-stream speech is "um", "oh" and other interjections). Taking the first stream voice "Today is..." as an example, if the machine dialogue content currently being output is the machine dialogue content of the previous voice data "Please help me plan a trip to City B" (that is, the dialogue content generated and output in response to the voice data), then because the third prediction category of the first stream voice is semantically incomplete and it is impossible to determine whether it is related to the context content, the machine dialogue content currently being output can continue the original topic from the interrupted dialogue. For example, the current dialogue content is generated and output to "...Mount C located in the north of City B is one of the famous attractions in the country", and the original topic can be continued at this interruption (that is, "one of"), such as continuing to generate and output machine dialogue content "It is one of the four famous mountains with Mountain D, Mountain E, and Mountain F. In addition,...", "We were just interrupted, let's continue. Just now we said that Mountain C located in the north of City B is one of the famous attractions in the country, and it is one of the four famous mountains with Mountain D, Mountain E, and Mountain F. In addition,...", and so on.

[0036] In one implementation scenario, in response to the third prediction category being semantically incomplete and related to the context content, the machine dialogue content currently being output can be paused and prepared to switch to a new topic. It should be noted that the new topic is the topic to be interacted with by the first stream voice. In addition, if it can be judged to be related to the context content, it can be considered that the first stream voice, although semantically incomplete, has a longer semantics, such as "Today's weather...", etc., can already express a certain degree of semantic information. If the machine dialogue content currently being output is the machine dialogue content in response to the previous voice data "Please help me plan a trip to City B" (i.e., the dialogue content generated and output in response to the voice data), and the current dialogue content is generated and output to "...Mount C in the north of City B is one of the famous attractions in the country", taking the first-stream voice "Is Mountain C in City B suitable for the elderly..." as an example, since the third prediction category of the first-stream voice is semantically incomplete (for example, in this example, there are the following possible semantic situations: whether Mountain C in City B is suitable for the elderly to climb, whether Mountain C in City B is suitable for the elderly to travel and live, whether Mountain C in City B is suitable for the elderly to escape the summer heat, etc.) but is related to the context content (all involve City B, and the latter "Mount C in City B is suitable for the elderly to climb, whether Mountain C in City B is suitable for the elderly to travel and live, etc.) ” is also one of the parts of the former “itinerary to City B”), so you can pause at “one” in the aforementioned generated and output machine dialogue content, and prepare to switch to a new topic (that is, “Is Mountain C in City B suitable for the elderly…”). At this time, when the first-stream voice continues to be collected over time and the semantics are complete at a certain moment, you can interrupt the currently output machine dialogue content and switch to a new topic. For example, at a certain moment, the first-stream voice is completely expressed as “Is Mountain C in City B suitable for climbing for the elderly?”, you can switch to the new topic and generate and output new machine dialogue content, such as “The temperature of Mountain C in City B is relatively low throughout the year, and the lowest temperature even drops below minus 20 degrees. It is very unsuitable for climbing for the elderly and the weak…”.

[0037] In one implementation scenario, in response to the third prediction category being semantically incomplete and irrelevant to the contextual content, the current topic can be maintained for the machine dialogue content currently being output and the dialogue content can be continued to be output. It should be noted that, in the case of being irrelevant to the contextual content, the first-stream voice collected at this time may be the speaker who is in human-computer interaction. The original intention is to use the first-stream voice collected at this time to have a dialogue with other people around him instead of a human-computer dialogue. If the machine dialogue content currently being output is the machine dialogue content of the previous voice data "Please help me plan the itinerary to City B" (that is, the dialogue content generated and output in response to the voice data), and the current dialogue content is generated and output to "... Mountain C located in the north of City B is one of the famous attractions in the country", taking the first-stream voice "Today's stock market trend..." as an example, its semantics is incomplete and irrelevant to the contextual content, so it can be considered that the first-stream voice collected at this time may be the speaker is having a human-to-human dialogue with others instead of a human-computer dialogue, then the current topic can be maintained for the machine dialogue content currently being output and the dialogue content can be continued to be output, such as the dialogue content "It is ranked among the four famous mountains with Mountain D, Mountain E, and Mountain F. In addition,..." can be continued to be output.

[0038] It should be noted that the above examples are only possible examples of target interaction operations when the third prediction category is semantically complete, semantically incomplete and cannot be determined whether it is related to the context, semantically incomplete and related to the context, and semantically incomplete and irrelevant to the context. Other possible situations will not be given one by one here. Figure 2 , Figure 2 1 is a process diagram of an embodiment of the streaming voice interaction method of the present application. Figure 2As shown, the target interaction operation can be triggered by the streaming voice interaction model, and the streaming voice interaction model can include at least a first classification network, a second classification network, a third classification network and a large language model, including a first classification network that can be used to perform a first classification, a second classification network that can be used to perform a second classification, a third classification network that can be used to perform a third classification, and a large language model that can be used to perform the target interaction operation. Here, the relevant connotations of the first classification network, the second classification network and the third classification network can refer to the aforementioned related descriptions, which will not be repeated here. In addition, the large language model can perform the corresponding target interaction operation according to the third prediction category. Exemplarily, a prompt instruction can be constructed in combination with the third prediction category and the recognition text of the first streaming voice, and the prompt instruction is used to instruct the large language model to perform the target interaction operation, and then the prompt instruction is input to the large language model, so that the large language model can respond to the first streaming voice in real time. For example, when the third prediction category is semantically incomplete and it is impossible to determine whether it is related to the context content, the prompt instruction may include but is not limited to "continue the original topic of the machine dialogue content currently being output from the interrupted dialogue"; when the third prediction category is semantically incomplete and related to the context content, the prompt instruction may include but is not limited to "pause the machine dialogue content currently being output and prepare to switch to a new topic [recognition text of the first stream speech]"; when the third prediction category is semantically incomplete and irrelevant to the context content, the prompt instruction may include but is not limited to "maintain the current topic and continue to output the dialogue content for the machine dialogue content currently being output"; when the third prediction category is semantically complete, the prompt instruction may include but is not limited to "interrupt the machine dialogue content currently being output and switch to a new topic [recognition text of the first stream speech]". Of course, the above examples are only several possible examples in actual application, and the specific content of the prompt instruction of the large language model is not limited here.

[0039] In the above scheme, a first classification is performed based on the first stream of speech currently collected to obtain a first predicted category of the first stream of speech, and the first predicted category is human, noise, or silence. In response to the first predicted category being human voice, a second classification is performed at least based on the first stream of speech to obtain a second predicted category of the first stream of speech, and the second predicted category is real human voice or background human voice. In response to the second predicted category being real human voice, a third classification is performed at least based on the first stream of speech to obtain a third predicted category of the first stream of speech, and the third predicted category involves whether the semantics are complete and whether it is related to the context content. Based on the third predicted category, the machine speech currently being output is classified. The target interactive operation is performed based on the content of the conversation. The target interactive operation is any one of several preset interactive operations, and several preset interactive operations include maintaining the current topic, switching to a new topic, and continuing the original topic from the interrupted conversation. Therefore, the first stream voice currently collected is hierarchically classified in the first classification, the second classification, and the third classification in sequence. In the first classification process, human voice, noise or silence are identified. In the second classification process, the human voice is further identified as a real human voice or a background human voice. In the third classification process, it is further identified whether the semantics of the real human voice are complete and whether the content is relevant, so as to minimize the false triggering of the stream voice through hierarchical and step-by-step progress. Therefore, the false triggering of the stream voice can be minimized as much as possible, and the accuracy of the stream voice interaction can be improved.

[0040] See also Figure 3 , Figure 3 It is a schematic diagram of the framework of an embodiment of a streaming voice interaction device of the present application. The streaming voice interaction device 30 includes: a first classification module 31, a second classification module 32, a third classification module 33 and an interactive operation module 34, wherein the first classification module 31 is used to perform a first classification based on the currently collected first streaming voice to obtain a first predicted category of the first streaming voice; wherein the first predicted category is human voice, noise, or silence; the second classification module 32 is used to perform a second classification based on at least the first streaming voice in response to the first predicted category being human voice, to obtain a second predicted category of the first streaming voice; wherein the second predicted category is real human voice or background human voice; the third classification module 33 is used to perform a third classification based on at least the first streaming voice in response to the second predicted category being real human voice, to obtain a third predicted category of the first streaming voice; wherein the third predicted category involves whether the semantics are complete and whether it is related to the context content; the interactive operation module 34 is used to perform a target interactive operation on the machine dialogue content currently being output based on the third predicted category; wherein the target interactive operation is any one of several preset interactive operations, and the several preset interactive operations include maintaining the current topic, switching to a new topic, and continuing the original topic from the interrupted conversation.

[0041] In the above scheme, the streaming voice interaction device 30 performs a first classification based on the currently collected first streaming voice to obtain a first predicted category of the first streaming voice, and the first predicted category is human, noise, or silence. In response to the first predicted category being human voice, a second classification is performed at least based on the first streaming voice to obtain a second predicted category of the first streaming voice, and the second predicted category is real human voice or background human voice. In response to the second predicted category being real human voice, a third classification is performed at least based on the first streaming voice to obtain a third predicted category of the first streaming voice, and the third predicted category involves whether the semantics are complete and whether it is related to the context content. Then, based on the third predicted category, the currently input voice is classified. The target interactive operation is performed on the machine dialogue content generated, and the target interactive operation is any one of several preset interactive operations, and several preset interactive operations include maintaining the current topic, switching to a new topic, and continuing the original topic from the interrupted conversation. Therefore, the first stream voice currently collected is hierarchically classified in the first classification, the second classification, and the third classification in sequence. In the first classification process, human voice, noise or silence are identified. In the second classification process, the human voice is further identified as a real human voice or a background human voice. In the third classification process, it is further identified whether the semantics of the real human voice are complete and whether the content is relevant, so as to minimize the false triggering of the streaming voice through hierarchical and step-by-step progress. Therefore, it is possible to minimize the false triggering of the streaming voice and improve the accuracy of the streaming voice interaction.

[0042] In some disclosed embodiments, the first predicted category is predicted by a first classification network, and the streaming voice interaction device 30 includes an endpoint prediction module for performing a first classification on the first sample voice based on the first classification network to obtain the predicted category of the first sample voice, and performing classification prediction on the sample original voice based on the voice endpoint detection network to obtain the voice classification of each sample voice segment in the sample original voice; wherein the sample original voice is the complete voice containing the first sample voice, the first sample voice is one of the sample voice segments, and the voice classification is human voice, noise, or silence; the streaming voice interaction device 30 includes a parameter adjustment module for adjusting the network parameters of the first classification network based on the difference between the predicted category and the labeled category of the first sample voice, and the difference between the predicted category and the voice classification of the first sample voice.

[0043] In some disclosed embodiments, the streaming voice interaction device 30 includes a voice acquisition module for acquiring the voice collected when the voice interaction is turned on as a reference voice; the second classification module 32 is specifically used to perform a second classification based on the first streaming voice and the reference voice to obtain a second predicted category of the first streaming voice.

[0044] In some disclosed embodiments, the second predicted category is predicted by a second classification network, which is trained based on a second sample voice, and the labeled category of the second sample voice is real human voice or background human voice. The streaming voice interaction device 30 includes a data enhancement module for performing at least one of the following: collecting voice from a sample source at a target distance from the location at the time of voice interaction to obtain a second sample voice with a labeled category of background human voice; wherein the sample source is a speaker or a player; and at least performing volume attenuation based on the second sample voice with a labeled category of real human voice to obtain a second sample voice with a labeled category of background human voice.

[0045] In some disclosed embodiments, the streaming voice interaction device 30 includes a text acquisition module for acquiring the recognized text of the second streaming voice as a historical text; wherein the second streaming voice is a real human voice that performs voice interaction before the first streaming voice; and the third classification module 33 is specifically used to perform a third classification based on the first streaming voice and the historical text to obtain a third predicted category of the first streaming voice.

[0046] In some disclosed embodiments, the interactive operation module 34 includes a first response submodule for, in response to the third prediction category being semantically incomplete and unable to determine whether it is related to the context content, continuing the original topic of the machine dialogue content currently being output from the interrupted dialogue point; the interactive operation module 34 includes a second response submodule for, in response to the third prediction category being semantically incomplete and related to the context content, pausing the machine dialogue content currently being output and preparing to switch to a new topic; the interactive operation module 34 includes a third response submodule for, in response to the third prediction category being semantically incomplete and irrelevant to the context content, maintaining the current topic and continuing to output the dialogue content for the machine dialogue content currently being output.

[0047] In some disclosed embodiments, the target interaction operation is triggered and executed by a streaming voice interaction model, which includes at least a first classification network, a second classification network, a third classification network and a large language model, including the first classification network for performing a first classification, the second classification network for performing a second classification, the third classification network for performing a third classification, and the large language model for performing the target interaction operation.

[0048] See also Figure 4 , Figure 4It is a schematic diagram of the framework of an embodiment of an electronic device of the present application. The electronic device 40 includes at least a memory 41 and a processor 42 coupled to each other, the memory 41 stores at least program instructions, and the processor 42 is used to execute the program instructions to implement the steps in any of the above-mentioned streaming voice interaction method embodiments. For details, please refer to the aforementioned disclosed embodiments, which will not be repeated here. As a possible example, the electronic device 40 may include but is not limited to an office notebook, a smart phone, a tablet computer, a learning machine, a server, etc., and the specific type of the electronic device 40 is not limited here.

[0049] Specifically, the processor 42 is used to control itself and the memory 41 to implement the steps in any of the above-mentioned streaming voice interaction method embodiments. The processor 42 can also be called a CPU (Central Processing Unit). The processor 42 may be an integrated circuit chip with signal processing capabilities. The processor 42 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 42 can be implemented by an integrated circuit chip.

[0050] In the above scheme, the electronic device 40 performs a first classification based on the currently collected first stream voice to obtain a first predicted category of the first stream voice, and the first predicted category is human, noise, or silence. In response to the first predicted category being human voice, the electronic device 40 performs a second classification based on at least the first stream voice to obtain a second predicted category of the first stream voice, and the second predicted category is real human voice or background human voice. In response to the second predicted category being real human voice, the electronic device 40 performs a third classification based on at least the first stream voice to obtain a third predicted category of the first stream voice, and the third predicted category involves whether the semantics are complete and whether it is related to the context content. Based on the third predicted category, the currently output The machine dialogue content performs the target interactive operation, which is any one of several preset interactive operations, and several preset interactive operations include maintaining the current topic, switching to a new topic, and continuing the original topic from the interrupted conversation. Therefore, the first stream voice currently collected is hierarchically classified in the first classification, the second classification, and the third classification in sequence. In the first classification process, human voice, noise or silence are identified. In the second classification process, the human voice is further identified as a real human voice or a background human voice. In the third classification process, it is further identified whether the semantics of the real human voice are complete and whether the content is relevant, so as to minimize the false triggering of the streaming voice through hierarchical and step-by-step progress. Therefore, it is possible to minimize the false triggering of the streaming voice and improve the accuracy of the streaming voice interaction.

[0051] See also Figure 5 , Figure 5 1 is a schematic diagram of a framework of an embodiment of a computer-readable storage medium 50 of the present application. The computer-readable storage medium 50 stores program instructions 51 that can be executed by a processor, and the program instructions 51 are used to implement the steps in any of the above-mentioned streaming voice interaction method embodiments.

[0052] In the above scheme, the computer-readable storage medium 50 performs a first classification based on the currently collected first stream speech to obtain a first predicted category of the first stream speech, and the first predicted category is human, noise, or silence. In response to the first predicted category being human voice, a second classification is performed at least based on the first stream speech to obtain a second predicted category of the first stream speech, and the second predicted category is real human voice or background human voice. In response to the second predicted category being real human voice, a third classification is performed at least based on the first stream speech to obtain a third predicted category of the first stream speech, and the third predicted category involves whether the semantics are complete and whether it is related to the context content. Based on the third predicted category, the currently input The target interactive operation is performed on the machine dialogue content generated, and the target interactive operation is any one of several preset interactive operations, and several preset interactive operations include maintaining the current topic, switching to a new topic, and continuing the original topic from the interrupted conversation. Therefore, the first stream voice currently collected is hierarchically classified in the first classification, the second classification, and the third classification in sequence. In the first classification process, human voice, noise or silence are identified. In the second classification process, the human voice is further identified as a real human voice or a background human voice. In the third classification process, it is further identified whether the semantics of the real human voice are complete and whether the content is relevant, so as to minimize the false triggering of the streaming voice through hierarchical and step-by-step progress. Therefore, it is possible to minimize the false triggering of the streaming voice and improve the accuracy of the streaming voice interaction.

[0053] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0054] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0055] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0056] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0057] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0058] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0059] If the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to the collection of his or her personal information; or on the device that processes personal information, the personal information processing rules are notified by obvious signs / information, and the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

Claims

1. A streaming voice interaction method, characterized in that: include: Performing a first classification based on the currently collected first stream speech to obtain a first predicted category of the first stream speech; wherein the first predicted category is human voice, noise, or silence; In response to the first predicted category being human voice, performing a second classification based at least on the first stream speech to obtain a second predicted category of the first stream speech; wherein the second predicted category is a real human voice or a background human voice; In response to the second predicted category being a real human voice, performing a third classification based at least on the first stream speech to obtain a third predicted category of the first stream speech; wherein the third predicted category involves whether the first stream speech is semantically complete and whether it is related to contextual content; Based on the third prediction category, a target interaction operation is performed on the machine dialogue content currently being output; wherein the target interaction operation is any one of several preset interaction operations, and the several preset interaction operations include maintaining the current topic, switching to a new topic, and continuing the original topic from the interrupted dialogue.

2. The method according to claim 1, characterized in that The first predicted category is predicted by a first classification network, and the training steps of the first classification network include: Performing a first classification on the first sample speech based on the first classification network to obtain a predicted category of the first sample speech, and performing classification prediction on the sample original speech based on the speech endpoint detection network to obtain a speech classification of each sample speech segment in the sample original speech; wherein the sample original speech is the complete speech where the first sample speech is located, the first sample speech is one of the sample speech segments, and the speech classification is human voice, noise, or silence; Based on the difference between the predicted category and the marked category of the first sample speech, and the difference between the predicted category and the speech classification of the first sample speech, the network parameters of the first classification network are adjusted.

3. The method according to claim 1, characterized in that Before performing a second classification based at least on the first stream speech to obtain a second predicted category of the first stream speech, the method further includes: Get the voice collected when voice interaction is enabled as the reference voice; The performing a second classification based at least on the first stream speech to obtain a second predicted category of the first stream speech includes: A second classification is performed based on the first stream speech and the reference speech to obtain a second predicted category of the first stream speech.

4. The method according to claim 1, characterized in that: The second predicted category is predicted by a second classification network, the second classification network is trained based on a second sample speech, the labeled category of the second sample speech is a real human voice or a background human voice, and the step of obtaining the second sample speech includes at least one of the following: At a target distance from the location at the time of voice interaction, voice is collected from a sample source to obtain a second sample voice with the labeled category being the background human voice; wherein the sample source is a speaker or a player; Based on the second sample speech whose labeled category is the real human voice, at least volume attenuation is performed to obtain the second sample speech whose labeled category is the background human voice.

5. The method according to claim 1, characterized in that Before performing a third classification based at least on the first stream speech to obtain a third predicted category of the first stream speech, the method further includes: Acquire the recognized text of the second stream voice as the historical text; wherein the second stream voice is a real human voice that interacts with the first stream voice before; The performing a third classification based at least on the first stream speech to obtain a third predicted category of the first stream speech includes: A third classification is performed based on the first stream speech and the historical text to obtain a third predicted category of the first stream speech.

6. The method according to claim 1, characterized in that The performing a target interactive operation on the machine dialogue content currently being output based on the third prediction category includes at least one of the following: In response to the third prediction category being semantically incomplete and unable to determine whether it is related to the context content, continuing the original topic of the machine dialogue content currently being output from the interrupted dialogue; In response to the third prediction category being semantically incomplete and relevant to the context content, pausing the machine dialogue content currently being output and preparing to switch to a new topic; In response to the third prediction category being semantically incomplete and irrelevant to contextual content, the current topic of the machine dialogue content currently being output is maintained to continue outputting the dialogue content.

7. The method according to any one of claims 1 to 6, characterized in that: The target interaction operation is triggered and executed by a streaming voice interaction model, and the streaming voice interaction model includes at least a first classification network, a second classification network, a third classification network and a large language model, wherein the first classification network is used to perform the first classification, the second classification network is used to perform the second classification, the third classification network is used to perform the third classification, and the large language model is used to perform the target interaction operation.

8. A streaming voice interaction device, characterized in that: include: A first classification module, configured to perform a first classification based on the currently collected first stream speech to obtain a first predicted category of the first stream speech; wherein the first predicted category is human voice, noise, or silence; a second classification module, configured to, in response to the first predicted category being human voice, perform a second classification based at least on the first stream speech to obtain a second predicted category of the first stream speech; wherein the second predicted category is a real human voice or a background human voice; A third classification module is configured to, in response to the second predicted category being a real human voice, perform a third classification based at least on the first stream speech to obtain a third predicted category of the first stream speech; wherein the third predicted category involves whether the first stream speech is semantically complete and whether the first stream speech is related to contextual content; An interactive operation module is used to perform a target interactive operation on the machine dialogue content currently being output based on the third prediction category; wherein the target interactive operation is any one of several preset interactive operations, and the several preset interactive operations include maintaining the current topic, switching to a new topic, and continuing the original topic from the interrupted conversation.

9. An electronic device, characterized in that: The method comprises at least a memory and a processor coupled to each other, wherein the memory at least stores program instructions, and the processor is used to execute the program instructions to implement the streaming voice interaction method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Program instructions that can be executed by a processor are stored, and the program instructions are used to implement the streaming voice interaction method described in any one of claims 1 to 7.

Images