Streaming voice interaction method, related device, equipment and storage medium

Abstract

The invention discloses a streaming voice interaction method, a related device, equipment and a storage medium, and the method comprises the steps: detecting whether streaming voice is semantically complete at a current moment; in response to a detection result that semantics is incomplete, obtaining target time delay based on basic time delay and additional time delay, analyzing whether effective voice is newly added in the streaming voice in a process from the current moment to waiting for the target time delay, if so, returning to detect whether semantics of the streaming voice is complete at the current moment, and if not, generating reply data; and in response to a detection result that the semantics is complete, analyzing whether effective voice is newly added in the streaming voice in a process from the current moment to waiting for the basic time delay, if so, returning to detect whether the semantics of the streaming voice is complete at the current moment, and if not, generating reply data. According to the scheme, the possibility that the speaking object is interrupted unexpectedly can be reduced on the premise that the response time of voice interaction is shortened.

Description

Technical Field

[0001] The present application relates to the field of voice interaction technology, and in particular to a streaming voice interaction method and related devices, equipment and storage media. Background Art

[0002] At present, voice interaction technology has been widely used in many scenarios such as smart driving and smart home. Thanks to the rapid development of voice interaction technology, the response time of voice interaction has also been greatly optimized, even reaching the millisecond level, which greatly improves the entire voice interaction system.

[0003] However, while the response time of voice interaction has been greatly optimized, a new problem has emerged: if the response is too fast, the speaker may be interrupted frequently or even unexpectedly. In view of this, how to reduce the possibility of the speaker being interrupted unexpectedly while shortening the response time of voice interaction has become an urgent problem to be solved. Summary of the invention

[0004] The main technical problem solved by the present application is to provide a streaming voice interaction method and related devices, equipment and storage media, which can reduce the possibility of the speaking object being unexpectedly interrupted while shortening the response time of the voice interaction.

[0005] In order to solve the above technical problems, the first aspect of the present application provides a streaming voice interaction method, including: detecting whether the streaming voice is semantically complete at the current moment; in response to the detection result being semantically incomplete, obtaining a target delay based on a basic delay plus an additional delay, and analyzing whether new valid voice is added to the streaming voice from the current moment until the target delay is reached, and if so, returning to detecting whether the streaming voice is semantically complete at the current moment, and otherwise generating reply data; in response to the detection result being semantically complete, analyzing whether new valid voice is added to the streaming voice from the current moment until the target delay is reached, and if so, returning to detecting whether the streaming voice is semantically complete at the current moment, and otherwise generating reply data.

[0006] In order to solve the above technical problems, the second aspect of the present application provides a streaming voice interaction device, including: a semantic detection module, a first response module and a second response module, the semantic detection module is used to detect whether the streaming voice is semantically complete at the current moment; the first response module is used to respond to the detection result of semantic incompleteness, based on the basic delay plus the additional delay, to obtain the target delay, and from the current moment to the waiting time of the target delay, analyze whether there is new valid voice in the streaming voice, if so, return to detect whether the streaming voice is semantically complete at the current moment, if otherwise, generate reply data; the second response module is used to respond to the detection result of semantic completeness, from the current moment to the waiting time of the basic delay, analyze whether there is new valid voice in the streaming voice, if so, return to detect whether the streaming voice is semantically complete at the current moment, if otherwise, generate reply data.

[0007] In order to solve the above technical problems, the third aspect of the present application provides an electronic device, which at least includes a memory and a processor coupled to each other, the memory at least stores program instructions, and the processor is used to execute the program instructions to implement the streaming voice interaction method in the above first aspect.

[0008] In order to solve the above technical problems, the fourth aspect of the present application provides a computer-readable storage medium, which stores program instructions that can be executed by a processor, and the program instructions are used to implement the streaming voice interaction method of the above first aspect.

[0009] The above scheme detects whether the streaming voice is semantically complete at the current moment, and in response to the detection result being semantically incomplete, obtains the target delay based on the basic delay plus the additional delay, and analyzes whether new valid voice is added to the streaming voice from the current moment until the target delay is reached. If so, return to detect whether the streaming voice is semantically complete at the current moment, otherwise generate reply data, and in response to the detection result being semantically complete, from the current moment until the target delay is reached, analyzes whether new valid voice is added to the streaming voice, and if so, return to detect whether the streaming voice is semantically complete at the current moment, otherwise generate reply data, and one party When it is detected that the semantics of the streaming speech are incomplete at the current moment, adding an extra delay to the basic delay compared to when the semantics are complete can extend the delay when the speaker is about to continue speaking, which helps reduce the possibility of the speaker being interrupted unexpectedly. On the other hand, regardless of whether the semantics are complete or not, as long as new valid speech is detected during the waiting process, the semantic integrity check will continue without replying temporarily, which also helps reduce the possibility of the speaker being interrupted unexpectedly. On the other hand, regardless of whether the semantics are complete or not, if no new valid speech is detected during the waiting process, a reply will be made, which can also shorten the response time of voice interaction as much as possible. Therefore, the possibility of the speaker being interrupted unexpectedly can be reduced while shortening the response time of voice interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 This is a flow chart of an embodiment of the streaming voice interaction method of the present application; Figure 2 This is a schematic diagram of the effect of an embodiment of the streaming voice interaction method of the present application; Figure 3 It is a schematic diagram of the framework of an embodiment of the streaming voice interaction device of the present application; Figure 4 It is a schematic diagram of the framework of an embodiment of the electronic device of the present application; Figure 5 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0011] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0012] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0013] The terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the fragment " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, "many" in this article means two or more than two.

[0014] See also Figure 1 , Figure 1 This is a flow chart of an embodiment of the streaming voice interaction method of the present application. Specifically, it may include the following steps: Step S11: Detect whether the streaming speech is semantically complete at the current moment.

[0015] In the disclosed embodiment, the streaming voice representation continuously collects audio signals during the interaction process. In actual application, the continuously collected audio signal (i.e., streaming voice) may contain both valid voice (i.e., audio signal containing substantial speech content) and invalid voice (i.e., audio signal without substantial speech content such as pauses). Valid voice and invalid voice may also exist intermittently and interspersed in the streaming voice, such as "[short pause] (i.e., this section is invalid voice) [How is the weather today] (i.e., this section is valid voice)...". Of course, the above examples are only a few possible examples of streaming voice, and the specific content of the streaming voice is not limited here, and no examples are given one by one.

[0016] In one implementation scenario, as a possible implementation example, in order to detect semantic integrity, a deep learning model can be selected as the first artificial intelligence model, and based on the first artificial intelligence model, whether the streaming speech is semantically complete at the current moment can be detected, that is, the deep learning model can be used to detect whether the streaming speech is semantically complete as of the current moment. It should be noted that the deep learning model can include but is not limited to: convolutional neural network, recurrent neural network, long short-term memory network, Transformer, etc., and the network structure of the deep learning model is not limited here.

[0017] In a specific implementation scenario, sample speech can be acquired in advance, and the sample speech can be annotated with a sample tag indicating whether it is actually semantically complete, and the sample speech is detected based on a deep learning model to obtain a predicted tag of the sample speech, and the predicted tag indicates whether the sample speech is semantically complete after prediction, and then the network parameters of the deep learning model are adjusted based on the difference between the sample tag and the predicted tag. The above process steps can be used to implement a round of training for the deep learning model. In actual application, the above process steps can be repeated to iteratively train the deep learning model until the training converges, and then the streaming speech can be detected based on the deep learning model to determine whether it is semantically complete.

[0018] In a specific implementation scenario, when using a deep learning model to perform semantic integrity detection on streaming speech, the speech segments in the streaming speech from the last effective interaction to the current moment can be sent to the deep learning model. The deep learning model can perform semantic integrity detection on the above speech segments to obtain detection results, such as the probability value of semantic completeness and the probability value of semantic incompleteness. When the former is higher than the latter, it can be considered that the semantics are complete. Conversely, when the former is lower than the latter, it can be considered that the semantics are incomplete.

[0019] In a specific implementation scenario, for ease of understanding, take the streaming voice "

short pause

How's the weather today

Long pause

Please help me check if there are any tickets to City A tomorrow

[0020] In another implementation scenario, as another possible implementation example, different from the aforementioned implementation, in order to detect semantic integrity, a large language model can also be selected as the first artificial intelligence model, and based on the first artificial intelligence model, whether the streaming speech is semantically complete at the current moment can be detected, that is, the large language model can be used to detect whether the streaming speech is semantically complete as of the current moment. It should be noted that the large language model may include but is not limited to open source large models such as Llama and Bloom, or a large language model obtained by fine-tuning parameters based on a specific corpus, or it can also be a custom large model, etc., and the network structure of the large language model is not limited here. Exemplarily, a prompt instruction can be constructed based on the speech segment in the streaming speech from the last effective interaction to the current moment, and the prompt instruction is used to instruct the large language model to perform semantic integrity detection on the speech segment, and the prompt instruction is input into the large language model to obtain the output result of the large language model, which can be used as the detection result of the semantic integrity of the streaming speech at the current moment. For ease of understanding, take the streaming voice "

short pause

How is the weather today

Long pause

Please help me check if there are any tickets to City A tomorrow

[0021] In another implementation scenario, as another possible implementation example, different from the aforementioned implementation mode, in order to detect semantic integrity, a deep learning model and a large language model can also be selected simultaneously as the first artificial intelligence model respectively. Based on this, the detection result of the deep learning model on the semantic integrity of the streaming speech at the current moment (including whether the semantics are complete and the corresponding confidence) can be obtained, and the detection result of the large language model on the semantic integrity of the streaming speech at the current moment (including whether the semantics are complete and the corresponding confidence) can be obtained, and then the one with a higher confidence level is selected to determine whether the streaming speech is semantically complete at present.

[0022] In one implementation scenario, in contrast to directly detecting whether the semantics of the streaming voice are complete at the current moment, voice activity detection (VAD) may be performed based on the streaming voice before that, and in response to the streaming voice being detected as a voice end endpoint (e.g., a short pause, etc.) at the current moment, the step of detecting whether the streaming voice is semantically complete at the current moment may be performed, and conversely, in response to the streaming voice being detected as not being a voice end endpoint at the current moment (e.g., still being a valid voice), the step of performing voice activity detection based on the streaming voice may be returned. In the above method, before performing the semantic integrity detection, voice activity detection is performed first, and when the current moment is detected as a voice end endpoint, the semantic integrity detection is performed. On the one hand, the detection burden of semantic integrity can be reduced, and on the other hand, the detection accuracy of whether the semantics are complete at the current moment can be improved by combining the two dimensions of voice activity detection and semantic integrity detection.

[0023] Step S12: In response to the detection result that the semantics are incomplete, the target delay is obtained based on the basic delay plus the additional delay, and from the current moment until the target delay, it is analyzed whether there is new valid voice in the streaming voice. If so, it returns to detect whether the streaming voice is semantically complete at the current moment. If not, reply data is generated.

[0024] In one implementation scenario, as a possible implementation example, the basic delay can be customized based on the actual application scenario, such as 200ms, 300ms, 400ms, 500ms, etc., and the specific value of the basic delay is not limited. In addition, the additional delay can also be customized based on the actual application scenario, such as 200ms, 300ms, 400ms, 500ms, etc., and the specific value of the additional delay is not limited.

[0025] In another implementation scenario, as another possible implementation example, different from the aforementioned implementation, the basic delay can also be adaptively acquired. Specifically, the basic delay can be determined based on the historical voice of the speaker to whom the streaming voice belongs before the current moment, and the basic delay is correlated with the speech state represented by the historical voice, and the speech state at least includes the speech speed. It should be noted that the historical voice can specifically be several rounds of voice that the speaker has interacted with at the current moment, such as five rounds of voice, ten rounds of voice, etc. that the speaker has interacted with before the current moment, and the number of voice rounds that have been interacted is not limited here. In the above method, the basic delay is determined based on the historical voice of the speaker to whom the streaming voice belongs before the current moment, and the basic delay is correlated with the speech state represented by the historical voice, and the speech state at least includes the speech speed, so the basic delay can be adaptively determined according to the actual situation of the speaker, which helps to improve the adaptability of the basic delay to the actual situation.

[0026] In a specific implementation scenario, as a possible implementation example, in order to adaptively determine the basic delay based on historical speech, statistics can be performed based on historical speech to obtain the speaking speed of the speaking object, and the speaking speed is expressed as the number of speaking words per standard time length. It should be noted that the standard time length can be set according to the statistical needs of the actual application, such as 500ms, etc., and the specific value of the standard time length is not limited here. Based on this, the unit delay is proportionally adjusted based on the comparison result of the speaking speed and the benchmark speaking speed to obtain the reference delay. It should be noted that the benchmark speaking speed can be a custom normal speaking speed, which can also be expressed as the number of speaking words per standard time length, such as 3 words per 500ms. In addition, the unit delay is expressed as the adjustment delay per unit text, such as 10ms per 0.1 speaking words, 15ms per 0.1 speaking words, etc., and the specific value of the unit delay is not limited here. The comparison result can specifically be the number of words spoken per standard delay between the speaking speed of the speaking object and the target speaking speed. The ratio of the difference in the number of words spoken to the unit word in the unit delay can be set as the adjustment ratio, and then the adjustment ratio is used to scale the adjustment delay in the unit delay to obtain the reference delay. Still taking the speaking speed of the speaking object as 2 words per 500ms as an example, if the target speaking speed is 3 words per 500ms, the difference in the number of words spoken is 1, and if the unit delay indicates that the adjustment delay for every 0.1 words spoken is 50ms, then the adjustment ratio is 1 / 0.1=10, and the adjustment delay of 50ms is scaled using the adjustment ratio 10 to obtain the reference delay of 500ms. Of course, the above example is only a possible example in practical applications, and other possible situations will not be given one by one here. After obtaining the reference delay, the reference delay can be further constrained based on the delay interval to obtain the basic delay. It should be noted that the delay interval can be set according to actual application needs, such as 400ms to 800ms, 300ms to 700ms, etc., and the specific range of the delay interval is not limited here. In addition, after obtaining the delay interval, the reference delay can be constrained. If the reference delay is in the delay interval, the reference delay can be directly determined as the basic delay. If the reference delay is lower than the lower limit of the delay interval, the lower limit of the delay interval can be determined as the basic delay. If the reference delay is higher than the upper limit of the delay interval, the upper limit of the delay interval can be determined as the basic delay. In the above method, the speaking speed of the speaker is obtained based on historical speech statistics, and the speaking speed is expressed as the number of words per standard time length, and the unit delay is proportionally adjusted based on the comparison result of the speaking speed and the benchmark speed to obtain the reference delay, and then the reference delay is constrained based on the delay interval to obtain the basic delay. The historical speech can be analyzed through relevant rules to determine the basic delay.

[0027] In a specific implementation scenario, as another possible implementation example, in order to adaptively determine the basic delay according to the historical speech, the historical speech can also be predicted based on the second artificial intelligence model to obtain the basic delay, and the second artificial intelligence model at least includes a deep learning model, and the deep learning model is trained based on the sample speech labeled with the sample delay. It should be noted that the deep learning model may include but is not limited to: convolutional neural network, recurrent neural network, long short-term memory network, Transformer, etc., and the network structure of the deep learning model is not limited here. Of course, in order to improve the accuracy of the deep learning model in predicting the basic delay, the deep learning model can be trained in advance, such as the deep learning model can be supervised based on the sample speech labeled with the sample basic delay. The training process can refer to the relevant description of the use of the deep learning model by the first artificial intelligence model mentioned above, which will not be repeated here. In the above method, the historical speech is predicted based on the second artificial intelligence model to obtain the basic delay, and the second artificial intelligence model at least includes a deep learning model, and the deep learning model is trained based on the sample speech labeled with the sample delay, and the historical speech can be predicted through the network model to determine the basic delay.

[0028] In a specific implementation scenario, the basic delay can also be adaptively updated, specifically, based on the recorded data of the streaming voice interaction on the current device, the usage rules of the streaming voice interaction on the current device can be obtained. It should be noted that the recorded data may include the total number of streaming voice interactions performed by the current device in each time period, based on which the usage rules of the streaming voice interaction on the current device can be analyzed, such as the possibility of the current device performing streaming voice interaction in each time period. Based on this, the update time of the basic delay can be determined based on the usage rules. For example, when the usage rules include the possibility of the current device performing streaming voice interaction in each time period, if it is found that the possibility of streaming voice interaction in a certain time period is high, this time period or a time period before this time period can be determined as the update time of the basic delay. Of course, the above example is only a possible example of the update time, and other possible situations are not given one by one here. After determining the update time, the acquisition step of the aforementioned basic delay can be performed according to the update time to adaptively update the basic delay. That is to say, once the update time of the basic delay is reached, the acquisition step of the aforementioned basic delay can be performed. The above method analyzes the recorded data of streaming voice interaction on the current device to obtain the usage pattern of streaming voice interaction on the current device, and then determines the update timing of the basic delay based on the usage pattern. Therefore, the basic delay acquisition step can be executed according to the update timing to adaptively update the basic delay, thereby improving the matching of the basic delay.

[0029] In an implementation scenario, after obtaining the basic delay, an additional delay can be added to the basic delay to obtain the target delay, and the process can wait for new valid voice to be added to the streaming voice from the current moment. If no new valid voice is added to the streaming voice until the target delay, reply data can be generated to respond to the streaming voice. Conversely, if new valid voice is added to the streaming voice at some moment before the target delay, the process can return to the aforementioned step of detecting whether the streaming voice is semantically complete at the current moment.

[0030] In a specific implementation scenario, in order to generate reply data, a voice segment from the last interaction completed to the current moment in the streaming voice can be obtained, and based on the voice segment, a prompt instruction is constructed, and the prompt instruction is used to instruct the large language model to generate reply data for the voice segment, so as to input the prompt instruction to the large language model, and obtain the output data of the large language model as reply data. It should be noted that the prompt instruction may include the required format of the reply data, such as text, voice, video, picture, etc., which is not limited here. In addition, after the reply data is generated, the reply data can be output on the current device.

[0031] In a specific implementation scenario, as described above, before detecting whether the streaming speech is semantically complete at the current moment, voice activity detection may be performed based on the streaming speech. In this case, in response to the addition of valid speech in the streaming speech, the step of performing voice activity detection based on the streaming speech may be returned.

[0032] Step S13: In response to the detection result being semantically complete, from the current moment until the waiting basic delay, analyze whether there is new valid speech in the streaming speech, if so, return to detect whether the streaming speech is semantically complete at the current moment, if not, generate reply data.

[0033] In one implementation scenario, the method for obtaining the basic delay can refer to the aforementioned related description, which will not be repeated here. In the case where the detection result is semantically complete, you can wait for the addition of valid voice to the streaming voice from the current moment. If no valid voice is added to the streaming voice after waiting until the basic delay, you can generate reply data for responding to the streaming voice. On the contrary, if valid voice is added to the streaming voice at a certain moment before waiting until the basic delay, you can return to the aforementioned step of detecting whether the streaming voice is semantically complete at the current moment. It should be noted that, as mentioned above, before detecting whether the streaming voice is semantically complete at the current moment, you can also perform voice activity detection based on the streaming voice. In this case, in response to the addition of valid voice to the streaming voice, you can return to the step of performing voice activity detection based on the streaming voice. In addition, the specific method of generating reply data can refer to the aforementioned related description, which will not be repeated here.

[0034] In addition, as an implementation example in the actual application process, after generating and outputting the reply data, the feedback data of the speaker on the reply data can also be obtained. The feedback data represents whether the reply data unexpectedly interrupts the speaker. The feedback data can be aggregated to the server, and the server can select the speaker who has not responded to the reply data feedback unexpected interruption from the user set composed of the various speaker objects as the reference object, and then form a speed-delay mapping relationship based on the basic delay and speech speed of the reference object. Exemplarily, the basic delay of the reference object (such as the average value of the basic delay during each interaction) and the speech speed (such as the average value of the speech speed during each interaction) can be fitted to obtain a mathematical function between the basic delay and the speech speed as the speed-delay mapping relationship. On this basis, in response to the fact that the speaking object is a newly registered user, a prompt message can be output to guide the newly registered user to perform speaking interaction in a normal state, and the voice data input by the newly registered user triggered by the prompt message can be analyzed to obtain the speaking speed of the newly registered user, and based on the mapping relationship between the speaking speed and speaking speed delay of the newly registered user, the initial basic delay of the newly registered user can be obtained. After the newly registered user starts streaming voice interaction, the basic delay can be recalculated and updated through the process steps described above. For details, please refer to the aforementioned description on the calculation and update of the basic delay, which will not be repeated here. The above method, in response to the speaking object being a newly registered user, outputs a prompt message for guiding the newly registered user to speak and interact in a normal state, and analyzes the voice data input by the newly registered user triggered by the prompt message to obtain the speaking speed of the newly registered user, and then obtains the initial basic delay of the newly registered user based on the speaking speed and speaking speed-delay mapping relationship of the newly registered user, and the speaking speed-delay mapping relationship is obtained based on the basic delay and speaking speed of each reference object. The reference object is the speaking object in the user set whose reply data feedback was not unexpectedly interrupted, so the basic delay that is as suitable as possible can also be initialized for the newly registered user.

[0035] Please refer to Figure 2 , Figure 2 This is a schematic diagram of the effect of an embodiment of the streaming voice interaction method of the present application. Figure 2As shown, in one implementation example, when the current moment is t1, since it is detected as semantically complete, it is possible to analyze whether new valid speech is added in the streaming voice from the current moment t1 until the process of waiting for the basic delay, and since it is detected that no new valid speech is added during this period, the reply data "The weather is very good today" can be obtained after waiting for the generation delay; or, in another implementation example, when the current moment is t2, since it is detected as semantically incomplete, an additional delay can be added to the basic delay to obtain the target delay, and then from the current moment t2 until the process of waiting for the target delay, it is possible to analyze whether new valid speech is added in the streaming voice, and since it is detected that new valid speech is added in the streaming voice after the current moment t2, it is possible to return to the step of detecting whether the streaming voice is semantically complete at the current moment, and this cycle is repeated until the speaker says "how" and is detected as semantically complete. Of course, the above examples are only two possible examples in the actual application process, and other possible situations are not given examples one by one here.

[0036] The above scheme detects whether the streaming voice is semantically complete at the current moment, and in response to the detection result being semantically incomplete, obtains the target delay based on the basic delay plus the additional delay, and analyzes whether new valid voice is added to the streaming voice from the current moment until the target delay is reached. If so, return to detect whether the streaming voice is semantically complete at the current moment, otherwise generate reply data, and in response to the detection result being semantically complete, from the current moment until the target delay is reached, analyzes whether new valid voice is added to the streaming voice, and if so, return to detect whether the streaming voice is semantically complete at the current moment, otherwise generate reply data, and one party When it is detected that the semantics of the streaming speech are incomplete at the current moment, adding an extra delay to the basic delay compared to when the semantics are complete can extend the delay when the speaker is about to continue speaking, which helps reduce the possibility of the speaker being interrupted unexpectedly. On the other hand, regardless of whether the semantics are complete or not, as long as new valid speech is detected during the waiting process, the semantic integrity check will continue without replying temporarily, which also helps reduce the possibility of the speaker being interrupted unexpectedly. On the other hand, regardless of whether the semantics are complete or not, if no new valid speech is detected during the waiting process, a reply will be made, which can also shorten the response time of voice interaction as much as possible. Therefore, the possibility of the speaker being interrupted unexpectedly can be reduced while shortening the response time of voice interaction.

[0037] See also Figure 3 , Figure 3It is a schematic diagram of a framework of an embodiment of a streaming voice interaction device of the present application. The streaming voice interaction device 30 includes: a semantic detection module 31, a first response module 32 and a second response module 33. The semantic detection module 31 is used to detect whether the streaming voice is semantically complete at the current moment; the first response module 32 is used to respond to the detection result that the semantics are incomplete, based on the basic delay plus the additional delay, to obtain the target delay, and from the current moment to the process of waiting for the target delay, analyze whether there is a new valid voice in the streaming voice, if so, return to detect whether the streaming voice is semantically complete at the current moment, if otherwise, generate reply data; the second response module 32 is used to respond to the detection result that the semantics are complete, from the current moment to the process of waiting for the basic delay, analyze whether there is a new valid voice in the streaming voice, if so, return to detect whether the streaming voice is semantically complete at the current moment, if otherwise, generate reply data.

[0038] In the above scheme, the streaming voice interaction device 30 detects whether the streaming voice is semantically complete at the current moment. In response to the detection result that the streaming voice is semantically incomplete, the target delay is obtained based on the basic delay plus the additional delay, and from the current moment until the target delay, it is analyzed whether there is new valid voice in the streaming voice. If so, it returns to detect whether the streaming voice is semantically complete at the current moment, otherwise it generates reply data, and in response to the detection result that the streaming voice is semantically complete, from the current moment until the target delay, it is analyzed whether there is new valid voice in the streaming voice. If so, it returns to detect whether the streaming voice is semantically complete at the current moment, otherwise it generates a reply data. Data, on the one hand, when it is detected that the semantics of the streaming speech are incomplete at the current moment, adding an additional delay to the basic delay compared to when the semantics are complete can extend the delay when the speaker is about to continue speaking, which helps reduce the possibility of the speaker being interrupted unexpectedly. On the other hand, regardless of whether the semantics are complete, as long as new valid speech is detected during the waiting process, the semantic integrity check will continue without replying temporarily, which also helps reduce the possibility of the speaker being interrupted unexpectedly. On the other hand, regardless of whether the semantics are complete, if no new valid speech is detected during the waiting process, a reply will be made, which can also shorten the response time of voice interaction as much as possible. Therefore, the possibility of the speaker being interrupted unexpectedly can be reduced while shortening the response time of voice interaction.

[0039] In some disclosed embodiments, the streaming voice interaction device 30 also includes an activity detection module for performing voice activity detection based on the streaming voice before detecting whether the streaming voice is semantically complete at the current moment; the semantic detection module 31 is specifically used to execute detection of whether the streaming voice is semantically complete at the current moment in response to the streaming voice being detected as a voice end endpoint at the current moment.

[0040] In some disclosed embodiments, the streaming voice interaction device 30 further includes a loop iteration module for returning to perform voice activity detection based on the streaming voice in response to the streaming voice being detected as not being a voice end endpoint at the current moment.

[0041] In some disclosed embodiments, the first response module 32 and the second response module 33 are further specifically configured to return to voice activity detection based on the streaming voice in response to newly added valid voice in the streaming voice.

[0042] In some disclosed embodiments, the semantic detection module 31 is specifically used to detect whether the streaming speech is semantically complete at the current moment based on a first artificial intelligence model; wherein the first artificial intelligence model includes: at least one of a deep learning model and a large language model.

[0043] In some disclosed embodiments, the streaming voice interaction device 30 also includes a delay determination module for determining a basic delay based on the historical voice of the speaker to which the streaming voice belongs before the current moment; wherein the basic delay is correlated with the speech state represented by the historical voice, and the speech state includes at least the speech speed.

[0044] In some disclosed embodiments, the delay determination module includes a historical analysis submodule, which is used to perform statistics based on historical speech to obtain the speaking rate of the speaker; wherein the speaking rate is expressed as the number of words spoken per standard time length; the delay determination module includes a proportional adjustment submodule, which is used to proportionally adjust the unit delay based on the comparison result of the speaking rate and the benchmark speaking rate to obtain a reference delay; the delay determination module includes an interval constraint submodule, which is used to constrain the reference delay based on the delay interval to obtain a basic delay.

[0045] In some disclosed embodiments, the delay determination module includes a model prediction submodule, which is used to predict historical speech based on a second artificial intelligence model to obtain a basic delay; wherein the second artificial intelligence model includes at least a deep learning model, and the deep learning model is trained based on sample speech labeled with sample delay.

[0046] In some disclosed embodiments, the delay determination module includes a record analysis submodule, which is used to analyze the recorded data of streaming voice interaction on the current device to obtain the usage pattern of streaming voice interaction on the current device; the delay determination module includes a timing determination submodule, which is used to determine the update timing of the basic delay based on the usage pattern; the delay determination module includes a delay update submodule, which is used to execute the basic delay acquisition step according to the update timing to adaptively update the basic delay.

[0047] In some disclosed embodiments, the streaming voice interaction device 30 also includes a speech guidance module for outputting a prompt message for guiding the newly registered user to perform speech interaction in a normal state in response to the speaking object being a newly registered user. The streaming voice interaction device 30 also includes a speech rate analysis module for analyzing the voice data input based on the prompt message triggered by the newly registered user to obtain the speaking rate of the newly registered user. The streaming voice interaction device 30 also includes a delay initialization module for obtaining the initial basic delay of the newly registered user based on the speaking rate of the newly registered user and the speech rate-delay mapping relationship. The speech rate-delay mapping relationship is obtained based on the basic delay and speech rate of each reference object, and the reference object is a speaking object in the user set whose reply data feedback was not unexpectedly interrupted.

[0048] See also Figure 4 , Figure 4 It is a schematic diagram of the framework of an embodiment of an electronic device of the present application. The electronic device 40 includes at least a memory 41 and a processor 42 coupled to each other, the memory 41 stores at least program instructions, and the processor 42 is used to execute the program instructions to implement the steps in any of the above-mentioned streaming voice interaction method embodiments. For details, please refer to the aforementioned disclosed embodiments, which will not be repeated here. As a possible example, the electronic device 40 may include but is not limited to smart phones, tablet computers, learning machines, office books, translation machines, smart watches, servers, etc., and the specific type of the electronic device 40 is not limited here.

[0049] Specifically, the processor 42 is used to control itself and the memory 41 to implement the steps in any of the above-mentioned streaming voice interaction method embodiments. The processor 42 can also be called a CPU (Central Processing Unit). The processor 42 may be an integrated circuit chip with signal processing capabilities. The processor 42 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 42 can be implemented by an integrated circuit chip.

[0050] In the above scheme, the electronic device 40 detects whether the streaming voice is semantically complete at the current moment, and in response to the detection result being semantically incomplete, the target delay is obtained based on the basic delay plus the additional delay, and from the current moment until the target delay, analyzes whether there is new valid voice in the streaming voice, and if so, returns to detect whether the streaming voice is semantically complete at the current moment, otherwise generates reply data, and in response to the detection result being semantically complete, from the current moment until the target delay, analyzes whether there is new valid voice in the streaming voice, and if so, returns to detect whether the streaming voice is semantically complete at the current moment, and if otherwise generates reply data On the one hand, when the semantics of the streaming speech are detected to be incomplete at the current moment, adding an additional delay to the basic delay compared to when the semantics are detected to be complete can extend the delay when the speaker is about to continue speaking, which helps reduce the possibility of the speaker being interrupted unexpectedly. On the other hand, regardless of whether the semantics are complete or not, as long as new valid speech is detected during the waiting process, the semantic integrity check will continue without replying temporarily, which also helps reduce the possibility of the speaker being interrupted unexpectedly. On the other hand, regardless of whether the semantics are complete or not, if no new valid speech is detected during the waiting process, a reply will be made, which can also shorten the response time of voice interaction as much as possible. Therefore, the possibility of the speaker being interrupted unexpectedly can be reduced while shortening the response time of voice interaction.

[0051] See also Figure 5 , Figure 5 1 is a schematic diagram of a framework of an embodiment of a computer-readable storage medium 50 of the present application. The computer-readable storage medium 50 stores program instructions 51 that can be executed by a processor, and the program instructions 51 are used to implement the steps in any of the above-mentioned streaming voice interaction method embodiments.

[0052] In the above scheme, the computer-readable storage medium 50 detects whether the streaming voice is semantically complete at the current moment, and in response to the detection result being semantically incomplete, the target delay is obtained based on the basic delay plus the additional delay, and from the current moment until the target delay, analyzes whether there is new valid voice in the streaming voice, and if so, returns to detect whether the streaming voice is semantically complete at the current moment, otherwise generates reply data, and in response to the detection result being semantically complete, from the current moment until the target delay, analyzes whether there is new valid voice in the streaming voice, and if so, returns to detect whether the streaming voice is semantically complete at the current moment, and if otherwise generates a reply Data, on the one hand, when it is detected that the semantics of the streaming speech are incomplete at the current moment, adding an additional delay to the basic delay compared to when the semantics are complete can extend the delay when the speaker is about to continue speaking, which helps reduce the possibility of the speaker being interrupted unexpectedly. On the other hand, regardless of whether the semantics are complete, as long as new valid speech is detected during the waiting process, the semantic integrity check will continue without replying temporarily, which also helps reduce the possibility of the speaker being interrupted unexpectedly. On the other hand, regardless of whether the semantics are complete, if no new valid speech is detected during the waiting process, a reply will be made, which can also shorten the response time of voice interaction as much as possible. Therefore, the possibility of the speaker being interrupted unexpectedly can be reduced while shortening the response time of voice interaction.

[0053] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0054] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0055] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0056] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0057] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0058] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0059] If the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to the collection of his or her personal information; or on the device that processes personal information, the personal information processing rules are notified by obvious signs / information, and the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

Claims

1. A streaming voice interaction method, characterized in that: include: Detect whether the streaming speech is semantically complete at the current moment; In response to the detection result being semantically incomplete, a target delay is obtained based on the basic delay plus the additional delay, and from the current moment until the target delay is reached, whether valid speech is newly added to the streaming speech is analyzed, and if so, whether the detected streaming speech is semantically complete at the current moment is returned, and if not, reply data is generated; In response to the detection result being semantically complete, from the current moment until the process of waiting for the basic delay, it is analyzed whether new valid voice is added to the streaming voice. If so, it is returned whether the detected streaming voice is semantically complete at the current moment. If not, reply data is generated.

2. The method according to claim 1, characterized in that Before detecting whether the streaming speech is semantically complete at the current moment, the method further includes: performing voice activity detection based on the streaming voice; The detecting whether the streaming speech is semantically complete at the current moment includes: In response to the streaming voice being detected as a voice end point at the current moment, the detecting of whether the streaming voice is semantically complete at the current moment is performed.

3. The method according to claim 2, characterized in that The method further comprises: In response to the streaming voice being detected as not being the voice end endpoint at the current moment, returning to perform the voice activity detection based on the streaming voice; And / or, the method further comprises: In response to valid voice being newly added to the streaming voice, returning to the voice activity detection based on the streaming voice.

4. The method according to claim 1, characterized in that The detecting whether the streaming speech is semantically complete at the current moment includes: Based on a first artificial intelligence model, it is detected whether the streaming speech is semantically complete at the current moment; wherein the first artificial intelligence model includes: at least one of a deep learning model and a large language model.

5. The method according to claim 1, characterized in that The step of obtaining the basic delay includes: The basic delay is determined based on the historical speech of the speaker to which the streaming speech belongs before the current moment; wherein the basic delay is correlated with the speech state represented by the historical speech, and the speech state at least includes the speech speed.

6. The method according to claim 5, characterized in that The determining the basic delay based on the historical speech of the speaker to which the streaming speech belongs before the current moment includes: Based on the historical speech, statistics are performed to obtain the speaking speed of the speaking object; wherein the speaking speed is expressed as the number of words per standard time length; Based on the comparison result between the speech speed and the target speech speed, the unit delay is proportionally adjusted to obtain a reference delay; The reference delay is constrained based on the delay interval to obtain the basic delay.

7. The method according to claim 5, characterized in that The determining the basic delay based on the historical speech of the speaker to which the streaming speech belongs before the current moment includes: The historical speech is predicted based on a second artificial intelligence model to obtain the basic delay; wherein the second artificial intelligence model at least includes a deep learning model, and the deep learning model is trained based on sample speech marked with sample delays.

8. The method according to claim 5, characterized in that The method further comprises: Analyze the recorded data of the streaming voice interaction on the current device to obtain the usage rules of the streaming voice interaction on the current device; Based on the usage pattern, determining an update timing of the basic delay; According to the update timing, the step of acquiring the basic delay is performed to adaptively update the basic delay.

9. The method according to claim 1, characterized in that: The method further comprises: In response to the speech object being a newly registered user, outputting a prompt message for guiding the newly registered user to perform speech interaction in a normal state, and analyzing voice data input by the newly registered user triggered by the prompt message to obtain the speech speed of the newly registered user; Based on the mapping relationship between the speaking speed and speaking speed delay of the newly registered user, the initial basic delay of the newly registered user is obtained; wherein the speaking speed delay mapping relationship is obtained based on the basic delay and speaking speed of each reference object, and the reference object is a speaking object in the user set whose response to the reply data feedback was not unexpectedly interrupted.

10. A streaming voice interaction device, characterized in that: include: A semantic detection module, used to detect whether the streaming speech is semantically complete at the current moment; A first response module is used for, in response to the detection result that the semantics are incomplete, obtaining a target delay based on the basic delay plus the additional delay, and analyzing whether a valid voice is added to the streaming voice from the current moment until the target delay is waited, and if so, returning whether the detected streaming voice is semantically complete at the current moment, and generating reply data if otherwise; The second response module is used to respond to the detection result being semantically complete, and analyze whether there is new valid voice in the streaming voice from the current moment to the process of waiting for the basic delay. If so, return whether the detected streaming voice is semantically complete at the current moment, and if not, generate reply data.

11. An electronic device, characterized in that: The method comprises at least a memory and a processor coupled to each other, wherein the memory at least stores program instructions, and the processor is used to execute the program instructions to implement the streaming voice interaction method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: Program instructions that can be executed by a processor are stored, and the program instructions are used to implement the streaming voice interaction method described in any one of claims 1 to 9.

Images