Streaming voice interaction method, related device, equipment and storage medium

Abstract

The invention discloses a streaming voice interaction method, a related device, equipment and a storage medium, and the method comprises the steps: carrying out the end point detection of streaming voice, responding to the detection of a starting end point of the streaming voice, executing the sliding window operation of the streaming voice, obtaining a voice frame, carrying out the feature extraction based on the voice frame, and obtaining a voice frame; voice features of the voice frames are obtained; inputting the speech features of the speech frames to a speech recognition system for recognizing streaming speech to obtain recognition results of the speech frames, and performing classification prediction based on the coding features of the speech frames to obtain classification results of the speech frames; in response to the classification result representing semantic ending, based on the recognition result of each voice frame from the starting endpoint to the ending endpoint, obtaining a recognition text, and processing at least based on the recognition text by the intelligent dialogue model to generate a reply text; and continuously returning to execute the sliding window operation in response to the classification result representing that the semantics is not finished. According to the scheme, the timeliness of streaming voice interaction can be improved.

Description

Technical Field

[0001] The present application relates to the field of voice interaction technology, and in particular to a streaming voice interaction method and related devices, equipment and storage media. Background Art

[0002] Streaming voice interaction is widely used in many scenarios such as customer service and business office. Unlike non-streaming voice interaction, which requires voice interaction after voice input is completed, streaming voice interaction requires timely voice interaction during the voice input process.

[0003] Existing voice interaction technologies mainly focus on non-streaming voice and are difficult to apply to streaming voice. In addition, although there are a few streaming voice interaction technologies, defects such as inappropriate voice interaction in the streaming voice process (e.g., interaction intervention too early, interaction intervention too late) have seriously restricted the promotion and application of existing streaming voice interaction technologies. In view of this, how to improve the timeliness of streaming voice interaction has become an urgent problem to be solved. Summary of the invention

[0004] The main technical problem solved by the present application is to provide a streaming voice interaction method and related devices, equipment and storage media, which can improve the timeliness of streaming voice interaction.

[0005] In order to solve the above technical problems, the first aspect of the present application provides a streaming voice interaction method, including: performing endpoint detection on the streaming voice, and in response to detecting the starting endpoint of the streaming voice, performing a sliding window operation on the streaming voice from the starting endpoint to obtain a voice frame, and performing feature extraction based on the voice frame to obtain the voice features of the voice frame; inputting the voice features of the voice frame into a voice recognition system for identifying the streaming voice to obtain a recognition result of the voice frame, and performing classification prediction based on the encoding features of the voice frame to obtain a classification result of the voice frame; wherein the encoding features are extracted by the encoding network in the intelligent dialogue model after the voice features are input into the intelligent dialogue model, and the classification result includes whether the voice frame represents an ending endpoint of a semantic end; in response to the classification result representing the semantic end, based on the recognition results of each voice frame from the starting endpoint to the ending endpoint, a recognition text is obtained, and at least based on the recognition text, the intelligent dialogue model is processed to generate a reply text; in response to the classification result representing that the semantics have not ended, continue to return to perform the sliding window operation.

[0006] In order to solve the above technical problems, the second aspect of the present application provides a streaming voice interaction device, including: a preparatory processing module, a recognition and classification module, a first response module and a second response module, the preparatory processing module is used to perform endpoint detection on the streaming voice, and in response to detecting the starting endpoint of the streaming voice, perform a sliding window operation on the streaming voice from the starting endpoint to obtain a voice frame, and perform feature extraction based on the voice frame to obtain the voice feature of the voice frame; the recognition and classification module is used to input the voice feature of the voice frame to a voice recognition system for recognizing the streaming voice, obtain the recognition result of the voice frame, and perform classification prediction based on the encoding feature of the voice frame to obtain the classification result of the voice frame; wherein the encoding feature is extracted by the encoding network in the intelligent dialogue model after the voice feature is input to the intelligent dialogue model, and the classification result includes whether the voice frame represents the ending endpoint of the semantic end; the first response module is used to represent the semantic end in response to the classification result, obtain the recognition text based on the recognition results of each voice frame from the starting endpoint to the ending endpoint, and generate a reply text based on at least the recognition text by processing by the intelligent dialogue model; the second response module is used to continue to return to perform the sliding window operation in response to the classification result representing that the semantics are not ended.

[0007] In order to solve the above technical problems, the third aspect of the present application provides an electronic device, which at least includes a memory and a processor coupled to each other, wherein the memory at least stores program instructions, and the processor is used to execute the program instructions to implement the streaming voice interaction method in the above first aspect.

[0008] In order to solve the above technical problems, the fourth aspect of the present application provides a computer-readable storage medium, which stores program instructions that can be executed by a processor, and the program instructions are used to implement the streaming voice interaction method of the above first aspect.

[0009] The above scheme performs endpoint detection on the streaming speech, and in response to detecting the starting endpoint of the streaming speech, performs a sliding window operation on the streaming speech from the starting endpoint to obtain a speech frame, and performs feature extraction based on the speech frame to obtain the speech features of the speech frame, and then inputs the speech features of the speech frame into a speech recognition system for identifying the streaming speech to obtain the recognition result of the speech frame, and performs classification prediction based on the encoding features of the speech frame to obtain the classification result of the speech frame, and the encoding features are extracted by the encoding network in the intelligent dialogue model after the speech features are input into the intelligent dialogue model, and the classification result includes whether the speech frame represents the ending endpoint of the semantic end, based on this, in response to the classification result representing the semantic end, based on the recognition results of each speech frame from the starting endpoint to the ending endpoint, a recognition text is obtained, and at least based on the recognition text, it is processed by the intelligent dialogue model to generate The reply text is obtained, and in response to the classification result representing that the semantics have not ended, the sliding window operation is continued to be performed. Therefore, on the one hand, in the process of streaming voice, the sliding window operation can be continuously performed after the start endpoint is detected to obtain the voice frame, and the relevant processing is performed on each voice frame in turn until the voice frame representing the semantic end is detected and the reply text is generated again, which can realize the closed loop of the process of streaming voice interaction. On the other hand, the voice features of the voice frame are sent to the voice recognition system for voice recognition in one way, and sent to the intelligent dialogue model in another way to be encoded by the encoding network in the intelligent dialogue model first and then classified and predicted whether the semantics have ended. With the help of the semantic understanding ability of the intelligent dialogue model, it is possible to judge as accurately as possible whether the voice frame has ended to determine whether to intervene in the interaction, and avoid intervening in the interaction too early or too late as much as possible, which helps to improve the timeliness of streaming voice interaction. Therefore, the timeliness of streaming voice interaction can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 This is a flow chart of an embodiment of the streaming voice interaction method of the present application; Figure 2 This is a process diagram of an embodiment of the streaming voice interaction method of the present application; Figure 3 It is a schematic diagram of the framework of an embodiment of the streaming voice interaction device of the present application; Figure 4 It is a schematic diagram of the framework of an embodiment of the electronic device of the present application; Figure 5 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0011] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0012] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0013] The terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the fragment " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, "many" in this article means two or more than two.

[0014] See also Figure 1 , Figure 1 This is a flow chart of an embodiment of the streaming voice interaction method of the present application. Specifically, it may include the following steps: Step S11: performing endpoint detection on the streaming speech, and in response to detecting the starting endpoint of the streaming speech, performing a sliding window operation on the streaming speech starting from the starting endpoint to obtain a speech frame, and performing feature extraction based on the speech frame to obtain speech features of the speech frame.

[0015] In the disclosed embodiments, streaming voice refers to an audio stream that is input or collected in real time, and the audio stream is processed during the real-time input or real-time collection process to perform voice interaction. Taking a business office scenario as an example, the real-time speech of a speaker in a meeting is streaming voice, such as "I think this project (short pause L seconds) is still under discussion"; or, taking a daily life scenario as an example, the real-time voice input of a user during a human-computer dialogue is streaming voice, such as "(short pause M seconds) We have arrived in City A now (long pause N seconds) Please report the local weather conditions tomorrow to the children in the tone of a kindergarten teacher". Of course, the above example is only one possible example in the actual application process, and other possible situations will not be given examples one by one here.

[0016] In an implementation scenario, the specific process of endpoint detection (Voice Activity Detection, VAD) can refer to the technical details of VAD, which will not be repeated here. Take the streaming voice "(short pause M seconds) We have now arrived in City A (long pause N seconds) Please report the local weather conditions tomorrow to the children in the tone of a kindergarten teacher" as an example. When the current voice progress is in the "short pause M seconds", since no valid voice is detected through endpoint detection, it is necessary to continue to perform endpoint detection on the streaming voice until the current voice progress is at the beginning of the pronunciation of "I", and the starting endpoint of the streaming voice can be detected. Of course, the above example is only a possible example in the actual application process, and other possible situations will not be given one by one here.

[0017] In one implementation scenario, the window length and window shift of the sliding window operation can be set according to application needs. For example, when more future horizons are needed, the window shift can be set appropriately smaller, or, when less future horizons are needed, the window shift can be set appropriately larger. Exemplarily, taking the window length as 200ms as an example, the window shift can be set to 120ms, that is, the future horizon is 80ms. In this case, taking 0s of the streaming voice as the starting endpoint, the first voice frame is 0~200ms of the streaming voice, and the second voice frame is 120~320ms of the streaming voice, that is, the last 80ms of the first voice frame is the future horizon. Of course, the above example is only a possible example of the sliding window operation in the actual application process, and other possible situations are not given one by one here. It should be noted that after determining the starting endpoint, the sliding window operation can be continuously performed on the streaming voice according to the window length and window shift to obtain voice frames one by one, and subsequent operations are performed on each voice frame in turn. For details, please refer to the subsequent related description, which will not be repeated here.

[0018] In one implementation scenario, after the speech frame is obtained through the sliding window operation, the speech frame can be feature extracted to obtain the speech features of the speech frame. As a possible example, acoustic features such as MFCC, FBank, etc. can be extracted from the speech frame as the speech features of the speech frame; or, as another possible example, the speech frame can also be feature extracted based on the speech coding network to obtain the speech features of the speech frame. The coding network can include but is not limited to convolutional neural networks, long short-term memory networks, recurrent neural networks, etc., and the specific structure of the coding network is not limited here. Of course, the above examples are only a few possible examples of extracting speech features, and other possible ways of extracting speech features are not limited thereto, and no examples are given one by one here.

[0019] Step S12: inputting speech features of the speech frame into a speech recognition system for recognizing streaming speech to obtain a recognition result of the speech frame, and performing classification prediction based on the coding features of the speech frame to obtain a classification result of the speech frame.

[0020] In an implementation scenario, after the speech features of the speech frame are extracted, one of the processing branches is sent to a speech recognition system for recognizing streaming speech, so that the recognition result of the speech frame can be obtained. It should be noted that the recognition result may include a number of characters, or the recognition result may include a number of phonemes, and the specific content of the recognition result is not limited here. In addition, the speech recognition system may include but is not limited to wenet, etc., and the specific structure of the speech recognition system is not limited here.

[0021] In one implementation scenario, after the speech features of the speech frame are extracted, another processing branch is sent to the intelligent dialogue model for processing by the intelligent dialogue model. It should be noted that the intelligent dialogue model may include a coding network, and after the speech features are encoded by the coding network, coding features can be obtained. On this basis, classification prediction can be performed based on the coding features of the speech frame to obtain a classification result of the speech frame. It should be noted that the classification result may include whether the speech frame represents the end endpoint of the semantic end, and the end endpoint is the end time of the speech frame representing the semantic end. In addition, the intelligent dialogue model may include but is not limited to a large language model, and the coding network may include but is not limited to Transformer, etc., and the specific structure of the intelligent dialogue model and the coding network is not limited here.

[0022] In a specific implementation scenario, the classification prediction can be performed by a neural network model. Exemplarily, the neural network model may include but is not limited to network layers such as convolutional layers and fully connected layers, and the specific structure of the neural network model is not limited here. After the encoding feature is input into the neural network model, the speech frame can be classified into a probability value representing the end of semantics and a probability value representing the end of semantics. On this basis, it can be determined whether the speech frame represents the end of semantics based on the larger probability value. If the probability value representing the end of semantics is greater than the probability value representing the end of semantics, it can be determined that the speech frame represents the end of semantics. On the contrary, if the probability value representing the end of semantics is less than the probability value representing the end of semantics, it can be determined that the speech frame represents the end of semantics; or, it can be determined whether the speech frame represents the end of semantics by taking a probability value greater than a probability threshold. If the probability value representing the end of semantics is greater than the probability threshold, it can be determined that the speech frame represents the end of semantics. On the contrary, if the probability value representing the end of semantics is greater than the probability threshold, it can be determined that the speech frame represents the end of semantics. Of course, the above examples are only several possible examples of determining whether a speech frame represents a semantic end, and do not limit other possible ways of determining whether a speech frame represents a semantic end. Other possible ways will not be given examples one by one here.

[0023] In a specific implementation scenario, for the sake of easy understanding, take the streaming speech "(short pause for M seconds) We have now arrived in City A (long pause for N seconds). Please announce the local weather tomorrow to the children in the tone of a kindergarten teacher." as an example. When the speech frame contains "la" in the above streaming speech (specifically, it can be the last phoneme containing "la"), since the meaning of "arriving in City A" has been completely expressed, the speech frame represents the end of semantics; or, take the streaming speech "I think this project (short pause for L seconds) remains to be discussed." as an example. When the speech frame contains "ne" in the above streaming speech (specifically, it can be the last phoneme containing "ne"), since the meaning of "project attitude" has not been completely expressed, the speech frame does not represent the end of semantics. Of course, the above examples are only specific examples given for the sake of easy understanding of the specific meaning of the end of semantics, and do not limit the streaming speech in the actual application scenario. No further examples will be given here.

[0024] Step S13: In response to the classification result representing the end of semantics, obtain the recognized text based on the recognition results of each speech frame from the start endpoint to the end endpoint, and at least process the recognized text by the intelligent dialogue model to generate a reply text.

[0025] In an implementation scenario, when the classification result represents the end of semantics, a control signal representing the stop of recognition can be transmitted to the speech recognition system, so that the speech recognition system pauses speech recognition after receiving the control signal until a new round of interaction is started and then restarts speech recognition. In the above manner, when the classification result represents the end of semantics, transmitting a control signal representing the stop of recognition to the speech recognition system can尽可能地减轻语音识别系统的处理负荷,并使得语音识别、语音交互等整体步调一致。

[0026] It should be noted that there is an unclear expression in the original text of . I have tried my best to translate it according to the context, but it may need to be adjusted according to the accurate meaning.In an implementation scenario, when the classification result represents a semantic end, the recognition results of each speech frame from the starting endpoint to the ending endpoint may be obtained first, and the above recognition results may be combined to obtain a recognized text. Exemplarily, still taking the streaming voice "(short pause M seconds) We have now arrived in City A (long pause N seconds) Please report the local weather conditions tomorrow to the children in the tone of a kindergarten teacher" as an example, as mentioned above, when the voice frame contains "la" in the above streaming voice (specifically, it can be the last phoneme containing "la"), the voice frame representation semantics ends, and the recognition results of each voice frame from the voice frame containing "I" (specifically, it can be the first phoneme containing "I") to the voice frame containing "la" (specifically, it can be the last phoneme containing "la") can be obtained. Taking the recognition result containing phonemes as an example, the recognition results of these voice frames can be listed in sequence as {wo men xian zai yi jing di daA shi la}, based on which the recognized text "We have now arrived in City A" can be obtained; or, taking the recognition result containing characters as an example, the recognition results of these voice frames can be listed in sequence as {we have now arrived in City A}, based on which the recognized text "we have now arrived in City A" can be obtained. Of course, the above examples are only a few possible examples in actual application, and other possible methods will not be given one by one here.

[0027] In one implementation scenario, the recognized text can be directly input into the intelligent dialogue model to obtain the output text of the intelligent dialogue model as the reply text. Exemplarily, still taking the intelligent dialogue model as a large language model, the large language model can generate a reply text for the recognized text based on the system instructions configured therein. Still taking the aforementioned recognized text "We have now arrived in City A" as an example, if the configuration information obtained by the system instructions includes "cute", "energetic", etc., the large language model can generate a reply text "That's great!" etc. based on the configuration information. Of course, the above example is only one possible example in the actual application process, and other possible situations will not be given one by one here.

[0028] In another implementation scenario, different from the aforementioned implementation, after obtaining the recognition text and before generating the reply text, a semantic analysis may be performed based on the recognition text to obtain an analysis result, and the analysis result indicates whether a query search is required before replying to the recognition text. In response to the analysis result indicating that a query search is required first, a query search may be performed based on the recognition text to obtain a search result, so that the intelligent dialogue model may process the recognition text and the search result to obtain a reply text. In the above method, after obtaining the recognition text and before replying to the recognition text, a semantic analysis is first performed on the recognition text to determine whether a query search is required before replying to the recognition text, and if a query search is required first, a query search is performed based on the recognition text to obtain a search result, so that the recognition text and the search result are combined and processed by the intelligent dialogue model to generate a reply text, which helps to improve the accuracy of the interaction.

[0029] In a specific implementation scenario, the semantic analysis of the recognized text can be specifically performed by a semantic analysis model. It should be noted that the semantic analysis model may include but is not limited to Transformer, etc., and the network structure of the semantic analysis model is not limited here. After the recognized text is semantically analyzed by the semantic analysis model, the analysis result can be obtained. Exemplarily, the semantic analysis model can be a binary classification model, and the analysis result can include: the probability value that the reply recognition text needs to be searched and retrieved first and the probability value that the reply recognition text does not need to be searched and retrieved first. Based on this probability value, it can be determined whether the reply recognition text needs to be searched and retrieved first. For details, please refer to the aforementioned description of determining whether the speech frame represents the semantic end based on the probability value, which will not be repeated here. Still taking the streaming speech "(short pause M seconds) We have arrived in City A now (long pause N seconds) Please report the local weather conditions tomorrow to the children in the tone of a kindergarten teacher" as an example, the recognized text "We have arrived in City A now" can be represented without first undergoing a query and retrieval analysis result after semantic analysis, while the second half of the recognized text "Please report the local weather conditions tomorrow to the children in the tone of a kindergarten teacher" needs to be combined with tomorrow's weather before replying after semantic analysis, so its analysis result representation needs to be first subjected to a query and retrieval. Of course, the above example is only one possible example in the actual application process, and other possible situations will not be given one by one here.

[0030] In a specific implementation scenario, as a possible example, before performing semantic analysis, the recognized text can be semantically regularized based on the context of the recognized text in the interaction process with the streaming voice to obtain a regularized text. Still taking the aforementioned recognized text "Please report the local weather conditions tomorrow to the children in the tone of a kindergarten teacher" as an example, combined with the context "We have arrived in City A now", the above recognized text can be semantically regularized to obtain a regularized text "Please report the weather conditions in City A tomorrow to the children in the tone of a kindergarten teacher", so that the recognized text (i.e., regularized text) after semantic regularization is semantically clear and unambiguous. On this basis, semantic analysis can be performed based on the regular text to obtain analysis results. For example, if semantic analysis is performed on the regular text "Please report the weather conditions of City A tomorrow to the children in the tone of a kindergarten teacher", it can be determined that a query search is required first, and a query search is performed based on the regular text to obtain a search result. For example, a query search is performed based on the regular text "Please report the weather conditions of City A tomorrow to the children in the tone of a kindergarten teacher", and a search result containing the weather conditions of City A tomorrow is obtained, such as "The weather is sunny". On this basis, the regular text "Please report the weather conditions of City A tomorrow to the children in the tone of a kindergarten teacher" and the search result "The weather is sunny" can be processed by the intelligent dialogue model to generate a reply text, such as "Children, tomorrow is a sunny day!". Of course, the above example is only a possible example in the actual application process, and other possible situations are not given one by one here. The above method first recognizes the context of the text in the process of interaction with the streaming voice, semantically regularizes the recognized text to obtain regularized text, then performs semantic analysis based on the regularized text to obtain analysis results, and performs query retrieval based on the regularized text to obtain retrieval results. The regularized text and the retrieval results are processed by an intelligent dialogue model to generate a reply text, which can improve the accuracy and pertinence of reply generation.

[0031] In a specific implementation scenario, after obtaining the search results and before generating the reply text, the padding text generated by the intelligent dialogue model can be obtained and output first to relieve the anxiety of waiting. Still taking the aforementioned recognition text "Please broadcast the local weather conditions tomorrow to the children in the tone of a kindergarten teacher" as an example, after obtaining the search result "The weather is sunny", the padding text can be generated by the intelligent dialogue model: "Querying, please wait" and so on, and other possible situations are not given one by one here. As a special example, in the case where the intelligent dialogue model adopts a large language model, a prompt instruction can be specifically constructed based on the recognition text (or regular text), and the prompt instruction is used to instruct the large language model to refer to the recognition text (or regular text) to generate a padding text that matches the style of the reply text. Exemplarily, still taking the aforementioned recognition text "Please broadcast the local weather conditions tomorrow to the children in the tone of a kindergarten teacher" as an example, a padding text that matches the style of the aforementioned reply text can be generated: "Well, let me check it carefully!". It should be noted that the padding text can be output in text form, or the synthesized speech of the padding text can be obtained by speech synthesis first, and then the synthesized speech of the padding text can be output in speech form. In the above method, after obtaining the search results and before generating the reply text, the padding text generated by the intelligent dialogue model is first obtained and output, which can help alleviate waiting anxiety.

[0032] In a specific implementation scenario, different from the aforementioned situation, in response to the analysis result representation, there is no need to first go through a query retrieval, and the intelligent dialogue model can be used to process the recognized text to generate a reply text. Still taking the aforementioned recognized text "We have now arrived in City A" as an example, it can be determined through semantic analysis that the reply to the recognized text does not need to go through a query retrieval first, so it can be directly processed based on the intelligent dialogue model to generate a reply text, such as "That's really great!". It should be noted that, similar to the aforementioned example, the reply text can be output in text form or in voice form, which is not limited here. In addition, the above example is only a possible example in the actual application process, and other possible situations are not limited thereto, and other situations are not given one by one here. In the above manner, in response to the analysis result representation, there is no need to first go through a query retrieval, and the intelligent dialogue model is used to process the recognized text to generate a reply text, which can improve the efficiency of voice interaction.

[0033] In another implementation scenario, as mentioned above, the intelligent dialogue model can be a large language model. In this case, when the classification result represents the end of semantics, before generating the reply text, it is also possible to perform intent recognition based on the encoding features of each speech frame from the starting endpoint to the ending endpoint to determine whether there is a control intention regarding the voice interaction method, and in response to the existence of the control intention, update the system instructions of the intelligent dialogue model based on the control intention, and then generate the reply text by processing the intelligent dialogue model after updating the system instructions based on at least the recognition text. The above method determines whether there is a control intention regarding the voice interaction method through intent recognition, and if there is a control intention, updates the system instructions of the intelligent dialogue model accordingly, so as to generate the reply text based on the intelligent dialogue model after updating the system instructions, which helps to improve the voice interaction quality and interaction satisfaction.

[0034] In a specific implementation scenario, still taking the aforementioned streaming voice "(short pause M seconds) We have now arrived in City A (long pause N seconds) Please tell the children about tomorrow's local weather conditions in the tone of a kindergarten teacher" as an example, when the voice frame contains "la" (specifically, it can be the last phoneme of "la"), it can be considered that the voice frame indicates the end of the semantics, and then the start endpoint of the aforementioned detection of the streaming voice can be returned. When the voice frame contains "please" (specifically, it can be the first phoneme of "please"), it can be considered that the start endpoint is detected, and when the voice frame contains "bar" (specifically, it can be the last phoneme of "bar"), it can be considered that the voice frame indicates the end of the semantics. At this time, based on the encoding features from the above-mentioned voice frame containing "please" to the voice frame containing "bar", it can be determined whether there is a control intention about the voice interaction method. It should be noted that the voice interaction method includes but is not limited to: the tone of voice interaction, the tone of voice interaction, the language of voice interaction, etc., and the specific connotations of the voice interaction method are not given one by one here. In the above example, there is a control intention about the voice interaction method "in the tone of a kindergarten teacher". Of course, the above example is only one possible example in actual application, and other possible situations will not be given one by one here.

[0035] In a specific implementation scenario, as mentioned above, the system instructions of the intelligent dialogue model can contain a lot of configuration information. If it is determined that there is a control intention, the configuration information in the system instructions can be updated based on the control intention. Exemplarily, in the above example, the configuration information related to the tone of voice in the system instructions can be updated to "kindergarten teacher". On this basis, the intelligent dialogue model after updating the system instructions can process the recognition text "Please report the local weather conditions tomorrow to the children in the tone of a kindergarten teacher" to generate a reply text, such as "Children, tomorrow morning in City A, the sun will rise from the east with a smile and give us a big, warm hug!". Of course, the above example is only a possible example in the actual application process, and other possible situations will not be given one by one here.

[0036] In one implementation scenario, when the classification result represents the end of semantics, after generating a reply text, control parameters for speech synthesis can be generated based on the reply text, and then speech synthesis is performed on the reply text based on the control parameters to obtain a reply speech. It should be noted that the control parameters may include but are not limited to pronunciation, pauses, etc., and the specific connotations of the control parameters are not given one by one here. In addition, the specific process of speech synthesis can refer to the technical details of speech synthesis, and examples are not given one by one here. The above method, after generating a reply text, further generates control parameters for speech synthesis based on the reply text, and then speech synthesis is performed on the reply text based on the control parameters to obtain a reply speech, which can enhance the fun and immersion of voice interaction.

[0037] In an implementation scenario, when the classification result represents the semantic end, after the response text is generated, the end endpoint can be returned from the end endpoint to perform the endpoint detection step for the streaming voice to start a new round of voice interaction. Still taking the aforementioned streaming voice "(short pause M seconds) We have now arrived in City A (long pause N seconds) Please broadcast the local weather conditions tomorrow to the children in the tone of a kindergarten teacher" as an example, when the voice frame contains "la" (specifically, it can be the last phoneme of "la"), it can be considered that the voice frame represents the semantic end, then after generating a response text (such as "It's great!") for the recognized text "We have now arrived in City A", it can be returned from the above end endpoint to the above start endpoint of detecting the streaming voice, and when the voice frame contains "please" (specifically, it can be the first phoneme of "please"), it can be considered that the start endpoint is detected, and so on and so forth, and no examples are given here one by one. In the above method, when the classification result represents the semantic end, after the reply text is generated, starting from the end endpoint, return to the step of performing endpoint detection on the streaming voice to start a new round of voice interaction, so that voice interaction can be automatically and timely performed during the input or collection process of streaming voice.

[0038] Step S14: In response to the classification result representation semantics not being completed, continue to return to perform the sliding window operation.

[0039] In one implementation scenario, when the semantic representation of the classification result has not ended, the sliding window operation can continue to be performed on the streaming speech to perform a series of operations such as feature extraction, speech recognition, and classification prediction on the continuously generated speech frames until it is detected that the semantic representation of the speech frame has ended. For details, please refer to the above-mentioned related description, which will not be repeated here.

[0040] In an implementation scenario, as a specific example, please refer to Figure 2 , Figure 2 1 is a process diagram of an embodiment of the streaming voice interaction method of the present application. Figure 2As shown, the streaming speech is firstly subjected to endpoint detection. When the starting endpoint of the streaming speech is detected, a sliding window operation can be performed on the streaming speech starting from the starting endpoint of the streaming speech to continuously obtain speech frames. The speech frames are firstly subjected to feature extraction to obtain speech features of the speech frames. The speech features of the speech frames are processed separately in two ways. One way is sent to a speech recognition system for identifying the streaming speech to obtain a recognition result, and the other way is sent to an intelligent dialogue model. The encoding features obtained by encoding the speech features by the encoding network in the intelligent dialogue model will be classified and predicted to determine whether the speech frame represents a semantic end. If it represents a semantic end, a control signal representing the stop of recognition will be transmitted to the speech recognition system to control the speech recognition system to stop speech recognition until a new round of speech interaction is started and then the speech recognition is restarted. When determining that the speech frame represents the semantic end, on the one hand, the recognition results of each speech frame from the starting endpoint to the semantic end can be combined to obtain the recognition text, and semantic regularization can be performed in combination with the context to obtain the regular text, and then the regular text can be semantically analyzed to obtain the analysis result, and the analysis result represents whether the reply to the regular text needs to be searched first. If no query retrieval is required, the query retrieval step can be skipped, and the intelligent dialogue model generates the reply. If it is necessary to search first, the regular text can be searched in the database, search engine, etc. to obtain the search result. At this time, the intelligent dialogue model can first generate the padding text to relieve the waiting anxiety, and then the search results and the regular text are handed over to the intelligent dialogue model for reply generation; on the other hand, the encoding features of each speech frame from the starting endpoint to the semantic end can be combined to perform intention recognition to determine whether there is a control intention about the voice interaction method. If so, the intelligent dialogue model first updates the system instructions of the intelligent dialogue model based on the control intention before generating a reply, and then the intelligent dialogue model after updating the system instructions generates a reply. After obtaining the reply text through the aforementioned process steps, speech synthesis can be performed based on the reply text to obtain the reply speech, thus completing a round of voice interaction, and then returning to the aforementioned step of performing endpoint detection on the streaming speech to start a new round of voice interaction. In addition, if the speech frame representation semantics has not ended, the sliding window operation can continue to be performed on the streaming speech and the subsequent process of the sliding window operation can be executed until the speech frame representation semantics ends.

[0041] The above scheme performs endpoint detection on the streaming speech, and in response to detecting the starting endpoint of the streaming speech, performs a sliding window operation on the streaming speech from the starting endpoint to obtain a speech frame, and performs feature extraction based on the speech frame to obtain the speech features of the speech frame, and then inputs the speech features of the speech frame into a speech recognition system for identifying the streaming speech to obtain the recognition result of the speech frame, and performs classification prediction based on the encoding features of the speech frame to obtain the classification result of the speech frame, and the encoding features are extracted by the encoding network in the intelligent dialogue model after the speech features are input into the intelligent dialogue model, and the classification result includes whether the speech frame represents the ending endpoint of the semantic end, based on this, in response to the classification result representing the semantic end, based on the recognition results of each speech frame from the starting endpoint to the ending endpoint, a recognition text is obtained, and at least based on the recognition text, it is processed by the intelligent dialogue model to generate The reply text is obtained, and in response to the classification result representing that the semantics have not ended, the sliding window operation is continued to be performed. Therefore, on the one hand, in the process of streaming voice, the sliding window operation can be continuously performed after the start endpoint is detected to obtain the voice frame, and the relevant processing is performed on each voice frame in turn until the voice frame representing the semantic end is detected and the reply text is generated again, which can realize the closed loop of the process of streaming voice interaction. On the other hand, the voice features of the voice frame are sent to the voice recognition system for voice recognition in one way, and sent to the intelligent dialogue model in another way to be encoded by the encoding network in the intelligent dialogue model first and then classified and predicted whether the semantics have ended. With the help of the semantic understanding ability of the intelligent dialogue model, it is possible to judge as accurately as possible whether the voice frame has ended to determine whether to intervene in the interaction, and avoid intervening in the interaction too early or too late as much as possible, which helps to improve the timeliness of streaming voice interaction. Therefore, the timeliness of streaming voice interaction can be improved.

[0042] See also Figure 3 , Figure 3The framework diagram of an embodiment of the streaming voice interaction device of the present application is shown in FIG. The streaming voice interaction device 30 includes: a preparatory processing module 31, a recognition and classification module 32, a first response module 33, and a second response module 34. The preparatory processing module 31 is used to perform endpoint detection on the streaming voice, and in response to detecting the starting endpoint of the streaming voice, performs a sliding window operation on the streaming voice from the starting endpoint to obtain a voice frame, and performs feature extraction based on the voice frame to obtain the voice features of the voice frame; the recognition and classification module 32 is used to input the voice features of the voice frame into a voice recognition system for recognizing streaming voice, obtain the recognition result of the voice frame, and perform feature extraction based on the encoding features of the voice frame. Classification prediction is performed to obtain the classification result of the speech frame; wherein, the coding feature is extracted by the coding network in the intelligent dialogue model after being input into the intelligent dialogue model by the speech feature, and the classification result includes whether the speech frame represents the end endpoint of the semantic end; the first response module 33 is used to respond to the classification result representing the semantic end, obtain the recognition text based on the recognition results of each speech frame from the starting endpoint to the ending endpoint, and generate a reply text based on at least the recognition text processed by the intelligent dialogue model; the second response module 34 is used to respond to the classification result representing that the semantics have not ended, and continue to return to perform the sliding window operation.

[0043] In the above scheme, the streaming voice interaction device 30 performs endpoint detection on the streaming voice, and in response to detecting the starting endpoint of the streaming voice, performs a sliding window operation on the streaming voice from the starting endpoint to obtain a voice frame, and extracts features based on the voice frame to obtain voice features of the voice frame, and then inputs the voice features of the voice frame into a voice recognition system for identifying the streaming voice to obtain a recognition result of the voice frame, and performs classification prediction based on the coding features of the voice frame to obtain a classification result of the voice frame, and the coding features are extracted by the coding network in the intelligent dialogue model after the voice features are input into the intelligent dialogue model, and the classification result includes whether the voice frame represents an ending endpoint of a semantic end, based on this, in response to the classification result representing the semantic end, based on the recognition results of each voice frame from the starting endpoint to the ending endpoint, a recognized text is obtained, and at least based on the recognized text, the intelligent dialogue model performs classification prediction based on the coding features of the voice frame to obtain a classification result. The processing is performed to generate a reply text, and in response to the classification result indicating that the semantics have not ended, the sliding window operation is continued to be performed. Therefore, on the one hand, in the process of streaming voice, the sliding window operation can be continuously performed after the start endpoint is detected to obtain a voice frame, and each voice frame is processed in turn until the voice frame indicating the semantics has ended is detected, and then the reply text is generated, which can realize the closed loop of the process of streaming voice interaction. On the other hand, the voice features of the voice frame are sent to the voice recognition system for voice recognition in one way, and sent to the intelligent dialogue model in another way to be encoded by the encoding network in the intelligent dialogue model first and then classified and predicted whether the semantics have ended. With the help of the semantic understanding ability of the intelligent dialogue model, it is possible to accurately judge whether the voice frame has ended semantically to determine whether to intervene in the interaction, and avoid intervening in the interaction too early or too late as much as possible, which helps to improve the timeliness of streaming voice interaction. Therefore, the timeliness of streaming voice interaction can be improved.

[0044] In some disclosed embodiments, the streaming voice interaction device 30 also includes a semantic analysis module, which is used to perform semantic analysis based on the recognized text to obtain an analysis result; wherein the analysis result indicates whether a query retrieval is required before replying to the recognized text; the streaming voice interaction device 30 also includes a query retrieval module, which is used to respond to the analysis result indicating that a query retrieval is required first, and perform a query retrieval based on the recognized text to obtain a retrieval result; the first response module 33 is also specifically used to generate a reply text by processing the recognized text and the retrieval result by an intelligent dialogue model.

[0045] In some disclosed embodiments, the streaming voice interaction device 30 further includes a filler control module for acquiring and outputting filler text generated by the intelligent dialogue model.

[0046] In some disclosed embodiments, the streaming voice interaction device 30 also includes a semantic regularization module, which is used to semantically regularize the recognized text based on the context of the recognized text during the interaction with the streaming voice to obtain a regularized text; the semantic analysis module is specifically used to perform semantic analysis based on the regularized text to obtain an analysis result; the query retrieval module is specifically used to perform query retrieval based on the regularized text to obtain a retrieval result; the first response module 33 is also specifically used to generate a reply text based on the regularized text and the retrieval result by processing by an intelligent dialogue model.

[0047] In some disclosed embodiments, the first response module 33 is further specifically configured to generate a reply text in response to the analysis result representation, based on the recognition text processed by the intelligent dialogue model without the need for query retrieval first.

[0048] In some disclosed embodiments, the intelligent dialogue model is a large language model, and the streaming voice interaction device 30 also includes an intention recognition module for performing intention recognition based on the encoding features of each voice frame from the starting endpoint to the ending endpoint when the classification result represents a semantic end, and determining whether there is a control intention regarding the voice interaction method; the streaming voice interaction device 30 also includes an instruction update module for updating the system instructions of the intelligent dialogue model based on the control intention in response to the existence of the control intention; the first response module 33 is also specifically used to generate a reply text by processing the intelligent dialogue model after updating the system instructions based on at least the recognized text.

[0049] In some disclosed embodiments, the streaming voice interaction device 30 also includes a parameter generation module for generating control parameters for speech synthesis based on the reply text when the classification result represents the semantic end; the streaming voice interaction device 30 also includes a speech synthesis module for performing speech synthesis on the reply text based on the control parameters to obtain a reply speech.

[0050] In some disclosed embodiments, the streaming voice interaction device 30 also includes a signal transmission module for returning to the step of performing endpoint detection on the streaming voice starting from the end endpoint when the classification result represents a semantic end, so as to start a new round of voice interaction.

[0051] See also Figure 4 , Figure 4: is a schematic diagram of the framework of an embodiment of an electronic device of the present application. The electronic device 40 includes at least a memory 41 and a processor 42 coupled to each other, the memory 41 stores at least program instructions, and the processor 42 is used to execute the program instructions to implement the steps in any of the above-mentioned streaming voice interaction method embodiments. For details, please refer to the aforementioned disclosed embodiments, which will not be repeated here. As a possible example, the electronic device 40 may include but is not limited to a server, a camera, a smart phone, a tablet computer, etc., and the specific type of the electronic device 40 is not limited here.

[0052] Specifically, the processor 42 is used to control itself and the memory 41 to implement the steps in any of the above-mentioned streaming voice interaction method embodiments. The processor 42 can also be called a CPU (Central Processing Unit). The processor 42 may be an integrated circuit chip with signal processing capabilities. The processor 42 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 42 can be implemented by an integrated circuit chip.

[0053] In the above scheme, the electronic device 40 performs endpoint detection on the streaming voice, and in response to detecting the starting endpoint of the streaming voice, performs a sliding window operation on the streaming voice from the starting endpoint to obtain a voice frame, and performs feature extraction based on the voice frame to obtain the voice features of the voice frame, and then inputs the voice features of the voice frame into a voice recognition system for identifying the streaming voice to obtain a recognition result of the voice frame, and performs classification prediction based on the coding features of the voice frame to obtain a classification result of the voice frame, and the coding features are extracted by the coding network in the intelligent dialogue model after the voice features are input into the intelligent dialogue model, and the classification result includes whether the voice frame represents an ending endpoint of a semantic end, based on this, in response to the classification result representing the semantic end, based on the recognition results of each voice frame from the starting endpoint to the ending endpoint, a recognized text is obtained, and at least based on the recognized text, the intelligent dialogue model performs processing The processing is performed to generate a reply text, and in response to the classification result representing that the semantics have not ended, the sliding window operation is continued to be performed. Therefore, on the one hand, in the process of streaming voice, the sliding window operation can be continuously performed after the start endpoint is detected to obtain the voice frame, and the relevant processing is performed on each voice frame in turn until the voice frame representing the semantics ends and the reply text is generated, which can realize the closed loop of the process of streaming voice interaction. On the other hand, the voice features of the voice frame are sent to the voice recognition system for voice recognition in one way, and sent to the intelligent dialogue model in another way to be encoded by the encoding network in the intelligent dialogue model first and then classified and predicted whether the semantics have ended. With the help of the semantic understanding ability of the intelligent dialogue model, it is possible to accurately judge whether the voice frame has ended as much as possible to determine whether to intervene in the interaction, and avoid intervening in the interaction too early or too late as much as possible, which helps to improve the timeliness of streaming voice interaction. Therefore, the timeliness of streaming voice interaction can be improved.

[0054] See also Figure 5 , Figure 5 1 is a schematic diagram of a framework of an embodiment of a computer-readable storage medium 50 of the present application. The computer-readable storage medium 50 stores program instructions 51 that can be executed by a processor, and the program instructions 51 are used to implement the steps in any of the above-mentioned streaming voice interaction method embodiments.

[0055] In the above scheme, the computer-readable storage medium 50 performs endpoint detection on the streaming speech, and in response to detecting the starting endpoint of the streaming speech, performs a sliding window operation on the streaming speech from the starting endpoint to obtain a speech frame, and extracts features based on the speech frame to obtain speech features of the speech frame, and then inputs the speech features of the speech frame into a speech recognition system for identifying the streaming speech to obtain a recognition result of the speech frame, and performs classification prediction based on the coding features of the speech frame to obtain a classification result of the speech frame, and the coding features are extracted by the coding network in the intelligent dialogue model after the speech features are input into the intelligent dialogue model, and the classification result includes whether the speech frame represents an ending endpoint of a semantic end, based on this, in response to the classification result representing the semantic end, based on the recognition results of each speech frame from the starting endpoint to the ending endpoint, a recognized text is obtained, and at least based on the recognized text, the intelligent dialogue model performs classification prediction based on the coding features of the speech frame to obtain a classification result. The processing is performed to generate a reply text, and in response to the classification result indicating that the semantics have not ended, the sliding window operation is continued to be performed. Therefore, on the one hand, in the process of streaming voice, the sliding window operation can be continuously performed after the start endpoint is detected to obtain a voice frame, and each voice frame is processed in turn until the voice frame indicating the semantics has ended is detected, and then the reply text is generated, which can realize the closed loop of the process of streaming voice interaction. On the other hand, the voice features of the voice frame are sent to the voice recognition system for voice recognition in one way, and sent to the intelligent dialogue model in another way to be encoded by the encoding network in the intelligent dialogue model first and then classified and predicted whether the semantics have ended. With the help of the semantic understanding ability of the intelligent dialogue model, it is possible to accurately judge whether the voice frame has ended semantically to determine whether to intervene in the interaction, and avoid intervening in the interaction too early or too late as much as possible, which helps to improve the timeliness of streaming voice interaction. Therefore, the timeliness of streaming voice interaction can be improved.

[0056] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0057] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0058] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0059] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0060] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0061] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0062] If the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to the collection of his or her personal information; or on the device that processes personal information, the personal information processing rules are notified by obvious signs / information, and the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

Claims

1. A streaming voice interaction method, characterized in that: include: Performing endpoint detection on the streaming voice, and in response to detecting the starting endpoint of the streaming voice, performing a sliding window operation on the streaming voice from the starting endpoint to obtain a voice frame, and performing feature extraction based on the voice frame to obtain a voice feature of the voice frame; Inputting the speech features of the speech frame into a speech recognition system for identifying streaming speech to obtain a recognition result of the speech frame, and performing classification prediction based on the coding features of the speech frame to obtain a classification result of the speech frame; wherein the coding features are extracted by the coding network in the intelligent dialogue model after the speech features are input into the intelligent dialogue model, and the classification result includes whether the speech frame represents an end endpoint of a semantic end; In response to the classification result representing the semantic end, based on the recognition results of each of the speech frames from the starting endpoint to the ending endpoint, a recognition text is obtained, and at least based on the recognition text, the intelligent dialogue model is processed to generate a reply text; In response to the classification result representing that the semantics are not finished, continue to return to execute the sliding window operation.

2. The method according to claim 1, characterized in that After obtaining the recognition text based on the recognition results of each of the speech frames from the starting endpoint to the ending endpoint, and before generating a reply text by processing the intelligent dialogue model at least based on the recognition text, the method further includes: Perform semantic analysis based on the recognized text to obtain an analysis result; wherein the analysis result indicates whether a query retrieval is required before replying to the recognized text; In response to the analysis result indicating that a query search is required first, a query search is performed based on the recognized text to obtain a search result; The step of processing the intelligent dialogue model at least based on the recognition text to generate a reply text includes: The intelligent dialogue model processes the recognized text and the search results to generate the reply text.

3. The method according to claim 2, characterized in that After performing a query search based on the recognition text to obtain a search result, and before the intelligent dialogue model processes the recognition text and the search result to generate the reply text, the method further includes: Acquire and output the pad text generated by the intelligent dialogue model.

4. The method according to claim 2, characterized in that: Before performing semantic analysis based on the recognized text to obtain an analysis result, the method further includes: Based on the context of the recognized text in the process of interaction with the streaming voice, semantically regularize the recognized text to obtain regularized text; The performing semantic analysis based on the recognized text to obtain analysis results includes: Performing semantic analysis based on the regularized text to obtain the analysis result; The query retrieval based on the recognized text to obtain the retrieval results includes: Performing a query search based on the regular text to obtain the search result; The processing based on the recognition text and the search result by the intelligent dialogue model to generate the reply text includes: The intelligent dialogue model is used to process at least the regular text and the search results to generate the reply text.

5. The method according to claim 2, characterized in that: The method further comprises: In response to the analysis result representation, the reply text is generated by processing the recognized text by the intelligent dialogue model without the need for query retrieval.

6. The method according to claim 1, characterized in that The intelligent dialogue model is a large language model. When the classification result represents a semantic end, before the intelligent dialogue model processes the recognized text at least based on the recognition text to generate a reply text, the method further includes: Performing intention recognition based on the coding features of each of the voice frames from the starting endpoint to the ending endpoint to determine whether there is a control intention regarding the voice interaction mode; In response to the existence of the control intention, updating the system instructions of the intelligent dialogue model based on the control intention; The step of processing the intelligent dialogue model at least based on the recognition text to generate a reply text includes: The reply text is generated based on at least the recognition text being processed by the intelligent dialogue model after updating the system instructions.

7. The method according to claim 1, characterized in that In the case where the classification result represents a semantic end, after the intelligent dialogue model processes the recognized text at least based on the generated reply text, the method further includes: Based on the reply text, generating control parameters for speech synthesis; The reply text is subjected to speech synthesis based on the control parameters to obtain a reply speech.

8. The method according to claim 1, characterized in that: In the case where the classification result represents a semantic end, before obtaining the recognized text based on the recognition results of each of the speech frames from the starting endpoint to the ending endpoint, the method further includes: A control signal indicating that recognition is to be stopped is transmitted to the speech recognition system.

9. The method according to claim 1, characterized in that: In the case where the classification result represents a semantic end, after the intelligent dialogue model processes the recognized text at least to generate a reply text, the method further includes: Starting from the end endpoint, return to execute the step of performing endpoint detection on the streaming voice to start a new round of voice interaction.

10. A streaming voice interaction device, characterized in that: include: A preparatory processing module, configured to perform endpoint detection on the streaming voice, and in response to detecting the starting endpoint of the streaming voice, perform a sliding window operation on the streaming voice from the starting endpoint to obtain a voice frame, and perform feature extraction based on the voice frame to obtain a voice feature of the voice frame; A recognition and classification module, used for inputting the speech features of the speech frame into a speech recognition system for identifying streaming speech, obtaining a recognition result of the speech frame, and performing classification prediction based on the coding features of the speech frame to obtain a classification result of the speech frame; wherein the coding features are extracted by the coding network in the intelligent dialogue model after the speech features are input into the intelligent dialogue model, and the classification result includes whether the speech frame represents an end endpoint of a semantic end; A first response module is used for obtaining a recognition text based on the recognition results of each of the speech frames from the starting endpoint to the ending endpoint in response to the semantic end represented by the classification result, and generating a reply text by processing the recognition text by the intelligent dialogue model at least based on the recognition text; The second response module is used for returning to execute the sliding window operation in response to the semantic representation of the classification result not being completed.

11. An electronic device, characterized in that: The method comprises at least a memory and a processor coupled to each other, wherein the memory at least stores program instructions, and the processor is used to execute the program instructions to implement the streaming voice interaction method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: Program instructions that can be executed by a processor are stored, and the program instructions are used to implement the streaming voice interaction method described in any one of claims 1 to 9.

Images