Voice interaction method and device, equipment, storage medium and product

Abstract

The invention discloses a voice interaction method and device, equipment, a storage medium and a product, and relates to the technical field of computers, the voice interaction method comprises the following steps: when a user voice stream is received, generating text information of the user voice stream in a streaming manner, and sending the text information to a preset large model; and receiving reply information returned by the large model, playing the reply information, and when the reply information is played, if a new round of user voice stream is received, returning to the step of generating the semantic information of the user voice stream in a streaming manner. According to the method, the character information of the user voice stream is replied by adopting the large model, when the output of the reply information is interrupted by the new round of user voice stream, the reply information is regenerated by returning to the step of generating the semantic information of the user voice stream in a stream mode, so that the voice interruption of the received new round of user voice stream is realized; therefore, the coherence of voice interaction is improved, and the fluency of voice interaction is finally improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to voice interaction methods, devices, equipment, storage media and products. Background Art

[0002] With the rapid development of artificial intelligence technology, voice interaction technology is becoming an indispensable part of people's daily life and work. Voice interaction can provide a more natural and convenient operation method, especially suitable for scenarios that require quick response and frequent operation.

[0003] Traditional voice interaction technology uses rule-based or template-based natural language generation technology for voice interaction. During the interaction process, it is difficult to receive a new round of voice input from the user in real time, and there is a possibility of interruption or information loss, resulting in low fluency of voice interaction.

[0004] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0005] The main purpose of this application is to provide a voice interaction method, device, equipment, storage medium and product, aiming to solve the technical problem of low fluency of voice interaction.

[0006] To achieve the above object, the present application proposes a voice interaction method, the method comprising:

[0007] When receiving a user voice stream, generating text information of the user voice stream in a streaming manner, and sending the text information to a preset large model;

[0008] Receive the reply information returned by the large model, and play the reply information, wherein when playing the reply information, if a new round of user voice stream is received, return to the step of streaming to generate text information of the user voice stream.

[0009] In one embodiment, the step of streaming the text information of the user voice stream and sending the text information to a preset large model includes:

[0010] Determine whether the current round is the first round;

[0011] If the current round is the first round, when the first punctuation mark of the text information is stream-generated, the generated text information is sent to the large model;

[0012] If the current round is not the first round, wait for the complete text information to be generated in a streaming manner, and send the complete text information to the large model.

[0013] In one embodiment, the step of waiting for the complete text information to be generated in a streaming manner and sending the complete text information to the large model comprises:

[0014] Extract the large model prompt words of all rounds of chat records;

[0015] Waiting for the complete text information of the current round to be generated by streaming;

[0016] The text information of the current round and the large model prompt words are sent to the large model.

[0017] In one embodiment, the step of receiving the reply information returned by the large model and playing the reply information comprises:

[0018] Receiving the reply information returned by the large model;

[0019] According to the preset voice setting, the reply information is streamed to generate reply voice information;

[0020] Play the reply voice message.

[0021] In one embodiment, when playing the reply information, if a new round of user voice stream is received, the step of returning to stream-generate text information of the user voice stream includes:

[0022] Record the timbre information of the user's voice stream;

[0023] If a user voice stream with different timbre information is received when outputting the reply information, confirm whether to return to the step of streaming to generate semantic information of the user voice stream according to the preset interruption setting;

[0024] If a new round of user voice stream with the same timbre information is received when the reply information is output, the process returns to the step of generating semantic information of the user voice stream in a streaming manner.

[0025] In one embodiment, the step of returning the semantic information of the user voice stream to be generated in a streaming manner includes:

[0026] When an interruption signal is detected in the user's voice stream, the current reply information output is stopped;

[0027] Identify verb features in the user's speech stream;

[0028] Selecting an interruption connection sentence corresponding to the verb feature from a preset interruption connection sentence library;

[0029] The interrupt connection statement is played.

[0030] In addition, to achieve the above purpose, the present application also proposes a voice interaction device, the voice interaction device comprising:

[0031] A receiving module, for, when receiving a user voice stream, generating text information of the user voice stream in a streaming manner, and sending the text information to a preset large model;

[0032] The interruption module is used to receive the reply information returned by the large model and play the reply information. When playing the reply information, if a new round of user voice stream is received, the step of streaming and generating text information of the user voice stream is returned.

[0033] In addition, to achieve the above-mentioned purpose, the present application also proposes a voice interaction device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the voice interaction method described above.

[0034] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the voice interaction method described above are implemented.

[0035] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the voice interaction method described above.

[0036] One or more technical solutions proposed in this application have at least the following technical effects:

[0037] Compared with the related art, the traditional voice interaction technology uses the natural language generation technology based on rules or templates for voice interaction. During the interaction process, it is difficult to receive the user's new round of voice input in real time, and there is the possibility of interruption or information loss, resulting in low fluency of voice interaction. In this application, when receiving the user voice stream, the text information of the user voice stream is generated in a streaming manner, and the text information is sent to the preset large model; the reply information returned by the large model is received, and the reply information is played, wherein when playing the reply information, if a new round of user voice stream is received, the step of streaming the text information of the user voice stream is returned. It can be understood that the present application uses a large model to reply to the text information of the user voice stream. When the output of the reply information is interrupted by a new round of user voice stream, the reply information is regenerated by returning the step of streaming the semantic information of the user voice stream, so as to achieve the interruption of the voice of the new round of user voice stream and timely generate the new round of reply content, so as to improve the coherence of voice interaction and ultimately improve the fluency of voice interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0040] Figure 1 A flowchart of the first embodiment of the voice interaction method of the present application is provided;

[0041] Figure 2 A brief flowchart of the voice interaction method provided in Example 2 of the present application;

[0042] Figure 3 A flowchart of the third embodiment of the voice interaction method of the present application is provided;

[0043] Figure 4 This is a schematic diagram of the module structure of the voice interaction device according to an embodiment of the present application;

[0044] Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the voice interaction method in the embodiment of the present application.

[0045] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0046] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0047] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0048] The main solutions of the embodiments of this application are:

[0049] When receiving a user voice stream, generating text information of the user voice stream in a streaming manner, and sending the text information to a preset large model;

[0050] Receive the reply information returned by the large model, and play the reply information, wherein when playing the reply information, if a new round of user voice stream is received, return to the step of streaming to generate text information of the user voice stream.

[0051] In this embodiment, the present application uses a voice interaction device as the execution subject. For the convenience of description, it is specifically described below as "device".

[0052] Since the existing technology uses rule-based or template-based natural language generation technology for voice interaction, it is difficult to receive a new round of user voice input in real time during the interaction process, and there is a possibility of interruption or information loss, resulting in low fluency of voice interaction.

[0053] The present application provides a solution, which adopts a large model to reply to the text information of the user voice stream. When the output of the reply information is interrupted by a new round of user voice stream, the reply information is regenerated by returning the step of streaming to generate the semantic information of the user voice stream, so as to achieve the reception of the voice interruption of the new round of user voice stream and the timely generation of the new round of reply content, thereby improving the coherence of voice interaction and ultimately improving the fluency of voice interaction.

[0054] Based on this, the present application embodiment provides a voice interaction method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the voice interaction method of the present application.

[0055] In this embodiment, the voice interaction method includes steps S10 to S20:

[0056] Step S10, when a user voice stream is received, streamingly generating text information of the user voice stream, and sending the text information to a preset large model;

[0057] It should be noted that the user voice stream is a sequence of voice signals emitted by the user in real time when interacting with the device. These voice signals are continuous and can be received and processed by the device in real time. Streaming generation is a technology for processing data. Especially when the amount of data is large or the data is continuously generated, streaming processing allows the device to process data while receiving it, rather than waiting for all the data to be collected before starting processing. In this scenario, it means that the device can convert the received voice stream into text information in real time.

[0058] The text information is text data converted from the user's voice stream. Through speech recognition technology, the system can convert the user's voice into corresponding text information in real time for further processing. The preset big model is a large artificial intelligence model that is pre-set and trained, usually a deep learning model, used to process and generate language-related tasks. In this scenario, the big model is responsible for receiving text information and generating corresponding replies.

[0059] Exemplarily, in a full-duplex voice interaction system, the speech recognition module (ASR) is responsible for receiving the user's voice input. The ASR module uses a streaming generation method to convert the user's voice signal into text information in real time. At the same time, the voice activity detection module (VAD) continuously monitors the user's speaking status and determines whether the user is speaking by analyzing the characteristics of the voice signal. Once the VAD module detects that the user's silent time ratio reaches a preset threshold, it is considered that the user has stopped speaking. It will immediately send the text information recognized by the ASR module to the preset large model through the websocket protocol. In this way, the large model can receive the user's input and generate a coherent response to achieve a real-time and efficient voice interaction experience.

[0060] It is understandable that the speech recognition module (ASR) is called the automatic speech recognition system and is a key component in the full-duplex voice interaction system. Its main function is to convert the user's voice input into readable text information. This module usually integrates advanced speech recognition technology and can handle different accents, speaking speeds and background noise to ensure high accuracy of speech-to-text services.

[0061] Streaming generation means that after receiving voice input, the ASR module does not wait for the entire voice segment to end before starting processing, but converts the voice signal into text in real time and continuously. This method allows the system to respond to the user's voice input instantly, improving the smoothness and real-time nature of the interaction.

[0062] The VAD module, or voice activity detection module, is responsible for monitoring the user's speaking status in real time. It determines whether the user is speaking, as well as the start and end of speaking by analyzing the characteristics of the voice signal. The VAD module is crucial to achieving full-duplex communication because it can identify the gaps in the user's speech, thereby triggering subsequent text sending actions.

[0063] The percentage of silent time refers to the proportion of time in a certain period of time when there is no voice signal or the voice signal is below a certain threshold. The VAD module will determine whether the user has paused speaking based on this indicator. If the percentage of silent time reaches the preset threshold, the VAD module will consider that the user has paused speaking.

[0064] Websocket is a network communication protocol that provides full-duplex communication capabilities over a single TCP connection. In this scenario, Websocket is used to implement real-time data transmission between the VAD module and the large model. Once the VAD module detects that the user has paused speaking, it immediately sends the text recognized by the ASR module to the large model via Websocket.

[0065] A big model refers to a pre-trained, complex AI model, usually a natural language processing model based on deep learning, that can understand the semantics of text messages and generate coherent and relevant responses.

[0066] Step S20, receiving the reply information returned by the large model, and playing the reply information, wherein when playing the reply information, if a new round of user voice stream is received, then returning to the step of streaming generation of text information of the user voice stream.

[0067] It should be noted that the reply information is a response generated by the large model based on the text information converted from the user's voice stream. This response is intended to be consistent with the user's voice input and provide useful information or perform a specific task. The playback is the device converting the generated reply information (usually in text form) into voice output so that the user can hear it. The new round of user voice stream is a new round of dialogue input that the user may continue after hearing the device's reply. This shows that the user can interrupt the device's reply, and the device needs to be able to handle such interruptions.

[0068] The step of returning the streaming generated semantic information of the user's voice stream means that when the device receives a new round of voice stream from the user while playing a reply message, the device will stop playing the current reply and re-enter the streaming generation process to process the user's new input in real time and generate a new reply.

[0069] It is understandable that during the playback process, the VAD module continues to monitor user input. Once new voice input is detected, it immediately interrupts the current playback and starts processing new issues, thereby realizing the call interruption function.

[0070] Exemplarily, in a full-duplex voice interaction system, when the device is playing the reply information generated by the large model, the voice activity detection module (VAD) does not stop working, but continues to monitor the user's voice input. The VAD module detects the user's speaking status in real time by analyzing the characteristics of the sound signal. Once the VAD module detects a new voice input, it immediately sends an interrupt signal to interrupt the current reply playback. This interrupt signal triggers the system to stop playing the current reply and start processing the user's new voice input. The device passes the new voice input to the speech recognition module (ASR), which converts it into text information. These text messages are then sent to the large model, and the large model generates a corresponding reply based on the new input. In this way, the device can respond to the user's new voice input at any time, realize the call interruption function, and provide a more natural and smooth interactive experience.

[0071] This embodiment provides a voice interaction method, which adopts a large model to reply to the text information of the user voice stream. When the output of the reply information is interrupted by a new round of user voice stream, the reply information is regenerated by returning the step of streaming to generate the semantic information of the user voice stream, so as to achieve the reception of the voice interruption of the new round of user voice stream and the timely generation of the new round of reply content, thereby improving the coherence of the voice interaction and ultimately improving the fluency of the voice interaction.

[0072] In a feasible implementation manner, the step of streaming generating text information of the user voice stream and sending the text information to a preset large model includes:

[0073] Determine whether the current round is the first round;

[0074] If the current round is the first round, when the first punctuation mark of the text information is stream-generated, the generated text information is sent to the large model;

[0075] If the current round is not the first round, wait for the complete text information to be generated in a streaming manner, and send the complete text information to the large model.

[0076] It should be noted that the current round is a round of dialogue or information exchange between the user and the device during the full-duplex voice interaction process. Each round may include the user's voice input and the device's response. The first round specifically refers to the first round of dialogue when the user and the system start to interact. In this round, the device needs to specially handle the user's input because it marks the beginning of the conversation.

[0077] Streaming refers to the process of converting the user's voice input into text information in real time. This processing method allows the device to convert while receiving the voice, without waiting for the entire voice input to end.

[0078] Text information is the text data converted from the user's voice input through the speech recognition module (ASR). These data will be used for subsequent processing and response generation.

[0079] In the text information generated by streaming, the first punctuation mark, such as a period, comma, etc., usually marks the end of a semantic unit and can be used as an appropriate time to send it to the large model for processing.

[0080] For example, in a full-duplex voice interaction device, the device needs to determine whether the current user voice input round is the first round of the conversation. If it is the first round, the device will specially process the user's input because it is the beginning of the conversation. In this case, the device will monitor the text information generated by the streaming, and when the first punctuation mark is detected, the generated part of the text information will be sent to the large model for processing and reply generation. This can quickly respond to the user's initial input and improve the real-time nature of the interaction.

[0081] If the current round is not the first round, the device will adopt a different strategy. In this case, the device will wait until the user's entire voice input is completely converted into text information, including all sentences and punctuation marks, to ensure that the information sent to the large model is complete and coherent. This ensures that the information received by the large model is complete, thereby generating a more accurate and coherent response. This strategy of distinguishing and processing different rounds helps the device provide a high-quality interactive experience while maintaining real-time performance.

[0082] It is understandable that after receiving the text, the big model starts to generate answers in a streaming manner. In order to reduce the delay of the first sentence and improve the response speed, for the first sentence, it starts generating when it encounters punctuation marks, and waits for the complete sentence for non-first sentences. The generated answers are sent to the TTS module via websocket streaming.

[0083] In a feasible implementation manner, the step of waiting for the complete text information to be generated in a streaming manner and sending the complete text information to the large model comprises:

[0084] Extract the large model prompt words of all rounds of chat records;

[0085] Waiting for the complete text information of the current round to be generated by streaming;

[0086] The text information of the current round and the large model prompt words are sent to the large model.

[0087] It should be noted that the chat records of all rounds refer to the records of all historical conversations between the user and the system during the full-duplex voice interaction process. These records include the voice input of the user and the system in each round and the converted text information.

[0088] Exemplarily, in a full-duplex voice interaction device, in order to ensure that the large model can generate coherent and relevant replies, the device extracts the large model prompt words in all rounds of chat records. These prompt words provide the large model with contextual information of the conversation, helping the model to better understand the user's needs and the history of the conversation. The device waits for the complete text information of the current round to be generated in a streaming manner, which means that the user's voice input will continue to be received until a logical end point, such as a complete sentence or a semantically complete expression is converted into text information. Once the text information of the current round is completely generated, the device sends this information together with the large model prompt words to the large model. This method of combining the text information of the current round and the prompt words in the historical chat records enables the large model to take into account the context of the entire conversation when generating a reply, thereby improving the accuracy and relevance of the reply.

[0089] It is understandable that the device will also record each round of chat records and write them into the prompt words of the large model for context understanding, so that the large model can know what the user said before and realize multiple rounds of dialogue.

[0090] In a feasible implementation manner, the step of receiving the reply information returned by the large model and playing the reply information includes:

[0091] Receiving the reply information returned by the large model;

[0092] According to the preset voice setting, the reply information is streamed to generate reply voice information;

[0093] Play the reply voice message.

[0094] It should be noted that the reply information returned by the large model is a textual reply generated by the large model after processing the user input. The large model generates one or more replies based on the user's voice input (which has been converted into text information) and possible contextual information, and these replies are intended to respond to the user's query or instruction.

[0095] The preset voice settings refer to the parameters and configurations used by the device when converting text to speech, including the speech speed, pitch, volume, gender, language, etc. These settings can be preset according to user preferences or device design requirements to ensure that the generated voice information conforms to the user's hearing habits.

[0096] Streaming here refers to the process by which the system converts the text reply information returned by the large model into speech in real time. This process is continuous, allowing the device to start playing the speech as it is generated, without having to wait for the entire text conversion to be completed, which can reduce the time the user waits for the reply.

[0097] The reply voice information refers to the voice output converted from the text reply information. This voice output is the final response of the device to the user input, and the user can obtain the required information or perform a specific operation by listening to this voice.

[0098] Playing the reply voice message refers to the process in which the system outputs the reply voice message to the user through a speaker or other audio output device. Playing is the last step in the interaction process, ensuring that the user can receive the system's reply.

[0099] Exemplarily, in a full-duplex voice interaction system, the device first receives the text reply information returned by the large model after processing the user input. Then, the device converts these text reply information into voice information in a streaming manner according to the preset voice settings. This conversion process is real-time, allowing the device to start playing as soon as the voice is generated, which improves the response speed. Finally, the device plays these reply voice information so that the user can hear the system's reply. This process ensures a smooth conversation between the user and the device, and the device can respond in a timely manner even when the user can interrupt and continue the conversation at any time.

[0100] It can be understood that the TTS module receives the output of the large model in real time, generates a PCM audio stream through websocket streaming, and plays the audio stream on the client. TTS is the abbreviation of "Text-to-Speech", which refers to the text-to-speech conversion module. The function of this module is to convert text information into audible voice output. In a full-duplex voice interaction system, the TTS module is a key component that converts the text reply generated by the large model into a voice that the user can hear. The TTS module is responsible for receiving the text output of the large model in real time. By using the websocket protocol, the TTS module is able to stream text in full-duplex communication and quickly convert it into a PCM audio stream. This conversion is continuous, allowing the audio stream to be generated and transmitted at the same time, thereby reducing latency. Once the PCM audio stream is generated, it will be sent to the client, and the client's audio playback system will play these audio streams so that the user can hear the device's reply. This real-time audio stream playback mechanism ensures that the user can receive the device's response instantly, even if the user can interrupt and continue the conversation at any time during the conversation.

[0101] In one possible implementation, refer to Figure 3 , when playing the reply information, if a new round of user voice stream is received, the step of returning to stream-generate semantic information of the user voice stream comprises:

[0102] Record the timbre information of the user's voice stream;

[0103] If a user voice stream with different timbre information is received when outputting the reply information, confirm whether to return to the step of streaming to generate semantic information of the user voice stream according to the preset interruption setting;

[0104] If a new round of user voice stream with the same timbre information is received when the reply information is output, the process returns to the step of generating semantic information of the user voice stream in a streaming manner.

[0105] It should be noted that the timbre information of the user's voice stream is the unique feature of the user's voice, including pitch, volume, sound quality, etc. These features make each person's voice recognizable. In a full-duplex voice interaction system, recording the user's timbre information can help the system identify and distinguish different speakers, and track the speech of a specific user in a multi-user environment.

[0106] Outputting the reply information is the process in which the system converts the text reply into voice through the TTS module and plays it to the user after processing the user's voice input.

[0107] User voice streams with different timbre information are when the system is outputting a reply message. If the new voice input received is different from the previously recorded timbre information, this may mean that a new speaker has joined the conversation. The system needs to identify this change and decide whether to interrupt the current reply based on the preset interruption settings.

[0108] Preset interruption settings are system-defined parameters and rules that determine when the system should interrupt the current reply to respond to new voice input. These settings may include sensitivity to voice changes, conversation turns, user priority, and other factors.

[0109] If the device receives a user voice stream with a different timbre, it needs to determine whether to interrupt the current reply based on the preset interruption settings and return to the step of streaming to generate semantic information of the new user voice stream. This means that the system needs to re-evaluate and process the new voice input.

[0110] When the device receives new voice input while outputting the reply message, the voice information is the same as the previously recorded voice information, which indicates that the same user continues to speak. The device regards these new voice inputs as a continuation of the current conversation.

[0111] For a new round of user voice stream with the same timbre, the device will interrupt the current response and return to the step of streaming the semantic information of the new voice stream. This allows the device to process the user's new input in real time and generate a coherent response.

[0112] Exemplarily, in a full-duplex voice interaction device, the device records the timbre information of the user voice stream in order to identify and distinguish different speakers. When the device is outputting a reply message, if it receives a user voice stream with a different timbre, it will confirm whether it is necessary to interrupt the current reply according to the preset interruption setting, and return to the step of streaming the semantic information of the new user voice stream. This is to ensure that the system can respond to new speakers in a timely manner. On the contrary, if the device receives a new round of user voice streams with the same timbre, it will directly return to the step of streaming the semantic information of the user voice stream to maintain the continuity and fluency of the conversation. This mechanism enables the device to flexibly handle conversations in a multi-user environment while maintaining sensitivity and responsiveness to each user's speech.

[0113] In a feasible implementation manner, before the step of returning the semantic information of the user voice stream to be generated in a streaming manner, the step includes:

[0114] When an interruption signal is detected in the user's voice stream, the current reply information output is stopped;

[0115] Identify verb features in the user's speech stream;

[0116] Selecting an interruption connection sentence corresponding to the verb feature from a preset interruption connection sentence library;

[0117] The interrupt connection statement is played.

[0118] It should be noted that the interruption signal in the user voice stream is a specific pattern or keyword that the device can recognize in the user voice input, indicating that the user wants to interrupt the current device response. This may include specific phrases, intonation changes or other voice features.

[0119] Stopping the current reply message output means that when the device detects an interruption signal, the currently playing reply message will be interrupted. This is to allow the device to immediately respond to the user's new input and maintain the fluency and interactivity of the conversation.

[0120] Identifying verb features in a user's voice stream involves the device analyzing the user's voice stream to identify and extract verbs therein. Verb features may indicate the user's intention or action, which is critical for the device to understand the user's needs and generate an appropriate response.

[0121] The preset interruption connection statement library is a database that stores a series of predefined statements designed to smoothly connect the conversation when the user interrupts. These statements can help maintain the continuity of the conversation while responding to the user's interruption.

[0122] Once the appropriate interruption phrase is selected, the device will play the phrase through the voice output device. This helps the device continue the conversation in a natural way after the user interrupts.

[0123] Exemplarily, in a full-duplex voice interaction device, the device is designed to detect an interruption signal in the user's voice stream. Once such a signal is detected, the device will immediately stop the current reply information output to prepare to respond to the user's new input. The device further analyzes the user's voice stream and identifies the verb features therein, which provide important clues about the user's intentions. Based on these features, the system selects the most appropriate interruption connection sentence from a preset interruption connection sentence library, which matches the user's verb features to ensure the continuity and relevance of the conversation. Finally, the device plays the selected interruption connection sentence to smoothly continue the conversation while maintaining sensitivity and responsiveness to user interruptions. This mechanism enables the full-duplex voice interaction system to effectively handle user interruptions while maintaining smooth conversation.

[0124] For example, to help understand the implementation process of the voice interaction method obtained by combining this embodiment with the above-mentioned embodiment 1, please refer to Figure 2 , Figure 2 A brief flow chart of a voice interaction method is provided, specifically:

[0125] The user starts speaking through a microphone or other audio input device, and the voice stream is received by the device in real time. The device uses streaming speech recognition technology to convert the user's voice stream received in real time into text information. This process allows the device to recognize the voice content while the user is speaking. The voice activity detection (VAD) module analyzes the voice stream to determine whether the user is speaking. The VAD module helps to distinguish between the user's voice and background noise, ensuring that only valid voice input is further processed. The device determines whether the current voice input is a new speech or an interruption. If it is detected that the user starts speaking while the device is playing a reply message, this may indicate that the user wants to interrupt the current reply. If the user speaks while the device is playing a reply message, the device stops the current reply message output and prepares to receive new voice input. The device sends the text information converted from the user's voice stream to the large language model. The large language model generates a coherent reply based on the received information. The text reply generated by the large language model is fed into the streaming speech synthesis module, which converts the text into voice information and prepares to play it to the user. The device plays the synthesized voice stream as a reply to the user input. If the user's voice input is detected again during this process, the device will determine whether it is an interruption again, and stop playing and process the new input accordingly. In order to maintain the continuity of the conversation, the device will store the user's voice stream and the system's reply information in the chat history. These records can be used for subsequent conversation analysis and optimization.

[0126] Through this process, the full-duplex voice interaction system enables users to interrupt the system response and enter new voice commands at any time, and the device can respond instantly and generate coherent responses, thus providing a smooth interactive experience.

[0127] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the voice interaction method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0128] This application also provides a voice interaction device, please refer to Figure 4 , the voice interaction device comprises:

[0129] The receiving module 10 is used for, when receiving a user voice stream, streamingly generating text information of the user voice stream and sending the text information to a preset large model;

[0130] The interruption module 20 is used to receive the reply information returned by the large model and play the reply information. When playing the reply information, if a new round of user voice stream is received, the step of streaming and generating text information of the user voice stream is returned.

[0131] And / or, the receiving module 10 includes:

[0132] A first judgment module, used to judge whether the current round is the first round;

[0133] A first sending module, configured to send the generated text information to the large model when the first punctuation mark of the text information is stream-generated if the current round is the first round;

[0134] The second sending module is used for waiting for the complete text information to be generated in a streaming manner if the current round is not the first round, and sending the complete text information to the large model.

[0135] And / or, the second sending module includes:

[0136] The first extraction module is used to extract the large model prompt words of all rounds of chat records;

[0137] A first waiting module, used for waiting for the complete text information of the current round to be generated in a streaming manner;

[0138] The third sending module is used to send the text information of the current round and the large model prompt word to the large model.

[0139] And / or, the interruption module 20 includes:

[0140] A first receiving module, used for receiving the reply information returned by the large model;

[0141] A first generating module, used for streaming the reply information to generate reply voice information according to a preset voice setting;

[0142] The first playing module is used to play the reply voice information.

[0143] And / or, the interruption module 20 includes:

[0144] The first recording module is used to record the timbre information of the user's voice stream;

[0145] A first confirmation module is used to confirm whether to return to the step of streaming to generate semantic information of the user voice stream according to a preset interruption setting when receiving a user voice stream with different timbre information when outputting the reply information;

[0146] The first returning module is used to return to the step of streamingly generating semantic information of the user voice stream when a new round of user voice stream with the same timbre information is received when the reply information is output.

[0147] And / or, the voice interaction device comprises:

[0148] A first stopping module, used for stopping the current reply information output when an interruption signal is detected in the user voice stream;

[0149] A first recognition module, used to recognize verb features in a user's voice stream;

[0150] A first interruption module, used to select an interruption connection sentence corresponding to the verb feature from a preset interruption connection sentence library;

[0151] The second playing module is used to play the interruption connection sentence.

[0152] The voice interaction device provided by the present application adopts the voice interaction method in the above embodiment, which can solve the technical problem of low fluency of voice interaction. Compared with the prior art, the beneficial effects of the voice interaction device provided by the present application are the same as the beneficial effects of the voice interaction method provided by the above embodiment, and other technical features in the voice interaction device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0153] The present application provides a voice interaction device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the voice interaction method in the above-mentioned embodiment one.

[0154] Reference below Figure 5 , which shows a schematic diagram of the structure of a voice interaction device suitable for implementing the embodiments of the present application. The voice interaction device in the embodiments of the present application may include but is not limited to mobile terminals such as mobile phones, tablet computers, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The voice interaction device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0155] like Figure 5 As shown, the voice interaction device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the voice interaction device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the voice interaction device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a voice interaction device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or provided alternatively.

[0156] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0157] The voice interaction device provided by the present application adopts the voice interaction method in the above embodiment, which can solve the technical problem of low fluency of voice interaction. Compared with the prior art, the beneficial effects of the voice interaction device provided by the present application are the same as the beneficial effects of the voice interaction method provided by the above embodiment, and the other technical features in the voice interaction device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0158] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0159] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0160] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the voice interaction method in the above-mentioned embodiment.

[0161] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0162] The computer-readable storage medium may be included in the voice interaction device; or may exist independently without being assembled into the voice interaction device.

[0163] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the voice interaction device, the voice interaction device: when receiving a user voice stream, generates text information of the user voice stream in a streaming manner, and sends the text information to a preset large model;

[0164] Receive the reply information returned by the large model, and play the reply information, wherein when playing the reply information, if a new round of user voice stream is received, return to the step of streaming to generate text information of the user voice stream.

[0165] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0166] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0167] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0168] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned voice interaction method, and can solve the technical problem of low fluency of voice interaction. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the voice interaction method provided in the above-mentioned embodiment, and will not be repeated here.

[0169] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned voice interaction method when executed by a processor.

[0170] The computer program product provided by the present application can solve the technical problem of low fluency of voice interaction. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the voice interaction method provided by the above embodiment, which will not be repeated here.

[0171] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A voice interaction method, characterized in that: The method includes: When receiving a user voice stream, generating text information of the user voice stream in a streaming manner, and sending the text information to a preset large model; Receive the reply information returned by the large model, and play the reply information, wherein when playing the reply information, if a new round of user voice stream is received, return to the step of streaming to generate text information of the user voice stream.

2. The method according to claim 1, characterized in that The step of streaming and generating text information of the user voice stream and sending the text information to a preset large model comprises: Determine whether the current round is the first round; If the current round is the first round, when the first punctuation mark of the text information is stream-generated, the generated text information is sent to the large model; If the current round is not the first round, wait for the complete text information to be generated in a streaming manner, and send the complete text information to the large model.

3. The method according to claim 2, characterized in that The step of waiting for the complete text information to be generated in a streaming manner and sending the complete text information to the large model comprises: Extract the large model prompt words of all rounds of chat records; Waiting for the complete text information of the current round to be generated by streaming; The text information of the current round and the large model prompt words are sent to the large model.

4. The method according to claim 1, characterized in that The step of receiving the reply information returned by the large model and playing the reply information comprises: Receiving reply information returned by the large model; According to the preset voice setting, the reply information is streamed to generate reply voice information; Play the reply voice message.

5. The method according to claim 1, characterized in that When playing the reply information, if a new round of user voice stream is received, the step of returning to stream-generate text information of the user voice stream includes: Record the timbre information of the user's voice stream; If a user voice stream with different timbre information is received when outputting the reply information, confirm whether to return to the step of streaming to generate semantic information of the user voice stream according to the preset interruption setting; If a new round of user voice stream with the same timbre information is received when the reply information is output, the process returns to the step of generating semantic information of the user voice stream in a streaming manner.

6. The method according to claim 1, characterized in that The step of returning the semantic information of the user voice stream to be generated in a streaming manner includes: When an interruption signal is detected in the user's voice stream, the current reply information output is stopped; Identify verb features in the user's speech stream; Selecting an interruption connection sentence corresponding to the verb feature from a preset interruption connection sentence library; The interrupt connection statement is played.

7. A voice interaction device, characterized in that: The device comprises: A receiving module, for, when receiving a user voice stream, generating text information of the user voice stream in a streaming manner, and sending the text information to a preset large model; The interruption module is used to receive the reply information returned by the large model and play the reply information. When playing the reply information, if a new round of user voice stream is received, the step of streaming and generating text information of the user voice stream is returned.

8. A voice interaction device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the voice interaction method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the voice interaction method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the voice interaction method according to any one of claims 1 to 6 are implemented.

Images