Voice interaction method and device, electronic equipment and storage medium

Abstract

The application provides a voice interaction method and device, electronic equipment and storage medium, and relates to the technical field of voice recognition. The method comprises the following steps: determining that a trigger signal exists, establishing a session connection to a session server; based on the session connection, sending a session audio stream of a user to the session server; feeding back response text audio data to the user; determining that a regulation signal exists, and regulating voice interaction based on the regulation signal. The application separates the session audio stream and the recognition audio stream, processes them independently, decouples the processing flow and processing resources of the session audio stream and the recognition audio stream, avoids resource competition and processing bottlenecks between the session audio stream and the recognition audio stream, and is beneficial to improving the accuracy of voice interaction. The recognition audio stream is used for recognizing the session intention of a user, and the session audio stream is used for voice interaction, which realizes the independence of session monitoring and session interaction, and improves the accuracy of voice interaction.

Description

Technical Field

[0001] This invention relates to the field of speech recognition technology, and in particular to a speech interaction method, device, electronic device, and storage medium. Background Technology

[0002] With the rapid development of artificial intelligence (AI) and the Internet of Things (IoT) technologies, voice interaction has become one of the most core and natural human-computer interaction methods in fields such as smart homes, in-vehicle systems, smart wearable devices, and personal assistant applications. Users expect to obtain a smooth, efficient, and intelligent dialogue experience, similar to human-to-human communication. To achieve this goal, voice interaction systems must possess high-precision speech recognition capabilities, low-latency response speeds, and the logic to handle complex multi-turn dialogues. Especially during continuous dialogues or task execution, users may need to interrupt, correct, or issue new commands at any time. Therefore, ensuring the system's response speed and recognition accuracy when playing downlink audio (e.g., broadcasting information, answering questions) is crucial.

[0003] Existing voice interaction systems typically employ an integrated processing architecture. In this architecture, the voice interaction recognition stream (uplink audio) and the conversation stream (downlink audio) usually share or compete for the same processing resources and audio channels.

[0004] Existing voice interaction methods suffer from resource contention and processing bottlenecks between the recognition stream and the conversation stream, which greatly reduces the accuracy of voice interaction. Summary of the Invention

[0005] This invention provides a voice interaction method, device, electronic device, and storage medium to address the shortcomings of low accuracy in existing voice interaction technologies and improve the accuracy of voice interaction.

[0006] This invention provides a voice interaction method applied to a front-end determination, comprising: Once a trigger signal is detected in the user's recognition audio stream, a session connection is established with the session server. Based on the session connection, send the user's session audio stream to the session server; Provide users with response text audio data of the conversation audio stream to enable voice interaction; The system identifies the presence of modulation signals in the audio stream and modulates the voice interaction based on these signals.

[0007] According to the voice interaction method provided by the present invention, determining that a modulation signal exists in the audio stream includes: The audio stream to be recognized is sent to the recognition server; Receive and recognize text recognition data from audio streams; The keyword database is retrieved based on text recognition data, and the keyword database includes preset control words; When the text recognition data matches the preset control words, it is determined that there is a control signal in the recognized audio stream.

[0008] According to the voice interaction method provided by the present invention, determining that a trigger signal exists in the user's recognition audio stream and establishing a session connection with the session server includes: Obtain text recognition data from the audio stream; When the text recognition data matches the preset trigger words, it is determined that there is a trigger signal in the recognized audio stream; Send a session connection establishment request to the session server; Receive the response result of the session connection establishment request to complete the establishment of the session connection; the session connection is used to transmit session audio streams or response text audio data between the decision frontend and the session server.

[0009] According to the voice interaction method provided by the present invention, voice interaction is controlled based on a control signal, including: The control signal is determined to be the termination signal; Send a termination command to the session server in response to the termination signal; the termination command carries the identifier of the session connection. Disconnect the session to terminate the voice interaction.

[0010] According to the voice interaction method provided by the present invention, voice interaction is controlled based on a control signal, including: The control signal is determined to be an interrupt signal; Send an interrupt command to the session server in response to the interrupt signal; the interrupt command carries the identifier of the session connection. Suspend the use of the session connection to interrupt voice interaction until a trigger signal is detected in the recognized audio stream, at which point the use of the session connection will be resumed.

[0011] The present invention also provides a voice interaction method applied to a session server, comprising: Based on a session connection, it receives the user's session audio stream; the session connection is established when a trigger signal is present in the user's recognized audio stream. Determine the response text audio data based on the session audio stream; Send response text and audio data to the front end for voice interaction; Determine the control of voice interaction; voice interaction is controlled when a control signal is recognized in the audio stream.

[0012] According to the voice interaction method provided by the present invention, determining response text audio data based on a conversation audio stream includes: Convert the audio stream into text-based session data; Perform semantic understanding on text conversation data, and determine the text response data based on the semantic understanding results; Convert the text response data into the target audio stream; Based on the target audio stream and text response data, determine the response text audio data.

[0013] The present invention also provides a voice interaction device, comprising: The trigger module is used to determine that a trigger signal exists in the user's recognition audio stream and establish a session connection with the session server; The first sending module is used to send the user's session audio stream to the session server based on the session connection; The feedback module is used to provide users with response text and audio data of the conversation audio stream in order to achieve voice interaction; The control module is used to determine the presence of control signals in the audio stream and to control the voice interaction based on the control signals.

[0014] The present invention also provides a voice interaction device, comprising: The receiving module is used to receive the user's session audio stream based on a session connection; the session connection is established when a trigger signal is present in the user's recognized audio stream. The first determining module is used to determine the response text audio data based on the session audio stream; The second sending module is used to send response text and audio data to the judgment front end to realize voice interaction; The second determining module is used to determine the control of voice interaction; the voice interaction is controlled when the presence of a control signal in the audio stream is recognized.

[0015] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the voice interaction methods described above.

[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the voice interaction methods described above.

[0017] The voice interaction method, apparatus, electronic device, and storage medium provided by this invention establish a session connection with a session server by determining the presence of a trigger signal in the user's recognized audio stream; based on the session connection, send the user's session audio stream to the session server; provide the user with response text audio data of the session audio stream to achieve voice interaction; and determine the presence of a control signal in the recognized audio stream to control the voice interaction based on the control signal. This invention separates the session audio stream and the recognized audio stream, processing them independently, thus decoupling the processing flow and resources of the session audio stream and the recognized audio stream. This avoids resource competition and processing bottlenecks between the session audio stream and the recognized audio stream, which is beneficial to improving the accuracy of voice interaction. The recognized audio stream is dedicated to recognizing and monitoring the user's session intent, while the session audio stream is dedicated to voice interaction, achieving independence between session monitoring and session interaction, thereby improving the accuracy of voice interaction. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is one of the flowcharts illustrating the voice interaction method provided by the present invention.

[0020] Figure 2 This is a flowchart illustrating the voice interaction method based on a voice interaction system provided by the present invention.

[0021] Figure 3 This is the second flowchart of the voice interaction method provided by the present invention.

[0022] Figure 4 This is one of the structural schematic diagrams of the voice interaction device provided by the present invention.

[0023] Figure 5 This is the second structural schematic diagram of the voice interaction device provided by the present invention.

[0024] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0026] The following is combined Figures 1 to 6 The present invention describes a voice interaction method, apparatus, electronic device, and storage medium.

[0027] Figure 1 This is one of the flowcharts illustrating the voice interaction method provided by the present invention, such as... Figure 1 As shown, the voice interaction method is applied to the front end for judgment and includes steps S100 to S400, each step of which is as follows.

[0028] S100: Determine that a trigger signal exists in the user's recognition audio stream, and establish a session connection with the session server.

[0029] The determination front end is used to determine whether there is a trigger signal or a control signal in the user's audio stream. Optionally, the determination front end can be set on the user terminal (including mobile phones, tablets, in-vehicle computers, etc.).

[0030] Before establishing a session connection, the user's voice is acquired, resulting in a recognized audio stream (Stream A). This recognized audio stream acts as a listener, monitoring the user's intent but not participating in the actual call transmission. Optionally, acquiring the recognized audio stream includes the following steps: The user opens a target application or webpage on their terminal. This application or webpage requests to record the user's voice using the user's terminal's recording device or camera. After the user agrees, the application or webpage uses the user's terminal's recording device or camera to record the user's voice, obtaining the recognized audio stream. The recognized audio stream is used for low-power continuous monitoring. Once a trigger signal is detected in the recognized audio stream, a high-power session connection is established to achieve efficient voice interaction.

[0031] Determine if a trigger signal exists in the user's recognized audio stream. Convert the recognized audio stream to text and perform keyword detection. If a preset trigger word is detected in the text of the recognized audio stream, a trigger signal is considered to exist. The trigger signal is used to initiate voice interaction between the user and the session server. For example, the name of the target application or target webpage is A. Optionally, the preset trigger word is the name of the target application or target webpage (including the full name, abbreviation, or nickname). For example, if the recognized audio stream is "Hello, A", or "Little A Little A", or "A", the recognized audio stream includes the preset trigger word.

[0032] Optionally, the front end performs semantic understanding on the recognized audio stream and determines the presence of a trigger signal based on the semantic understanding results.

[0033] When a trigger signal is detected in the user's audio stream, it indicates that the user wishes to interact with the session server of the target application or webpage via voice. At this point, the front-end determines whether to establish a session connection with the session server of the target application or webpage. This session connection is used exclusively for voice interaction between the user and the session server.

[0034] S200: Based on the session connection, send the user's session audio stream to the session server.

[0035] A session connection includes the session resources allocated by the session server to the decision-making frontend, as well as the transmission method of the session audio stream. Session connections are used for high-quality transmission of session audio streams.

[0036] The conversational audio stream comprises the user's audio stream acquired during voice interaction between the user and the conversation server. The conversational audio stream (Stream B) acts as a mediator, used for real-time voice interaction between the user and the conversation server.

[0037] During the process of establishing a session connection, the transmission method of the session resources and session audio streams that the front end obtains and interacts with the session server is determined.

[0038] During voice interaction, the front-end determines whether to record the user's voice questions and obtain the user's conversational audio stream. The front-end then sends the user's conversational audio stream to the conversation server according to the conversational resources and the transmission method of the conversational audio stream. The conversational connection is dedicated to real-time communication. Optionally, the conversational connection includes Web Real-Time Communication (WebRTC). Optionally, the transmission method of the conversational audio stream includes User Datagram Protocol (UDP) transmission. UDP transmission can significantly reduce the latency of the conversational audio stream in the network.

[0039] The front-end determines how to efficiently and accurately send the user's session audio stream to the session server based on the session resources allocated by the session server and the transmission method of the session audio stream, while also efficiently and accurately receiving the response text audio data of the session audio stream from the session server, so as to achieve efficient and accurate voice interaction between the user and the session server.

[0040] S300: Provides users with response text audio data of the conversational audio stream to enable voice interaction.

[0041] The session server performs speech parsing on the session audio stream to identify the user's voice interaction intent. Based on the identified voice interaction intent, the session server determines the response text audio data and sends the response text audio data to the decision front end.

[0042] The system determines the response text audio data received by the front end and plays it on the user's terminal to provide feedback to the user with the response text audio data of the conversation audio stream, thereby realizing voice interaction.

[0043] Furthermore, the system determines that the front-end sends the user's session audio stream to the session server in real time and receives the response text audio data of the session audio stream returned by the session server in real time. The system also determines that the front-end provides real-time feedback of the session audio stream response text audio data to the user, enabling real-time voice interaction between the user and the session server.

[0044] S400: Determines the presence of a modulation signal in the audio stream and modulates the voice interaction based on the modulation signal.

[0045] The front-end determines whether to acquire the recognition audio stream corresponding to the session audio stream while facilitating real-time voice interaction between the user and the session server. Optionally, the recognition audio stream and the session audio stream are acquired based on the same voice prompt (or the user's voice). The recognition audio stream is acquired simultaneously with the session audio stream.

[0046] Optionally, the recognition audio stream and the session audio stream have different formats. The recognition audio stream is used for low-power listening to the user's intent and does not participate in the actual call transmission. Both the recognition audio stream and the session audio stream are obtained based on the same user voice. Before the session connection is established, there is no voice interaction; only the recognition audio stream is generated based on the user's voice, and the user's voice interaction intent is listened to at low power based on the recognition audio stream. After the session connection is established, both the session audio stream (Stream B) and the recognition audio stream (Stream A) are obtained simultaneously based on the user's voice. The session audio stream is transmitted separately through the session connection with high quality to achieve efficient voice interaction between the user and the session server. The recognition audio stream is processed by additional resources and a server (e.g., a recognition server) to achieve real-time, low-power continuous listening to the user's voice interaction intent during voice interaction. When a modulation signal is detected in the recognition audio stream, the voice interaction is adjusted based on the modulation signal.

[0047] Optionally, the identified audio streams are all Pulse Code Modulation (PCM) data.

[0048] The audio stream is converted into text and keyword detection is performed. If a preset control word is detected in the text of the audio stream, it is considered that a control signal exists in the audio stream. The control signal is used to control the voice interaction between the user and the target application or target webpage. Controlling the voice interaction includes terminating the voice interaction, interrupting the voice interaction, and adjusting the mode of voice interaction.

[0049] The reason this invention processes the identification audio stream and the session audio stream separately is because their functions and transmission requirements differ. The session audio stream requires complete, error-free, and efficient transmission to achieve accurate and efficient voice interaction between the user and the session server. Therefore, this invention establishes an independent session connection for the session audio stream to achieve efficient and error-free transmission.

[0050] The audio stream is used to continuously monitor the user's voice interaction intent (including starting, pausing, and terminating voice interaction). The audio stream can be transmitted using low-power transmission methods. Before a session connection is established, the audio stream is monitored in real-time to construct the user's voice interaction intent with low power consumption. After the session connection is established, the audio stream continues to be monitored in real-time to control the user's voice interaction intent with low power consumption.

[0051] The voice interaction method provided in this invention establishes a session connection with a session server by determining that a trigger signal exists in the user's recognized audio stream; based on the session connection, the user's session audio stream is sent to the session server; response text audio data of the session audio stream is fed back to the user to achieve voice interaction; and a control signal is determined in the recognized audio stream, and the voice interaction is controlled based on the control signal. This invention separates the session audio stream and the recognized audio stream, processing them independently, thus decoupling the processing flow and resources of the session audio stream and the recognized audio stream. This avoids resource competition and processing bottlenecks between the session audio stream and the recognized audio stream, which is beneficial to improving the accuracy of voice interaction. The recognized audio stream is dedicated to recognizing and monitoring the user's session intent, while the session audio stream is dedicated to voice interaction, achieving independence between session monitoring and session interaction, and improving the accuracy of voice interaction.

[0052] Based on the above embodiments, determining that a trigger signal exists in the user's recognition audio stream and establishing a session connection with the session server includes the following steps: Obtain text recognition data from the audio stream; When the text recognition data matches the preset trigger words, it is determined that there is a trigger signal in the recognized audio stream; Send a session connection establishment request to the session server; Receive the response result of the session connection establishment request to complete the establishment of the session connection; the session connection is used to transmit session audio streams or response text audio data between the decision frontend and the session server.

[0053] The front-end determines whether to convert the audio stream into PCM data and transmit the PCM data to the recognition server for recognition via a WebSocket. Optionally, the front-end can transmit the PCM data of the audio stream to the recognition server through a recognition resource.

[0054] The recognition server obtains the audio stream based on PCM data, converts the audio stream into text, and obtains the text recognition data. The recognition server then returns the text recognition data to the judgment front end.

[0055] Optionally, the front-end performs text feature extraction on the received text recognition data to obtain text features. The front-end then sends the text features to the retrieval end.

[0056] The search engine maintains a local keyword database. Based on text characteristics, the search engine retrieves keywords from this database. The keyword database includes preset trigger words. These preset trigger words are determined based on the name of the target application or webpage. Optionally, preset trigger words may include the full name, abbreviation, or nickname of the target application or webpage. For example, if the name of the target application or webpage is A, preset trigger words might include "Hello, A," or "Little A Little A," or simply "A." Figure 2 As shown, if the retrieval end finds a text feature that matches a preset trigger word, the matching preset trigger word will be sent to the judgment front end.

[0057] After the front end receives a matching trigger word, it determines that the text recognition data matches the preset trigger word, and thus determines that there is a trigger signal in the recognized audio stream.

[0058] like Figure 2 As shown, when a trigger signal is detected in the audio stream, the front end determines to send a session connection establishment request to the session server in response to the trigger signal. Optionally, the session connection establishment request includes a WebRTC connection request.

[0059] Optionally, after receiving a session connection establishment request, the session server allocates session resources to the decision-making frontend and determines the transmission method of the session audio stream. The session server uses the allocated session resources and the determined transmission method of the session audio stream as the response result to the session connection establishment request, and sends this response result to the decision-making frontend. At this point, the session connection between the decision-making frontend and the session server is established.

[0060] This invention achieves accurate identification of trigger signals by matching preset trigger words. Upon confirming the presence of a trigger signal in the audio stream, a session connection is established, facilitating the transition between the speech recognition process and the speech conversation process. This invention allocates separate session resources to the session audio stream for independent transmission, ensuring accurate and efficient transmission of the session audio stream, thus improving the accuracy and efficiency of voice interaction.

[0061] Based on the above embodiments, determining the presence of a modulation signal in the audio stream includes the following steps: The audio stream to be recognized is sent to the recognition server; Receive and recognize text recognition data from audio streams; The keyword database is retrieved based on text recognition data, and the keyword database includes preset control words; When the text recognition data matches the preset control words, it is determined that there is a control signal in the recognized audio stream.

[0062] After the session connection is established, the session audio stream and response text audio data are transmitted through the session connection to enable voice interaction between the user and the session server. During the voice interaction, the front-end monitors and recognizes the audio stream in real time. The front-end then sends the recognized audio stream to the recognition server. The front-end converts the recognized audio stream into PCM data and transmits the PCM data of the recognized audio stream to the recognition server for recognition via a WebSocket.

[0063] The recognition server obtains the audio stream based on PCM data and converts it into text to obtain text recognition data. The recognition server then returns the text recognition data to the judgment frontend. Optionally, a local keyword database can be set up on the judgment frontend.

[0064] The front-end determines whether a control signal exists by searching a local keyword database based on the acquired text recognition data. This database includes preset control words, which may be manually set. Optionally, preset control words may include phrases like "don't speak," "quiet," "step back," "exit," and "goodbye." For example, the front-end performs text feature extraction on the text recognition data to obtain text features. It then searches the local keyword database based on these text features. If the text features match preset control words in the local keyword database, it is determined that a control signal exists in the recognized audio stream.

[0065] This invention uses a keyword database to retrieve preset control terms, enabling accurate identification of control signals. By expanding or narrowing the scope of the preset control terms, this invention achieves both precise and coarse retrieval of control signals, allowing for flexible adjustment of the detection requirements based on user needs.

[0066] Based on the above embodiments, controlling voice interaction based on control signals includes the following steps: The control signal is determined to be the termination signal; Send a termination command to the session server in response to the termination signal; the termination command carries the identifier of the session connection. Disconnect the session to terminate the voice interaction.

[0067] Preset control words include preset termination words. Preset termination words include words that contain the semantics of terminating a session connection. For example, preset termination words include "step down," "exit," and "goodbye."

[0068] When the front end determines that a pre-defined terminator is found in the keyword database based on the text recognition data, it is confirmed that there is a termination signal in the recognized audio stream.

[0069] The front-end sends a termination command to the session server in response to the termination signal. The termination command carries the session connection identifier. The session connection identifier may include the session connection number or the user terminal number, etc.

[0070] Furthermore, upon receiving the termination command, the session server disconnects the session connection based on the session connection identifier carried in the termination command.

[0071] After determining that the front end has sent a termination command, the session connection is disconnected to terminate the voice interaction.

[0072] This invention, upon detecting a termination signal, can promptly notify the session server to terminate the voice interaction by sending a termination command. Timely disconnection of the session connection upon detecting a termination signal helps improve the utilization rate of the session connection.

[0073] Based on the above embodiments, controlling voice interaction based on control signals includes the following steps: The control signal is determined to be an interrupt signal; Send an interrupt command to the session server in response to the interrupt signal; the interrupt command carries the identifier of the session connection. Suspend the use of the session connection to interrupt voice interaction until a trigger signal is detected in the recognized audio stream, at which point the use of the session connection will be resumed.

[0074] Preset control words include preset interruption words. Preset interruption words include words containing the semantics of interrupting the session connection. For example, preset interruption words include "quiet," "don't talk," "wait a moment," and "stop."

[0075] When the front end determines that a pre-defined interruption word is found in the keyword database based on the text recognition data, it is determined that there is an interruption signal in the audio stream.

[0076] The front end determines whether to send an interrupt command to the session server in response to the interrupt signal. The interrupt command carries an identifier of the session connection.

[0077] Furthermore, upon receiving an interruption command, the session server suspends the use of the session connection based on the session connection identifier carried in the interruption command.

[0078] After determining that the front end has sent an interrupt command, the session connection is suspended to interrupt (or pause) the voice interaction.

[0079] Furthermore, after the voice interaction is interrupted, it is determined that the front end is still continuously listening to the user's voice and acquiring the recognition audio stream. Optionally, if it is determined that the front end detects the recognition audio stream, it sends the recognition audio stream to the recognition server in real time.

[0080] The recognition server performs semantic understanding on the audio stream and converts it into text recognition data. The recognition server then sends the text recognition data to the judgment front end.

[0081] The front-end determines whether to obtain text recognition data from the audio stream sent by the recognition server. It then searches the keyword database based on this text recognition data. If the front-end finds a preset trigger word in the text recognition data, it confirms the presence of a trigger signal in the audio stream. The front-end then sends a recovery command to the session server, carrying an identifier of the session connection, informing the session server that voice interaction can be resumed based on the session connection. After sending the recovery command, the front-end restores the session connection to resume voice interaction.

[0082] This invention promptly interrupts voice interaction upon detecting an interruption signal, flexibly adapting to the user's voice interaction needs. Even after the interruption, it continues to monitor and recognize the audio stream, capturing the user's conversational needs in real time. Upon recognizing a trigger signal in the audio stream, it promptly restores the conversation connection, responding to the user's conversational requests in a timely manner.

[0083] like Figure 2 and Figure 3 As shown, the present invention also provides a voice interaction method applied to a session server, including steps S500 to S800. The specific steps are as follows.

[0084] S500: Based on session connection, it receives the user's session audio stream.

[0085] A session connection is established when a trigger signal is present in the user's recognized audio stream.

[0086] Before establishing a session connection, the audio stream generated by the user is acquired to obtain the recognition audio stream (Stream A). The recognition audio stream is used for low-power continuous monitoring. Once a trigger signal is detected in the recognition audio stream, a high-power session connection is established to achieve efficient voice interaction.

[0087] Determine if a trigger signal exists in the user's audio stream. Convert the audio stream to text and perform keyword detection. If a preset trigger word is detected in the text of the audio stream, it is considered that a trigger signal exists in the audio stream.

[0088] Optionally, the front end performs semantic understanding on the recognized audio stream and determines the presence of a trigger signal based on the semantic understanding results.

[0089] When a trigger signal is detected in the user's audio stream, it indicates that the user wants to interact with the session server of the target application or webpage via voice. At this point, the front-end determines whether to establish a session connection with the session server of the target application or webpage.

[0090] During the process of establishing a session connection, the transmission method of the session resources and session audio streams that the front end obtains and interacts with the session server is determined.

[0091] During voice interaction, the front-end determines whether to record the user's voice questions and obtain the user's conversation audio stream. The front-end then determines whether to send the user's conversation audio stream to the conversation server using the appropriate conversation resources and transmission method.

[0092] The session server efficiently and accurately receives and judges the session audio stream sent by the front end through session connections.

[0093] S600: Determine response text audio data based on the session audio stream.

[0094] The session server performs speech parsing on the session audio stream to identify the user's voice interaction intent. Based on the identified voice interaction intent, the session server determines the response text audio data.

[0095] S700: Sends response text and audio data to the front end to enable voice interaction.

[0096] The session server sends response text and audio data to the front end through a session connection to enable voice interaction.

[0097] S800: Determines and controls voice interaction.

[0098] Voice interaction is regulated when the presence of a modulation signal in the audio stream is recognized.

[0099] The front-end determines whether to acquire the recognition audio stream corresponding to the session audio stream while facilitating real-time voice interaction between the user and the session server. Optionally, the recognition audio stream and the session audio stream are acquired based on the same voice prompt (or user voice). The recognition audio stream is acquired simultaneously with the session audio stream.

[0100] When the front end detects a control signal in the audio stream, it sends a control command (including a termination command or an interrupt command) to the session server.

[0101] The session server controls the voice interaction based on the received control commands. Optionally, when the session server receives a termination command, it disconnects the session connection. Optionally, when the session server receives an interrupt command, it suspends the use of the session connection (including temporarily blocking the session audio stream transmitted through the session connection and suspending the transmission of response text audio data through the session connection).

[0102] The voice interaction method provided in this invention receives a user's conversational audio stream based on a session connection. The session connection is established when a trigger signal is present in the user's recognized audio stream. Response text audio data is determined based on the session audio stream. The response text audio data is sent to a determination front-end to achieve voice interaction. Voice interaction is controlled when a control signal is present in the recognized audio stream. This invention separates the session audio stream and the recognized audio stream, processing them independently. This decouples the processing flow and resources of the session audio stream and the recognized audio stream, avoiding resource competition and processing bottlenecks between them, and improving the accuracy of voice interaction. The recognized audio stream is dedicated to recognizing and monitoring the user's conversational intent, while the session audio stream is dedicated to voice interaction, achieving independence between session monitoring and session interaction, thus improving the accuracy of voice interaction.

[0103] Based on the above embodiments, determining the response text audio data based on the session audio stream includes the following steps: Convert the audio stream into text-based session data; Perform semantic understanding on text conversation data, and determine the text response data based on the semantic understanding results; Convert the text response data into the target audio stream; Based on the target audio stream and text response data, determine the response text audio data.

[0104] During voice interaction, it is determined that the front end transmits the session audio stream to the session server through the session connection.

[0105] The session server converts the session audio stream into text session data. The session server performs semantic understanding on the text session data and determines the text response data based on the semantic understanding results. Optionally, the session server inputs the text session data into a language model and performs semantic understanding on the text session data through the language model. The session server obtains the text response data output by the language model. The session server converts the text response data into a target audio stream and determines the response text audio data based on the target audio stream (audio format) and the text response data (text format).

[0106] This invention utilizes semantic understanding of text conversation data to determine text response data, enabling accurate extraction of user voice interaction content. Simultaneously, based on the target audio stream and text response data, it determines the response text audio data, allowing the results of the voice interaction to be presented to the user in both audio and text formats. This enables users to more accurately understand the content of the voice interaction, improving the accuracy and efficiency of voice interaction.

[0107] like Figure 2As shown, this invention also provides a voice interaction system, including a determination front-end, a recognition server, a session server, and a retrieval terminal including a local keyword database. The determination front-end is located on a user terminal (e.g., a mobile phone, computer, etc.). The voice interaction process according to the voice interaction system includes local monitoring (first stage), establishing a session connection (second stage), dual-stream collaborative interaction (third stage), and controlling voice interaction (fourth stage). The dual-stream collaborative interaction includes the interactive processing of the session audio stream and the interactive processing of the recognition audio stream. The specific voice interaction process is as follows.

[0108] Step 1: Determine if the front end is listening to the user's voice in real time and collecting and recognizing the audio stream in real time.

[0109] Step 2: Determine whether the front end will send the recognized audio stream to the recognition server.

[0110] Step 3: The recognition server converts the audio stream into text recognition data and returns the text recognition data to the judgment front end.

[0111] Step 4: The front-end determines whether to extract text features from the text recognition data. Optionally, the front-end may extract keywords from the text recognition data to obtain text features.

[0112] Step 5: Determine whether the front end will send the text features to the retrieval end.

[0113] Step 6: The retrieval end searches its local keyword database based on text features. The keyword database includes preset trigger words. If the text features match the preset trigger words, the matching preset trigger words are sent to the judgment frontend.

[0114] Step 7: After the front end receives the matching preset trigger word, it is determined that there is a trigger signal in the audio stream.

[0115] This completes the first phase of the local monitoring process.

[0116] Step 8: After determining that the front end has identified a trigger signal in the audio stream, it sends a session connection establishment request to the session server.

[0117] Step 9: The session server returns a response to the session connection establishment request to the front end (including the transmission method of the session resources and session audio streams that were interacted with by the session server).

[0118] This completes the second phase of establishing a session connection.

[0119] Step 10: After the session connection is established, the user sends a voice question to start the voice interaction.

[0120] Step 11: Determine if the front-end obtains the conversation audio stream based on the user's voice question.

[0121] Step 12: Determine that the front end is transmitting the session audio stream to the session server in high quality through the session connection.

[0122] Step 13: The session server performs speech understanding on the session audio stream and determines the response text audio data of the session audio stream. The session server sends the response text audio data of the session audio stream to the decision frontend through the session connection.

[0123] Furthermore, the front end determines the response text audio data (including playing the target audio stream and displaying text response data) to the user to enable voice interaction between the user and the session server.

[0124] Step 14: During the voice interaction process, determine the front end to obtain the recognized audio stream based on the user's voice question.

[0125] Step 15: Determine whether the front end will send the recognized audio stream to the recognition server.

[0126] Step 16: The recognition server converts the audio stream into text recognition data and sends the text recognition data to the judgment front end.

[0127] Step 17: Determine if the front end performs text feature extraction on the text recognition data.

[0128] Step 18: Determine whether the front end will send the extracted text features to the retrieval end.

[0129] Step 19: The retrieval end searches its local keyword database based on text features. The keyword database includes preset control words. If the text features match the preset control words, the matching preset control words are sent to the judgment front end.

[0130] Step 20: The front end determines whether there is a control signal in the audio stream based on the matched preset control words.

[0131] This completes the third phase of the dual-flow collaborative interaction process.

[0132] Step 21: Determine whether the front end sends a termination command or an interrupt command to the session server based on the control signal.

[0133] Step 22: Determine whether the front end terminates or interrupts the session connection based on the control signal.

[0134] This completes the fourth stage of the process of regulating voice interaction.

[0135] This invention physically separates the recognition audio stream from the conversational audio stream, enabling the voice interaction system to maintain high command sensitivity while playing audio. The judgment front-end of this invention, through local AI judgment combined with a keyword database, eliminates the need for cloud-based operations such as wake-up, interruption, and mute, achieving millisecond-level response times and improving the efficiency and accuracy of monitoring the recognition audio stream. When matching text recognition data or text features of the recognition audio stream, local AI fuzzy matching can be used, effectively handling environmental noise and speech deviations. This invention reduces data (recognition audio stream) transmission overhead and enhances privacy and security by processing the recognition audio stream through the judgment front-end.

[0136] The voice interaction device provided by the present invention will be described below. The voice interaction device described below can be referred to in correspondence with the voice interaction method described above.

[0137] like Figure 4 As shown, the present invention provides a voice interaction device, comprising: Trigger module 401 is used to determine that there is a trigger signal in the user's recognition audio stream and establish a session connection with the session server; The first sending module 402 is used to send the user's session audio stream to the session server based on the session connection; Feedback module 403 is used to provide the user with response text audio data of the conversation audio stream in order to achieve voice interaction; The control module 404 is used to determine the presence of a control signal in the audio stream and to control the voice interaction based on the control signal.

[0138] The voice interaction device provided in this invention establishes a session connection with a session server by determining that a trigger signal exists in the user's recognized audio stream; based on the session connection, it sends the user's session audio stream to the session server; it provides the user with response text and audio data of the session audio stream to achieve voice interaction; and it determines that a control signal exists in the recognized audio stream and controls the voice interaction based on the control signal. This invention separates the session audio stream and the recognized audio stream, processing them independently, thus decoupling the processing flow and resources of the session audio stream and the recognized audio stream. This avoids resource competition and processing bottlenecks between the session audio stream and the recognized audio stream, which is beneficial to improving the accuracy of voice interaction. The recognized audio stream is dedicated to recognizing and monitoring the user's session intent, while the session audio stream is dedicated to voice interaction, achieving independence between session monitoring and session interaction, thereby improving the accuracy of voice interaction.

[0139] In one embodiment, the control module 404 is configured to: send the recognized audio stream to the recognition server; receive text recognition data of the recognized audio stream; search a keyword library based on the text recognition data, the keyword library including preset control words; and determine that there is a control signal in the recognized audio stream if the text recognition data and the preset control words match.

[0140] In one embodiment, the triggering module 401 is used to: acquire text recognition data of the recognized audio stream; determine that a trigger signal exists in the recognized audio stream if the text recognition data matches a preset trigger word; send a session connection establishment request to the session server; receive the response result of the session connection establishment request to complete the establishment of the session connection; the session connection is used to transmit the session audio stream or the response text audio data between the determination front-end and the session server.

[0141] In one embodiment, the control module 404 is configured to: determine that the control signal is a termination signal; send a termination instruction to the session server in response to the termination signal, the termination instruction carrying an identifier of the session connection; and disconnect the session connection to terminate the voice interaction.

[0142] In one embodiment, the control module 404 is configured to: determine that the control signal is an interrupt signal; send an interrupt command to the session server in response to the interrupt signal, the interrupt command carrying an identifier of the session connection; suspend the use of the session connection to interrupt voice interaction until a trigger signal is determined to exist in the recognized audio stream, and then resume the use of the session connection.

[0143] like Figure 5 As shown, the present invention provides a voice interaction device, comprising: The receiving module 501 is used to receive the user's session audio stream based on a session connection; the session connection is established when a trigger signal is present in the user's recognized audio stream. The first determining module 502 is used to determine the response text audio data based on the session audio stream; The second sending module 503 is used to send response text audio data to the judgment front end to realize voice interaction; The second determining module 504 is used to determine the control voice interaction; the voice interaction is controlled when the presence of a control signal in the audio stream is recognized.

[0144] The voice interaction device provided in this invention receives a user's conversational audio stream based on a session connection. The session connection is established when a trigger signal is present in the user's recognized audio stream. Response text audio data is determined based on the session audio stream. The response text audio data is sent to a determination front-end to achieve voice interaction. Voice interaction is controlled when a control signal is present in the recognized audio stream. This invention separates the session audio stream and the recognized audio stream, processing them independently. This decouples the processing flow and resources of the session audio stream and the recognized audio stream, avoiding resource competition and processing bottlenecks between them, and improving the accuracy of voice interaction. The recognized audio stream is dedicated to recognizing and monitoring the user's conversational intent, while the session audio stream is dedicated to voice interaction, achieving independence between session monitoring and session interaction, thus improving the accuracy of voice interaction.

[0145] In one embodiment, the first determining module 502 is configured to: convert the conversational audio stream into text conversational data; perform semantic understanding on the text conversational data, and determine the text response data of the text conversational data based on the semantic understanding result; convert the text response data into a target audio stream; and determine the response text audio data based on the target audio stream and the text response data.

[0146] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include a processor 610, a communications interface 620, a memory 630, and a communication bus 640. The processor 610, communications interface 620, and memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a voice interaction method. This method includes: determining that a trigger signal exists in the user's recognized audio stream and establishing a session connection with a session server; based on the session connection, sending the user's session audio stream to the session server; providing the user with response text audio data of the session audio stream to achieve voice interaction; and determining that a control signal exists in the recognized audio stream and controlling the voice interaction based on the control signal.

[0147] Alternatively, a voice interaction method may be executed, comprising: receiving a user's session audio stream based on a session connection; the session connection being established when a trigger signal is present in the user's recognized audio stream; determining response text audio data based on the session audio stream; sending the response text audio data to a determination front end to achieve voice interaction; and determining the control of the voice interaction; the voice interaction being controlled when a control signal is present in the recognized audio stream.

[0148] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0149] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When executed by a processor, the computer program implements the voice interaction method provided by the above methods. The method includes: determining that a trigger signal exists in the user's recognized audio stream, and establishing a session connection with a session server; based on the session connection, sending the user's session audio stream to the session server; feeding back response text audio data of the session audio stream to the user to realize voice interaction; determining that a control signal exists in the recognized audio stream, and controlling the voice interaction based on the control signal.

[0150] Alternatively, when the computer program is executed by a processor, it implements the voice interaction method provided by the methods described above, the method comprising: receiving a user's session audio stream based on a session connection; the session connection being established when a trigger signal is present in the user's recognized audio stream; determining response text audio data based on the session audio stream; sending the response text audio data to a determination front end to realize voice interaction; determining and controlling the voice interaction; the voice interaction being controlled when a control signal is present in the recognized audio stream.

[0151] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0152] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0153] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A voice interaction method, characterized in that, Applied to the front end for decision-making, including: Once a trigger signal is detected in the user's recognition audio stream, a session connection is established with the session server. Based on the session connection, the user's session audio stream is sent to the session server; The system provides the user with response text audio data of the conversation audio stream to enable voice interaction. The presence of a modulation signal in the identified audio stream is determined, and the voice interaction is modulated based on the modulation signal.

2. The voice interaction method according to claim 1, characterized in that, The step of determining that the identified audio stream contains a modulation signal includes: The identified audio stream is sent to the identification server; Receive the text recognition data of the identified audio stream; A keyword database is retrieved based on the text recognition data; the keyword database includes preset control words. If the text recognition data matches the preset control word, it is determined that the recognized audio stream contains the control signal.

3. The voice interaction method according to claim 1, characterized in that, The step of determining that a trigger signal exists in the user's recognized audio stream and establishing a session connection with the session server includes: Obtain the text recognition data of the audio stream being recognized; If the text recognition data matches the preset trigger word, it is determined that the recognized audio stream contains the trigger signal; Send a session connection establishment request to the session server; The system receives a response to a session connection establishment request to complete the establishment of the session connection. The session connection is used to transmit the session audio stream or the response text audio data between the determination front-end and the session server.

4. The voice interaction method according to claim 1, characterized in that, The control of the voice interaction based on the control signal includes: The control signal is determined to be a termination signal; Send a termination instruction to the session server in response to the termination signal; the termination instruction carries an identifier of the session connection. Disconnect the session connection to terminate the voice interaction.

5. The voice interaction method according to claim 1, characterized in that, The control of the voice interaction based on the control signal includes: The control signal is determined to be an interrupt signal; Send an interrupt command to the session server in response to the interrupt signal; the interrupt command carries an identifier of the session connection. The session connection is suspended to interrupt the voice interaction until the trigger signal is detected in the recognized audio stream, at which point the session connection is resumed.

6. A voice interaction method, characterized in that, Applied to session servers, including: Based on a session connection, the system receives the user's session audio stream; the session connection is established when a trigger signal is present in the user's identified audio stream. Determine the response text audio data based on the session audio stream; The response text and audio data are sent to the front end for judgment to achieve voice interaction; Determine to regulate the voice interaction; the voice interaction is regulated when a regulation signal is present in the identified audio stream.

7. The voice interaction method according to claim 6, characterized in that, The step of determining the response text audio data based on the session audio stream includes: Convert the audio stream into text session data; Perform semantic understanding on the text conversation data, and determine the text response data of the text conversation data based on the semantic understanding results; Convert the text response data into a target audio stream; The response text audio data is determined based on the target audio stream and the text response data.

8. A voice interaction device, characterized in that, include: The trigger module is used to determine that a trigger signal exists in the user's recognition audio stream and establish a session connection with the session server; The first sending module is used to send the user's session audio stream to the session server based on the session connection; The feedback module is used to provide the user with response text audio data of the conversation audio stream in order to achieve voice interaction; The control module is used to determine that there is a control signal in the recognized audio stream, and to control the voice interaction based on the control signal.

9. A voice interaction device, characterized in that, include: The receiving module is used to receive the user's session audio stream based on the session connection; The session connection is established when a trigger signal is present in the user's identified audio stream; The first determining module is used to determine response text audio data based on the session audio stream; The second sending module is used to send the response text audio data to the judgment front end to realize voice interaction; The second determining module is used to determine the regulation of the voice interaction; the voice interaction is regulated when a regulation signal exists in the identified audio stream.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the voice interaction method as described in any one of claims 1 to 7.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the voice interaction method as described in any one of claims 1 to 7.

Images