Audio interaction processing method and system, server, client and electronic equipment
Through direct interaction between the audio processing server and the text processing server, real-time conversion and transmission of reply text and audio streams, the real-time text and voice reply synchronization problem of audio input in human-computer voice interaction is solved, and the user experience and reply efficiency are improved.
Patent Information
- Application Number
- CN202480002169.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-09
- Publication Date
- 2024-12-06
AI Technical Summary
In human-computer voice interaction scenarios, it is difficult for the prior art to realize real-time text conversion of input audio and synchronize voice reply, resulting in poor user experience.
Through the audio processing server, the audio inputted by the client is converted into the text to be replied, and directly interacts with the text processing server to generate a reply text stream and a reply audio stream, which is transmitted to the client in real time for display and playback, improving text and The efficiency and timeliness of voice reply.
It realizes synchronous processing of real-time text and voice reply of input audio, improves user experience, improves the efficiency and timeliness of reply, and enhances the timeliness and efficiency of display and playback on the client.
Smart Images

Figure CN119096295A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer technology, and in particular to a method, system, server, client, and electronic device for processing audio interaction. Background Art
[0002] With the development of Internet technology, human-computer voice interaction technology is being applied in more and more scenarios, such as smart home, smart customer service, smart assistant, etc.
[0003] In some scenarios of human-computer voice interaction, when a user inputs voice, not only will the robot respond with voice, but the robot will also generate corresponding text based on the user's voice input, and the robot's reply text will also be displayed accordingly, allowing users to receive replies through text and voice simultaneously, better understand the content of the reply and improve user experience. Summary of the invention
[0004] This summary is provided to introduce concepts in a brief form that will be described in detail in the detailed description below. This summary is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.
[0005] According to some embodiments of the present disclosure, a method for processing audio interaction is provided, which is executed by an audio processing server, including: converting input audio sent by a client into text to be replied; sending the text to be replied to a text processing server to generate a reply text stream; receiving a request from the client for a reply audio stream corresponding to the reply text stream, wherein the request includes a message identifier corresponding to the reply text stream, and the message identifier is sent by the text processing server to the client; obtaining the reply text stream from the text processing server according to the message identifier; generating a reply audio stream according to the reply text stream; and sending the reply audio stream to the client for playback.
[0006] According to some other embodiments of the present disclosure, a method for processing audio interaction is provided, which is executed by a client, and includes: sending input audio to an audio processing server, converting it into text to be replied, wherein the text to be replied is sent to a text processing server to generate a reply text stream; receiving a message identifier corresponding to the reply text stream sent by the text processing server; sending a request for a reply audio stream corresponding to the reply text stream to the audio processing server, wherein the request includes the message identifier; receiving the reply audio stream sent by the audio processing server; and playing the reply audio stream.
[0007] According to some other embodiments of the present disclosure, an audio processing server is provided, including: a conversion module, configured to convert the input audio sent by the client into a text to be replied; a first sending module, configured to send the text to be replied to the text processing server, and generate a reply text stream; a receiving module, configured to receive the client's request for a reply audio stream corresponding to the reply text stream, wherein the request includes a message identifier corresponding to the reply text stream, and the message identifier is sent by the text processing server to the client; an acquisition module, configured to acquire the reply text stream from the text processing server according to the message identifier; a generation module, configured to generate a reply audio stream according to the reply text stream; and a second sending module, configured to send the reply audio stream to the client for playback.
[0008] According to some further embodiments of the present disclosure, a client is provided, comprising: a first sending module, configured to send input audio to an audio processing server to convert it into text to be replied, wherein the text to be replied is sent to a text processing server to generate a reply text stream; a first receiving module, configured to receive a message identifier corresponding to the reply text stream sent by the text processing server; a second sending module, configured to send a request for a reply audio stream corresponding to the reply text stream to the audio processing server, wherein the request includes the message identifier; a second receiving module, configured to receive the reply audio stream sent by the audio processing server; and a playing module, configured to play the reply audio stream.
[0009] According to some further embodiments of the present disclosure, there is provided an audio interaction processing system, comprising: an audio processing server of any embodiment of the present disclosure and a client of any embodiment of the present disclosure.
[0010] According to some further embodiments of the present disclosure, there is provided an electronic device, comprising: a processor; and a memory coupled to the processor, for storing instructions, which, when executed by the processor, causes the processor to execute the audio interaction processing method of any embodiment of the present disclosure.
[0011] According to some further embodiments of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method for processing audio interaction of any embodiment of the present disclosure is performed.
[0012] According to some further embodiments of the present disclosure, a computer program product is provided, comprising: instructions, which, when executed by a processor, implement the audio interaction processing method of any embodiment of the present disclosure.
[0013] According to some further embodiments of the present disclosure, a computer program is provided, comprising: instructions, which, when executed by a processor, implement the audio interaction processing method of any embodiment of the present disclosure.
[0014] Other features, aspects and advantages of the present disclosure will become apparent from the following detailed description of exemplary embodiments of the present disclosure with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] The preferred embodiments of the present disclosure are described below with reference to the accompanying drawings. The drawings described herein are used to provide a further understanding of the present disclosure. Each of the drawings together with the following detailed description is included in this specification and forms a part of the specification, and is used to explain the present disclosure. It should be understood that the drawings described below only relate to some embodiments of the present disclosure and do not constitute a limitation on the present disclosure. In the drawings:
[0016] Figure 1 A schematic diagram showing a flow chart of a method for processing audio interaction in some embodiments of the present disclosure;
[0017] Figure 2 A schematic diagram showing a flow chart of a method for processing audio interaction according to other embodiments of the present disclosure;
[0018] Figure 3 A schematic diagram showing retransmission of audio data packets by an audio processing server in some embodiments of the present disclosure;
[0019] Figure 4 A schematic diagram showing a flow chart of a method for processing audio interaction according to yet other embodiments of the present disclosure;
[0020] Figure 5 A schematic diagram showing audio data packet retransmission by a client in some embodiments of the present disclosure;
[0021] Fig. 6A A schematic diagram showing the information interaction process in an ASR message link of some embodiments of the present disclosure;
[0022] Figure 6B A schematic diagram showing the information interaction process in a TTS message link of some embodiments of the present disclosure;
[0023] Figure 6C A schematic diagram showing the information interaction flow of the audio interaction processing method of some embodiments of the present disclosure;
[0024] Figure 7 A schematic diagram showing the structure of an audio processing server in some embodiments of the present disclosure;
[0025] Figure 8 A schematic diagram showing the structure of a client in some embodiments of the present disclosure;
[0026] Fig. 9 A schematic diagram showing the structure of an electronic device according to some embodiments of the present disclosure;
[0027] Fig.10A schematic diagram showing the structure of a computer system according to some embodiments of the present disclosure;
[0028] Fig.11 A schematic diagram showing the structure of an audio interaction processing system according to some embodiments of the present disclosure.
[0029] It should be understood that, for ease of description, the sizes of the various parts shown in the drawings are not necessarily drawn according to the actual proportional relationship. The same or similar reference numerals are used in the various drawings to represent the same or similar parts. Therefore, once an item is defined in one drawing, it may not be further discussed in subsequent drawings. DETAILED DESCRIPTION
[0030] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure, but it is obvious that the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. The following description of the embodiments is actually only illustrative and is by no means intended to limit the present disclosure and its application or use. It should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein.
[0031] It should be understood that the various steps recorded in the method embodiments of the present disclosure can be performed in different orders, and / or performed in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown in the execution. The scope of the present disclosure is not limited in this respect. Unless otherwise specifically stated, the relative arrangement, numerical expressions and numerical values of the parts and steps set forth in these embodiments should be interpreted as being merely exemplary and do not limit the scope of the present disclosure.
[0032] The term "include" and its variations used in the present disclosure mean an open term that includes at least the following elements / features but does not exclude other elements / features, that is, "including but not limited to". In addition, the term "include" and its variations used in the present disclosure mean an open term that includes at least the following elements / features but does not exclude other elements / features, that is, "including but not limited to". Therefore, include is synonymous with include. The term "based on" means "based at least in part on".
[0033] References throughout this specification to "one embodiment," "some embodiments," or "an embodiment" mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. For example, the term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; and the term "some embodiments" means "at least some embodiments." Furthermore, the appearances of the phrases "in one embodiment," "in some embodiments," or "in an embodiment" in various places throughout this specification do not necessarily all refer to the same embodiment, but may do so.
[0034] It should be noted that the concepts of "first", "second", etc. mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units. Unless otherwise specified, the concepts of "first", "second", etc. are not intended to imply that the objects described in this way must be in a given order in time, space, ranking, or any other manner.
[0035] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".
[0036] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.
[0037] The embodiments of the present disclosure are described in detail below in conjunction with the accompanying drawings, but the present disclosure is not limited to these specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. In addition, in one or more embodiments, specific features, structures or characteristics can be combined in any suitable manner that will be clear from the present disclosure by a person of ordinary skill in the art.
[0038] When users interact with robots, they can interact based on text, voice, or a combination of the two. In some scenarios, the user's voice (audio) input can be converted into text in real time for display, and the robot can reply with text and voice at the same time, improving the user's interactive experience.
[0039] For the robot to reply with text and voice at the same time, it is necessary to configure a text processing server that can process text and an audio processing server that can process audio. The timeliness and synchronization of the robot's text and voice replies directly affect the user experience.
[0040] The present invention provides a method for processing audio interaction to improve the efficiency and timeliness of text and voice replies to input audio.
[0041] Combine the following Figure 1-6C Some embodiments of the audio interaction processing method disclosed in the present invention are described.
[0042] Figure 1 FIG. 1 is a structural diagram of some embodiments of the audio interaction processing method disclosed in the present invention. Figure 1 As shown, the method of this embodiment includes: steps S102 to S112. The method of this embodiment can be executed by an audio processing server. The audio processing server can be implemented by hardware or software, such as a server or application that can implement audio processing.
[0043] In step S102, the input audio sent by the client is converted into text to be replied.
[0044] The client can be implemented by hardware or software, for example, it can implement a terminal or an application set in the terminal. In the human-computer interaction scenario, the user can input audio through the client, and the client can transcode the audio file into multiple audio segments, and then compress the multiple audio segments and use a queue for local storage, and then use streaming to send them to the audio processing server for audio recognition and conversion into text to be replied. The audio processing server can be implemented based on a machine learning model. For example, a machine learning model can realize two-way conversion between speech and text. For example, the audio processing server is a server with ASR (Automatic Speech Recognition) and TTS (Text To Speech) functions.
[0045] In step S104, the text to be replied is sent to the text processing server to generate a reply text stream.
[0046] The audio processing server sends the text to be replied directly to the text processing server. Compared with the method in which the text to be replied is sent to the client and then sent to the text processing server by the client, the efficiency of the text processing server in obtaining the text to be replied is improved, thereby improving the overall processing efficiency.
[0047] The text processing server is responsible for generating a reply text stream based on the text to be replied. The text processing server can be implemented based on a machine learning model, such as a large language model (LLM) or a foundation model. For example, the text processing server can stream out reply text fragments generated in real time, that is, generate a reply text stream, which can improve the timeliness and efficiency of reply text generation. There is no need to wait until the entire content of the reply text is generated before providing feedback, thereby improving the user experience.
[0048] In step S106, a request from the client for a reply audio stream corresponding to the reply text stream is received.
[0049] For example, the request includes a message identifier corresponding to the reply text stream, and the message identifier is sent by the text processing server to the client. After receiving the text to be replied, the text processing server can generate a corresponding message identifier and send it to the client. The message identifier is used to identify the reply text stream generated for the text to be replied. After receiving the message identifier, the client generates a request for a reply audio stream, which is used to request the audio processing server to reply the audio stream.
[0050] In step S108, a reply text stream is obtained from the text processing server according to the message identifier.
[0051] The audio processing server can directly obtain the reply text stream from the text processing server according to the message identifier. Compared with the text server sending the reply text stream to the client, and then the client sending the reply text stream to the audio processing server, this improves the efficiency of the audio processing server in obtaining the reply text stream, and thus improves the efficiency of the audio processing server in subsequently generating reply audio.
[0052] The audio processing server obtains the reply text stream from the text processing server. That is, the text processing server uses streaming transmission to transmit the generated reply text fragments to the audio processing server in real time, instead of generating the entire reply text and then sending it to the audio processing server. This improves the transmission efficiency and the subsequent processing efficiency of the audio processing server.
[0053] In step S110, a reply audio stream is generated according to the reply text stream.
[0054] The audio processing server converts each reply text segment in the received reply text stream into audio in real time, and streams the reply audio stream.
[0055] In step S112, the reply audio stream is sent to the client for playing.
[0056] The audio processing server streams out the reply audio segments generated in real time to form a reply audio stream that is transmitted to the client for real-time playback. There is no need to generate all the reply audio and then return it to the client for playback.
[0057] The method of the above embodiment provides an overall implementation scheme for audio interaction. The audio processing server receives the audio input by the client and converts it into the text to be replied, and then directly interacts with the text processing server. On the one hand, the text to be replied is directly sent to the text processing server to generate a reply text stream. On the other hand, the reply text stream is directly obtained from the text processing server, and a reply audio stream is generated and returned to the client for playback. The audio processing server communicates directly with the text processing server. Compared with the method of forwarding the text to be replied and the reply text stream through the client, the transmission efficiency of the text to be replied and the reply text stream is improved, so that the text processing server can obtain the text to be replied in time, and the audio processing server can obtain the reply audio stream in time, thereby improving the generation efficiency of the reply text stream and the reply audio stream. In addition, the reply text and reply audio are generated and transmitted in a streaming manner throughout the process, so that feedback can be given in time for the input audio, and the efficiency and timeliness of text and voice replies to the input audio are improved.
[0058] In addition, since the audio processing server obtains the reply audio stream from the text processing server in real time and generates a reply audio stream, the synchronization effect of text and voice replies on the client can be improved, thereby improving the user experience.
[0059] The following further describes the audio interaction processing method of the present disclosure in conjunction with some specific embodiments.
[0060] In some embodiments, the reply text stream includes multiple text data packets, and the audio processing server periodically pulls the current text data packet generated by the text processing server from the text processing server based on the message identifier, or receives the current text data packet generated and pushed by the text processing server, wherein each of the multiple text data packets includes a message identifier.
[0061] The text processing server generates text data packets in a streaming manner and uses streaming transmission to form a reply text stream. For example, the entire reply text generated by the text processing server is "Hello, it's nice to chat with you". By using streaming to generate text data packets, multiple data packets can be generated for this sentence, for example, "Hello", "Nice", "With you", and "Chatting" are respectively used as the contents of different data packets. The streaming generation and transmission method makes the reply more timely, more efficient, and provides a better user experience. Each text data packet includes a message identifier (message_id), so that the audio processing server can accurately identify each text data packet that is a reply to the input audio. The audio processing server can use the message identifier to poll to obtain each text data packet, or the text processing server can push each text data packet to the audio processing server after generating it.
[0062] Pulling the reply audio stream by the audio processing server can reduce the intrusion into the message link and improve the stability of the overall link. The audio processing server relatively isolates the ASR and TTS links. If the reply text stream is pushed through the text processing server, it is necessary to cooperate with the audio processing server to provide active push capabilities, and the cost of modifying the existing link is high.
[0063] In some embodiments, the audio processing server obtains a first preset amount of text before the reply text stream from the text processing server, generates a current audio segment corresponding to the current text data packet based on the text segment in the current text data packet, the text segment in one or more text data packets before the current text data packet, and the first preset amount of text, and generates the current audio data packet in the reply audio stream.
[0064] Each round of interaction process includes client input audio, conversion to text to be replied, generation of reply text stream, and generation of reply audio stream. For each round of interaction process, the audio processing server can obtain the first preset amount of text from the previous round or multiple rounds of conversations, and combine the text fragments in the current text data packet and the text fragments before the current text data packet to generate the current audio fragment corresponding to the current text data packet. The audio processing server can generate the current audio fragment in combination with the context of multiple rounds of conversations, which can better understand the semantics and context, improve the accuracy of audio generation, and improve the accuracy of reading aloud.
[0065] Since the reply text is generally stored on the text processing server, the text of one or more rounds of historical interactions can be obtained from the text processing server. If the audio processing server can store the text to be replied and the reply text in each round of historical interactions, it can be stored after obtaining the reply text stream each time, without having to repeatedly obtain it from the text processing server. By opening up the communication between the text processing server and the audio processing server, the latency of obtaining the reply text stream and the text of historical interactions can be reduced, and the efficiency can be improved, thereby improving the efficiency of the audio processing server in generating the reply audio stream.
[0066] In some embodiments, the audio processing server determines whether to perform sentence segmentation after the text segment in the current text data packet based on the text segment in the current text data packet and the text segment in one or more text data packets before the current text data packet; in response to performing sentence segmentation after the text segment in the current text data packet, a current audio segment corresponding to the current text data packet is generated based on the text segment in the current text data packet, the text segment in the text data packet before the current text data packet, and a first preset number of texts.
[0067] The audio processing server makes a sentence segmentation judgment based on the text fragments in the current text data packet and the text fragments in one or more text data packets before the current text data packet. If sentence segmentation is required, the current audio segment is generated. If sentence segmentation is not required, the audio segment is generated when sentence segmentation is required after continuing to receive text data packets. The audio processing server makes sentence segmentation judgments based on the context, which improves the fluency of the generated reply audio and the fluency of the reading. In addition, compared with the method in which the text processing server sends the reply text stream to the client, the client performs sentence segmentation and then sends it to the audio processing server to generate audio, the efficiency of generating audio is improved.
[0068] The text processing server generates a reply text stream in a streaming manner and sends it to the client for display. The audio processing server generates a reply audio stream in a streaming manner and sends it to the client for playback. In addition, after the client inputs audio, the audio processing server can also generate a reply text in a streaming manner and send it to the client for display. During the interaction process, the content that needs to be sent to the client for display or playback is generated and transmitted in a streaming manner, which reduces the delay of display and playback on the client and improves the timeliness and efficiency of client display and playback.
[0069] In some embodiments, the audio processing server performs speech recognition on the input audio and converts it into a text stream of the text to be replied; and sends the text stream of the text to be replied to the client for display.
[0070] The audio processing server adopts ASR technology to generate a text stream of the text to be replied, and outputs it to the client for display, which improves the timeliness of displaying the text to be replied and enhances the user experience.
[0071] In some embodiments, the audio processing server obtains a second preset amount of text before the text to be replied from the text processing server, performs speech recognition based on the input audio and the second preset amount of text, and converts the input audio into a text stream of the text to be replied.
[0072] For each round of interaction, the audio processing server can obtain a second preset amount of text from the previous round or rounds of conversations, and generate a text stream of the text to be replied in combination with the input audio. The audio processing server can generate a text stream of the text to be replied in combination with the context of multiple rounds of conversations, which can better understand the semantics and context and improve the accuracy of the generated text to be replied.
[0073] If the audio processing server can store the text to be replied and the reply text in each round of historical interaction, it does not need to repeatedly obtain it from the text processing server. By opening up the communication between the text processing server and the audio processing server, the acquisition delay of the text of historical interactions can be reduced, and the efficiency can be improved, thereby improving the efficiency of the audio processing server in generating the text to be replied.
[0074] In some embodiments, the audio processing server and the text processing server perform data transmission via remote procedure calls.
[0075] For example, the audio processing server and the text processing server can communicate through RPC (google Remote Procedure Call) or gRPC to achieve the transmission of the above-mentioned reply text and reply text stream, thereby improving transmission efficiency and the efficiency of subsequent replies to the client.
[0076] In some embodiments, the audio processor transmits data with the client through the audio service gateway, and in response to the start of the audio interactive application, a long connection is established between the audio service gateway and the client. Correspondingly, in response to the start of the audio interactive application, the client establishes a long connection with the audio service gateway corresponding to the audio processing server, and the client transmits data with the audio processor through the audio service gateway.
[0077] When the audio interaction application is started, a long connection is established, such as a Websocket connection. The long connection can be reused to reduce the delay of repeated connection establishment and improve the efficiency of subsequent transmission and audio interaction.
[0078] During the communication between the client and the audio server or text server, network switching or disconnection may occur, which may cause the loss of audio data packets sent by the audio processing server to the client. When the network is restored, problems may occur with the audio played by the client.
[0079] Combine the following Figure 2 Other embodiments of the audio interaction processing method disclosed in the present invention are described.
[0080] Figure 2 FIG. 1 is a structural diagram of some other embodiments of the audio interaction processing method disclosed in the present invention. Figure 2 As shown, the method of this embodiment includes: steps S202 to S206.
[0081] In step S202, in response to the network interruption, the client sends a retransmission request to the audio processing server, and correspondingly, the audio server receives the retransmission request sent by the client.
[0082] In some embodiments, the reply audio stream includes multiple audio data packets, each of which includes a sequence number identifier of the audio data packet. By setting the sequence number identifier, the order between the audio data packets can be determined, so that when the video data packet is lost or retransmitted, the order of the audio data packets can be accurately identified and whether there is a missing.
[0083] For example, the retransmission request includes the sequence number identifier of the last audio data packet received by the client before the network interruption.
[0084] In step S204, the audio processing server determines whether there is an audio data packet to be retransmitted based on the sequence number identifier of the last audio data packet received by the client before the network is disconnected.
[0085] For example, the data packet to be retransmitted is an audio data packet that has been sent to the client after the last audio data packet received by the client before the network was interrupted. The audio processing server can determine the sequence number of the last audio data packet received by the client according to the retransmission request. If the audio data packet after the sequence number has been sent, it may be lost, that is, the audio data packet to be retransmitted.
[0086] In step S206, the audio processing server sends the audio data packet to be retransmitted to the client when there is an audio data packet to be retransmitted.
[0087] The client receives a retransmitted audio data packet sent by the audio processing server, where the retransmitted data packet is a data packet that has been sent to the client by the audio processing server after the last audio data packet received by the client before the network is interrupted.
[0088] like Figure 3 As shown, the text processing server sends each text data packet of the generated reply text stream, Text T1, Text T2, etc., to the audio processing server through gRPC. The audio processing server generates multiple audio data packets (Stream block, streaming data blocks) in a streaming manner, S1, S2, etc., and transmits them to the client through the external network. If a network interruption occurs after S1, the client sends a retransmission request to the audio processing server. The retransmission request may include the sequence number identifier of S1, and the audio processing server determines whether retransmission is required and which audio data packets to retransmit based on the sequence number identifier.
[0089] For example, each audio data packet in the reply audio stream includes the reply text, audio data, and reply text after Markdown filtering corresponding to the audio data packet. Each audio data packet may include a key field Seq_id to represent the sequence number. If a network outage occurs, the client can send a retransmission request to tell the audio processing server that a network error has occurred and that it needs to retransmit all audio data packets after the Seq_id of the last audio data packet it received. The client can identify the Seq_id of each received audio data packet to avoid receiving duplicate audio data packets.
[0090] According to the method of the above embodiment, in the event of a network interruption, the client only needs to notify the audio processing server of the serial number identifier to pull the subsequent audio for playback. If the client receives the reply text stream from the text processing server, breaks it into sentences, and then sends it to the audio processing server to generate a reply audio stream, the client will continue to send the reply text data packet to the audio processing server while waiting for the network to recover after a network interruption occurs. However, the audio processing server will not be able to distinguish whether the received data packet is used to continue synthesizing the audio. The audio processing server and the text processing server communicate through the intranet, the network is stable, and recovery from network errors is easier to handle, and the cache of reply text and audio is increased, making retransmission and continued transmission simpler and more efficient. The method of the above embodiment reduces the failure of reply audio playback caused by interruption of the reply audio stream, thereby improving the user experience.
[0091] Combine the following Figure 4 Some other embodiments of the method for processing audio interaction of the present disclosure are described. The method of this embodiment can be executed by a client.
[0092] Figure 4 FIG. 1 is a structural diagram of some embodiments of the audio interaction processing method disclosed in the present invention. Figure 4 As shown, the processing method of this embodiment includes: steps S402 to S410. The method of this embodiment can be executed by the client.
[0093] In step S402, the input audio is sent to the audio processing server and converted into text to be replied.
[0094] The text to be replied is sent by the audio processing server to the text processing server to generate a reply text stream. For details, please refer to the above embodiment and will not be described again here.
[0095] In step S404, a message identifier corresponding to the reply text stream sent by the text processing server is received.
[0096] After receiving the text to be replied, the text processing server may generate a message identifier and send it to the client. For details, please refer to the above embodiment and will not be described in detail here.
[0097] In step S406, a request for a reply audio stream corresponding to the reply text stream is sent to the audio processing server.
[0098] The request sent by the client includes a message identifier.
[0099] In step S408, a reply audio stream sent by the audio processing server is received.
[0100] The reply audio stream is generated by the audio processing server after obtaining the reply text stream from the text processing server according to the message identifier. The process of the audio processing server obtaining the reply audio stream from the text processing server can be referred to in the signing implementation, which will not be repeated here.
[0101] In step S410, the reply audio stream is played.
[0102] In the method of the above embodiment, the client sends the input audio to the audio processing server to convert it into the text to be replied, and then directly interacts with the text processing server. On the one hand, the text to be replied is directly sent to the text processing server to generate a reply text stream. On the other hand, the reply text stream is directly obtained from the text processing server to generate a reply audio stream, and then returned to the client for playback. The audio processing server communicates directly with the text processing server, which improves the transmission efficiency of the text to be replied and the reply text stream compared to the method of forwarding the text to be replied and the reply text stream through the client, so that the text processing server can obtain the text to be replied in time, and the audio processing server can obtain the reply audio stream in time, thereby improving the generation efficiency of the reply text stream and the reply audio stream. In addition, the reply text and reply audio are generated and transmitted in a streaming manner throughout the process, so that feedback can be given in time for the input audio, thereby improving the efficiency and timeliness of text and voice replies to the input audio.
[0103] In addition, since the audio processing server obtains the reply audio stream from the text processing server in real time and generates a reply audio stream, the synchronization effect of text and voice replies on the client can be improved, thereby improving the user experience.
[0104] The following further describes the audio interaction processing method of the present disclosure in conjunction with some embodiments.
[0105] In some embodiments, the client divides the input audio into multiple audio segments; compresses each of the multiple audio segments and stores them in order into a first queue; takes out the first audio segment from the first queue, and sends it to the audio processing server for speech recognition; in response to receiving a confirmation message sent by the audio processing server, repeatedly takes out the first audio segment from the first queue, and sends it to the audio processing server for speech recognition, until the first queue is empty.
[0106] The client stores the compressed audio clips in the first queue, and can send them to the audio processing server in order for voice recognition and conversion into text to be replied. The reply confirmation from the audio processing server can ensure that each audio clip is sent to the audio processing server, reducing the loss of audio clips due to network interruption and other situations, and improving the accuracy of voice interaction.
[0107] In some embodiments, in response to not receiving a confirmation message sent by the audio processing server after waiting for a preset period of time, the client resends a most recently sent audio segment.
[0108] If the client does not receive a confirmation message from the audio processing server after waiting for a preset period of time, it may be that the corresponding audio segment is lost. Resending the corresponding audio segment reduces the probability that the audio processing server cannot accurately recognize the input audio and improves the accuracy of subsequent voice interactions.
[0109] like Figure 5 As shown, the client sends each audio data packet of the input audio to the audio processing server. If the network is interrupted after the client sends the S1 data packet, and the client waits for a preset time and cannot receive the confirmation information sent by the audio processing server, the S1 data packet will be resent.
[0110] The client transcodes the input audio file and divides it into multiple audio segments, which are stored locally in the first queue. The client streams the audio file to the audio processing server for audio recognition. After receiving the confirmation message from the audio processing server, the subsequent audio segments are dequeued and subsequently transmitted. When abnormal situations such as network interruption are detected, the configuration determines whether to resume the transmission and wait for a preset time each time to retry. The audio segment at the head of the most recently sent team can be retried. This ensures that the streaming recognition of the input audio can run normally in the case of network switching or poor network quality, and obtains correct recognition results.
[0111] In some embodiments, the reply audio stream includes multiple audio data packets, each of the multiple audio data packets includes a serial number identifier of the audio data packet, and the client stores the current audio data packet in the second queue according to the serial number identifier of the received current audio data packet; and plays the audio data packets in the order of the audio data packets in the second queue.
[0112] Adding sequence number identifiers to audio data packets allows the client to play each audio data packet in sequence, and even if a network transmission error occurs, the reply audio stream can be accurately played.
[0113] In some embodiments, the client determines whether an audio data packet with the same serial number identifier as the current audio data packet already exists in the second queue based on the serial number identifier of the current audio data packet received; if an audio data packet with the same serial number identifier as the current audio data packet already exists in the second queue, the current audio data packet is discarded; if an audio data packet with the same serial number identifier as the current audio data packet does not exist in the second queue, the current audio data packet is stored in the queue.
[0114] The client can determine whether there are duplicate audio data packets based on the sequence number identifier of the audio data packet. In particular, in cases of network interruption, retransmission, and resumption, there may be duplicate transmission of audio data packets, which improves the accuracy of reply audio stream transmission and the accuracy of reply audio stream playback.
[0115] In some embodiments, the client receives a text stream of text to be replied sent by the audio processing server; displays the text stream of text to be replied; receives a reply text stream sent by the text processing server; and displays the reply text stream.
[0116] The reply text stream is generated by the text processing server after receiving the reply text sent by the audio processing server. The client will receive the text stream of the reply text sent by the audio processing server, the reply text stream sent by the text processing server, and the reply audio stream sent by the audio processing server for display and playback.
[0117] The method of the disclosed embodiment can obtain the reply text stream in real time for the input audio, quickly convert it into a reply audio stream, and push it to the client in real time for playback. In scenarios where the real-time streaming output is more real-time, the reply text to audio playback experience is greatly improved, and the generation of the reply audio stream in combination with the context can improve accuracy. By retrying and retransmitting the reply audio stream and the input audio stream, it can be ensured that even in the event of network quality degradation, network disconnection, jitter, etc. during the ASR recognition process of long texts or users for a long time, ASR and TTS can still be accurately recognized or played normally.
[0118] The following describes the audio interaction processing method of the present disclosure in conjunction with some application examples.
[0119] Fig. 6A It is the information exchange process in the ASR message link. Fig. 6AAs shown, (0) the audio module (APP Audio) in the audio interaction application of the client (client) establishes a long connection with the audio service gateway (GateWay) when the audio interaction application is turned on. The user sends a voice, starts ASR message transmission, and inputs it into the audio module in the audio interaction application of the client. (1) The user voice (input audio) is sent to the audio service gateway through the audio module. The audio service gateway belongs to the service side device. (2) The audio service gateway sends the user voice to the audio processing module. The audio processing module and the ASR module can form the audio processing server in the aforementioned embodiment. The ASR module can also be deployed separately, and the audio processing module serves as the audio processing server in the aforementioned embodiment. The audio processing module calls the ASR module through gRPC to realize user voice recognition and obtains the voice recognition text (text to be replied). (3) The audio processing module returns the voice recognition text to the audio service gateway. (4) The audio service gateway returns the voice device text to the audio module in the client, and (5) the audio module in the client sends the voice recognition text to the message module (APP message) in the client for display. (6) After the audio processing module obtains the speech recognition text, it sends the speech recognition text to the interaction processing module. (7) The interaction processing module sends a text message to LLM, requesting LLM to process the text message. The interaction processing module and LLM can form the text processing server in the aforementioned embodiment. LLM can also be deployed separately, and the interaction processing module serves as the text processing server in the aforementioned embodiment. (8) The interaction processing module sends the full text of the user message (speech recognition text) to LLM, and LLM returns the key for polling to the interaction processing module. (9) The interaction processing module polls LLM for incremental replies based on the key. LLM generates replies in a streaming manner, and the interaction processing module only needs to obtain the incremental reply text each time. (10) The interaction processing module sends the reply text to the Frotier (border) device, and (11) the Frotier device sends the reply text to the message module in the client for display.
[0120] The audio service gateway and Frotier device can be used as the WebSocket access layer to implement transmission between the external network and the internal network.
[0121] Figure 6B It is the information exchange process in the TTS message link. Figure 6BAs shown, (1) the audio processing module sends the speech recognition text (text to be replied) to the interaction processing module, and (2) the interaction processing module sends the speech recognition text to the LLM. (3) The interaction processing module sends the Message id (message identifier) of the reply text stream to the Frotier device. (4) The Frotier device sends the Message id of the reply text stream to the message module in the client. (5) The message module sends the Message id of the reply text stream to the audio module in the client to start the TTS session. (6) The audio module sends a request to subscribe to the TTS audio stream (reply audio stream) to the audio service gateway. (7) The audio service gateway sends a request to subscribe to the TTS audio stream (reply audio stream) to the audio processing module. (8) The audio processing module requests the reply text from the interaction processing module. (9) The audio processing module generates a TTS audio stream (reply audio stream) based on the reply text and sends it to the audio service gateway. (10) The audio service gateway sends the TTS audio stream to the audio module in the client for playback.
[0122] like Figure 6C This is the information interaction flow of the audio interaction processing method. Figure 6C As shown, in order to implement the solution of the present disclosure, an audio SDK can be added to the client for interacting with the audio processing server. The audio SDK can be located in the client or set up independently. (1) The client sends input audio to the audio SDK (i.e., Audio uplink). (2) The audio SDK sends input audio to the audio processing server. (3) The audio processing server requests context content (i.e., the text of historical interactions before this round of interaction) from the text processing server. (4) The text processing server returns the context content to the audio processing server. (5) The audio processing server performs ASR recognition and obtains ASR recognized text (text to be replied). The audio processing server combines the context content when performing ASR recognition, which is more accurate. (6) The audio processing server sends the ASR recognized text to the audio SDK. (7) The audio SDK sends the ASR recognized text to the client. (8) The client displays the ASR recognized text.
[0123] (9) The audio processing server sends the ASR recognized text (text to be replied) to the text processing server. (10) The text processing server sends the message ID to the client. (11) The text processing server generates the reply text and returns the reply text stream to the client in streaming mode. (12) The client displays the reply text.
[0124] (13) The client sends a request for TTS audio data (reply audio stream) to the audio SDK, which carries the MessageID. (14) The audio SDK sends a request for TTS audio data to the audio processing server. (15) The audio processing server requests the text processing server to reply the text. (16) The text processing server sends the reply text stream to the audio processing server. (17) The audio processing server generates a TTS audio stream based on the reply text stream. (18) The audio processing server sends the TTS audio stream to the audio SDK. (19) The audio SDK sends the TTS audio stream to the client. (20) The client plays the TTS audio.
[0125] The above steps (1) to (8) are the process of generating and displaying the text to be replied, and (9) to (12) are the process of generating and displaying the reply text. (13) to (20) are the process of generating and playing the reply audio stream. For ease of understanding, the steps in the same process are described together. In the actual processing process, the order of these steps is not limited to the order and numbering shown in the figure. For example, after the audio processing server generates the ASR recognized text, it can be sent to the text processing service segment, that is, (6) and (9) can be synchronized or (9) can be after (6).
[0126] The present disclosure also provides an audio processing server. Figure 7 Give a description.
[0127] Figure 7 FIG. 1 is a structural diagram of some embodiments of the audio processing server disclosed in the present invention. Figure 7 As shown, the audio processing server 70 of this embodiment includes: a conversion module 710 , a first sending module 720 , a receiving module 730 , an acquisition module 740 , a generation module 750 , and a second sending module 760 .
[0128] The conversion module 710 is configured to convert the input audio sent by the client into a text to be replied.
[0129] The first sending module 720 is configured to send the text to be replied to to the text processing server to generate a reply text stream.
[0130] The receiving module 730 is configured as a request of the client for the reply audio stream corresponding to the reply text stream, wherein the request includes a message identifier corresponding to the reply text stream, and the message identifier is sent by the text processing server to the client.
[0131] The acquisition module 740 is configured to acquire a reply text stream from the text processing server according to the message identifier.
[0132] The generating module 750 is configured to generate a reply audio stream according to the reply text stream.
[0133] The second sending module 760 is configured to send the reply audio stream to the client for playing.
[0134] In some embodiments, the reply audio stream includes multiple audio data packets, each of the multiple audio data packets includes a serial number identifier of the audio data packet, the audio processing server 70 also includes a determination module 770, and the receiving module 730 is also configured to receive a retransmission request sent by the client, wherein the retransmission request includes the serial number identifier of the last audio data packet received by the client before the network interruption; the determination module 770 is configured to determine whether there is an audio data packet to be retransmitted based on the serial number identifier of the last audio data packet, wherein the data packet to be retransmitted is an audio data packet sent to the client after the last audio data packet; the second sending module 760 is configured to send the audio data packet to be retransmitted to the client if there is an audio data packet to be retransmitted.
[0135] In some embodiments, the reply text stream includes multiple text data packets, and the acquisition module 740 is configured to periodically pull the current text data packet generated by the text processing server from the text processing server according to the message identifier; or receive the generated current text data packet pushed by the text processing server, wherein each of the multiple text data packets includes a message identifier.
[0136] In some embodiments, the acquisition module 740 is also configured to acquire a first preset amount of text before the reply text stream from the text processing server, and the generation module 750 is configured to generate a current audio segment corresponding to the current text data packet based on the text segment in the current text data packet, the text segment in one or more text data packets before the current text data packet, and the first preset amount of text, and generate the current audio data packet in the reply audio stream.
[0137] In some embodiments, the generation module 750 is configured to determine whether to perform sentence segmentation after the text segment in the current text data packet based on the text segment in the current text data packet and the text segment in one or more text data packets before the current text data packet; in response to performing sentence segmentation after the text segment in the current text data packet, generate a current audio segment corresponding to the current text data packet based on the text segment in the current text data packet, the text segment in the text data packet before the current text data packet, and a first preset number of texts.
[0138] In some embodiments, the conversion module 710 is configured to perform speech recognition on the input audio and convert it into a text stream of the text to be replied; the second sending module 760 is also configured to send the text stream of the text to be replied to the client for display.
[0139] In some embodiments, the acquisition module 740 is also configured to acquire a second preset amount of text before the text to be replied from the text processing server, and the conversion module 710 is configured to perform speech recognition based on the input audio and the second preset amount of text, and convert the input audio into a text stream of the text to be replied.
[0140] In some embodiments, the audio processing server and the text processing server perform data transmission via remote procedure calls.
[0141] In some embodiments, the audio processor transmits data with the client through the audio service gateway, and in response to the start of the audio interaction application, a long connection is established between the audio service gateway and the client.
[0142] The present disclosure also provides a client, Figure 8 Give a description.
[0143] Figure 8 FIG. 1 is a structural diagram of some embodiments of the client of the present disclosure. Figure 8 As shown, the client 80 of this embodiment includes: a first sending module 810 , a first receiving module 820 , a second sending module 830 , a second receiving module 840 , and a playing module 850 .
[0144] The first sending module 810 is configured to send the input audio to the audio processing server to convert it into a text to be replied, wherein the text to be replied is sent to the text processing server to generate a reply text stream.
[0145] The first receiving module 820 is configured to receive a message identifier corresponding to the reply text stream sent by the text processing server.
[0146] The second sending module 830 is configured to send a request for a reply audio stream corresponding to the reply text stream to the audio processing server, wherein the request includes a message identifier.
[0147] The second receiving module 840 is configured to receive a reply audio stream sent by the audio processing server.
[0148] The playing module 850 is configured to play the reply audio stream.
[0149] In some embodiments, the first sending module 810 is configured to divide the input audio into multiple audio segments; compress each of the multiple audio segments and store them in order into a first queue; take out the first audio segment from the first queue, and send it to the audio processing server for speech recognition; in response to receiving a confirmation message sent by the audio processing server, repeatedly take out the first audio segment from the first queue, and send it to the audio processing server for speech recognition until the first queue is empty.
[0150] In some embodiments, the first sending module 810 is configured to resend a most recently sent audio segment in response to not receiving a confirmation message sent by the audio processing server after waiting for a preset period of time.
[0151] In some embodiments, the reply audio stream includes multiple audio data packets, each of the multiple audio data packets includes a serial number identifier of the audio data packet, and the playback module 850 is configured to store the current audio data packet into a second queue according to the serial number identifier of the received current audio data packet; and play the audio data packets in the order of the audio data packets in the second queue.
[0152] In some embodiments, the playback module 850 is configured to determine whether an audio data packet with the same serial number identifier as the current audio data packet already exists in the second queue based on the serial number identifier of the received current audio data packet; if an audio data packet with the same serial number identifier as the current audio data packet already exists in the second queue, the current audio data packet is discarded; if an audio data packet with the same serial number identifier as the current audio data packet does not exist in the second queue, the current audio data packet is stored in the queue.
[0153] In some embodiments, the second sending module 830 is also configured to send a retransmission request to the audio processing server in response to a network interruption, wherein the retransmission request includes a sequence number identifier of the last audio data packet received by the client before the network interruption; the second receiving module 840 is also configured to receive a retransmitted audio data packet sent by the audio processing server, wherein the retransmitted data packet is a data packet that has been sent by the audio processing server to the client after the last audio data packet.
[0154] In some embodiments, the client 80 further includes a third receiving module 860 configured to receive a text stream of text to be replied sent by the audio processing server; and a display module 870 configured to display the text stream of text to be replied.
[0155] In some embodiments, the client 80 also includes a fourth receiving module 880, which is configured to receive a reply text stream sent by the text processing server, wherein the reply text stream is generated after the text processing server receives the text to be replied sent by the audio processing server; the display module 870 is also configured to display the reply text stream.
[0156] In some embodiments, in response to the start of the audio interaction application, the client 80 establishes a long connection with the audio service gateway corresponding to the audio processing server, wherein the client transmits data with the audio processor through the audio service gateway.
[0157] It should be noted that the above-mentioned units (modules) are only logical modules divided according to the specific functions they implement, and are not used to limit the specific implementation methods. For example, they can be implemented in software, hardware, or a combination of software and hardware. In actual implementation, the above-mentioned units can be implemented as independent physical entities, or can also be implemented by a single entity (for example, a processor (CPU or DSP, etc.), an integrated circuit, etc.). In addition, the above-mentioned units are shown with dotted lines in the drawings to indicate that these units may not actually exist, and the operations / functions they implement can be implemented by the processing circuit itself.
[0158] In addition, although not shown, the device may also include a memory, which may store various information generated by the device, various units contained in the device during operation, programs and data for operation, data to be sent by the communication unit, etc. The memory may be a volatile memory and / or a non-volatile memory. For example, the memory may include, but is not limited to, a random access memory (RAM), a dynamic random access memory (DRAM), a static random access memory (SRAM), a read-only memory (ROM), and a flash memory. Of course, the memory may also be located outside the device. Optionally, although not shown, the device may also include a communication unit, which may be used to communicate with other devices. In one example, the communication unit may be implemented in a suitable manner known in the art, such as including communication components such as an antenna array and / or a radio frequency link, various types of interfaces, communication units, etc. It will not be described in detail here. In addition, the device may also include other components not shown, such as a radio frequency link, a baseband processing unit, a network interface, a processor, a controller, etc. It will not be described in detail here.
[0159] Some embodiments of the present disclosure also provide an electronic device. Fig. 9 Block diagrams of some embodiments of the electronic device of the present disclosure are shown. For example, in some embodiments, the electronic device 90 can be various types of devices, for example, can include but are not limited to mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. For example, the electronic device 90 may include a display panel for displaying data and / or execution results used in the scheme of the present disclosure. For example, the display panel can be in various shapes, such as a rectangular panel, an elliptical panel, or a polygonal panel, etc. In addition, the display panel can be not only a flat panel, but also a curved panel, or even a spherical panel.
[0160] like Fig. 9 As shown, the electronic device 90 of this embodiment includes: a memory 91 and a processor 92 coupled to the memory 91. It should be noted that Fig. 9 The components of the electronic device 90 shown are only exemplary and non-limiting. The electronic device 90 may also have other components according to actual application requirements. The processor 92 may control other components in the electronic device 90 to perform desired functions.
[0161] In some embodiments, the memory 91 is used to store one or more computer-readable instructions. When the processor 92 is used to execute the computer-readable instructions, the computer-readable instructions are executed by the processor 92 to implement the method according to any of the above embodiments. For the specific implementation of each step of the method and the related explanation content, please refer to the above embodiments, and the repeated parts will not be repeated here.
[0162] For example, the processor 92 and the memory 91 may communicate with each other directly or indirectly. For example, the processor 92 and the memory 91 may communicate with each other through a network. The network may include a wireless network, a wired network, and / or any combination of a wireless network and a wired network. The processor 92 and the memory 91 may also communicate with each other through a system bus, which is not limited by the present disclosure.
[0163] For example, the processor 92 may be embodied as various appropriate processors, processing devices, etc., such as a central processing unit (CPU), a graphics processing unit (GPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The central processing unit (CPU) may be an X86 or ARM architecture, etc. For example, the memory 91 may include any combination of various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The memory 91 may include, for example, a system memory, which may store, for example, an operating system, an application, a boot loader (Boot Loader), a database, and other programs. Various applications and various data may also be stored in the storage medium.
[0164] In addition, according to some embodiments of the present disclosure, various operations / processes according to the present disclosure, when implemented by software and / or firmware, can be transmitted from a storage medium or a network to a computer system having a dedicated hardware structure, such as Fig.10 The computer system (or electronic device) 100 shown is installed with the programs constituting the software, and when the various programs are installed, the computer system can execute various functions, including the functions described above. Fig.10 is a block diagram illustrating an example structure of a computer system that can be employed in an embodiment according to the present disclosure.
[0165] exist Fig.10 In the embodiment, a central processing unit (CPU) 1001 performs various processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage part 1008 to a random access memory (RAM) 1003. In RAM 1003, data required when CPU 1001 performs various processes, etc., is also stored as needed. The central processing unit is merely exemplary, and it may also be other types of processors, such as the various processors described above. ROM 1002, RAM 1003, and storage part 1008 may be various forms of computer-readable storage media, as described below. It should be noted that although Fig.10 ROM 1002, RAM 1003 and storage device 1008 are shown separately in FIG, but one or more of them may be combined or located in the same or different memory or storage modules.
[0166] The CPU 1001, the ROM 1002, and the RAM 1003 are connected to one another via a bus 1004. To the bus 1004, an input / output interface 1005 is also connected.
[0167] The following components are connected to the input / output interface 1005: an input portion 1006, such as a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output portion 1007, including a display, such as a cathode ray tube (CRT), a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage portion 1008, including a hard disk, a magnetic tape, etc.; and a communication portion 1009, including a network interface card such as a LAN card, a modem, etc. The communication portion 1009 allows communication processing to be performed via a network such as the Internet. It is easy to understand that although Fig.10 The various devices or modules in the computer system 100 are shown to communicate via the bus 1004, but they may also communicate via a network or other means, wherein the network may include a wireless network, a wired network, and / or any combination of a wireless network and a wired network.
[0168] A drive 1010 is also connected to the input / output interface 1005 as needed. A removable medium 1011 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is mounted on the drive 1010 as needed so that a computer program read therefrom is installed into the storage section 1008 as needed.
[0169] In the case where the above-described series of processing is realized by software, a program constituting the software can be installed from a network such as the Internet or a storage medium such as the removable medium 1011 .
[0170] According to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 1009, or installed from a storage device 1008, or installed from a ROM 1002. When the computer program is executed by the CPU 1001, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.
[0171] It should be noted that in the context of the present disclosure, a computer-readable medium may be a tangible medium that may contain or store a program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. A computer-readable storage medium may be, for example, but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that may be used by an instruction execution system, device or device or used in combination with it. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.
[0172] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.
[0173] In some embodiments, the present disclosure further provides a computer program, including: instructions, which, when executed by a processor, cause the processor to execute any of the methods of the above embodiments. For example, the instructions may be embodied as computer program codes.
[0174] The present disclosure also provides an audio interaction processing system, Fig.11 Give a description.
[0175] Fig.11 FIG. 1 is a structural diagram of some embodiments of the audio interaction processing system of the present disclosure. Fig.11 As shown, the audio interaction processing system 11 of this embodiment includes: the audio processing server 70 of any embodiment of the present disclosure and the client 80 of any embodiment of the present disclosure.
[0176] In some embodiments, the system 11 further includes a text processing server 110 configured to generate a reply text stream based on the text to be replied converted from the input audio, and send the reply text stream to the audio processing server 70 and the client 80 .
[0177] In some embodiments, the text processing server 110 is further configured to receive the text to be replied sent by the audio processing server 70 , and send a message identifier corresponding to the reply audio stream to the client 80 .
[0178] In embodiments of the present disclosure, computer program codes for performing the operations of the present disclosure may be written in one or more programming languages or combinations thereof, including but not limited to object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In situations involving a remote computer, the remote computer may be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet).
[0179] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0180] The modules, components or units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the names of the modules, components or units do not, in some cases, limit the modules, components or units themselves.
[0181] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.
[0182] According to some embodiments of the present disclosure, a method for processing audio interaction is provided, which is executed by an audio processing server, and includes: converting the input audio sent by the client into text to be replied; sending the text to be replied to the text processing server to generate a reply text stream; receiving a request from the client for a reply audio stream corresponding to the reply text stream, wherein the request includes a message identifier corresponding to the reply text stream, and the message identifier is sent by the text processing server to the client; obtaining the reply text stream from the text processing server according to the message identifier; generating a reply audio stream according to the reply text stream; and sending the reply audio stream to the client for playback.
[0183] In some embodiments, the reply audio stream includes multiple audio data packets, each of the multiple audio data packets includes a sequence number identifier of the audio data packet, and the processing method also includes: receiving a retransmission request sent by the client, wherein the retransmission request includes the sequence number identifier of the last audio data packet received by the client before the network interruption; determining whether there is an audio data packet to be retransmitted based on the sequence number identifier of the last audio data packet, wherein the data packet to be retransmitted is an audio data packet sent to the client after the last audio data packet; and sending the audio data packet to be retransmitted to the client if there is an audio data packet to be retransmitted.
[0184] In some embodiments, the reply text stream includes multiple text data packets. According to the message identifier, obtaining the reply text stream from the text processing server includes: according to the message identifier, periodically pulling the current text data packet generated by the text processing server from the text processing server; or receiving the generated current text data packet pushed by the text processing server, wherein each of the multiple text data packets includes a message identifier.
[0185] In some embodiments, the processing method also includes: obtaining a first preset amount of text before the reply text stream from the text processing server, wherein generating a reply audio stream based on the reply text stream includes: generating a current audio segment corresponding to the current text data packet based on a text segment in the current text data packet, a text segment in one or more text data packets before the current text data packet, and the first preset amount of text, and generating a current audio data packet in the reply audio stream.
[0186] In some embodiments, generating a current audio segment corresponding to a current text data packet based on a text segment in a current text data packet, a text segment in a text data packet before the current text data packet, and a first preset number of texts includes: determining whether to perform sentence segmentation after the text segment in the current text data packet based on the text segment in the current text data packet and the text segments in one or more text data packets before the current text data packet; in response to performing sentence segmentation after the text segment in the current text data packet, generating a current audio segment corresponding to the current text data packet based on the text segment in the current text data packet, the text segment in the text data packet before the current text data packet, and the first preset number of texts.
[0187] In some embodiments, converting the input audio sent by the client into the text to be replied includes: performing speech recognition on the input audio and converting it into a text stream of the text to be replied; the processing method also includes: sending the text stream of the text to be replied to the client for display.
[0188] In some embodiments, the processing method also includes: obtaining a second preset amount of text before the text to be replied from a text processing server, wherein performing speech recognition on the input audio and converting it into a text stream of the text to be replied includes: performing speech recognition based on the input audio and the second preset amount of text, and converting the input audio into a text stream of the text to be replied.
[0189] In some embodiments, the audio processing server and the text processing server perform data transmission via remote procedure calls.
[0190] In some embodiments, the audio processor transmits data with the client through the audio service gateway, and in response to the start of the audio interaction application, a long connection is established between the audio service gateway and the client.
[0191] According to other embodiments of the present disclosure, a method for processing audio interaction is provided, which is executed by a client, and includes: sending input audio to an audio processing server, converting it into text to be replied, wherein the text to be replied is sent to a text processing server to generate a reply text stream; receiving a message identifier corresponding to the reply text stream sent by the text processing server; sending a request for a reply audio stream corresponding to the reply text stream to the audio processing server, wherein the request includes the message identifier; receiving the reply audio stream sent by the audio processing server; and playing the reply audio stream.
[0192] In some embodiments, sending the input audio to the audio processing server includes: dividing the input audio into multiple audio segments; compressing each of the multiple audio segments and storing them in order into a first queue; taking out the first audio segment from the first queue, and sending it to the audio processing server for speech recognition; in response to receiving a confirmation message sent by the audio processing server, repeatedly taking out the first audio segment from the first queue, and sending it to the audio processing server for speech recognition, until the first queue is empty.
[0193] In some embodiments, sending the input audio to the audio processing server further includes: in response to not receiving a confirmation message sent by the audio processing server after waiting for a preset period of time, resending a most recently sent audio segment.
[0194] In some embodiments, the reply audio stream includes multiple audio data packets, each of the multiple audio data packets includes a serial number identifier of the audio data packet, and playing the reply audio stream includes: storing the current audio data packet in a second queue according to the serial number identifier of the received current audio data packet; and playing according to the order of the audio data packets in the second queue.
[0195] In some embodiments, storing the current audio data packet in the second queue according to the serial number identifier of the received current audio data packet includes: determining whether an audio data packet with the same serial number identifier as the current audio data packet already exists in the second queue according to the serial number identifier of the received current audio data packet; if an audio data packet with the same serial number identifier as the current audio data packet already exists in the second queue, discarding the current audio data packet; if an audio data packet with the same serial number identifier as the current audio data packet does not exist in the second queue, storing the current audio data packet in the queue.
[0196] In some embodiments, the processing method also includes: in response to a network interruption, sending a retransmission request to an audio processing server, wherein the retransmission request includes a sequence number identifier of the last audio data packet received by the client before the network interruption; receiving a retransmitted audio data packet sent by the audio processing server, wherein the retransmitted data packet is a data packet that has been sent by the audio processing server to the client after the last audio data packet.
[0197] In some embodiments, the processing method also includes: receiving a text stream of text to be replied sent by an audio processing server; displaying the text stream of text to be replied; receiving a reply text stream sent by a text processing server, wherein the reply text stream is generated after the text processing server receives the text to be replied sent by the audio processing server; and displaying the reply text stream.
[0198] In some embodiments, the processing method further includes: in response to the start of the audio interaction application, establishing a long connection with an audio service gateway corresponding to the audio processing server, wherein the client transmits data with the audio processor through the audio service gateway.
[0199] According to some other embodiments of the present disclosure, an audio processing server is provided, including: a conversion module, configured to convert the input audio sent by the client into a text to be replied; a first sending module, configured to send the text to be replied to the text processing server, and generate a reply text stream; a receiving module, configured to receive the client's request for a reply audio stream corresponding to the reply text stream, wherein the request includes a message identifier corresponding to the reply text stream, and the message identifier is sent by the text processing server to the client; an acquisition module, configured to acquire the reply text stream from the text processing server according to the message identifier; a generation module, configured to generate a reply audio stream according to the reply text stream; and a second sending module, configured to send the reply audio stream to the client for playback.
[0200] According to some further embodiments of the present disclosure, a client is provided, comprising: a first sending module, configured to send input audio to an audio processing server to convert it into text to be replied, wherein the text to be replied is sent to a text processing server to generate a reply text stream; a first receiving module, configured to receive a message identifier corresponding to the reply text stream sent by the text processing server; a second sending module, configured to send a request for a reply audio stream corresponding to the reply text stream to the audio processing server, wherein the request includes the message identifier; a second receiving module, configured to receive the reply audio stream sent by the audio processing server; and a playing module, configured to play the reply audio stream.
[0201] According to some further embodiments of the present disclosure, there is provided an audio interaction processing system, comprising: an audio processing server of any embodiment of the present disclosure and a client of any embodiment of the present disclosure.
[0202] In some embodiments, the processing system further includes: a text processing server configured to generate a reply text stream based on the text to be replied converted from the input audio, and send the reply text stream to the audio processing server and the client.
[0203] In some embodiments, the text processing server is further configured to receive the text to be replied to sent by the audio processing server, and send a message identifier corresponding to the reply audio stream to the client.
[0204] According to some further embodiments of the present disclosure, there is provided an electronic device, comprising: a processor; and a memory coupled to the processor, for storing instructions, which, when executed by the processor, causes the processor to execute the audio interaction processing method of any embodiment of the present disclosure.
[0205] According to some further embodiments of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method for processing audio interaction of any embodiment of the present disclosure is performed.
[0206] According to some further embodiments of the present disclosure, a computer program product is provided, comprising: instructions, which, when executed by a processor, implement the audio interaction processing method of any embodiment of the present disclosure.
[0207] According to some further embodiments of the present disclosure, a computer program is provided, comprising: instructions, which, when executed by a processor, implement the audio interaction processing method of any embodiment of the present disclosure.
[0208] The above descriptions are only some embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, a technical solution formed by replacing the above features with the technical features with similar functions disclosed in the present disclosure (but not limited to).
[0209] In the description provided herein, many specific details are set forth. However, it is understood that embodiments of the present invention can be implemented without these specific details. In other cases, well-known methods, structures, and techniques are not presented in detail in order not to obscure the understanding of the description.
[0210] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.
[0211] Although some specific embodiments of the present disclosure have been described in detail by way of example, it should be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present disclosure. It should be understood by those skilled in the art that the above embodiments may be modified without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.
Claims
1. A method for processing audio interaction, executed by an audio processing server, comprising: Convert the input audio sent by the client into text to be replied; Sending the text to be replied to to a text processing server to generate a reply text stream; Receiving a request from the client for a reply audio stream corresponding to the reply text stream, wherein the request includes a message identifier corresponding to the reply text stream, and the message identifier is sent by the text processing server to the client; According to the message identifier, obtaining the reply text stream from the text processing server; Generate the reply audio stream according to the reply text stream; The reply audio stream is sent to the client for playing.
2. The processing method according to claim 1, wherein: The reply audio stream includes a plurality of audio data packets, each of the plurality of audio data packets includes a sequence number identifier of the audio data packet, and the processing method further includes: Receiving a retransmission request sent by the client, wherein the retransmission request includes a sequence number identifier of the last audio data packet received by the client before the network is interrupted; Determine, according to the sequence number identifier of the last audio data packet, whether there is an audio data packet to be retransmitted, wherein the data packet to be retransmitted is an audio data packet that has been sent to the client after the last audio data packet; In the case where the audio data packet to be retransmitted exists, the audio data packet to be retransmitted is sent to the client.
3. The processing method according to claim 1 or 2, wherein: The reply text stream includes a plurality of text data packets, and obtaining the reply text stream from the text processing server according to the message identifier includes: periodically pulling from the text processing server a current text data packet generated by the text processing server according to the message identifier; or The generated current text data packet pushed by the text processing server is received, wherein each of the multiple text data packets includes the message identifier.
4. The processing method according to claim 3, further comprising: Acquire a first preset amount of text before the reply text stream from the text processing server, The step of generating the reply audio stream according to the reply text stream comprises: Based on the text segment in the current text data packet, the text segment in one or more text data packets before the current text data packet, and the first preset number of texts, a current audio segment corresponding to the current text data packet is generated, and a current audio data packet in the reply audio stream is generated.
5. The processing method according to claim 4, wherein: The step of generating the current audio segment corresponding to the current text data packet according to the text segment in the current text data packet, the text segment in the text data packet before the current text data packet, and the first preset number of texts comprises: Determining whether to perform sentence segmentation after the text segment in the current text data packet according to the text segment in the current text data packet and the text segments in one or more text data packets before the current text data packet; In response to sentence segmentation after the text segment in the current text data packet, a current audio segment corresponding to the current text data packet is generated based on the text segment in the current text data packet, the text segment in the text data packet before the current text data packet, and the first preset amount of text.
6. The processing method according to any one of claims 1 to 5, wherein: The step of converting the input audio sent by the client into a text to be replied includes: Performing speech recognition on the input audio to convert it into a text stream of the text to be replied; The processing method also includes: The text stream of the text to be replied is sent to the client for display.
7. The processing method according to claim 6, further comprising: Acquire a second preset number of texts before the text to be replied from the text processing server, The step of performing speech recognition on the input audio and converting the input audio into a text stream of the text to be replied includes: Speech recognition is performed based on the input audio and the second preset amount of text, and the input audio is converted into a text stream of the text to be replied.
8. The processing method according to any one of claims 1 to 7, wherein: The audio processing server and the text processing server perform data transmission via remote procedure call.
9. The processing method according to any one of claims 1 to 8, wherein: The audio processor performs data transmission with the client through the audio service gateway, and in response to the start of the audio interaction application, a long connection is established between the audio service gateway and the client.
10. A method for processing audio interaction, executed by a client, comprising: Sending the input audio to the audio processing server to convert it into a text to be replied, wherein the text to be replied is sent by the audio processing server to the text processing server to generate a reply text stream; Receiving a message identifier corresponding to the reply text stream sent by the text processing server; Sending a request for a reply audio stream corresponding to the reply text stream to the audio processing server, wherein the request includes the message identifier; Receiving the reply audio stream sent by the audio processing server, wherein the reply audio stream is generated by the audio processing server after obtaining the reply text stream from the text processing server according to the message identifier; The reply audio stream is played.
11. The processing method according to claim 10, wherein: The sending of the input audio to the audio processing server includes: Dividing the input audio into a plurality of audio segments; Compress each of the multiple audio clips and store them in a first queue in order; Taking out the first audio clip from the first queue and sending it to the audio processing server for speech recognition; In response to receiving a confirmation message sent by the audio processing server, repeatedly taking out the first audio segment from the first queue and sending it to the audio processing server for speech recognition until the first queue is empty.
12. The processing method according to claim 11, wherein: The sending of the input audio to the audio processing server also includes: In response to not receiving a confirmation message sent by the audio processing server after waiting for a preset time period, resending a most recently sent audio segment.
13. The processing method according to claim 11 or 12, wherein: The reply audio stream includes a plurality of audio data packets, each of the plurality of audio data packets includes a sequence number identifier of the audio data packet, and playing the reply audio stream includes: According to the sequence number identifier of the received current audio data packet, storing the current audio data packet into a second queue; The audio data packets in the second queue are played in order.
14. The processing method according to claim 13, wherein: According to the sequence number of the received current audio data packet, storing the current audio data packet in the second queue comprises: Determine, according to the received sequence number identifier of the current audio data packet, whether an audio data packet having the same sequence number identifier as the current audio data packet already exists in the second queue; If an audio data packet with the same sequence number as the current audio data packet already exists in the second queue, the current audio data packet is discarded; If there is no audio data packet with the same sequence number as the current audio data packet in the second queue, the current audio data packet is stored in the queue.
15. The processing method according to any one of claims 10 to 14, further comprising: In response to a network interruption, sending a retransmission request to the audio processing server, wherein the retransmission request includes a sequence number identifier of the last audio data packet received by the client before the network interruption; Receive a retransmitted audio data packet sent by the audio processing server, wherein the retransmitted data packet is a data packet that has been sent by the audio processing server to the client after the last audio data packet.
16. The processing method according to any one of claims 10 to 15, further comprising: Receiving a text stream of the text to be replied sent by the audio processing server; Displaying the text stream of the text to be replied; Receiving the reply text stream sent by the text processing server, wherein the reply text stream is generated after the text processing server receives the text to be replied sent by the audio processing server; The reply text stream is displayed.
17. The processing method according to any one of claims 10 to 16, further comprising: In response to the start of the audio interaction application, a long connection is established with the audio service gateway corresponding to the audio processing server, wherein the client performs data transmission with the audio processor through the audio service gateway.
18. An audio processing server, comprising: The conversion module is configured to convert the input audio sent by the client into a text to be replied; A first sending module is configured to send the text to be replied to a text processing server to generate a reply text stream; A receiving module, configured to receive a request from the client for a reply audio stream corresponding to the reply text stream, wherein the request includes a message identifier corresponding to the reply text stream, and the message identifier is sent from the text processing server to the client; An acquisition module, configured to acquire the reply text stream from the text processing server according to the message identifier; A generating module, configured to generate the reply audio stream according to the reply text stream; The second sending module is configured to send the reply audio stream to the client for playing.
19. A client, comprising: A first sending module is configured to send the input audio to the audio processing server to convert it into a text to be replied, wherein the text to be replied is sent to the text processing server to generate a reply text stream; A first receiving module is configured to receive a message identifier corresponding to the reply text stream sent by the text processing server; A second sending module is configured to send a request for a reply audio stream corresponding to the reply text stream to the audio processing server, wherein the request includes the message identifier; A second receiving module is configured to receive the reply audio stream sent by the audio processing server; The playing module is configured to play the reply audio stream.
20. An audio interaction processing system, comprising: The audio processing server as described in claim 18 and the client as described in claim 19.
21. The processing system of claim 20, further comprising: The text processing server is configured to generate a reply text stream based on the to-be-reply text converted from the input audio, and send the reply text stream to the audio processing server and the client.
22. The processing system of claim 21, wherein: The text processing server is further configured to receive the text to be replied sent by the audio processing server, and send a message identifier corresponding to the reply audio stream to the client.
23. An electronic device, comprising: processor; as well as A memory coupled to the processor, for storing instructions, wherein when the instructions are executed by the processor, the processor executes the audio interaction processing method as described in any one of claims 1-9, or the audio interaction processing method as described in any one of claims 10-17.
24. A computer-readable storage medium having a computer program stored thereon, wherein: When the program is executed by a processor, it implements the audio interaction processing method described in any one of claims 1-9, or the audio interaction processing method described in any one of claims 10-17.
25. A computer program product comprising: An instruction, which, when executed by a processor, implements the audio interaction processing method described in any one of claims 1-9, or the audio interaction processing method described in any one of claims 10-17.
26. A computer program comprising: An instruction, which, when executed by a processor, implements the audio interaction processing method described in any one of claims 1-9, or the audio interaction processing method described in any one of claims 10-17.
Citation Information
Cited By
Audio task processing method and audio agent system
CN121478914A
An audio task processing method and an audio agent system
CN121478914B
Real-time streaming speech synthesis playing method and system based on WebSocket
CN121686999A
Real-time processing method, device, gateway equipment and system for converting streaming text into voice of AI (Artificial Intelligence) large model
CN122050359A
